Let's make the email demo really good

[gvanrossum-ms]

So good that I'll actually want to use it for searching my own email instead of GMail's own search (which is okay but not amazing, and recent tweaks just make it more chaotic).

But not so good that it becomes a service that everyone can just subscribe to -- I don't want to deal with running a service that might go down, etc.

[gvanrossum-ms]

@umeshma

[bmerkle]

small idea: for the email data so that we do not have to use gmail accounts for demos...

we could use public available datasets and download them e.g. from huggingface or kaggle,
or download them a priori and store it in the demo git repo. (so if would be selfcontained)

Using the Enron Email Dataset contains ~500,000 messages from ~150 users at Enron.
from datasets import load_dataset
ds = load_dataset("enron_email")
print(ds["train"][0])

Available also at Kaggle or Carnegie Mellon
.

[gvanrossum]

Good idea -- though I currently expect that ingesting 500,000 messages could take weeks to index.

I was focusing on my own emails because that's my actual use case. :-)

[bmerkle]

yes, lets take only a fraction of them :) or maybe i find a smaller public set
if its ok, i could create a PR for it.

the approach would be similar to sklearn
from sklearn.datasets import fetch_20newsgroups

the old times when there were newsgroups :) transferred with uucp and cnews

[gvanrossum]

I like including some of that in testdata or having a test that downloads and caches it.

But beware that both those corpora are over 20 years old, and the way people write and mark up their emails have changed. In part because the tools are imposing new conventions for quoting -- what Outlook and GMail do is quite different from what we did around 2000.

[bmerkle]

@gvanrossum small question: are there specific points in which direction we should improve the demo ?
We have discussed about the (mail) data, but to do concrete steps, i would need a few concrete topics/plan

I also looked at TODO.md and would be open to take some tasks: maybe first start with a easy task from

• Code Quality & Cleanup
• Documentation
• Development Infra
  and via this I get also more insight into the codebase

[gvanrossum]

Here are some things wrong with the current email parsing.

• It uses an overly-simple heuristic for deciding where messages previously in the conversation are quoted. For example for GMail messages it essentially triggers on "On , wrote:", but if there are unquoted lines following that (meaning someone did inline responses) it will miss those. Conversely, if someone removes the "On ..." line and uses inline responses, it will appear as if the lines quoted with ">" are spoken by the message author. Also, that "On ..." line is often localized, so e.g. from some German correspondents I receive messages containing something like "Auf , sprach:".

• For long messages we should probably do some kind of chunking (e.g. by paragraph) rather than putting an entire 1000-line email in a single message chunk.

• There's no unification of email addresses / users; many users use multiple accounts with different spellings or orderings of their name.

• Forum software (e.g. Discourse, GitHub issues/PRs) often sends email from a single fixed address (e.g. noreply@example.com) with the "human name" showing the human sender, but often still in some kind of weird augmented form. E.g. Discourse sends emails from addresses like this: "John Doe via Discussions on Python.org notifications@python1.discoursemail.com".

I realize that some of these can't be solved perfectly or can only be solved with yet another LLM call, but it would be nice if our heuristics covered more of the common habits of email users (and email tools).

---

If you want to try your hand at some items from TODO.md, maybe these make decent starter issues:

Code Quality & Cleanup

P2 - Medium Priority

• [P2, small] Avoid most inline imports
• [P2, medium] Break cycles by moving things to their own file if necessary

I'd start with the former but back out whenever moving an inline import reveals a circular dependency -- for those we often need to do the latter, which is much more subtle and requires a deep understanding of the architecture in order to move the right things into separate files.

I'm not in a hurry with the development infra items (all P3s anyways). Feel free to document the test/build process. The release process I'd rather document myself because I think there are actually some improvements needed for that.

[bmerkle]

Hi again,

this is a good plan, thanks for the outline.
I will start with Code Quality & Cleanup and create PR accordingly.

Following I will try to cover the email aspects.
I call the problem we want to solve here with the email aspect “the lost meta information”
or “unfortunately, everything is now just text without any reference to the knowledge graph”....

What I mean by this is that quoting text in an email actually has a meta type and reference, and if this information were still there, much of what we now have to do would be much easier or completely unnecessary.

In the past, I have worked a lot with MPS from Jetbrains (and language-oriented programming). It uses a projecting editor, and you never lose context. Some things (e.g., refactorings) become trivial as a result; it's a ONE-CLICK rename, and all places where the variable is used (even in comments) update automatically (or magically). 😄

[gvanrossum]

For moving inline imports to the top of the file, I'd try one file at a time under typechat. There are some traps where these are needed to avoid circular imports, but many are undoubtedly due to Claude's preference for that style, which is not mine.

For the tests I think you should be able to move all imports out, since there are no circular imports between tests and everything else ought to be importable in any order.

...

I think I understand what you mean about "lost meta information", but it's not typeagent's fault -- it's that email tools (because they predate the Internet) work using conventions that are easy for humans to understand but less so for simplistic software. An LLM would have no problem figuring out who said what, but that's yet another LLM roundtrip.

...

You can use whatever tools you like, including LLMs, but for those I'd like to know about it (and ideally which one and version), and I urge you to review their output carefully before submitting the PR. (You can also make it a Draft PR so I'll know you're still working on it.)

[gvanrossum-ms]

@robgruen and I brainstormed a bit, and we came up with the need to parse message with inline quotations from previous messages. We prefer not to update the data structures (message ~~ list[str]) but we came up with a way to represent quotations as actions: X quotes Y saying Z, where Z is an entire quoted block (which must be followed by non-quoted, original text). We need to create an entity to represent the quoted block Z.

Separately, I've tried to investigate (with LLMs' help :-) if there are 3rd party packages that parse email messages. The recommendation is quotequail. I will try to use this in a new email ingesting script.

[gvanrossum]

FWIW it doesn't look like quotequail does much for us -- it just recognizes leading ">" and groups blocks by presence or absence thereof? In fact it looks like it only ever returns up to two blocks -- one unquoted, the text until the first quotation header ("On ..., ... wrote:"), and one (optional) quoted block -- the rest of the text. Let me see if instead we can do this part ourselves without being constrained by quotequail's idiosyncracies. (It doesn't even seem to recognize Google's current reply format which doesn't have a comma between date and author.)

Sorry quotequail. You're a lovely little library.

[bmerkle]

@gvanrossum i will continue the discussion about code refactorings in a separate issue #112 and copy over some of the comments (so we can better trac it there)