Another experiment with using Jev on two ends of an email classification system (a Jev sandwich?) to figure out what’s worth my attention based on how well I slept and my current vibes

This was partly inspired by @danshipper’s thought experiment on “urgency” at the top his piece on seeing like a language model. Worth a read!

I was like “cool, cool” might try this THEN you demoed the way your health data gets used and now I’m ready to buy every biometric device to reach my potential.

A classifier that adjusts to your sleep and vibes is either the future of inbox triage or a very polite way to let fatigue run the company. How are you evaluating false negatives?

great use case, great video Jack!

Thanks Steve :)

Very unique use case!!! Loved it. Especially the part where Opus generated a new set of evaluation questions against the data set based on user intent. I wonder if you have prompted it to consider the questions to be of system 1 type?

More or less! I’ve found it helps to have typesafe’s official skill installed to help since LLMs don’t seem to have decision model best practices as part of their training data. In this case I also ran some evals to help me tune the prompt; the current one asks the reasoning model to write concrete situations for jev to evaluate instead of broad topics

Have you faced the issue that the model assigned different weights to parameters after asking the same custom question formulated differently? Ex “I need to focus on reading” vs “today I want to prioritize some reading only”. If yes, how do you handle this?

Ah good question! I disabled caching for the purposes of the demo to show how long it actually took, but what you could do is cache the answers to the custom questions, the have jev look at whether or not the question has been answered before. Useful especially for frequent context notes like “I have 30 minutes free and want to jam on some email”

oh this makes sense! and later if you repeat the same question but in different words the system could make a shortcut for you Thanks for the explanation!

Are we watching the end of spam and email marketing in real time?

I don’t think those will end anytime soon but imagine owning your personal algorithm instead of leaving it up to the platforms to decide for you

The Jev sandwich is a nice pattern. Using a second pass to challenge the first one feels useful, but I'm curious about drift: does that second pass need retuning more often as user state changes, even if the inbox distribution stays fairly stable?