New work with Edward Grefenstette at Google DeepMind: 🚨Interaction Dynamics as a Reward Signal for LLMs🚨 When it comes to interactions, the "how" is just as important as the "what" There is a signal in how we interact with a model that text analysis misses: hesitation, drift, friction

@egrefen @GoogleDeepMind Love the idea of a geometry of conversations. Reminds me a lot of Cohen, Frank and Ippolito's "Argument Drawings" series.

To capture this, we treated dialogue as a trajectory in semantic space—not just a transcript. By mapping the geometry of the conversation, we can measure the "flow" of collaboration. (Visuals created with the incredible #manim library! cc @3blue1brown)

Our method, TRACE, analyzes the features of this dialogue trajectory. This allows us to diagnose interaction failures—like "Mismatched Effort" or "Goal Drift"—that standard evaluations overlook. It turns implicit cues into a concrete, measurable signal.

The best part? It's not "either/or". When we combine our interaction dynamics with an LLM judge, we reach 80.17% accuracy—significantly better than either approach alone. We capture the behavioral information that text models miss.

This framework is privacy-preserving, scalable, and computationally efficient. Read the full paper here:

@egrefen @GoogleDeepMind What great work and beautifully illustrated 🤌 We've been looking at similar signals for understanding trajectories in LLM-assisted data exploration- I'll finally have a term to call it now!

@egrefen @GoogleDeepMind

@egrefen @GoogleDeepMind Really fascinating work, whoever can best accommodate human preference will likely be more successful than those who top benchmarks.

@EkdeepL @egrefen @GoogleDeepMind Wow, this is very interesting! Any thoughts on which dynamics could be most promising?

@egrefen @GoogleDeepMind this is super interesting! I've been looking at thematically coding user follow-up patterns in conversations -- will have to test this!

@egrefen @GoogleDeepMind very promising, 42

@egrefen @GoogleDeepMind Interesting work! Any plans to release the dataset or code? Also, I didn't spot which embedding model was used for computing the trajectory features, would be interested to know.

@egrefen @GoogleDeepMind Thanks! We use Gemini embeddings & are working on a formal submission that will include data