Skip navigation
New work with Edward Grefenstette at Google DeepMind:
0:00 / 1:53

New work with Edward Grefenstette at Google DeepMind:

Sian Gooding

@SianGooding2.7K followers

Visit on X
166 likesDownloadOriginal post

36,722 views

New work with Edward Grefenstette at Google DeepMind: 🚨Interaction Dynamics as a Reward Signal for LLMs🚨 When it comes to interactions, the "how" is just as important as the "what" There is a signal in how we interact with a model that text analysis misses: hesitation, drift, friction

13 Comments

Comments from the original post on X (Twitter)

  • David Pfau's profile picture

    @pfau

    @egrefen @GoogleDeepMind Love the idea of a geometry of conversations. Reminds me a lot of Cohen, Frank and Ippolito's "Argument Drawings" series.

    29 likes

  • Sian Gooding's profile picture

    @SianGooding

    To capture this, we treated dialogue as a trajectory in semantic space—not just a transcript. By mapping the geometry of the conversation, we can measure the "flow" of collaboration. (Visuals created with the incredible #manim library! cc @3blue1brown)

    13 likes

  • Sian Gooding's profile picture

    @SianGooding

    Our method, TRACE, analyzes the features of this dialogue trajectory. This allows us to diagnose interaction failures—like "Mismatched Effort" or "Goal Drift"—that standard evaluations overlook. It turns implicit cues into a concrete, measurable signal.

    10 likes

  • Sian Gooding's profile picture

    @SianGooding

    The best part? It's not "either/or". When we combine our interaction dynamics with an LLM judge, we reach 80.17% accuracy—significantly better than either approach alone. We capture the behavioral information that text models miss.

    13 likes

  • Sian Gooding's profile picture

    @SianGooding

    This framework is privacy-preserving, scalable, and computationally efficient. Read the full paper here:

    11 likes

  • Dylan Wootton's profile picture

    @WoottonDylan

    @egrefen @GoogleDeepMind What great work and beautifully illustrated 🤌 We've been looking at similar signals for understanding trajectories in LLM-assisted data exploration- I'll finally have a term to call it now!

    3 likes

  • Stanislav Nikolov's profile picture

    @snikolov

    @egrefen @GoogleDeepMind

    2 likes

  • ✨Myrddin✨'s profile picture

    @myrddin_neuro

    @egrefen @GoogleDeepMind Really fascinating work, whoever can best accommodate human preference will likely be more successful than those who top benchmarks.

    2 likes

  • josh :)'s profile picture

    @jfonsecarivera

    @EkdeepL @egrefen @GoogleDeepMind Wow, this is very interesting! Any thoughts on which dynamics could be most promising?

    1 like

  • shaily's profile picture

    @shailyfozdar

    @egrefen @GoogleDeepMind this is super interesting! I've been looking at thematically coding user follow-up patterns in conversations -- will have to test this!

    1 like

  • AInthusiast's profile picture

    @ainthusiast

    @egrefen @GoogleDeepMind very promising, 42

    1 like

  • Sander Land's profile picture

    @magikarp_tokens

    @egrefen @GoogleDeepMind Interesting work! Any plans to release the dataset or code? Also, I didn't spot which embedding model was used for computing the trajectory features, would be interested to know.

  • Sian Gooding's profile picture

    @SianGooding

    @egrefen @GoogleDeepMind Thanks! We use Gemini embeddings & are working on a formal submission that will include data

    2 likes