One feature I got to implement in OpenCode 2.0 was allowing for HOT CONTEXT¹ without busting the cache. Here's me explaining how it works ↓ ¹ ⁿᵒᵗ ᵃ ᵗᵉʳᵐ

obligatory @samwhoo link

@opencode This is smart but also I'm distracted by this really nice looking demo ui

@opencode 🥰 aw, thank ye zeb!

@opencode wait HOT LOCAL CONTEXT??? in my AREA???

@opencode crazy they hired the same narrator as effect institute

@opencode great explanation

@opencode your demos are always the best holy

@opencode 😳 😍

@opencode How did you evaluate that this way of updating context values doesn’t affect the quality of the LLM’s replies?

@opencode pretty cool! though makes you wonder in what cases you might just want to intentionally change the prefix rather than append deltas, e.g. if you are on a provider with known low cache ttl, or if returning to a session after a long time

@opencode Yeah! If we have that information about the provider/model we can squash everything back into the prompt. I’ll look into this :)

@opencode We built something very similar for our own internal harness. Lovely representation though, will share with my team bc it makes understanding the concept a lot easier lol

@opencode Very intuitive demo

@opencode Nice UI is that openTUI ?

@opencode

@thdxr @opencode I wish opencode app would make cache easier to understand in general. Like if messages hit or miss cache, why (ttl, change of system prompt). Also knowing how much requests cost at message level would be sooo good to self improve.

I use a simpler version of this for my personal agent, I keep the system prompt stable, then inject dynamic context right before the latest user turn. That gives me cache hits up to that point. It also works well for reminders and temporary instructions, since the context is recent and the LLM is more likely to follow them. The tradeoff is that on the next turn, I pay again for the previous turn’s new content, but I think it’s a nice tradeoff for the simplicity and cache efficiency.

@opencode very interesting

@opencode very interesting. learned something new today

@opencode Cc @gubatron

@opencode I always thought prepending that at the end of the transcript would work

@opencode 😮🔥

@opencode im stealing the term hot context and using it in my next meeting 😂

@thdxr @opencode Urgently add method in app to delete project. Stuck with a dead project path. Cant add new.

@thdxr @opencode Sorry, could you elaborate?

@thdxr @opencode Desktop app doesnt have a project delete button option. I changed project location, now even after app restart it is stuck with old path. I deleted config and session files, didnt work. I tried deleting project using oc server too. This issue is open on gh. Many people requested

@thdxr @opencode Got it! I’ll pass this along to the desktop ppl

@opencode When the hot context itself changes between turns, which parts of the cache do you actually recover on the next request?

@opencode cool implementation. have you run evals on this? i wonder if there are losses from the prompt structure

@opencode @grok 具体是如何实现的

@opencode The cache invalidation boundary is the architecture decision most inference implementations never make explicit. Once you do, cost and latency stop being mysteries and become variables you can control.

@opencode i prefer to let my contexts bust as they please, keeping the hot ones from doing it seems kinda discriminatory

@opencode This works non-deterministically, right? You rely on the model understanding what you want to do with the "context" tags, or understanding that it should change the date when they see several "today is". Not bashing, just wanting to understand! Looks cool :)

@opencode Is it getting hot in here?

@opencode Separating mutable context from cacheable structure is the right split. Most implementations just re-run everything. Isolating what changes from what doesn't — that's correct architecture.

@opencode but what if the diff is large? doesn't keeping both the before and after versions bloat the context?

@opencode does this piecemeal approach have any effect when handling models with mid-thread dynamic tool injection like k3? ( I presume not )

@opencode Very nice!!

@opencode Question modal need new flow.. submit -> keep planning or submit -> build.

@opencode Is this a new gui?

@opencode nice!