How do we cache prompts with LLMs? Interview with Tanishq Singh. This is a real interview question from a big tech company, asked to a candidate in their technical interview round. The video explains the answer in roughly 6 minutes. 00:00 Question - Prompt Caching with LLMs 00:39 Exact... Query Caching 02:04 Semantic Caching 04:14 Tradeoffs 05:52 Blooper :p AI engineering program for software engineers: #LLMs #AI #InterviewReadyshow more

Is it right answer .. for the question .. question was how are you caching.. ?? .. But answer was how caching works.. only extra point added was about age out and LRU..

hi gaurav, can you please update your ai engineering Course on interview ready also, so far loving your system design simplified course will go for ai course next

have you developed any production grade agent?

In most of the cases no one stores the prompt embedding to do a redis data send, it happens with pre-fix caching(KV) which are stored near to the GPU on premise, such that no two people get same responses even at enterprise level, and whatever places has cache, they use raw high-speed memory and exact-prefix hashing right on their GPU clusters, they don't use vector similarity to give the exact same response, as on enterprise level vector similarity can't be trusted. @gkcs_

Hey gaurav, I really enjoy your informative videos. Was going thru above link, What is the fees for the program?

semantic caching looks so clean on a whiteboard. in prod it's the thing that quietly serves the wrong cached answer and nobody notices for a week.

Yes caching great for repeated tasks wanting the same information

prompt caching breaks fast when user-specific context changes inside the supposedly stable prefix

Wish you taught me the whole 4 year engineering course. 🥹💯🙏

One nuance worth making explicit: cache hit rate is a product metric, not just an infra metric. Stable system-prompt prefixes, deterministic tool schemas, and avoiding volatile fields can matter as much as cache size.

What a neat way of answering @gkcs_ .

One subtle issue with semantic caching is that people often optimize the similarity threshold instead of validating correctness. Threshold tuning changes the precision recall tradeoff, but it doesn’t solve incorrect cache hits.

Wonderful explanation. My question is at what point does having a cache becomes costlier than an LLM? What if I have SLMs in this case? At what point of LLM size, can we avoid cache completely?

Naive

why not use llm cache?

KV cache

This is a fantastic question to ask! It's great that they're focusing on practical aspects like prompt caching. Thanks for sharing!

Prompt caching is one of those topics that sounds trivial until you realize cache invalidation across distributed inference servers is its own nightmare. Semantic similarity based caching sounds great until two nearly identical prompts need completely different outputs.

现象背后有更深的逻辑。高手和普通人的区别，就在于愿不愿意多问几个为什么。大多数人停在第一个答案，真正的高手会一直追问到根因。

Really useful

prompt caching dropped our API costs 40% almost overnight. if you have a heavy system prompt you're repeating every call and you're not caching, that's just money you're leaving on the table.