跳过导航
LLMs require more GPU memory as they generate longer responses.
0:00 / 0:13

LLMs require more GPU memory as they generate longer responses.

Ke Li 🍁

@KL_Div6481 位关注者

在 X 上查看

21,163 次观看

LLMs require more GPU memory as they generate longer responses. Can we make GPU memory constant without significantly sacrificing accuracy? IceCache is a new method for managing KV caches that leverages Dynamic Continuous Indexing (DCI) to efficiently group and retrieve tokens by semantics. Joint work w/ Yuzhen Mao, Qitong Wang and Martin Ester. For details, check out the links below.

评论

评论来自 X (Twitter) 上的原帖

暂无评论