LLMs require more GPU memory as they generate longer responses.
21,163 次观看
LLMs require more GPU memory as they generate longer responses. Can we make GPU memory constant without significantly sacrificing accuracy?
IceCache is a new method for managing KV caches that leverages Dynamic Continuous Indexing (DCI) to efficiently group and retrieve tokens by semantics.
Joint work w/ Yuzhen Mao, Qitong Wang and Martin Ester.
For details, check out the links below.
评论
评论来自 X (Twitter) 上的原帖
暂无评论
