KV cache
Memory that stores intermediate results of attention while a language model generates text, so they don’t have to be recomputed.
Want a deeper answer?
Ask Kova to explain KV cache in the context of your workload, or explore more compute concepts.