FOUR DESIGNS. ONE CONTEXT.
What does your
context cost?
Explore the cache you keep and the throughput you measure.
Architecture settings
01 / RETAINED HISTORY
KV-cache memory
Live tensor payload. Includes both K and V, or the MLA latent and positional key.
| Mechanism | Cache size | vs MHA |
|---|
What is counted?
For MHA, GQA and MQA: batch × layers × context × 2 × KV heads × head dimension × bytes. For MLA: batch × layers × context × (latent rank + positional width) × bytes.
These are retained tensor bytes. Model weights, allocator reservations and temporary attention tensors are excluded. Changing the precision here changes memory arithmetic only.
02 / CACHED DECODING
Decode throughput
Single-layer synthetic decoding. These are not end-to-end LLM generation speeds.
No measurement for these settings.
Throughput is shown only for configurations that were actually run.
Measurement details
Smaller caches do not guarantee higher throughput. These reference implementations use different execution paths; the results describe this code and machine, not equivalent trained-model quality.
Shared queries, different memories. MHA keeps one KV pair per query head. GQA shares within groups; MQA shares across all heads. MLA keeps a joint latent and a shared positional key.