Attention ExplorerMEMORY & THROUGHPUT

FOUR DESIGNS. ONE CONTEXT.

What does your
context cost?

Explore the cache you keep and the throughput you measure.

128 tokens128K tokens
Architecture settings

01 / RETAINED HISTORY

KV-cache memory

Calculated

Live tensor payload. Includes both K and V, or the MLA latent and positional key.

Calculated cache payload at the selected context
MechanismCache sizevs MHA
What is counted?

For MHA, GQA and MQA: batch × layers × context × 2 × KV heads × head dimension × bytes. For MLA: batch × layers × context × (latent rank + positional width) × bytes.

These are retained tensor bytes. Model weights, allocator reservations and temporary attention tensors are excluded. Changing the precision here changes memory arithmetic only.

02 / CACHED DECODING

Decode throughput

Measured · CPU

Single-layer synthetic decoding. These are not end-to-end LLM generation speeds.

Measurement details

Smaller caches do not guarantee higher throughput. These reference implementations use different execution paths; the results describe this code and machine, not equivalent trained-model quality.

Shared queries, different memories. MHA keeps one KV pair per query head. GQA shares within groups; MQA shares across all heads. MLA keeps a joint latent and a shared positional key.