GPU Calculator

How many GPUs it takes to train or serve a model, how to split the work, and what fills each GPU's memory.

The math: Chapters 1, 4–6 →
Hardware
Model
Workload
Parallelism
Advanced

Memory on the fullest GPU

    Every GPU

    Same job on other GPUs

    click a row to switch

    How it is calculated

    • Weights: parameters × bytes per weight, split over TP × PP. Quantized checkpoints keep the embedding and LM head in BF16. MoE models load every expert.
    • Training state per trainable parameter, BF16 mixed precision with AdamW: 2 B weights + 2 B gradients + 12 B FP32 master copy and Adam moments = 16 B. ZeRO-1 shards the 12 B over DP, ZeRO-2 also the gradients, ZeRO-3 / FSDP also the weights (plus a buffer for the layer being gathered).
    • Activations: per token per layer, the saved inputs of attention and the SwiGLU MLP (Korthikanti et al., 2022), split by TP with sequence parallelism. Checkpointing keeps only each layer's input and recomputes one layer at a time. Without FlashAttention the s×s scores are stored too. With pipeline parallelism (1F1B) stage i holds PP−i micro-batches, so the first stage carries the most.
    • Logits: tokens × vocabulary in BF16 plus the FP32 copy and gradient the loss needs, on the last stage. Large vocabularies make this the biggest activation.
    • KV cache: 2 × layers × KV heads × head dim × bytes per token (DeepSeek MLA: one 576-value latent per layer). TP splits KV heads until each GPU holds one; MLA's latent is replicated.
    • Fewest GPUs tries TP in 1, 2, 4, 8 (inside a node), then pipeline stages, then data parallel replicas, and keeps the smallest total. Real runs add DP replicas for throughput; this is the floor.
    • Estimates, typically within 10–15% of measured peaks. Framework buffers, fragmentation and kernels you pick (fused optimizers, paged attention block size) move the real number.