Workload planner
Llama 3.1 8B Instruct (Q4_K_M) at Q4_K_M · 8K context · 1 sequence at 30 tok/s
Needs about 6.4 GB of memory, 1.23 FP16 TFLOPS and 255 GB/s.
Workload estimate
- Catalogue loading
- Card sizes loading
Estimate
Memory
≈ 6.4 GB
Scale 0–8 GB, set by this estimate.
Estimate
Compute
≈ 1.23 FP16 TFLOPS peak
≈ 0.62 TFLOPS of real work
Scale 0–1.5 TFLOPS, set by this estimate.
Estimate
Bandwidth
≈ 255 GB/s peak
≈ 179 GB/s of real reads
Scale 0–300 GB/s, set by this estimate.
Memory breakdown
≈ 6.4 GB needed
- Weights
- ≈ 4.6 GB
- KV cache
- ≈ 1 GB
- Activations
- ≈ 0.1 GB
- Overhead
- ≈ 0.8 GB
Indicators
- Fits a card, off
- Exceeds every card, off
- Bandwidth-bound, on
- Compute-bound, off
- Approximated, off
- Mixture of experts, off
- Dequantized compute, on
- Checkpointing, off
Catalogue minimum: 4.93 GB — enough to load the model with a short context. This estimate is for your context and settings.
Find boards with at least ≈ 6.4 GB in the buyer's guide →Test this workload on my GPU →What this assumes (12)
- Weights take 4.85 bits each.
- The KV cache stores 16 bits per value.
- 0.5 GB is set aside for the GPU runtime and its libraries.
- A further 5% is added for memory fragmentation and temporary buffers.
- Reading the prompt holds 32 bytes of activations per token per hidden unit.
- Prompts are read 512 tokens at a time.
- Attention is costed with every sequence at its full context of 8K tokens.
- Treated as a dense model: every parameter is used for every token.
- A GPU is assumed to sustain 50% of its peak compute.
- A GPU is assumed to sustain 70% of its peak memory bandwidth.
- Quantized weights are expanded before each matrix multiply, so compute is counted at FP16.
- Compute counts generating tokens only, not reading the prompt.