How much VRAM does Mixtral 8x7B need?
Mixtral 8x7B needs about 95.4 GB of GPU memory at FP16 and about 27.9 GB at 4-bit. Pick a precision below to see which ComputeBD AI Computer fits and what it costs per hour in taka.
You need about 95.4 GB of GPU memory to run Mixtral 8x7B at FP16 / BF16.
- Model weights 87 GB
- Conversation memory (KV cache) 0.5 GB
- Runtime & working space 8 GB
NVIDIA H200 is the most affordable AI Computer that fits — ৳1,390/hour, billed by the minute.
| AI Computer | Fits? | Speed (1 user) | Price |
|---|---|---|---|
| NVIDIA L424 GB | Too small |
≈ 7 tokens/s | ৳147/hr |
| NVIDIA A1024 GB | Too small |
≈ 14 tokens/s | ৳196/hr |
| NVIDIA L40S48 GB | Too small |
≈ 20 tokens/s | ৳343/hr |
| NVIDIA A10080 GB | Too small |
≈ 47 tokens/s | ৳638/hr |
| NVIDIA H10080 GB | Too small |
≈ 78 tokens/s | ৳1,047/hr |
| NVIDIA H200141 GB | Fits |
≈ 112 tokens/s | ৳1,390/hr |
| NVIDIA B200180 GB | Fits |
≈ 186 tokens/s | ৳1,652/hr |
| NVIDIA B300288 GB | Fits |
≈ 186 tokens/s | ৳1,880/hr |
Mixtral 8x7B at each precision
| Precision | GPU memory needed | Cheapest ComputeBD AI Computer |
|---|---|---|
| FP32full precision — rarely needed | 189.4 GB | NVIDIA B300 · ৳1,880/hr |
| FP16 / BF16standard — original quality | 95.4 GB | NVIDIA H200 · ৳1,390/hr |
| 8-bitalmost no quality loss | 51.4 GB | NVIDIA A100 · ৳638/hr |
| 4-bit (GGUF / AWQ / GPTQ)small quality loss, ~4× less memory | 27.9 GB | NVIDIA L40S · ৳343/hr |
- Made by
- Mistral AI
- Parameters
- 46.7B (12.9B active per token)
- Max context
- 32,768 tokens
- LoRA / QLoRA fine-tune
- ≈ 102.4 GB / 39.9 GB
- Hugging Face
- mistralai/Mixtral-8x7B-Instruct-v0.1
How the estimate works
- Weights = parameters × bytes per parameter (FP16 = 2 bytes, 8-bit ≈ 1, 4-bit ≈ 0.56).
- KV cache (conversation memory) grows with context length and the number of people at once: 2 × layers × KV-heads × head size × tokens × 2 bytes.
- Runtime: about 1 GB for CUDA plus ~8 % of the weights for working space.
- Fine-tuning: LoRA keeps an FP16 copy of the model and trains ~1 % extra parameters; QLoRA keeps a 4-bit copy; full fine-tuning needs ~16 bytes per parameter. Gradient checkpointing is assumed.
- Speed: generating text for one user is limited by memory bandwidth, so tokens/second ≈ bandwidth × 60 % ÷ model size.
These are good planning estimates, not guarantees: the real figure depends on the software (vLLM, llama.cpp, Transformers…), settings and your data. Leave ~10 % spare.
VRAM needed, model by model
Frequently asked questions
How much VRAM does Mixtral 8x7B need?
About 95.4 GB at FP16 (original quality) and about 27.9 GB at 4-bit, for one user with a 4k context.
Which GPU can run Mixtral 8x7B?
At full quality it fits on an NVIDIA H200 (141 GB), ৳1,390/hour on ComputeBD.
How much VRAM to fine-tune Mixtral 8x7B?
About 102.4 GB with LoRA and 39.9 GB with QLoRA (sequence length 2,048, batch 1, gradient checkpointing). Estimate the time and cost with the fine-tuning cost estimator.
Try it on a real AI Computer
JupyterLab opens in about a minute. Pay in taka with bKash, billed by the minute — stop whenever you like.