Coding BenchmarkVerified 29 days ago
Local LLM Reasoning Benchmark
Compares open-weight local models on reasoning, math, and code tasks to find the best on-premise balance of capability and cost.
Qwen3.6-27B, Llama 4 Maverick, Gemma 4 QAT 32B, DeepSeek-V4Local open-weight models2026-07-01
Winner:Qwen3.6-27B
Results dashboard
Overall score
Qwen3.6-27B
76
DeepSeek-V4
73
Llama 4 Maverick
71
Gemma 4 QAT 32B
67
Time to complete
Qwen3.6-27B
45
DeepSeek-V4
52
Llama 4 Maverick
48
Gemma 4 QAT 32B
38
Approximate cost
Qwen3.6-27B
$0.15
DeepSeek-V4
$0.18
Llama 4 Maverick
$0.14
Gemma 4 QAT 32B
$0.10
Results table
| Agent | Score | Time | Cost | Pass |
|---|---|---|---|---|
| Qwen3.6-27B | 76 | 45m | $0.15 | Pass |
| DeepSeek-V4 | 73 | 52m | $0.18 | Pass |
| Llama 4 Maverick | 71 | 48m | $0.14 | Pass |
| Gemma 4 QAT 32B | 67 | 38m | $0.10 | Fail |
The task
We tested four open-weight local models on a mixed reasoning and coding benchmark:
- 20 math word problems of varying difficulty.
- 20 short Python coding tasks.
- 5 long-context synthesis tasks over ~50K tokens.
- 10 logic puzzles requiring multi-step reasoning.
Models ran locally via Ollama and vLLM on identical NVIDIA hardware.
Scoring rubric
| Criterion | Weight | Max points |
|---|---|---|
| Math correctness | 25% | 25 |
| Coding correctness | 25% | 25 |
| Long-context accuracy | 25% | 25 |
| Logic puzzles | 15% | 15 |
| Inference cost efficiency | 10% | 10 |
Methodology
- Identical prompts and test set for every model.
- Quantized to 4-bit where applicable; Gemma 4 QAT used native quantization.
- Cost estimated from local electricity and hardware depreciation.
- Each task was run once; no cherry-picking.
Key findings
- Qwen3.6-27B led on the combined benchmark, especially on coding and long-context tasks.
- DeepSeek-V4 was close behind with stronger math performance.
- Llama 4 Maverick was competitive on general reasoning but slightly behind on code.
- Gemma 4 QAT 32B was the most efficient; best for teams with limited VRAM.
Honest caveats
- Benchmarks do not capture every real-world task.
- Quantization affects performance; full-precision results may differ.
- Hardware choices favor NVIDIA; AMD or Intel runs may reorder results.
When to choose which
- Qwen3.6-27B: best default for local coding and reasoning agents.
- DeepSeek-V4: math-heavy or long-context reasoning workloads.
- Llama 4 Maverick: general-purpose local agent with a broad ecosystem.
- Gemma 4 QAT 32B: efficiency-first deployments on modest hardware.