Coding BenchmarkVerified 44 days ago
Local Model Code Generation Benchmark
Can local models keep up with cloud APIs on everyday coding tasks? We tested Qwen, Llama, and Mistral against Claude.
Qwen 3.5:9b via Ollama, Llama 3.1 8B via Ollama, Mistral Small via Ollama, Claude 3.5 Haiku via APIVarious local + cloud2026-06-16
Winner:Claude 3.5 Haiku via API
Results dashboard
Overall score
Claude 3.5 Haiku via API
88
Qwen 3.5:9b via Ollama
79
Llama 3.1 8B via Ollama
71
Mistral Small via Ollama
64
Time to complete
Claude 3.5 Haiku via API
12
Qwen 3.5:9b via Ollama
18
Llama 3.1 8B via Ollama
21
Mistral Small via Ollama
24
Tokens used
Claude 3.5 Haiku via API
42,000
Qwen 3.5:9b via Ollama
0
Llama 3.1 8B via Ollama
0
Mistral Small via Ollama
0
Approximate cost
Claude 3.5 Haiku via API
$0.35
Qwen 3.5:9b via Ollama
$0.00
Llama 3.1 8B via Ollama
$0.00
Mistral Small via Ollama
$0.00
Results table
| Agent | Score | Time | Cost | Pass |
|---|---|---|---|---|
| Claude 3.5 Haiku via API | 88 | 12m | $0.35 | Pass |
| Qwen 3.5:9b via Ollama | 79 | 18m | $0.00 | Pass |
| Llama 3.1 8B via Ollama | 71 | 21m | $0.00 | Pass |
| Mistral Small via Ollama | 64 | 24m | $0.00 | Fail |
The task
Five representative coding tasks, from a simple utility function to a small refactor with tests. Each model/agent ran the same prompt set.
Scoring rubric
| Criterion | Weight | Max points |
|---|---|---|
| Correctness | 35% | 35 |
| Test quality | 25% | 25 |
| Speed | 15% | 15 |
| Cost | 15% | 15 |
| Code clarity | 10% | 10 |
Methodology
- Same prompts, no follow-ups.
- Local models ran on an RTX 4090 with Ollama.
- Cloud runs used the same agent framework with Claude 3.5 Haiku.
- We scored on first-pass correctness, not after repeated refinement.
Key findings
- Claude 3.5 Haiku is still the practical winner for quality and speed, but local models are closing the gap for routine work.
- Qwen 3.5:9b is the best local option we tested for general coding. It is good enough that many teams can default to it for private tasks.
- Llama 3.1 8B handles simple functions well but loses coherence across multiple files.
- Mistral Small is fast but less reliable. Best for drafts and exploration.
Honest caveats
- Hardware matters. A slower GPU changes the time scores significantly.
- We tested coding only. Reasoning, writing, and multi-modal tasks may rank differently.
- Local model quality is improving quickly; this benchmark will age.
Cost note
Cloud cost per task was small but real. Local cost is hardware amortization plus electricity. For high-volume work, local wins. For sporadic work, cloud is simpler.