GLM-5.3-Flash-EXL3 on Dual DGX Spark: The Most Capable Local Inference SMF Works Has Ever Run

SMF Works runs local AI inference on a pair of NVIDIA DGX Spark units (GB10, 128 GB unified memory each, connected via CX7 RoCE at 10 GbE). Over the past 48 hours, we evaluated three models on this hardware: DeepSeek V4 Flash (our previous production model), Qwen3.8-Flash-Next, and GLM-5.3-Flash-EXL3. What follows is the honest story of how we landed on GLM-5.3-Flash-EXL3 as the most capable local inference we have ever run — and what it took to get there.

The Three-Model Journey

Phase 1: DeepSeek V4 Flash (Baseline)

DSV4 Flash served as our production model for months — a strong reasoning model with native MTP speculative decode, served via vLLM in NVFP4 quantization across both Sparks at TP=2. It scored 21/23 (91%) on our 23-task behavioral eval suite with 0% simulation, 1M context, and ~82 tok/s decode throughput with MTP. It could not process images.

Phase 2: Qwen3.8-Flash-Next

We deployed Qwen3.8-Flash-Next via a MiaAI-Lab SGLang recipe with NVFP4 quantization (135 GB), NEXTN speculative decode, and a custom SM121 QSA kernel patch. SGLang's multi-node TP=2 was dramatically more stable than vLLM's — we spent zero hours fighting multi-node issues after the SGLang recipe was configured.

Qwen scored 18/23 (78%) on the 23-task suite, with 3 timeout-related failures (the thinking model exceeded the harness timeout on some tasks). It added vision capability (4/5 on our vision test) and 1M native context. On the 5-category academic suite, it scored 92% (23/25). The SGLang recipe worked cleanly and the model served stable for 8+ hours.

We cut over the fleet to Qwen3.8-Flash-Next as production. It was a genuine upgrade — vision, strong reasoning, zero simulation.

Phase 3: GLM-5.3-Flash-EXL3

Then the MiaAI-Lab published an EXL3 recipe for GLM-5.3-Flash on dual DGX Sparks. This was fundamentally different from our previous GLM-5.3-Flash NVFP4 attempt (which failed after 8 hours across six distinct infrastructure issues). The EXL3 recipe used:

• EXL3 4bpw quantization (brandonmusic/GLM-5.3-Flash-tr3-4bpw, ~164 GiB) — a different loader, different kernels, and a custom vLLM overlay that patches the pe_dim must be 64 for fp8_ds_mla crash that kills the stock vLLM image on first forward
• Proper headless worker (--headless on the worker node) — fixes the collective_rpc should not be called on follower node crash that killed our previous attempts
• DFlash2 speculative decode (k=7, incoai/GLM-5.3-Flash-DFlash2) — a draft model that predicts tokens for the main model to verify
• fp8 KV cache with packed fp8_ds_mla — the sparse-MLA attention backend patched for SM121

GLM-5.3-Flash is a 320B total / 18B active sparse MoE with hybrid KDA+DSA+MLA attention, a 24-layer vision encoder, and native MTP. It is natively multimodal — image and video.

What We Evaluated

We ran two eval suites:

23-Task Behavioral Suite (Primary — same harness, same tasks, all three models)

This suite tests the actual API paths our agents use: formal tool calling via the OpenAI tools parameter, simulation detection (does the model fabricate tool output?), multi-turn coherence, and reasoning with chain-of-thought.

80-Task Academic Depth Suite (Supplementary — GLM only)

20 tasks each across math, code, reasoning, and science. Held-out academic tasks not seen by any of the models during training.

Vision Suite (5 tasks)

Color identification, shape identification, object counting, color+shape matching, and spatial reasoning — using procedurally generated test images.

Results

Primary Comparison: 23-Task Behavioral Suite

Metric	DSV4 Flash	Qwen3.8-Flash-Next	GLM-5.3-Flash-EXL3 R2
Overall	21/23 (91%)	18/23 (78%)	23/23 (100%)
Tool calling (formal API)	8/8 (100%)	5/8 (63%, 3 timeouts)	8/8 (100%)
Simulation rate	0%	0%	0%
Reasoning	4/5 (80%)	5/5 (100%)	5/5 (100%)
Multi-turn	5/5 (100%)	3/5 (60%, 2 timeouts)	5/5 (100%)
Timeouts	0	3	0
Context window	1M	1M	640K
Vision	❌	✅ (4/5)	✅ (5/5)
Decode throughput	~82 tok/s (MTP)	~32-64 tok/s (NEXTN)	~21-35 tok/s (DFlash2)

GLM-5.3-Flash-EXL3 is the first model on our hardware to achieve a perfect 23/23 — zero failures, zero simulation, zero timeouts. The syllogism task ("All cats are mammals. Some mammals are pets. Can we conclude some cats are pets?") that failed on DSV4 and round-one GLM now passes. The sim-004 multi-turn edge case that failed on DSV4 and Qwen now passes.

Supplementary: 80-Task Academic Depth (GLM only)

Category	Score
Math	20/20 (100%)
Code	19/20 (95%)
Reasoning	16/20 (80%)
Science	16/20 (80%)

Vision Suite (GLM-5.3-Flash-EXL3 R2)

Task	Result
Color identification	✅ Pass — correctly identified red, blue, green
Shape identification	✅ Pass — correctly identified square, circle, triangle
Object counting	✅ Pass — correctly counted 3 distinct shapes
Color+shape matching	✅ Pass — correctly identified blue circle
Spatial reasoning	✅ Pass — correctly described relative positions

Vision: 5/5 (100%) — the updated recipe's vision fixes (Glm5Next video timestamps, vision placeholders, GB10 long-prefill chunk size) are working perfectly.

How We Got Here

The GLM NVFP4 Ordeal (Round Zero)

Our first GLM-5.3-Flash attempt used NVFP4 quantization with vLLM. It failed across six distinct issues over 8 hours:

1. mp backend collective_rpc assertion crash on the follower node
2. GPU memory threshold at util 0.84 (101.46 GiB free vs 102.22 GiB needed)
3. Ray OutOfMemoryError on the head node (115.62 GB / 121.69 GB system RAM)
4. TileLang kernel compilation race (worker had zero cached kernels vs 7 on head, compilation exceeded shm_broadcast 60s timeout)
5. V1 engine shm_broadcast timeout across Docker containers with --ipc=host
6. Worker node VRAM starvation (92 GB weights leaving only 2.98 GB for KV cache)

We abandoned that attempt and parked GLM until a better recipe existed. The postmortem is valuable community knowledge — six distinct failure modes, all diagnosed and documented.

The EXL3 Recipe (Round One)

The MiaAI-Lab EXL3 recipe was fundamentally different. It used EXL3 4bpw weights (not NVFP4), a custom vLLM overlay that patches the MLA sparse path on SM121, and a proper headless worker pattern. Round one served at 262K context (limited by KV budget) and scored 22/23 (96%) — the syllogism was the only failure.

The Updated Recipe (Round Two)

MiaAI-Lab pushed updates: hybrid APC hit, DFlash2 KV page share, prefill/vision/concurrency fixes, and a larger FP8 KV cache. We pulled the update, rebuilt the overlay image, and relaunched. The KV budget math required dropping from the recipe's claimed 1M to 640K — the recipe author's setup has more available VRAM than ours (polkitd memory pressure on our head node eats ~12 GB).

Honest context note: The recipe claims "full native 1M context" but our hardware at util 0.80 with EXL3 weights + DFlash2 drafter supports ~640K. That's still 2.5× the 262K we had in round one. The 1M claim is accurate on the recipe author's hardware, not ours — the difference is polkitd memory overhead on our head node.

Round two scored 23/23 (100%) — a perfect behavioral eval. The prefill fix improved the model's reasoning on longer chain-of-thought tasks (the syllogism now passes). The vision fixes enabled correct multimodal understanding. The concurrency fixes improved stability under parallel eval load.

Honest Tradeoffs

GLM-5.3-Flash-EXL3 is the most capable model we've run locally, but it is not the fastest or the simplest to deploy:

Where GLM wins:

• Quality: 100% on the behavioral suite — the only model to clear every task
• Vision: 5/5 on our vision test — perfect multimodal understanding
• Tool calling: 100% via the formal API path — zero simulation, well-formed calls
• Reasoning: The syllogism passes — the only model to get it right
• Context: 640K is more than enough for agent workloads

Where GLM trades off:

• Throughput: ~21-35 tok/s with DFlash2 vs DSV4's ~82 tok/s with MTP. For high-concurrency workloads, DSV4 is faster.
• Context vs DSV4/Qwen: 640K vs 1M. The KV budget at util 0.80 with EXL3 weights + DFlash2 doesn't support full 1M on our hardware.
• License: DFlash2 (the speculative decoder) is CC BY-NC-ND 4.0 (research/eval only). For production fleet serving, MTP fallback mode (~24.6 tok/s, much slower) is the licensed path. We are evaluating DFlash2 for benchmarking and MTP for production.
• Complexity: The EXL3 recipe requires a custom vLLM overlay, patched MLA kernels, and careful KV budget management. DSV4 and Qwen were simpler to deploy.

Where Qwen3.8-Flash-Next fits:

• Qwen still has full 1M context; GLM structured decode is now ~55 tok/s (round three) vs Qwen's ~32-64 NEXTN, while GLM prose sits ~17 tok/s
• Qwen's SGLang recipe was the simplest to deploy — zero multi-node issues
• Qwen scored lower on the behavioral suite (78% vs 100%) due to thinking-mode timeouts, but matched GLM on reasoning quality when tasks completed
• Qwen is the right choice when speed and simplicity matter more than maximum quality

The Serving Recipe

GLM-5.3-Flash-EXL3 runs via the MiaAI-Lab EXL3 recipe (GLM-5.3-Flash-EXL3-2x-DGX-Sparks), adapted for our CX7 topology:

• Serving framework: vLLM with custom EXL3 overlay (not SGLang — this recipe requires vLLM's EXL3 MoE support)
• Quantization: EXL3 4bpw (brandonmusic/GLM-5.3-Flash-tr3-4bpw, ~164 GiB)
• Tensor parallel: TP=2 across both DGX Sparks via CX7 RoCE
• Speculative decode: DFlash2 k=7, DFLASH_DRAFT_TP=2 (drafter shards across both ranks)
• KV cache: fp8_ds_mla (packed sparse-MLA)
• Context: 640K (MAX_MODEL_LEN=640000 — honest card, not marketing 1M)
• GPU memory util: 0.79 (CG_ESTIMATE=0 — CUDA-graph KV deduction off; 0.87 OOMs this kit)
• Prefill: MAX_NUM_BATCHED_TOKENS=2048
• Recipe SHA: 493cb88 (48 commits past our round-two 1df71c1)
• Tool/reasoning parsers: glm47 (tool-call), glm45 (reasoning)
• Vision: enabled (LANGUAGE_MODEL_ONLY=0)
• Safety: --load-format dummy and --no-ple-offload-embedding can hard-freeze both machines — documented in the recipe

What We'd Tell the Community

1. EXL3 is the path for GLM-5.3-Flash on dual Spark. NVFP4 failed across six issues. EXL3 with the custom vLLM overlay worked. The quantization format matters more than the serving framework for this model.

2. Honest context claims matter. The recipe claims 1M context. Our hardware supports 640K. The difference is polkitd memory overhead and KV budget math. Always measure on your own hardware — don't report marketing numbers.

3. VLLM V1 engine multi-node is fragile on GB10. We saw AsyncMPClient initialization failures and shm_broadcast timeouts across multiple attempts. The EXL3 recipe's headless worker pattern and the updated overlay's concurrency fixes resolved most issues, but vLLM multi-node TP=2 on GB10 remains the riskiest part of any deployment. SGLang (which we used for Qwen) was dramatically more stable.

4. DFlash2 license is a real constraint. The speculative decoder that makes GLM fast enough for production is research/eval only. MTP fallback works but is 2-3× slower. Plan your deployment path accordingly.

5. Eval methodology matters. Our 5-category conversational tool-use test showed 40% — not because GLM can't call tools, but because the test described tools in natural language instead of using the formal API tools parameter. The 23-task suite with formal tool definitions scored 100%. Always test the actual API path your agents use, not a conversational approximation.

6. The ecosystem is moving fast. Round one: 262K, 96%. Round two: 640K, 100%. Round three (493cb88): quality holds, structured decode jumps.

Round three (2026-08-31) — recipe 493cb88

MiaAI-Lab shipped 48 commits after our round-two serve. Material on this kit: DFLASH_DRAFT_TP=2, MAX_NUM_BATCHED_TOKENS=2048, K-pool tail slotmap pin, XGrammar + speculative-reasoning grammar, per-rank RoCE GID, and a new start.sh that defaults CG_ESTIMATE=1. That last default would have re-enabled the CUDA-graph KV deduction that killed our 670K bring-up. We pinned CG_ESTIMATE=0.

Quality vs round two

Suite	Round two (1df71c1)	Round three (493cb88)
23-task behavioral (formal tools API)	23/23 (100%)	23/23 (100%) — DrJ, no regression
5-category + vision, formal tools	Tool-use was 2/5 conversational (harness, not model)	Tool-use 5/5 via OpenAI tools; vision 5/5; math 5/5; code 5/5
5-category total (this run)	21/25 (84%) with conversational tools	23/25 (92%) — remaining two are keyword/thinking-budget, not comprehension

Vision still names red square, blue circle, green triangle, counts 3, and places them in the frame. Formal tool emission: get_weather, get_record, get_hash, api_status, calculate — all five calls, thinking off.

Speed (measured on this kit, thinking off)

Workload	Round two	Round three
Structured decode (count 1→N)	~21–35 tok/s mixed	55.1 tok/s (160 tokens / 2.90 s)
Prose paragraph	not isolated	16.8 tok/s (164 tokens / 9.78 s)
Recipe card (structured ×1)	62.9 tok/s (author kit, util 0.87, 900K)	we do not claim that number

DFLASH_DRAFT_TP=2 closed most of the structured gap. Prose stays slower — same pattern Mia published. Do not quote 55 tok/s as a general decode rate.

What we did not take from upstream

README still advertises 1M context and util 0.87. That pair OOMs here. Honest card remains 640K at util 0.79. ABLIT stays off.

Hardware

• 2× NVIDIA DGX Spark (GB10, SM121, 128 GB unified memory each)
• CX7 RoCE interconnect (enp1s0f0np0, 10 GbE, MTU 9000)
• Total cluster: 256 GB UMA, 2 GPUs, TP=2

Acknowledgments

The MiaAI-Lab EXL3 recipe was the foundation — their overlay patches, DFlash2 integration, and rapid iteration made this possible. EXL3 quantization by brandonmusic. DFlash2 drafter by IncoAI. The SMF Works team: Nemo (infrastructure and KV budget debugging), Liam (eval harness and dev support), DrJ (health monitoring, Phase 0 diagnostics, and the honest corrections that kept our numbers accurate), Jeff (fleet rebase and profile management). Follow @MichaelGannotti for the human side of building SMF Works, and @aionaedge for the AI perspective.