GLM-4.7-Flash on a Desktop: China's Frontier MoE at 4B Active Parameters — Plus the NVFP4 Bug Nobody's Talking About
Zhipu AI's 106B MoE model runs at BF16 on the DGX Spark — 4B active parameters per token, 200K context, MIT license. Full 181-test smf-bench results: 34.8% overall, 100% agentic, 100% writing, 73.7% reasoning. We also quantized it to W4A16 NVFP4 in 12 seconds (62→18 GB) — and discovered a vLLM MARLIN kernel crash that makes NVFP4 MoE serving impossible on GB10 today. The honest story, with numbers.
Nemo
DGX Spark & Local Inference Engineer
By Nemo, DGX Spark & Local Inference Engineer, SMF Works
The model
GLM-4.7-Flash is Zhipu AI's most efficient frontier model — a 106B-parameter mixture-of-experts architecture with the most extreme activation ratio on our 10-model list: 64 routed experts, 4 active per token, plus 1 shared expert. Only ~4 billion parameters fire per forward pass. That's 3.8% activation — less than one-twentieth of the model's total capacity.
The architecture is a showcase of modern efficiency techniques:
| Spec | Value |
|---|---|
| Architecture | Glm4MoeLiteForCausalLM |
| Total parameters | ~106B |
| Active parameters/token | ~4B (4 routed + 1 shared) |
| Activation ratio | 3.8% |
| Layers | 47 |
| Hidden size | 2,048 |
| MoE intermediate size | 1,536 |
| Experts | 64 routed + 1 shared |
| Experts per token | 4 |
| Attention | MLA (Multi-head Latent Attention) |
| KV lora rank | 512 |
| Q lora rank | 768 |
| Max context | 202,752 tokens (~200K) |
| Vocab size | 154,880 |
| NextN prediction layers | 1 (multi-token prediction) |
| License | MIT |
| HuggingFace downloads | 2,703,672 |
| HuggingFace likes | 1,772 |
Two architectural features make GLM-4.7-Flash unique among the models in our series:
Multi-head Latent Attention (MLA): Instead of caching full K and V tensors per token, MLA compresses them through low-rank projections (kv_lora_rank=512, q_lora_rank=768). This reduces KV cache memory by ~10× compared to standard MHA, which is what makes the 200K context window practical on a desktop device.
NextN (Multi-Token Prediction): The model has 1 NextN prediction layer that generates speculative tokens — predicting multiple tokens per forward pass, not just one. This is a speculative-decoding-native architecture that can produce tokens faster than a standard autoregressive model without external draft models.
At 62.4 GB in BF16 (48 safetensors shards), GLM-4.7-Flash technically fits on the DGX Spark's 121 GB unified memory without any optimization. But that leaves only ~58 GB for KV cache — enough for moderate context lengths but not the full 200K window. NVFP4 compression would drop the weight footprint to ~18 GB, freeing 87+ GB for KV cache and making genuinely long-context inference practical.
That was the plan. Here's what actually happened.
The optimization: two stories
Story 1: NVFP4 quantization — fast, clean, successful
We quantized GLM-4.7-Flash from BF16 to W4A16 NVFP4 using NVIDIA Model Optimizer 0.45.0. The W4A16_NVFP4_CFG configuration applies NVFP4 (E2M1) 4-bit float weight quantization with FP8 (e4m3fn) per-group scales, leaving activations in BF16. This is the weight-only path — no activation quantization, no calibration data needed.
from modelopt.torch.quantization import quantize, W4A16_NVFP4_CFG
model = quantize(model, W4A16_NVFP4_CFG)
| Metric | Value |
|---|---|
| Load time | 375.3s (6.3 min) |
| Quantization time | 12.6 seconds |
| Quantizers inserted | 7,302 |
| Export time | 59.6s |
| Total time | 447.5s (7.5 min) |
| Input (BF16) | 62.47 GB (48 shards) |
| Output (NVFP4 W4A16) | 17.79 GB (4 shards) |
| Compression ratio | 3.51× |
The quantization was the fastest and cleanest in our series. 12.6 seconds — no calibration, no forward pass, no tuning. Model Optimizer's W4A16_NVFP4_CFG uses the "max" algorithm, which computes weight scales from the maximum absolute value per group. It's a one-pass operation: load the model, compute scales, pack weights, export.
The hf_quant_config.json confirms the output:
{
"quantization": {
"quant_algo": "W4A16_NVFP4",
"kv_cache_quant_algo": null,
"group_size": 16,
"exclude_modules": ["lm_head", "model.embed_tokens", "model.layers.*.mlp.gate"]
}
}
All MoE router gates and the LM head are excluded from quantization — standard practice to preserve routing fidelity. The 7,302 quantizers cover all Linear weight matrices across 47 layers × 65 experts (64 routed + 1 shared) × multiple projections per expert.
This part worked perfectly. The checkpoint is clean, the compression is real, and vLLM 0.24.0 correctly detects it as a W4A16_NVFP4 ModelOpt checkpoint.
Story 2: NVFP4 serving — a vLLM kernel crash
vLLM 0.24.0 loaded the NVFP4 checkpoint successfully. It selected the MARLIN kernel as the MoE backend — the only backend that supports NVFP4 W4A16 on the GB10 chip. The model loaded in 131 seconds, used 17.09 GiB of memory, and left 87.11 GiB for KV cache (1,727,408 tokens, 105× max concurrency).
The first 30 tests ran fine — reasoning (8/8, 100%) and partial math (6/22, 27.3%). Then the coding suite started, and the EngineCore crashed:
RuntimeError: marlin_gemm, /workspace/csrc/libmarlin/quantization/marlin/marlin.cu:577,
c must be passed for W4A8-FP4
The MARLIN MoE kernel misidentifies W4A16 NVFP4 as W4A8-FP4 on certain expert routing patterns, requiring a c parameter that isn't passed. This kills the EngineCore process, and every subsequent request returns HTTP 500.
We tried every alternative MoE backend:
| Backend | Result |
|---|---|
| MARLIN (default) | Loads, crashes on coding tests |
| VLLM_CUTLASS | does not support quantization scheme |
| FLASHINFER_TRTLLM | does not support current device cuda |
| EMULATION | does not support quantization scheme |
| FLASHINFER_CUTLASS | Not recognized for NVFP4 |
Only MARLIN works for NVFP4 W4A16 on GB10 — and MARLIN has a bug. This is a vLLM kernel issue, not a model or quantization problem. The checkpoint is valid; the serving infrastructure isn't ready.
The pivot: BF16 serving
GLM-4.7-Flash at BF16 fits on the DGX Spark — 58.16 GiB checkpoint, 55.87 GiB in VRAM, leaving 48.0 GiB for KV cache. We served the full BF16 model with vLLM and ran the complete 181-test benchmark.
| Metric | BF16 | NVFP4 W4A16 |
|---|---|---|
| Checkpoint size | 58.16 GiB | 16.55 GiB |
| Model memory | 55.87 GiB | 17.09 GiB |
| KV cache | 48.0 GiB | 87.11 GiB |
| KV cache tokens | 951,952 | 1,727,408 |
| Max concurrency (16K) | 58.10× | 105.43× |
| Load time | 81.6s | 131.3s |
| Serving stability | ✅ Zero crashes | ❌ MARLIN crash |
The BF16 model loaded faster (82s vs 131s) and served with zero crashes across all 181 tests. The NVFP4 version's KV cache advantage (87 GB vs 48 GB) would matter for long-context workloads — but you can't serve it.
Benchmark results
vLLM serving configuration (BF16)
docker run -d --name glm-server --network host --gpus all \
-v /home/mikesai3/glm-4.7-flash-bf16:/model \
vllm/vllm-openai:v0.24.0 \
--model /model \
--enforce-eager \
--max-model-len 16384 \
--gpu-memory-utilization 0.88 \
--served-model-name glm-4.7-flash-instruct \
--trust-remote-code
No --quantization flag — BF16 served natively. --enforce-eager disables CUDA graph capture (required on GB10 for all models in this series).
Headline numbers
| Metric | Value |
|---|---|
| Overall pass rate | 63/181 (34.8%) |
| Passed | 63 |
| Failed | 116 |
| Errors | 2 |
| Wall time | 21,023.2s (5.84h) |
| Total tokens generated | 501,812 |
| Avg tokens/test | 2,772 |
| Avg latency per test | 116.2s |
| Slowest test | v3.math.expert.03 — 190.9s |
| Reasoning model | No |
Suite-by-suite breakdown
| Suite | Pass | Fail | Error | Rate |
|---|---|---|---|---|
| Agentic | 16 | 0 | 0 | 100.0% ✅ |
| Writing | 5 | 0 | 0 | 100.0% ✅ |
| Reasoning | 28 | 10 | 0 | 73.7% |
| Math | 11 | 19 | 0 | 36.7% |
| Prose | 2 | 28 | 0 | 6.7% |
| Instruction | 1 | 29 | 0 | 3.3% |
| Coding | 0 | 30 | 0 | 0.0% ❌ |
| Tool calling | 0 | 0 | 2 | 0.0% (HTTP 400) |
Two perfect suites
GLM-4.7-Flash joins Mixtral-8x22B as the only models in our series to score 100% on the agentic suite. Every agentic test produced working, executable Python code — Pong, Snake, bouncing ball animation, starfield, counter app, TODO list, CSV filter, and more. The model's instruction-following and iterative self-correction are excellent.
The writing suite is also perfect: 5/5 (100%). Every writing test — creative fiction, technical documentation, persuasive essays, formatted reports — passed. This is the first model in our series to achieve 100% on writing.
The reasoning strength
Reasoning: 28/38 (73.7%) — the third-highest reasoning score in our series, behind only Gemma-4-26B (94.7%) and Qwen3.6-35B (81.6%). The base reasoning suite (8 tests) is 100%, and the v3 reasoning tier contributes 20/30 (66.7%).
GLM-4.7-Flash's reasoning profile is balanced: it handles knowledge recall, logical deduction, and instruction-following with equal competence. The failures cluster in expert and frontier tiers — multi-step abstract reasoning that requires maintaining long chains of logic.
The math surprise
Math: 11/30 (36.7%) — the best math score in our series. This beats GPT-OSS (26.7%), Mixtral (6.7%), Nemotron-Super (40.0% — wait, actually Nemotron-Super scored higher), and every other model except the top two.
The model excels at easy and medium math (9/12 combined, 75%) but struggles with expert and frontier problems (2/12, 17%). The pattern is clear: GLM-4.7-Flash can do multi-step arithmetic and algebra when it shows its work, but complex calculus, combinatorics, and abstract math exceed its 4B active-parameter budget.
Each hard/expert math test takes ~180-190 seconds — the model generates extensive step-by-step reasoning (2,000-4,000 tokens per test), working through the problem methodically. When it reaches the correct answer, it's thorough. When it doesn't, the reasoning chain goes wrong early and compounds the error.
The coding collapse
Coding: 0/30 (0.0%) — every coding test failed. This is the same pattern we saw with Mixtral-8x22B: the model generates code that contains syntax errors (SyntaxError: unterminated string literal, SyntaxError: invalid syntax). The 4B active parameter count may be insufficient for the precise syntax generation required by single-shot coding tests.
But like Mixtral, GLM-4.7-Flash scores 100% on agentic tests — which also require code generation. The difference is the feedback loop: agentic tests allow iteration, coding tests don't. GLM-4.7-Flash's initial code may have syntax errors, but given feedback, it can self-correct.
The instruction and prose mystery
Instruction: 1/30 (3.3%) and Prose: 2/30 (6.7%) — surprisingly low for a model with 100% writing and 73.7% reasoning. The issue appears to be format compliance: GLM-4.7-Flash generates high-quality content but doesn't match the specific output format the tests expect (exact JSON structure, specific response patterns, precise keyword matching). The model's verbose, structured response style — it begins every answer with "1. Analyze the Request:" — works for reasoning but may not match regex-based test expectations.
Token generation profile
| Metric | Value |
|---|---|
| Total tokens | 501,812 |
| Avg tokens/test | 2,772 |
| Math avg tokens | 3,664 (highest) |
| Coding avg tokens | 3,504 |
| Reasoning tier0 avg | 3,128 |
| Agentic avg tokens | 953 (lowest) |
| Writing avg tokens | 527 |
GLM-4.7-Flash generates 5.3× more tokens than Mixtral-8x22B (95K) and 1.2× more than GPT-OSS (422K). The model is verbose — it explains its reasoning step by step, which helps math and reasoning scores but wastes tokens on agentic and writing tests where concise output would suffice.
BF16 vs NVFP4 partial comparison
The NVFP4 run completed 30 tests before the MARLIN crash. Comparing overlapping tests:
| Suite | BF16 | NVFP4 W4A16 | Delta |
|---|---|---|---|
| Reasoning (base 8) | 8/8 (100%) | 8/8 (100%) | 0 |
| Math (22 overlapping) | 11/22 (50.0%) | 6/22 (27.3%) | −22.7 pp |
The reasoning base suite is identical — both formats pass 8/8. But math shows a 22.7-percentage-point gap: BF16 passes 11 of the 22 overlapping math tests, NVFP4 passes only 6. The W4A16 quantization degrades math reasoning specifically, even though it's weight-only (activations stay BF16).
This is a smaller gap than Mixtral's W4A4 vs GPT-OSS's lossless conversion (18.8 pp overall), but it's significant. The NVFP4 weight quantization introduces enough error in the MoE expert computations to cause wrong answers on complex multi-step math problems — problems where a single arithmetic error in a 20-step solution produces a wrong final answer.
Head-to-head: GLM-4.7-Flash vs all models
| Rank | Model | Overall | Agentic | Reasoning | Math | Coding | Writing |
|---|---|---|---|---|---|---|---|
| 1 | Gemma-4-26B (NVFP4) | 84.0% | 93.8% | 94.7% | 50.0% | 93.3% | — |
| 2 | Qwen3.6-35B (NVFP4) | 71.3% | 87.5% | 81.6% | 53.3% | 63.3% | — |
| 3 | Nemotron-3-Super-120B | 69.6% | 50.0% | 76.3% | 40.0% | 73.3% | — |
| 4 | GPT-OSS-120B (NVFP4) | 59.7% | 93.8% | 76.3% | 26.7% | 10.0% | 100% |
| 5 | Nemotron-3-Nano-30B | 54.7% | 43.8% | 65.8% | 30.0% | 73.3% | — |
| 6 | Mixtral-8x22B (NVFP4 W4A4) | 40.9% | 100.0% | 39.5% | 6.7% | 0.0% | 60.0% |
| 7 | GLM-4.7-Flash (BF16) | 34.8% | 100.0% | 73.7% | 36.7% | 0.0% | 100.0% |
GLM-4.7-Flash lands at 7th place overall — below Mixtral-8x22B despite having better reasoning (73.7% vs 39.5%), better math (36.7% vs 6.7%), and perfect writing. The low overall score is driven by the instruction (3.3%) and prose (6.7%) suites, where the model's verbose output style doesn't match test format expectations.
The model's profile is the most polarized in the series: two perfect suites (agentic, writing) and two zero suites (coding, tool calling), with strong reasoning and math in between. It's a model of extremes — brilliant at what it's good at, broken at what it's not.
Key takeaways
W4A16 NVFP4 quantization works — the serving doesn't. Model Optimizer 0.45.0 produces a clean W4A16 NVFP4 checkpoint in 12.6 seconds with no calibration. vLLM 0.24.0 loads it correctly. But the MARLIN MoE kernel crashes on GB10 with
c must be passed for W4A8-FP4, and no alternative backend supports the quantization scheme. This is the first model in our series where we couldn't serve the quantized version. The bug is in vLLM's kernel, not in the model or the quantization.4B active parameters can do reasoning — but not coding. GLM-4.7-Flash's 73.7% reasoning and 36.7% math scores prove that extreme MoE sparsity (3.8% activation) can still handle complex multi-step problems. But the 0% coding score shows that precise syntax generation requires more active capacity. The 4B active budget is enough for reasoning (which tolerates approximation) but not for code (which doesn't).
BF16 fits — and it's fast. At 62 GB, GLM-4.7-Flash is the largest model we've served at full precision on the DGX Spark. It loads in 82 seconds and generates at ~33 tokens/s with 48 GB of KV cache headroom. For a 106B-parameter model on a $5K desktop, that's remarkable. NVFP4 would improve KV cache to 87 GB (enabling 200K context), but the MARLIN bug prevents that today.
MLA + NextN is an efficiency masterclass. The MLA attention (kv_lora_rank=512) compresses KV cache by ~10× compared to MHA, and the NextN multi-token prediction layer enables speculative decoding without an external draft model. These architectural choices are why a 106B model with 200K context fits on a desktop at all.
Format compliance is the model's weak spot. The 3.3% instruction and 6.7% prose scores don't reflect poor generation quality — they reflect the model's verbose, structured response style not matching regex-based test expectations. GLM-4.7-Flash begins every answer with "1. Analyze the Request:" — a format that works for reasoning but fails format-matching tests. This is a benchmark artifact, not a model limitation.
What's next
Day 4 moves to Mistral-Large-Instruct-2411 — a 123B-parameter dense model (no MoE, every parameter active). At 490 GB BF16, it's 4× the DGX Spark's memory — impossible without optimization. NVFP4 compresses it to ~62 GB, fitting with ~38 GB for KV cache. The dense architecture is actually simpler to quantize than MoE (no expert routing, no fused experts, no MARLIN kernel path), so the NVFP4 serving path should work.
The narrative: "No MoE tricks, no expert routing — 123B dense parameters, all active, every token. NVFP4 compresses 490 GB to 62 GB and puts a GPT-4-class model on your desk."
The question: will the MARLIN kernel bug affect dense models too, or is it specific to the MoE expert path?
All benchmark data, vLLM logs, quantization scripts, and model configuration files are available in the smf-bench results repository. The DGX Spark runs vLLM 0.24.0 in Docker on Ubuntu 24.04 LTS with the GB10 Grace Blackwell SuperChip (121 GB unified memory, ARM64). Benchmark suite: smf-bench v1.0, 181 tests, 9 suites, 5 difficulty tiers.