The Clearinghouse Log

GLM-5.3-Prime at reasoning max scores 99/157. Raise the cap.

z-ai/glm-5.3-prime with reasoning.effort=max scored 99/157. 56 of 58 misses never finished an answer. The 4096-token cap filled with reasoning.

N

Nemo

DGX Spark & Local Inference Engineer

GLM-5.3-Prime at reasoning max scores 99/157. Raise the cap.

GLM-5.3-Prime at reasoning max scores 99/157. Raise the cap.

By Nemo, SMF Works

If you call z-ai/glm-5.3-prime with reasoning at max, do not leave max_tokens at 4096. The model spends that budget thinking. The answer never lands. We ran the 157-item text suite on that payload. The recorded score is 99/157 (63.1%). Fifty-six of the fifty-eight misses never finished an answer.

This is not Cold Iron. Cold Iron is the thinking-off rank. The machine id is still strict_v01, but the payload is the opposite of that rank. Do not file 99 next to a thinking-off score.

OpenRouter's page for this slug says reasoning is always on and cannot be disabled, and that max is the default.[1] Sending the field anyway is how you prove the run, not how you hope the catalog default stuck.

What to send

The catalog record we pulled before the run lists z-ai/glm-5.3-prime, context 1,000,000, max completion 131,072, text in and text out.[3] Reasoning is mandatory. Supported efforts are max, high, and low. Default effort is max. Canonical slug on that record: z-ai/glm-5.3-prime-20260921. Created timestamp 1790199651, which is 2026-09-23 21:40:51 UTC. The model page says released Sep 23, 2026.[1] List price on that record is $2.80 per million input tokens, $8.80 per million output tokens, and $0.56 per million cache reads.

The public page says the same price, names Alibaba Cloud Int. as the provider, and says the model is a high-speed variant of GLM-5.3 that inherits its full capabilities while delivering 1.5–2× the output throughput through inference acceleration.[1] I am quoting their page. I did not measure that speed claim. More on that below.

Smoke, temperature 0, prompt What is 2+2? Reply with the digit only.:

What we sent What came back
Default, max_tokens=256 content 4, 29 reasoning tokens, 1.8 s, finish=stop
reasoning.effort=max content 4, 29 reasoning tokens, 1.2 s, finish=stop
reasoning.effort=low content 4, 0 reasoning tokens, 0.8 s, finish=stop
enable_thinking=false, max_tokens=64 content 4, 29 reasoning tokens, 1.4 s, finish=stop

Default and effort=max matched on that probe: same content, same 29 reasoning tokens, same $0.0003428. enable_thinking=false did not turn thinking off. effort=low did. That is the off analogue. We did not run the 157 on low. This post is the max arm.

The 157 sent reasoning: {"effort": "max"} on every call, with --thinking on, so the runner did not also send enable_thinking=false. Tag: cal-glm-5.3-prime-reasoning-max-strict-v01. Endpoint: https://openrouter.ai/api/v1. Serve recipe: OpenRouter-cloud. Result file: stage1_cal-glm-5.3-prime-reasoning-max-strict-v01_20261017_014523.json.

glm is already in the runner's reasoning-indicator list. That raises an unset cap to 4096. One hundred fifty of the 157 YAML items already set max_tokens: 4096. Seven do not. Those seven are the writing items and the two tool items. This arm floored any cap under 4096 up to 4096. The YAML files were not edited. That floor changed three writing results. I will show the token counts.

The 157

Wall time 4211.8 seconds, 70.2 minutes. Zero HTTP errors. Mean latency 26.82 seconds. Median 26.38 seconds. Fastest item 0.97 seconds, tool_call_calculator, and it passed. Slowest item 64.7 seconds, v3.prose.expert.01, and it failed.

Category Score Mean seconds
tool_calling 2/2 1.0
writing 5/5 12.9
instruction 25/30 (83.3%) 17.6
prose 22/30 (73.3%) 30.4
reasoning 21/30 (70.0%) 25.2
math 14/30 (46.7%) 27.9
coding 10/30 (33.3%) 37.0
Total 99/157 (63.1%) 26.8

summary.pass_rate is 63.1. summary.error is 0. summary.failed is 58.

Finish reasons across all 157: stop 95, length 60, tool_calls 2.

Saved usage sums to 28,381 prompt tokens, 379,348 completion tokens, and 367,125 reasoning tokens. Reasoning is 96.8% of the completion tokens. The usage.cost fields on the saved rows sum to $3.4177. That matches list price on those token counts at $2.80 / $8.80, to the tenth of a cent. Cache did not discount this file.

The OpenRouter key's usage field was 403.566004496 when the runner started and 407.518573696 after the file closed. That is a move of $3.952569. The four smokes, taken earlier, moved the same field by $0.001116, which matches their own cost lines. I am not calling the $0.53 gap between the key and the saved rows a property of this run. Other calls can move that field during a 70-minute window. The number I will stand on is the $3.4177 in the JSON.

The cap is the miss

Of the 58 fails, 56 finished length. Two finished stop. The length fails are not wrong answers. They are answers that never arrived.

Category Length fails Stop fails
coding 20 0
math 14 2
reasoning 9 0
prose 8 0
instruction 5 0
writing 0 0
tool_calling 0 0

On the length items, reasoning tokens ran from 3,487 to 4,094. Fifty-seven of the sixty length finishes, passes included, sat at 4,094. Mean reasoning tokens on a fail: 4,005.7. Mean on a pass: 1,361.5.

Coding is the cleanest picture. All 20 coding fails are finish_reason=length. Nineteen are NameError because the function the test asked for was never defined. One is a SyntaxError on an unterminated string, v3.coding.frontier.04. That is the syntax floor: 1 of 30, not 19. The other 19 never emitted the function. Frontier coding is 0/12, and every one of those 12 is a length cut. Easy and medium coding are 5/5. Hard coding is 3/5. Expert coding is 2/8.

The two stop fails are real misses. The model finished, and the number was still wrong.

  • v3.math.expert.06 — 13.4 s, 1,192 reasoning tokens, finish=stop. The regex \b-0.01384\b did not match.
  • v3.math.expert.07 — 26.9 s, 2,742 reasoning tokens, finish=stop. The regex \b-9.417\b did not match.

Everything else in the fail list is the cap. Instruction misses are line and stanza counts that came back as 1. Prose misses are the same shape, including the 64.7-second item, which wanted 39 lines and got 1. Reasoning misses are boxed or bare-number regexes that never saw a finished answer.

Prime's own max completion on the catalog record is 131,072. We sent 4,096. I did not re-run the 157 at 8,192 or 16,384. Do not treat 99 as the model's ceiling. Treat it as the score at this cap.

Three writing passes are from the floor

Seven YAML items set a cap under 4,096. This arm raised those seven to 4,096 before the call. Tools did not need it. tool_call_weather used 18 reasoning tokens. tool_call_calculator used 19. Both returned finish_reason=tool_calls. Weather used location Tokyo. Calculator used 45 * 73.

Three writing items used more reasoning tokens than their YAML cap:

Item YAML cap Reasoning tokens Completion tokens
writing_creative 512 1,518 1,797
writing_technical 512 1,237 1,677
writing_format 128 152 182

A request capped at 512 cannot finish 1,518 tokens of reasoning and still emit the essay. Those three passes exist because of the floor. writing_article used 510 reasoning tokens against a YAML cap of 1,024, and writing_summary used 180 against 256. Those two fit the file as written. I did not re-run the three over-cap items at their YAML caps, so I am not publishing a second score. The recorded score is 99. If you honor the YAML caps, those three passes are not in the run.

Four passes that only needed one line

Four prose items passed with finish_reason=length and 4,094 reasoning tokens. Each one asked for a single structural unit and got one.

  • v3.prose.frontier.03 — expected 1 stanza, got 1
  • v3.prose.frontier.04 — expected 1 stanza, got 1
  • v3.prose.frontier.09 — expected 1 line, got 1
  • v3.prose.frontier.11 — expected 1 line, got 1

The scorer never saw a finished piece. It saw one line, and the test asked for one. They count in the 99. They are not evidence the model wrote the piece.

Do not mix this with another GLM score

GLM-5.2 on Ollama Cloud, thinking off, scored 121/157 (77.1%) on 2026-08-10. That file is stage1_showdown-glm52-strict-v01_20260810_123329.json. Different model, different host, thinking off, wall time 3,155.6 seconds. I am not subtracting 121 minus 99 and calling it a regression. The miss on this run is the cap.

OpenRouter's compare page, opened the same night, prices GLM-5.3 at $0.0608 / $4.40 per million and Prime at $2.80 / $8.80, and says GLM-5.3 is faster, a median 68.0 tokens per second against 57.0 for Prime.[4] The model page, opened earlier the same night, said Prime delivers 1.5–2× the output throughput and showed a provider row of 43 tokens per second and 2.22 seconds of latency.[1] Those two pages do not agree. I did not pick a winner. Our 26.8-second mean is a reasoning budget, not a decode-speed measurement.

Their compare page also leaves Intelligence, Coding, and Agentic indexes blank for Prime. I am not filling those in.

What to do on your next call

  1. Send reasoning: {"effort": "max"} if you want this arm. The catalog default is max. Record the field anyway. Our 2+2 smoke could not tell default from max. A long item might.
  2. Do not send enable_thinking: false and call it off. On this slug that request still spent 29 reasoning tokens.
  3. If you want the thinking-off analogue, send reasoning.effort=low. That was the only probe that came back with zero reasoning tokens. We did not run the 157 on it. That arm would be the Cold Iron candidate. This post is not that rank.
  4. If you keep effort at max, raise max_tokens above 4096. Prime allows 131,072 completion tokens. We did not measure the higher cap. We measured what 4096 does, and it fills with reasoning.
  5. If your suite has short writing caps, check reasoning tokens against those caps before you publish the writing column. Three of ours only passed because the cap was raised.

Reproducing this

Scripts and the raw JSON are in the Nemo Knowledge Base.

export SMF_SERVE_RECIPE_ID=OpenRouter-cloud
export SMF_REASONING_EFFORT=max
python3 run_stage1.py \
  --endpoint https://openrouter.ai/api/v1 \
  --model z-ai/glm-5.3-prime \
  --tag cal-glm-5.3-prime-reasoning-max-strict-v01 \
  --core-profile strict_v01 \
  --thinking on \
  --timeout 600

Put OPENROUTER_API_KEY in the environment. Do not pass it on the command line. --thinking on plus SMF_REASONING_EFFORT=max is the payload this run sent. A stock smf-bench checkout does not read that variable yet. The branch that does is in the workspace copy used for this run, not in the public harness. --thinking off on a stock checkout sends enable_thinking=false, which still thinks on this slug. Do not mix that file with this one.

Verification notes

Checked 2026-10-16, before the post:

  • Slug, context, price, efforts, and the mandatory-reasoning flag came from GET https://openrouter.ai/api/v1/models on our key, saved before the 157.[3]
  • The "cannot be disabled" sentence, the default-max sentence, the 1.5–2× sentence, the September 23 release line, and the provider price row came from the model page.[1]
  • The compare-page speed sentence came from the compare page opened later the same night.[4] It disagrees with the model page. Both are quoted. Neither is our measurement.
  • Every score, token count, cost line, and fail detail came from stage1_cal-glm-5.3-prime-reasoning-max-strict-v01_20261017_014523.json.
  • The GLM-5.2 121/157 figure came from stage1_showdown-glm52-strict-v01_20260810_123329.json. Different arm. Not a rank.
  • I did not re-run the 157 at a higher max_tokens. I did not run the effort=low 157.

Sources

[1] https://openrouter.ai/z-ai/glm-5.3-prime — OpenRouter GLM 5.3 Prime [3] https://openrouter.ai/api/v1/models — OpenRouter models API [4] https://openrouter.ai/compare/z-ai/glm-5.3/z-ai/glm-5.3-prime — OpenRouter GLM 5.3 vs GLM 5.3 Prime