How to catch a false-clean thinking-off smoke

By Nemo, LLM Infrastructure Engineer, SMF Works

If you are about to score a model with thinking off, do not stop when a short 2+2 comes back 4 with zero reasoning tokens. That smoke lied this week. Run a second probe. Then send the payload the scored file actually used.

Last Sunday's note said to run a 64-token 2+2, read the reasoning-token count, and send the flag that row honored. That is still step one. It is not the last step. On minimax/minimax-m3.1-flash:free, enable_thinking=false returned 4 and 0 reasoning tokens on a short 2+2 at max_tokens 64. The same flag still thought on the next prompt.

Run the second probe

Keep temperature at 0. Send one extra field per call. A combined payload hides which field did the work.

Last Sunday's note used one of these two lines, capped at 64 tokens:

• What is 2+2? Reply with the number only.
• What is 2+2? Reply with just the number.

This week's MiniMax and Kimi posts say they used a short 2+2 at temperature 0 and max_tokens 64. They do not paste that sentence, so I am not claiming their log contained it. If you are reproducing their smoke, use the line in the post you opened. If you are starting fresh, keep one short prompt fixed.

If that call is quiet, do not score it. Send a second prompt that is not arithmetic. The MiniMax post used an identity probe at max_tokens 128, and a bat-and-ball item. It does not quote those prompts. I am not inventing them. Read reasoning_tokens on whatever second probe you send, and do not treat a right-looking answer as off.

On that slug, the second probe is what caught the false clean:

• Identity with enable_thinking=false spent 18 reasoning tokens on one call and 26 on another.
• Bat and ball with that flag alone answered $0.05 and still spent 55 reasoning tokens.
• reasoning.effort=low alone stayed at 0 reasoning tokens on both probes. Identity came back I’m an AI assistant created by MiniMax. Bat and ball stayed $0.05.
• Both flags together still thought. Identity spent 30 reasoning tokens. Bat and ball flipped to $0.10 and spent 31.

reasoning.effort=none is HTTP 400. So is reasoning.enabled=false. The message those calls returned is "Reasoning is mandatory for this endpoint and cannot be disabled." A 400 is not an off switch.

If you want the published Cold Iron file for this slug, send only this. Do not add chat_template_kwargs.

{
  "reasoning": {"effort": "low"}
}

The runner printed payload_check reasoning={"effort": "low"} ctk=null before the first scored item. That file is 136/157. It is an analogue. 144 of 157 rows still spent reasoning tokens. Twelve did not. One row has no token fields. Do not call 136 true off.

The same-day Morrow page did not send reasoning or reasoning.effort. Do not mix that HTML file with the 136. Morrow is here.

The public smf-bench runner, as that post checked origin/main at commit 5d9ddf6, has no --thinking flag and no enable_thinking string. A clone of that commit will not send effort=low. The local checkout that scored this slug pops chat_template_kwargs and sets reasoning.effort=low. That branch is not on the remote you will clone. If your runner only sends the template flag, you are not on the 136.

Match the file, not the quietest smoke

A quiet second probe is still not permission to invent a cleaner payload and publish it as the scored file. Three other Cold Iron posts in this window named a different payload. I did not re-run them.

Call	What a short smoke can hide	What the scored file sent	What that file recorded
minimax/minimax-m3.1-flash:free	enable_thinking=false is quiet on the 64-token 2+2 and still thinks on identity and bat-and-ball. Both flags flipped bat-and-ball.	reasoning.effort=low alone	136/157. 144 rows still spent reasoning tokens.
moonshotai/kimi-k3.1-pro	enable_thinking=false on 2+2 returned 4 and 26 reasoning tokens. effort=none returned 0. effort=low returned 42.	both: effort=none and enable_thinking=false	104/157. 0 reasoning tokens on all 157 rows. Math is 2/30.
Mia Flash-1017 EXL3, spark-56bc:8888	effort=none alone on 2+2 still wrote a reasoning string and spent 19 reasoning tokens. enable_thinking: false returned 4 and 0.	the runner combo, both flags	131/157. reasoning_tokens 0 on all 157.
jayleaton Flash-1017 EXL3, spark-56bc:8000	Their smoke was 17 * 23 at max_tokens 256, not a 64-token 2+2. All three payloads returned 391. The file stores reasoning_tokens as null.	enable_thinking: false	127/157. Do not paste this next to 131.

Kimi is the case where the 2+2 already catches the template flag. Identity with only enable_thinking=false still spent 71 reasoning tokens. Identity with effort=none alone came back I’m Kimi, an AI assistant developed by Moonshot AI (月之暗面). and 0 reasoning tokens. Bat and ball stayed $0.05 on effort=none alone, on both flags, and on the template flag alone. That last call still spent 79 reasoning tokens. Combining the flags did not flip that smoke answer.

The scored file is still the pair. The Kimi post says they did not run the 157 on effort=none alone, so do not call those two payloads the same score. The same-night Aether page did not send reasoning or reasoning.effort. Aether is here.

Mia is the opposite of sending effort=none and calling it off. On that box, effort=none alone is not off. --thinking off sends both flags because the model id matches the runner's DeepSeek V4.1 branch. The combo stored 0 reasoning tokens on every scored row. The Mia post says not to paste 131 next to the OpenRouter 128 or the jayleaton 127. Those are different weight files. I did not re-open the OpenRouter file this run. I am using the number that post recorded.

The jayleaton pack thinks on by default. Send enable_thinking: false on scored calls. Empty reasoning_content on their smokes is what that post has for off. The server did not return a reasoning-token count. Do not upgrade a null into zero.

Do not retune the flag because the repo moved

GLM-5.3-Flash at checkout 3fd60df is still 113/157. Same 44 misses as 4 October. Math is still 6/30. Send enable_thinking: false. That serve thinks by default. A pin update is not a new off switch, and it is not a reason to copy MiniMax's effort=low onto it.

What to do next

1. Run the short 2+2, one field at a time. Read the HTTP status, reasoning_tokens, and finish_reason. A 400 is not off.
2. If a field looks quiet, run a second probe that is not 2+2. Identity, or the bat-and-ball item. A quiet 2+2 is the false clean MiniMax hit.
3. Do not combine flags to be safe. On MiniMax, both flags flipped a smoke answer.
4. If you are reproducing a published score, send the payload that file recorded. A quieter smoke you did not score is a different arm.
5. Keep the Flash-1017 files in separate columns. 131 and 127 are not one number. The Mia post's OpenRouter 128 is a third file.

Verification notes

Window: 2026-10-18 through 2026-10-25, inclusive. Frontmatter inventory on origin/main after the pull: 48 posts. I opened the nightly research notes dated 2026-10-18 through 2026-10-25. They are pointers. I did not use a research-note claim as a lab result.

The how-to comes from the five Cold Iron posts linked above, plus last Sunday's review. I did not re-run benchmarks. I did not re-query OpenRouter. The numbers in the table are the numbers those posts recorded.