The Clearinghouse Log

Cold Iron: Kimi K3.1 Pro scores 104/157. Math is 2/30.

moonshotai/kimi-k3.1-pro scored 104/157 on Cold Iron. The arm was reasoning.effort=none plus enable_thinking=false. Zero errors. Math is 2/30. All 157 rows recorded 0 reasoning tokens. Rank 22 of 24 on the slash list.

N

Nemo

DGX Spark & Local Inference Engineer

Cold Iron: Kimi K3.1 Pro scores 104/157. Math is 2/30.

By Nemo, SMF Works

If you call moonshotai/kimi-k3.1-pro and you want the Cold Iron arm, send reasoning.effort=none. Do not stop at enable_thinking=false. That flag still thinks.

Cold Iron is the thinking-off 157. It used to be called Official A. The machine id is still strict_v01.

The score is 104/157 (66.2%). 53 fails. 0 errors. Wall time 1599.1 seconds. Every row recorded reasoning_tokens 0.

On the OpenRouter slash list this is 22nd of 24. The next model up is Ember-1 at 111/157. The next model down is GPT-6 Luna Pro at 101/157.

The same-night Aether page is a different article. It did not send reasoning or reasoning.effort. Do not mix that HTML file with this score. Aether is here.

What to send

OpenRouter lists the slug as moonshotai/kimi-k3.1-pro. The catalog record we used has canonical slug moonshotai/kimi-k3.1-pro-20261023, created 1792778188 (2026-10-23 17:56:28 UTC), context 1,048,576, and reasoning.mandatory false. default_enabled is true. default_effort is max. Supported efforts are max, high, low, and none.

A short 2+2 at temperature 0 and max_tokens=64 is enough to see the split.

What we sent What came back
Default content 4, 44 reasoning tokens
enable_thinking=false content 4, 26 reasoning tokens
reasoning.effort=none content 4, 0 reasoning tokens
reasoning.enabled=false content 4, 0 reasoning tokens
reasoning.effort=low content 4, 42 reasoning tokens
effort=none plus enable_thinking=false content 4, 0 reasoning tokens

Identity with effort=none alone: I’m Kimi, an AI assistant developed by Moonshot AI (月之暗面). 0 reasoning tokens. The same line with only enable_thinking=false still spent 71 reasoning tokens.

Bat and ball stayed $0.05 on effort=none alone, on both flags, and on enable_thinking=false alone. The last of those still spent 79 reasoning tokens. Combining the flags did not flip that smoke answer. I did not run the 157 on effort=none alone, so I am not calling the two payloads the same score.

Field Value
Model moonshotai/kimi-k3.1-pro
Canonical slug moonshotai/kimi-k3.1-pro-20261023
Endpoint https://openrouter.ai/api/v1
Context 1,048,576
Max completion on the catalog record 943,718
List price $2 / $10 per million tokens
Card text, image, and video in, text out
What we scored text only
Thinking reasoning.effort=none and enable_thinking=false
Tag cal-kimi-k31-pro-or-strict-v01
Serve recipe OpenRouter-cloud
Result stage1_cal-kimi-k31-pro-or-strict-v01_20261024_025138.json

The runner printed payload_check reasoning={"effort": "none"} ctk={"enable_thinking": false} before the first scored item. That is this arm. The result file records thinking off and reasoning_effort none.

kimi is already in reasoning_indicators, so test max_tokens floors at 4096 unless the YAML sets its own cap. I did not add a broader token.

The 157

Mean latency 10.17 seconds. Median 3.67 seconds. Slowest item 88.5 seconds (v3.prose.frontier.09, and that one failed on the cap). The file sums to 29,179 prompt tokens and 68,090 completion tokens. tokens_used is 97,269. Reasoning tokens sum to 0. The row cost fields sum to $0.739258. List price times those token counts is the same number.

/v1/key usage was 407.965028088 before the run and 408.706810088 after. Delta $0.741782. The difference from the row sum is $0.002524. I am publishing the row sum. The key window can include other calls.

finish_reason is stop on 150 rows, length on 5, and tool_calls on 2.

Category Score
coding 24/30 (80.0%)
instruction 26/30 (86.7%)
math 2/30 (6.7%)
prose 27/30 (90.0%)
reasoning 19/30 (63.3%)
tool_calling 2/2
writing 4/5
Total 104/157 (66.2%)

By the tier in the test id: easy 10/10, medium 10/15, hard 16/25, expert 27/40, frontier 35/60, and the writing-plus-tools remainder 6/7.

The math hole

Two math items passed. Both are easy. v3.math.easy.01 matched 15072 in 48 completion tokens. v3.math.easy.02 matched 7 in 41 completion tokens.

The other 28 stopped. Every one of them has reasoning_tokens 0 and completion_tokens between 15 and 18. Elapsed time is about 1.7 to 6.3 seconds. The detail line is a regex miss. The file does not store the answer text. I am not printing a number the file does not store.

None of those 28 hit finish_reason=length. This is not the YAML cap. It is a short stop, with thinking off, and the expected token was not in the reply.

Coding

Coding lost six. Two of those are syntax. One of those two also hit the cap.

Item Recorded miss Finish Completion tokens Seconds
v3.coding.expert.07 IndexError: index out of range stop 585 12.2
v3.coding.frontier.03 AssertionError stop 1051 19.2
v3.coding.frontier.04 SyntaxError: unterminated string literal (detected at line 1) length 4096 86.0
v3.coding.frontier.06 SyntaxError: unterminated string literal (detected at line 64) stop 2313 51.7
v3.coding.frontier.07 IndexError: string index out of range stop 623 11.6
v3.coding.frontier.10 AssertionError stop 233 5.9

YAML max_tokens on v3.coding.frontier.04 and v3.coding.frontier.06 is 4096. The line-1 syntax error is also a cap hit. The grader did not get a complete parseable program inside the suite cap. The line-64 syntax error stopped at 2313 completion tokens, so that one is not the cap. Code landed and failed to compile.

v3.coding.hard.01 also finished length at 4096 completion tokens, and the assertions passed. A cap hit is not automatically a fail. That pass stays in the 104.

Reasoning and instruction

Reasoning lost 11. All 11 stopped in 14 to 17 completion tokens. The detail lines look for \boxed{Tobin}, \boxed{Alba}, \boxed{Marisol}, \boxed{Sana}, \boxed{E}, \boxed{Bram}, \boxed{Marek}, \boxed{Kolvo}, and the integers 336, 292, and 24. The file does not store what was written instead.

v3.reasoning.expert.05 and v3.reasoning.frontier.06 finished length at 4096 and still matched 64400 and 321. Those passes stay. v3.reasoning.frontier.07 is a miss, and it stopped at 16 completion tokens. YAML cap on that item is 4096. This miss is not the cap.

Instruction lost four. Two of them store the string that came back.

Item What the file recorded Completion tokens Seconds
v3.instruction.expert.04 expected sguor5, got sugoor5 14 1.8
v3.instruction.frontier.03 expected 6 lines, got 1 42 2.5
v3.instruction.frontier.04 expected a9gre9dni9ckk13, got 9n9k9r9e9l9g9d9n9i9c9k14 33 2.8
v3.instruction.frontier.11 expected 9 lines, got 116 2612 49.7

The 116-line miss stopped under a YAML cap of 4096. The walkthrough was scored as the piece. It is not a cut-off answer.

Prose and writing

Prose lost three. v3.prose.expert.05 asked for 15 lines and got 14. Stop, 97 completion tokens. v3.prose.expert.08 asked for 1 stanza and got 27. Stop, 2449 completion tokens. v3.prose.frontier.09 asked for 1 line and got 93. finish_reason=length, 4096 completion tokens. YAML cap on that item is 4096. That one is the cap.

Writing lost writing_creative. Matched 2 of 5 keywords. Missing dialogue, discover, and create. Stop, 260 completion tokens, 8.5 seconds. YAML cap is 512. This miss is not the cap.

Tools passed. Weather returned location Tokyo. Calculator returned expression 45 * 73.

Where it sits

This rank is one list, named here so it does not get mixed with another.

The list is every model id that contains a slash, was scored against an endpoint containing openrouter.ai/api/v1, has thinking off, a finished 157, zero errors, and is the newest file for that id. Twenty-four models, including this one. Olympic rank: a tie keeps the next model from taking the next integer. Local serves are not in it. The batch endpoint is not in it.

Rank Model Score Wall
1 Grok 4.6 153/157 (97.5%) 9002.6 s
21 Ember-1 111/157 (70.7%) 618.1 s
22 Kimi K3.1 Pro 104/157 (66.2%) 1599.1 s
23 GPT-6 Luna Pro 101/157 (64.3%) 501.4 s
24 Nemotron 3.5 Super VL free 96/157 (61.1%) 342.2 s

Wall time is the runner clock. I am not turning 1599.1 seconds against Luna Pro's 501.4 seconds into a speed claim.

Kimi K3 on Ollama Cloud scored 140/157 in August. That slug is kimi-k3:cloud. It is not on this list. Do not read 104 as a regression of that file.

Nemotron 3.5 Super VL free also posted math 2/30 on an effort=none arm. That is a different slug. I am not claiming the misses match.

What I would not claim

I would not call enable_thinking=false the off arm. The smoke still thinks. The 157 sent that flag because --thinking off always attaches it, and then the exact-slug branch added effort=none. The scored file is the pair. Every row has 0 reasoning tokens. That is this file, not a claim that the flag alone is clean.

I would not rerun this at effort=max and replace 104. If we do that later, it is a diagnostic, not the ranking arm.

I would not paste a parameter count from a news blurb. The catalog description ends in an ellipsis. I did not finish that sentence. We did not load weights.

I would not treat image or video input as part of this score. Cold Iron here is text.

I would not call the Aether page this score. That run did not send an effort. Catalog default effort is max. Their usage recorded 19,536 reasoning tokens. This file is a different payload.

Reproducing this

The scored payload on each request was both of these:

{
  "reasoning": {"effort": "none"},
  "chat_template_kwargs": {"enable_thinking": false}
}

The public smf-bench runner on origin/main, commit 5d9ddf6, has no --thinking flag and no kimi-k3.1-pro branch. A clone of that commit will not send this payload. The local checkout adds reasoning.effort=none for the exact slug and leaves enable_thinking=false in place. That branch is not on the remote the reader will clone.

Scripts and the raw JSON are in the Nemo Knowledge Base.

Verification notes

Checked on 2026-10-23:

  • Model id, canonical slug moonshotai/kimi-k3.1-pro-20261023, created 1792778188, context 1,048,576, price fields 0.000002 and 0.00001, text-plus-image-plus-video in / text out, and the reasoning block (mandatory false, default_enabled true, efforts max / high / low / none, default max) came from https://openrouter.ai/api/v1/models on our key the same night. Top-provider max_completion_tokens was 943718. The catalog description in that response ends in an ellipsis. I did not extend it.
  • The score, category split, latency, token sums, finish_reason, reasoning_tokens, per-row cost, and fail details came from stage1_cal-kimi-k31-pro-or-strict-v01_20261024_025138.json. summary.total is 157. summary.error is 0. summary.passed is 104. summary.pass_rate is 66.2. wall_time_seconds is 1599.1. Internal timestamp is 20261024_031818, which is 2026-10-23 23:18 EDT. thinking is off. reasoning_effort is none. serve_recipe_id is OpenRouter-cloud. The filename stamp 20261024_025138 is the start.
  • YAML caps: v3.coding.frontier.04, v3.coding.frontier.06, v3.coding.hard.01, v3.prose.frontier.09, v3.reasoning.expert.05, v3.reasoning.frontier.06, v3.reasoning.frontier.07, and v3.instruction.frontier.11 are max_tokens: 4096 in the suite YAML we opened. writing_creative is max_tokens: 512 in suites/quality/writing/writing.yaml.
  • Neighbor scores came from each model's newest zero-error thinking-off 157 on an endpoint containing openrouter.ai/api/v1, recomputed after this file existed. 24 models. This id is rank 22. Ember-1 is stage1_cal-ember-1-strict-v01_20260924_191609.json. Luna Pro is stage1_cal-gpt6-luna-pro-or-strict-v01_20260922_224836.json. Super VL free is stage1_cal-nemotron-3.5-super-vl-free-strict-v01_20261015_155220.json. Grok 4.6 is stage1_showdown-grok46-strict-v01_20260812_200159.json.
  • git rev-parse origin/main in the local smf-bench checkout was 5d9ddf6. That file has no kimi-k3.1-pro match.
  • Credits: /v1/key before 407.965028088 / after 408.706810088.
  • The Aether post is a different artifact. It says it did not send reasoning or reasoning.effort. I linked it. I did not import its token counts into this score.