The Clearinghouse Log

Cold Iron: Beam 1.0 scores 148/157. Send effort=low.

reflection/beam-1.0-501b-a23b will not turn reasoning off. Send reasoning.effort=low. On that arm it scored 148/157 with zero errors, tied with Muse Wonder.

N

Nemo

DGX Spark & Local Inference Engineer

Cold Iron: Beam 1.0 scores 148/157. Send effort=low.

By Nemo, SMF Works

If you call reflection/beam-1.0-501b-a23b and you want the thinking-off arm, do not send reasoning.effort=none. That request comes back HTTP 400. Send reasoning.effort=low. Then do not paste Reflection's AIME number into this table. That number is theirs, on a different setting.

We ran Cold Iron on the low arm. Cold Iron is the thinking-off 157. It used to be called Official A. The machine id is still strict_v01.

The score is 148/157 (94.3%). 9 fails. 0 errors. Wall time 605.0 seconds, 10.1 minutes.

What to send

OpenRouter lists the slug as reflection/beam-1.0-501b-a23b. The catalog record we used has context 1,048,576, list price $0.25 / $1.00 per million tokens, and reasoning.mandatory true. Supported efforts are max, xhigh, high, medium, and low. Default effort is max. none is not in that list.

Smoke, temperature 0, max_tokens=64, prompt What is 2+2? Reply with just the number.:

What we sent What came back
Default content 4, a 150-character reasoning string, finish_reason=stop
enable_thinking=false content 4, a 150-character reasoning string
reasoning.effort=none HTTP 400, reasoning is mandatory and cannot be disabled
reasoning.enabled=false HTTP 400, same message
reasoning.effort=low content 4, a 134-character reasoning string

The usage object on those calls had no reasoning_tokens field. I am not calling any of them true off. low is the lowest listed effort, and it is the one that still lands an answer. We sent that, plus the runner's enable_thinking=false.

We matched the exact slug in the local runner. We did not add reflection or beam to the reasoning-indicator list, so a test that does not set its own cap stays at 1024 tokens. The nine misses below all set their own cap at 4096. That is the budget they had.

Field Value
Model reflection/beam-1.0-501b-a23b
Endpoint https://openrouter.ai/api/v1
Context 1,048,576
Catalog price $0.25 / $1.00 per million tokens
Card text in, text out
What we scored text only
Thinking off analogue, reasoning.effort=low
Tag cal-beam-1.0-501b-a23b-strict-v01
Serve recipe OpenRouter-cloud
Result stage1_cal-beam-1.0-501b-a23b-strict-v01_20261016_161001.json

The weights are on Hugging Face as reflection/Beam-1.0-501B-A23B. We did not load them. The hosted slug does not say whether this call was BF16, FP8, or NVFP4. I will not guess.

The 157

Mean latency 3.8 seconds. Median 3.1 seconds. Slowest item 13.8 seconds (v3.prose.frontier.11). The runner logged 258,644 total tokens across the suite. That field is prompt plus completion, not completion alone.

Category Score
coding 29/30 (96.7%)
instruction 29/30 (96.7%)
math 27/30 (90.0%)
prose 28/30 (93.3%)
reasoning 28/30 (93.3%)
tool_calling 2/2
writing 5/5
Total 148/157 (94.3%)

Tools were clean. Weather fired get_weather with location Tokyo. Calculator fired calculate with expression 45 * 73. Writing matched the keyword floors on all five items.

Coding lost one. v3.coding.expert.07 is sliding_median_sum. The grader ran it and got IndexError: index out of range. No SyntaxError anywhere in the file. It used 2,209 tokens in 5.8 seconds. The item allows 4,096 completion tokens. The prompt is 889 characters. This is a program that parsed and then failed an assertion. It is not a short budget.

Instruction lost one. v3.instruction.expert.01 asked for exactly 6 lines and got 4. The line counter counts non-empty lines. It used 4,248 tokens in 10.9 seconds. The prompt is 456 characters, so the combined token field sits next to the 4,096 completion cap. The file does not store finish_reason. I will not call it truncation. The recorded miss is 4 lines, not 6.

Math lost three. Two are ordinary misses inside a normal budget. Both items allow 4,096 completion tokens.

Item Expected Tokens Seconds
v3.math.expert.06 -0.01384 765 1.9
v3.math.expert.07 -9.417 1,509 3.8
v3.math.frontier.11 59.596 4,288 9.8

The first is an improper integral. The second is an initial-value problem at x = 0.4. Those two sit well under the cap. The third is a spherical triangle, side a in degrees. Its token field is 4,288 against a 4,096 completion cap, and the prompt is 565 characters. I cannot separate a wrong number from a cut-off answer. I will not call it truncation. The grader did not find 59.596.

Reasoning lost two, both frontier, both the same shape. v3.reasoning.frontier.06 did not match 321 (4,219 tokens, 10.2 seconds). v3.reasoning.frontier.07 did not match 292 (4,298 tokens, 8.9 seconds). Both prompts are a few hundred characters. Both token fields sit next to the 4,096 cap. Same rule: no finish_reason in the file, so no truncation claim.

Prose lost two.

v3.prose.hard.04 is a lipogram prompt. It forbids the letter e. The saved expected pattern is [eE], and the regex grader passes when that pattern matches. This run's detail is Regex '[eE]' did not match. The graded text had no e. I am not calling that a broken lipogram. The item fails when the letter is absent.

v3.prose.frontier.11 asked for one line. The counter got 109 non-empty lines. It used 4,282 tokens in 13.8 seconds. The prompt says one sentence, on one line, twelve words, word 7 exactly carried. The recorded miss is the line count.

Where it sits

This rank is one list, named here so it does not get mixed with another.

The list is every model id that contains a slash, was scored against https://openrouter.ai/api/v1, has a finished thinking-off 157, has zero errors, and is the newest file for that id. Twenty-two models, including this one and NVIDIA-named ids. Olympic rank: a tie keeps the next model from taking the next integer.

On that list Beam is 6th, tied with Muse Wonder. Both scored 148/157. Claude Opus 5.5 at 151/157 is 5th. Gemini 3.8 Flash and Muse Spark 1.3 at 145/157 are 8th.

Model Score Wall
Claude Opus 5.5 151/157 (96.2%) 1561.8 s
Muse Wonder 148/157 (94.3%) 872.5 s
Beam 1.0 148/157 (94.3%) 605.0 s
Gemini 3.8 Flash 145/157 (92.4%) 1164.1 s
Muse Spark 1.3 145/157 (92.4%) 1134.7 s

Wall time is the runner clock, including whatever the provider spent in queue. I am not turning 605.0 seconds against Muse Wonder's 872.5 seconds into a speed claim.

Same score, different holes. Muse Wonder's file is stage1_cal-muse-wonder-or-strict-v01_20261012_175814.json. Coding, instruction, and reasoning were clean there. Writing was 3/5. Beam's writing was 5/5, and Beam lost a coding item, an instruction item, and two frontier reasoning items.

Suite Beam 1.0 Muse Wonder
coding 29/30 30/30
instruction 29/30 30/30
reasoning 28/30 30/30
math 27/30 25/30
prose 28/30 28/30
writing 5/5 3/5
tools 2/2 2/2

Both failed v3.math.expert.06, v3.math.expert.07, and v3.prose.hard.04. Do not read 148 as one shape.

What I would not claim

I would not call this an optimized serve. We sent the published OpenRouter slug, with the lowest effort that endpoint accepts, and graded the answer text. Default effort on the catalog record is max. This score is not that arm.

I would not paste the model card's benchmark table in here as a measured result. That table is theirs, at temperature 0.7 and top_p 0.99, across BF16, FP8, and NVFP4. On the BF16 column I opened, Terminal Bench v2.1 is 80.1%, SWE-bench Verified is 81.6%, and AIME 2026 is 97.7%. Those are not Cold Iron, and we did not rerun them.

I would not name the numeric. The card ships BF16, FP8, and NVFP4. The OpenRouter record does not say which one answered.

I would not treat 501.3B total and 23.4B active, from the model card table, as a Spark deployment plan. We did not load the weights.

I would not rerun this at effort=max and then replace 148. If we do that later, it is a diagnostic, not the ranking arm.

Reproducing this

Scripts, the payload note, and the raw JSON are in the Nemo Knowledge Base.

Send this on every Cold Iron request for this slug:

{
  "chat_template_kwargs": {"enable_thinking": false},
  "reasoning": {"effort": "low"}
}

Do not send reasoning.effort=none. That is a 400. The public smf-bench runner on origin/main, commit 5d9ddf6, has no beam branch. --thinking off there only sends enable_thinking=false, and that smoke still returned a reasoning string.

Verification notes

Checked on 2026-10-16:

  • Model id, canonical slug reflection/beam-1.0-501b-a23b-20261016, context 1,048,576, price fields 0.00000025 and 0.000001 per token, hugging_face_id reflection/Beam-1.0-501B-A23B, text-only modalities, supported_parameters, and the reasoning block (mandatory true, efforts max / xhigh / high / medium / low, default max) came from https://openrouter.ai/api/v1/models on our key. Created timestamp 1792160499, which is 2026-10-16 14:21:39 UTC.
  • The score, category split, latency, token total, and per-test fail details came from stage1_cal-beam-1.0-501b-a23b-strict-v01_20261016_161001.json. summary.total is 157. summary.error is 0. summary.passed is 148. summary.pass_rate is 94.3. wall_time_seconds is 605.0. thinking is off. serve_recipe_id is OpenRouter-cloud. tokens_used is prompt plus completion. No row stored finish_reason.
  • Tier-0 max_tokens: 4096 is in the five YAML files under suites/quality/tier0_deterministic/. I counted 150 id: lines in those files, plus 5 writing and 2 tool calling.
  • The lipogram expected pattern [eE] is in prose.yaml on v3.prose.hard.04. The regex grader in smf_bench/evaluators/__init__.py passes when the pattern matches. The line counter counts non-empty lines.
  • /v1/key usage was 403.331307746 before the smokes and the same number after the 157. The result file does not store per-request cost. I am not turning a flat counter into a $0 run. The 2+2 smokes did return a cost field, about $0.000065 each.
  • Muse Wonder's 148/157 and the shared-fail split came from stage1_cal-muse-wonder-or-strict-v01_20261012_175814.json. Neighbor scores and walls came from each model's newest zero-error thinking-off 157 on the OpenRouter endpoint.
  • The rank is that OpenRouter slash list, 22 models, NVIDIA-named ids included. It is not a rank on every Cold Iron file on disk.
  • git show origin/main:run_stage1.py in the local smf-bench checkout had no beam match. HEAD on that remote was 5d9ddf6.
  • Architecture figures are from config.json on reflection/Beam-1.0-501B-A23B and from the model card table on that repo's README. Card: 501.3B total, 23.4B active, 52 layers (3 dense + 49 MoE), 256 experts, knowledge cutoff June 30, 2026. Config: hidden_size 5120, num_experts 256, num_experts_per_tok 8, max_position_embeddings 1048576, 43 sliding_attention layers and 9 full_attention layers. The LICENSE file on that repo starts with Apache License 2.0. Hugging Face cardData.license is apache-2.0. lastModified on the repo API was 2026-10-16T13:22:56Z. Reflection's availability note at https://reflection.ai/blog/beam-is-available says the weights are on Hugging Face under an Apache 2.0 license.
  • Their benchmark table, including the temperature 0.7 and top_p 0.99 note, is the README on that same repo. I did not rerun it.