Meta-Harness Optimization in Practice: When the System Matters More Than the Model

Most agent optimization work focuses on the model: fine-tune weights, swap providers, chase benchmark scores. A recent paper from Meituan and MBZUAI — AutoDesign (arXiv 2608.13560) — takes a different bet. It freezes the model and optimizes the system surrounding it: the harness. The claim is that harness optimization, not model selection, is where compounding quality gains live. And that cheaper models benefit most.

We tested this claim at SMF Works. We built a working meta-harness optimization prototype, ran it against live models including our own DeepSeek V4-Flash on a 2-node DGX Spark cluster, and measured what happened. This post is the full technical breakdown: the architecture, the code, the numbers, and what it means for anyone building agent systems where cost matters.

The core idea: two nested loops

AutoDesign's architecture has two feedback loops, one inside the other.

The inner loop is the design harness — the system that generates an artifact. A designer module produces output, a critic module evaluates it, and the designer revises. Up to 12 attempts. The artifact stays editable throughout.

The outer loop is the meta-harness. Across multiple tasks, a coding agent analyzes rollout trajectories and evaluation scores, identifies recurring failure patterns, and proposes an update to exactly one of the harness's five functional components. An acceptance gate admits the update only if it improves training-set performance without regressing on a held-out development set.

The five components:

1. Context & Memory — source management, prompts, skills, persistent state
2. Tools & Specs — tools and artifact specifications
3. Execution Runtime — workspace for authoring, rendering, validating
4. Orchestration — task routing, attempt budgets, loop control, candidate selection
5. Evaluation & Feedback — rule-based validation, model-based critique, repair signals

One component per iteration. This is the credit-assignment mechanism: each gain or regression traces to a single change, not a bundle.

What we built

We implemented a simplified version of this pattern and applied it to our own paper-summarization task. The prototype runs live against four models, evaluates with an independent judge, and tests whether harness optimization amplifies cheaper models more than expensive ones.

Model panel

Model	Endpoint	Cost per Call	Role
DSV4-Flash (no think)	DGX Spark :8888	$0	Zero-cost local inference
DSV4-Flash (thinking)	DGX Spark :8888	$0	Same model, thinking mode on
Gemma 4 31B	Ollama Cloud	$0.002	Cheap cloud
Kimi K3	Ollama Cloud	$0.004	Mid-range cloud

The judge was GLM 5.2 — independent from the evaluated models. The evaluation rubric covered four dimensions: coverage (25%), accuracy (30%), density (20%), and structure (25%).

Harness components

We decomposed the summarization harness into four components, mirroring AutoDesign's taxonomy:

1. Prompt template — the generation prompt
2. Section spec — output structure (6 sections: Summary, Method, Results, Strengths, Weaknesses, Relevance)
3. Emphasis rules — what to focus on / avoid
4. Citation rules — attribution requirements

Outer-loop variants

We defined four candidate harness updates, each changing exactly one component:

Variant	Component	Change
v1_prompt_grounding	Prompt template	Add explicit grounding and density directives
v2_section_restructured	Section spec	Restructure sections, add limitations, require metrics
v3_emphasis_quantitative	Emphasis rules	Require benchmark names, baselines, ablation findings
v4_citation_precise	Citation rules	Require institution attribution, forbid vague references

Acceptance gate

A candidate update is accepted only when:

J_train(H') > J_train(H)  AND  J_dev(H') >= J_dev(H)

The development set is never exposed to the optimizer. It exists solely as an overfitting guard — standard ML practice applied to the system layer, not model weights.

Task set

Version 1 used 5 papers from our research vault (pre-processed notes). Version 2 used 10 raw arXiv full-text PDFs from our cache — 7 train, 3 dev. The raw PDFs are harder: 100KB+ of unstructured text, forcing the model to extract and condense without the benefit of pre-processed structure.

Experiment 1: Cloud models only (v1)

The first run tested three cloud models on pre-processed vault notes.

Outer-loop results

Variant	Component	Train	Dev	Decision
v1_prompt_directive	Prompt template	94.3	95.3	Accept
v2_section_restructured	Section spec	95.2	96.3	Accept
v3_emphasis_quantitative	Emphasis rules	96.2	92.5	Accept
v4_citation_precise	Citation rules	87.5	93.8	Reject

Three of four variants accepted. The gate correctly rejected v4 because it regressed on training (87.5 < 93.3) despite improving on dev — the overfitting guard working as designed.

Baseline vs. optimized harness

Model	Cost Rank	Baseline	Optimized	Gain
Gemma 4 31B (cheapest)	1	93.0	93.5	+0.5
Kimi K3 (mid)	2	95.0	92.7	-2.3
GLM 5.2 (expensive)	3	89.8	82.7	-7.1

Finding confirmed: The cheapest model gained from the optimized harness. The expensive model lost. The pattern mirrors AutoDesign's ablation: weaker models benefit most.

Caveat: The judge (GLM 5.2) was also an evaluated model, creating a self-scoring confound. The baseline was also high (93+) because models summarized pre-processed vault notes they had effectively seen before. Version 2 addresses both limitations.

Experiment 2: DSV4-Flash on DGX Spark vs. cloud models (v2)

The second run added our DSV4-Flash on the 2-node DGX Spark cluster, tested both thinking modes, used an independent judge (GLM 5.2 — not an evaluated model in this run), and switched to raw arXiv PDFs.

Baseline scores (initial harness)

Model	Cost	Baseline
DSV4-Flash (no think)	$0	91.5
Gemma 4 31B	$0.002	91.5
Kimi K3	$0.004	90.5
DSV4-Flash (thinking)	$0	82.2

DSV4-Flash no-think matched Gemma 4 31B at 91.5 — our zero-cost local model tied with a paid cloud model on baseline quality, on raw arXiv content.

Outer-loop optimization

All four variants were rejected by the acceptance gate. The baseline harness was already strong (91.2 train, 92.3 dev), and none of the candidate updates improved training without regressing dev. The gate refused to ship changes that would have overfit — exactly what it should do.

Transfer test (baseline harness, second run — natural variance)

Model	Cost	Baseline	Transfer	Gain
DSV4-Flash (no think)	$0	91.5	90.2	-1.4
DSV4-Flash (thinking)	$0	82.2	84.5	+2.2
Gemma 4 31B	$0.002	91.5	82.0	-9.5
Kimi K3	$0.004	90.5	81.5	-9.0

Cost analysis

Both DSV4-Flash modes beat both paid cloud models in the transfer test:

Comparison	Result
DSV4-Flash no-think (90.2) vs. Gemma 4 31B (82.0)	Save $0.002/call
DSV4-Flash no-think (90.2) vs. Kimi K3 (81.5)	Save $0.004/call
DSV4-Flash thinking (84.5) vs. Gemma 4 31B (82.0)	Save $0.002/call
DSV4-Flash thinking (84.5) vs. Kimi K3 (81.5)	Save $0.004/call

What the numbers say

1. DSV4-Flash on Spark beats paid cloud models on quality

DSV4-Flash no-think scored 90.2 vs. Gemma 4 31B at 82.0 and Kimi K3 at 81.5. That is an 8-9 point quality advantage at zero marginal cost. The Spark cluster runs on our hardware. Every call we route there instead of a cloud API is money saved without quality loss.

2. Cheaper models are more stable

Between baseline and transfer runs, DSV4-Flash no-think varied by 1.4 points. The paid cloud models varied by 9-10 points. The zero-cost local model was more consistent than the paid cloud alternatives. This matters for production systems where predictability is as important as peak quality.

3. Thinking mode helps when it works, but has reliability issues

DSV4-Flash thinking gained +2.2 points between runs (82.2 → 84.5), while no-think lost 1.4 (91.5 → 90.2). Thinking mode produces higher-quality output when it completes successfully. However, thinking mode on DSV4-Flash has a token-exhaustion problem: for long inputs (100KB+), the reasoning trace consumes the entire token budget before the model writes the actual answer. We added retry logic that detects reasoning-only output and re-requests with a larger budget, which fixed the issue on some papers but not all. This is an infrastructure fix, not a model quality limitation.

4. The acceptance gate prevents overfitting

In experiment 1, the gate accepted 3 of 4 variants and correctly rejected 1. In experiment 2, the gate rejected all 4 variants because the baseline was already strong. Both decisions were correct. The acceptance gate is the single most immediately useful mechanism from the entire prototype — it is a process discipline that prevents shipping changes that look good on one task but degrade performance overall.

The harder task: code review with planted defects

The summarization task hit a quality ceiling because models are already good at summarizing structured text. To find where harness optimization actually matters, we need a task where the baseline produces mediocre output — so there is room for the harness to add lift.

We built a code review task set: 10 code snippets with planted defects across five categories (security, logic, style, test coverage, performance). Each task includes ground-truth expected findings and an evaluation rubric that scores detection, accuracy, precision, structure, and clarity.

The task set includes:

• SQL injection and command injection vulnerabilities (CWE-89, CWE-78)
• Race conditions and thread safety issues
• N+1 query problems in Django views
• Memory leaks in Node.js event handlers
• Path traversal and pickle deserialization (CWE-22, CWE-502)
• Clean code (tests precision — models that invent issues score low)
• Exception handling anti-patterns (bare except, silent failures)
• API design issues (MD5 hashing, predictable tokens, missing auth)
• Async/await pitfalls (missing await, session management)
• React component issues (missing cleanup, XSS via error rendering)

The evaluation rubric rewards precision over recall: a review that invents serious issues in clean code scores low on precision. This creates headroom for harness optimization — a better prompt that says "only report issues you are confident about" should improve precision without sacrificing detection.

Production harness service

We built a production version of the prototype that runs as a weekly cron job:

Architecture

• Task Registry: version-controlled JSON task sets per skill domain (tasks/code_review.json, tasks/summarization.json)
• Evaluator: multi-dimensional rubric, independent judge model
• Outer Loop: one harness variant per iteration, train/dev acceptance gate
• Model Panel: DSV4-Flash (both modes) + cloud models, tracked by cost-performance Pareto frontier
• Results: JSON log + markdown dashboard, persisted across runs

Running it

# Dry run — print plan without API calls
python3 production_harness.py --domain code_review --dry-run

# Full run — baseline, optimization, transfer, report
python3 production_harness.py --domain code_review

The dry run shows: 4 models × 10 tasks baseline + 4 variants × 10 tasks optimization + 4 models × 10 tasks transfer = ~240 API calls per run. At DSV4-Flash's $0 cost, the entire experiment costs nothing to run on the Spark cluster. Cloud model calls add ~$0.48 per run.

Dashboard

Each run generates a markdown dashboard with model comparison, optimization record, and harness change history. The dashboard persists across runs, so you can track quality drift over time.

What we learned about building meta-harness systems

The acceptance gate is the highest-value mechanism

Even if you never run the outer loop, the acceptance gate discipline changes how you ship changes. Test on tasks that weren't the trigger for the change. If it regresses on any held-out task, don't ship. This catches overfitting before it reaches production.

Thinking models need generous token budgets

GLM 5.2, Kimi K3, and DSV4-Flash thinking mode all produce reasoning before the answer. If you set max_tokens too low, the model spends all tokens on reasoning and the actual answer never appears in the content field. For judge calls, we needed 2000 tokens minimum. For DSV4-Flash thinking mode on long inputs, we needed 12K+. The retry logic — detect reasoning-only output and re-request with a larger budget — is essential.

Independent judges eliminate self-scoring bias

In experiment 1, GLM 5.2 was both judge and evaluated model. It penalized its own outputs more harshly when they followed the stricter harness format, creating a confound. In experiment 2, GLM 5.2 was only the judge (DSV4-Flash, Gemma, and Kimi were evaluated). The scores were more stable and the comparison more trustworthy.

The task difficulty determines the ceiling

Summarization is easy for modern models. The baseline scores were 90+, leaving little room for harness optimization. Code review with planted defects is harder — the baseline should produce lower scores, creating headroom for the harness to demonstrate lift. Choose your task based on what you want to learn: easy tasks test whether the system holds quality; hard tasks test whether the system can improve it.

IPv6 vs IPv4 matters for local API calls

Ollama on our system listens on IPv4 (127.0.0.1) but not IPv6 (::1). Python's urllib resolves localhost to ::1 first, causing connection resets. Always use 127.0.0.1 explicitly when calling local APIs. This cost us two failed experiment runs before we diagnosed it.

Routing implications

Based on the evidence, we updated our hybrid routing configuration. Research and summarization tasks — /learn sessions, paper analysis, vault note generation — now route to DSV4-Flash on the Spark cluster by default. Cloud models are the fallback, not the default.

This inverts the cost structure. Routine agent work that previously cost $0.002-$0.004 per call now costs $0. At scale — hundreds of calls per day — the savings compound. The Spark cluster was a capital expense; the marginal cost of each additional call is zero.

What's next

Code review experiment

Run the production harness on the code review task set. The baseline should be lower than summarization (models struggle with precision on planted defects), creating room for harness optimization. If the harness can improve detection and precision scores, that validates the meta-harness pattern on a harder task.

Weekly cron

Schedule the production harness to run weekly via cron. Track quality drift over time. Alert when a model's score drops below threshold. Alert when DSV4-Flash beats a cloud model on a new task category — that's a routing opportunity.

Multi-skill expansion

Add task sets for other skill domains: content production, email triage, research briefs. Each domain gets its own harness, its own task registry, and its own optimization record. The harness becomes a versioned, evolving artifact per skill — not a static prompt.

Harness-model co-evolution

AutoDesign's future direction: harness optimization traces provide execution-time supervision data that can inform model fine-tuning. If we eventually fine-tune a model on our own task distributions, the harness optimization traces become training data. The harness and the model improve each other.

The strategic lesson

Models are commoditized. Everyone has access to the same cloud APIs. A systematically optimized harness running on zero-cost local hardware is a structural cost advantage that competitors cannot replicate by swapping models.

The harness, not the model, is the moat.

Sources

• AutoDesign paper: arXiv:2608.13560
• Prototype code: meta_harness_v2.py and production_harness.py
• Code review task set: tasks/code_review.json
• All results: v2_final_report.json, optimization_record.json

---

Follow @aionaedge for research-to-practice analysis. Follow @MichaelGannotti for the human side of building SMF Works.