What 34 Passing Tests Did Not Prove: Taking the SMF Regulatory Assurance PoC from Green Suite to Evidence Candidate

The SMF Regulatory Assurance proof of concept has 34 passing Python tests. Ruff passes. Strict MyPy passes across all 13 source modules. The deterministic core can take a normalized operation, select applicable rules, resolve layered policy, bind an approval to the exact proposed action, and claim that approval once through an atomic SQLite transition.

The PoC has not passed.

That distinction is the reason for this build log. A green component suite proves specific local properties. It does not prove that Hermes or OpenClaw intercepts a real consequential action, that an adapter hashes the final executable parameters, that a connector executes at most once, that streamed output cannot escape before review, or that SMF Swarm cites only rules selected from the governed release.

Aiona Edge's July 24 architecture and review account explains why SMF moved from a tool-shaped rules feed to a five-plane regulatory control architecture. This follow-up takes the maker's view: what the code now binds, what each test actually proves, what adversarial review found after the suite went green, and what must exist before Aiona can inspect an immutable implementation candidate.

The current status is exact:

The architecture passed peer review for a bounded design-validation experiment. The Python control-plane core is substantive but incomplete. No runtime has passed end-to-end conformance, and no immutable candidate exists.

The review cycle changed the implementation, not just the prose

Michael started with a strategic question: could SMF preserve the regulated behavior developed in Praxis while letting upstream communities maintain general-purpose runtimes such as Hermes and OpenClaw?

My first design said yes, but it made runtime tools carry too much regulatory meaning. It grouped native actions into broad risk classes, treated block-and-escalate too much like staged approval, expected one physical rules feed to serve deterministic enforcement and Swarm analysis, and gave signatures an anti-copying role they cannot provide.

Michael required Aiona to review the proposal before he made a build decision. Her first architecture review accepted the product direction and rejected the implementation model. She identified runtime coupling, Hermes fail-open and outbound-surface gaps, insufficient fidelity to Praxis's exact-action protocol, an ungrounded Swarm path, weak release and provenance contracts, and the false equivalence between blocking an action and preserving it for later approval.

I accepted those findings and rewrote the architecture around normalized operations, deterministic policy, separate enforcement and analysis projections, exact-action approval, explicit assurance levels, and a defined trusted computing base. Aiona then reviewed the revised artifacts and returned:

PASS WITH NON-BLOCKING REVISIONS

That verdict authorized architecture, not implementation. It meant Michael could approve a narrow PoC without another design rewrite. The remaining concerns became implementation gates.

The governed sequence is important:

Stage	Decision	What it established
Liam v1	Rejected for implementation	The strategy had merit, but the policy and runtime abstractions were too weak
Aiona first review	Conditional GO for revision	The experiment needed runtime-neutral operations, stronger approval semantics, provenance, and conformance boundaries
Liam v2	Material architecture rewrite	Five governed planes, dual projections, deterministic evaluation, exact-action binding, and bounded assurance claims
Aiona final architecture review	PASS WITH NON-BLOCKING REVISIONS	The narrow PoC could be built; eight gate families still required code and evidence
Joint Liam-Aiona recommendation	GO for four scenarios only	No customer pilot, production legal reliance, Praxis equivalence, or retirement
Michael authorization, July 24	Go	Build the bounded design-validation PoC
Current implementation evidence review	Not acceptance-complete	Working Python components, open defects, failing coverage, and no runtime conformance
Final exact-SHA review	Not started	There is no committed candidate to review

The current four scenarios remain narrow:

1. require licensed-attorney approval before an external client or court transmission;
2. deny deletion while legal hold applies;
3. deny or condition confidential or injection-tainted external egress;
4. produce one provenance-grounded SMF Swarm legal-analysis report from a separate analysis projection.

Nothing in the current work authorizes a customer deployment or legal reliance.

One operation through the current Python code path

The cleanest way to understand the implementation is to follow one synthetic court transmission through the control plane.

OperationEnvelope
  -> typed fact registry
  -> bounded predicate selection
  -> layered policy resolution
  -> deterministic decision + operation hash
  -> ApprovalBinding in SQLite
  -> licensed-role resolution
  -> atomic one-shot claim
  -> consumed | execution_unknown
  -> minimized evidence record

Every arrow before a real runtime adapter is implemented at the Python-library level. The adapter and connector arrows are not.

1. Normalize the proposed action

models.py defines strict, frozen Pydantic contracts for the operation, destination, data, professional context, workflow, runtime evidence, rules, policies, requirements, and decision.

A proposed external transmission contains regulatory facts rather than a native tool category:

{
  "capability": "external.transmit",
  "effects": {
    "egress": true,
    "persistence": "external_persistent",
    "reversibility": "limited"
  },
  "destination": {
    "channel": "electronic_filing",
    "locality": "external",
    "target_identity_hash": "sha256:..."
  },
  "data": {
    "classification": "privileged",
    "redaction_status": "pending",
    "injection_taint": "clear",
    "content_hash": "sha256:..."
  },
  "professional": {
    "domain": "legal",
    "tenant_id": "tenant-test",
    "matter_id": "matter-test",
    "actor_roles": ["paralegal"],
    "governing_laws": ["US-CT"]
  },
  "workflow": {
    "legal_hold": false,
    "citation_verification": "pending"
  },
  "runtime": {
    "runtime_id": "test",
    "adapter_id": "smf-test-adapter",
    "canonical_args_hash": "sha256:..."
  }
}

This is the portability boundary. A web query containing client facts may have egress=true even though a runtime labels the native tool as a read. A terminal invocation may describe a harmless local read or a destructive evidence deletion. The adapter must map effects and context. Canonical policy never decides from the string terminal, web_search, or send_message alone.

The model tests establish schema strictness. They reject unknown fields, require evidence before redaction can be marked complete, and keep unknown separate from complete. They do not prove that any runtime mapper supplies correct facts.

2. Validate facts and predicates

The versioned fact registry defines 29 allowed paths, seven operators, type and cardinality rules, and budgets for expression depth, node count, and list size.

The predicate engine supports a deliberately closed language:

eq | neq | in | not_in | contains | exists | not_exists

It rejects unknown fact paths, wrong scalar/list semantics, Boolean-to-integer coercion, excessive expression depth, and oversized lists. Missing facts evaluate false unless the rule explicitly asks not_exists.

This is not a universal legal reasoning language. It is a bounded decision language for the four scenarios. No model interprets policy at enforcement time.

3. Compile two projections from one governed source

compiler.py validates and sorts rules and policies, computes a projection recipe identity, and emits two artifacts:

• a compact enforcement projection for deterministic runtime decisions;
• a richer analysis projection for Swarm selection and provenance.

The focused reproducibility test reverses input order and requires byte-identical canonical outputs. Both projections carry the same rule identities and source hashes.

One governed source can therefore support two query shapes without forcing runtime enforcement and LLM analysis to consume the same artifact.

The compiler currently has a referential-integrity hole. It validates rule and policy shape but does not verify that every policy.rule_id exists in the supplied rule set. An adversarial probe compiled a policy referencing urn:smf:rule:not-supplied with zero rules. The resulting evidence chain can name a rule that was never in the release.

The repair is straightforward in concept: reject dangling rule IDs, duplicate identities, and inconsistent release metadata before projection emission. The regression must prove the compiler fails before it writes either projection.

4. Resolve layered policy monotonically

policy.py orders controls from strongest to weakest operating layer:

binding_authority
smf_safety_floor
customer
matter
user

A lower-layer allow cannot erase an existing restriction. Customer policy must bind to the current tenant. Hard denial dominates. Compatible requirements accumulate.

The current requirement merger takes the largest approval count, unions required roles, and takes the shortest TTL. Those are tightening operations.

Two Boolean fields use the wrong operation:

exact_action_binding=all(item.exact_action_binding for item in approvals),
one_shot=all(item.one_shot for item in approvals),

If binding authority requires True and a weaker customer layer supplies False, all(...) returns False. The lower layer weakens the approval contract even though the terminal disposition stays conditional.

This defect matters more than its two lines suggest. Monotonicity applies to requirements, not only allow, conditional, and deny. The regression needs a mixed-strength policy stack in which stronger exact-binding and one-shot flags survive every weaker layer.

5. Produce a deterministic decision

evaluator.py selects active and effective rules, resolves applicable policies, hashes the canonical operation, and emits a frozen Decision containing:

• operation and decision identities;
• release sequence and projection identity;
• policy fingerprint;
• terminal disposition;
• normalized requirements;
• applicable rule IDs and source hashes;
• applied-policy evidence and reason codes;
• evaluation time from an injected clock.

If no policy applies to a consequential operation, the evaluator creates an implicit SMF safety-floor denial:

no_applicable_policy_for_consequential_operation

The three core scenario tests currently produce:

Synthetic envelope	Python evaluator result
Privileged court transmission	conditional, with attorney approval and egress controls
Deletion under legal hold	deny
Injection-tainted external egress	deny
Uncovered consequential operation	implicit fail-closed deny

These results prove deterministic policy behavior over in-memory envelopes. They do not prove that a live Hermes or OpenClaw action reached the evaluator, that the mapper preserved the final parameters, or that the runtime honored the result.

6. Bind the exact action for approval

approvals.py persists an ApprovalBinding over the decision, operation, arguments, content, destination, release, projection, policy, tenant, organization, matter, actor, required role, creation time, and expiry.

The store accepts only allow-once or deny. It enforces the required resolver role, rejects self-approval, and caps the PoC TTL at 600 seconds.

Before execution, claim_once() rechecks the complete binding:

expected = {
    "decision_id": binding.decision_id,
    "operation_hash": binding.operation_hash,
    "canonical_args_hash": binding.canonical_args_hash,
    "content_hash": binding.content_hash,
    "target_identity_hash": binding.target_identity_hash,
    "release_id": binding.release_id,
    "projection_id": binding.projection_id,
    "policy_fingerprint": binding.policy_fingerprint,
    "tenant_id": binding.tenant_id,
    "organization_id": binding.organization_id,
    "matter_id": binding.matter_id,
    "actor_id": binding.actor_id,
}

A changed argument, destination, content, release, projection, policy fingerprint, tenant, matter, or actor invalidates reuse.

7. Claim once atomically

The central state transition runs inside BEGIN IMMEDIATE and uses a conditional update:

UPDATE approvals
SET state='executing', execution_count=1, idempotency_key=?
WHERE approval_id=? AND state='approved' AND execution_count=0

The code requires exactly one affected row. If another claimant wins first, the loser receives approval one-shot claim lost.

The concurrency test sends eight threads after one synthetic approval. One thread wins. Seven lose. The database reports one execution claim.

That test proves one SQLite winner for one stored binding. It does not prove one external side effect. A connector can complete an action and lose its acknowledgment. The state machine therefore distinguishes:

consumed
execution_unknown

A confirmed result becomes consumed. An ambiguous result becomes execution_unknown, a terminal fail-closed state. A process restart invalidates pending and approved work and moves an in-flight execution to execution_unknown.

A real OpenClaw test must still prove that the native held call resumes once and that the idempotency key reaches a connector capable of honoring it.

8. Record minimized evidence

audit.py stores operation and decision identities, release and projection IDs, disposition, reason codes, status, pseudonymous tenant identity, and time. It does not define columns for raw prompts, arguments, content, recipients, or matter IDs.

SQLite triggers reject ordinary updates and deletes. The focused audit test proves delete rejection.

The next test needs to attack the invariant rather than repeat the happy path: INSERT OR REPLACE, duplicate insertion, and conflict upsert. SQLite replacement behavior can bypass a design that only blocks explicit UPDATE and DELETE statements.

What the 34 tests prove, and what they do not

A test count is not an assurance statement. The useful unit is claim-to-evidence mapping.

Test evidence	What it proves	What it does not prove
Canonical key-order and Unicode tests	Equivalent structured values produce stable bytes and hashes	Python and TypeScript agree; all cross-language edge cases are covered
Unknown-fact and expression-budget tests	Invalid policy expressions fail before evaluation	A runtime mapper supplies complete, truthful facts
Projection reproducibility test	Reordered fixture inputs produce identical enforcement and analysis bytes	The source corpus is legally reviewed or complete
External-transmission evaluator test	The fixture returns conditional approval and egress requirements	A real court transmission is held before side effect
Legal-hold deletion test	The fixture returns hard denial	Every delete-capable runtime surface is intercepted
Tainted-egress test	The fixture returns hard denial	Direct replies, streaming, delegation, and web queries are gated
Changed-argument approval test	A stored approval cannot be claimed with a different argument hash	The runtime hashes final executable arguments after rewrites
Eight-way claim test	One SQLite transaction wins	The external connector executes once
Ambiguous-result test	The store reaches terminal execution_unknown	The runtime and connector report ambiguity correctly
Restart test	Pending work invalidates and executing work becomes unknown	Crash recovery works across a real runtime process boundary
DSSE and release-state tests	Focused signature, digest, sequence, prior-hash, lease, and clock paths fail closed	Production update security, revocation operations, or outage recovery is complete
Evidence-ledger test	Raw sensitive payload columns are absent and delete is blocked	Every replacement or upsert bypass is closed; telemetry separation is exercised

This mapping is why the current suite can be green while the PoC remains open.

The current verification record

The latest local run produced:

$ .venv/bin/ruff check .
All checks passed!

$ .venv/bin/mypy
Success: no issues found in 13 source files

$ .venv/bin/pytest -q
.................................. [100%]
34 passed

The configured branch-coverage gate failed:

TOTAL  817 statements  64 missed  202 branches  49 partial  88.91%
FAIL Required test coverage of 90.0% not reached.
34 passed

The missing branches cluster in approval expiry and mismatch paths, predicate errors, release verification, artifact handling, and time-state failures. We will raise coverage by exercising those contracts, not by excluding code from measurement.

The Python package also declares this command:

[project.scripts]
smf-assurance = "smf_assurance.cli:main"

src/smf_assurance/cli.py does not exist. The installed command fails with ModuleNotFoundError. A package cannot pass its release gate while its advertised entry point is broken.

The OpenClaw directory contains package metadata, a lockfile, and strict TypeScript configuration. It has no src/**/*.ts or test/**/*.ts files. Its test command ends at:

TS18003: No inputs were found in config file 'openclaw/tsconfig.json'.

No result above is bound to Git provenance. The repository has no commit, no HEAD, and no remote. All 33 non-ignored project files are untracked. A working-tree digest can help detect inspection drift, but it is not a commit SHA and cannot support an exact-snapshot verdict.

What the coordinated evidence review found

Aiona coordinated and synthesized a three-lane review of the current evidence:

1. an acceptance matrix against the PoC contract;
2. a red-team review of claims we must not make;
3. a chronology of architecture decisions, withdrawn assumptions, and authorization boundaries.

I separately inspected the live repository and returned the current test, coverage, TypeScript, CLI, and Git state. This was a work-in-progress evidence review, not Aiona's final exact-SHA implementation review.

Adversarial probes during that review surfaced four reproducible code defects:

Finding	Implementation consequence	Required regression
Approval Booleans merge with all(...)	A weaker layer can remove exact binding and one-shot protection	Strong requirements survive mixed weaker overlays
Compiler does not validate policy-to-rule references	Decision evidence can name a rule absent from the release	Dangling and duplicate identities fail compilation
keyed_identifier uses ambiguous plain SHA-256 concatenation	Domain/value boundaries can collide; empty domains are accepted	HMAC-SHA-256 with strict, length-framed domain and value inputs
CLI module is missing	The advertised installed command fails	Build, clean install, smf-assurance --help, and a deterministic command fixture

The identity defect is concrete. The helper currently computes plain SHA-256 over:

key || NUL || domain || NUL || normalized-value

It labels the output hmac-profile-sha256 without using HMAC and does not reject NUL inside the domain. The pairs (domain="a\0b", value="c") and (domain="a", value="b\0c") reach the same byte stream. The replacement must use actual HMAC-SHA-256 with strict domain validation and explicit length framing.

The review also decomposed the acceptance contract into twelve checks. Two had bounded Python-library evidence, nine were partial, and one was missing. None of the four scenarios crossed a real runtime adapter. The correct verdict was not acceptance-complete.

Runtime conformance is now the critical path

The largest remaining risks no longer sit in the architecture diagram. They sit at runtime seams.

OpenClaw: prove the exact current call

OpenClaw is the experimental exact-action target. Its native approval seam can hold a current call and resume it on an allow-once decision. The SMF adapter must still prove:

• exact runtime version and package integrity;
• normalization of final executable parameters after permitted rewrites;
• no mutation after hashing;
• binding of arguments, content, actor, destination, release, projection, overlay, and preconditions;
• resolver identity tied to a governed synthetic licensed-attorney role;
• ten-minute maximum TTL;
• timeout and restart fail-closed behavior;
• one held call resumes once;
• duplicate resolution and replay cannot execute twice;
• connector retry uses idempotency where available;
• ambiguity becomes execution_unknown rather than an automatic retry.

If the pinned runtime cannot expose final parameters or resolver-role evidence, the result is a documented negative finding. We will not weaken the control object to manufacture a pass.

Hermes: prove block-only enforcement without relabeling it approval

Hermes has useful pre-tool and final-output seams. The inspected host behavior also has known limitations: callback exceptions may fail open, final-output transform failure preserves the original output, and unbuffered streaming can expose tokens before a later gate.

The bounded Hermes adapter must:

• map native calls into the operation envelope;
• turn deny into a deterministic block;
• turn approval requirements into an explicit unsupported-approval block;
• disable or buffer streaming;
• gate the completed final response before delivery;
• attest adapter registration, compatible runtime version, verified projection, output mode, and evidence-store availability at startup;
• test tool calls, direct replies, gateway delivery, cron, delegation, background work, and dynamic MCP tools;
• fail visibly when the host cannot support the required seam.

A tested block is an L2-style experimental result. It is not exact-action approval and not Praxis parity.

SMF Swarm: bind analysis to selected rule identities

The PoC compiler already emits a separate analysis projection. The Swarm consumer is missing.

The clean integration seam is after _parse_llm_report() in the inspected Swarm analysis engine. The parser preserves extension fields, but later PredictiveReport construction discards unknown metadata. The PoC adapter must insert deterministic selection and claim validation before provenance disappears.

The Swarm scenario needs:

• selected, unresolved, and excluded rule sets with reason codes;
• effective-time and applicability decisions;
• normative claims restricted to selected rule IDs;
• rejection of invented IDs and release/projection mismatches;
• explicit labeling of uncited prose as model analysis;
• release, projection, rule, citation, and source identities in the report;
• complete separation between persona output and enforcement decisions;
• manual semantic-faithfulness review.

Rule-ID validation closes identity fabrication. It does not prove that every sentence semantically follows from the cited rule, and it does not eliminate hallucination.

The evidence matrix we will use from here

A binary task list is too weak for the next phase. Every gate needs five evidence fields:

Gate	Implementation artifact	Regression	Runtime/conformance test	Independent review	Immutable SHA
Canonical identities	Python and TypeScript canonical modules	Shared golden vectors, malformed inputs, framed HMAC	Adapter hashes final parameters identically	Pending	Pending
Monotonic policy	Corrected requirement merge	Weaker overlays cannot relax any field	Same decision across runtimes	Pending	Pending
Projection integrity	Compiler reference checks	Dangling, duplicate, and inconsistent IDs fail	Both consumers load one release	Pending	Pending
Exact-action approval	SQLite state machine plus OpenClaw adapter	Changed bindings, expiry, replay, restart, ambiguity	One native held call resumes once	Pending	Pending
Hermes block-only	Hermes adapter and output gate	Hook failure, adapter absence, streaming and direct output	Denials block across bounded surfaces	Pending	Pending
Release state	Signed manifest and persistent sequence state	Signature, digest, rollback, lease, clock, outage	Adapters reject unsafe releases	Pending	Pending
Protected evidence	Append-only protected ledger	Replace/upsert/delete and store outage	Runtime decisions write minimized evidence	Pending	Pending
Swarm traceability	Selector, validator, provenance report	Invented IDs and mismatches fail	Persona report cites selected rules	Pending	Pending

A row is not closed because its first column contains code. It closes when implementation, regression, runtime behavior, independent review, and immutable provenance all point to the same candidate.

The build sequence from here

The implementation plan follows the dependencies in the evidence matrix.

1. Repair the Python defects

We will first add failing regressions and fix requirement monotonicity, compiler referential integrity, HMAC framing, and the CLI. We will expand approval, release, predicate, trusted-time, and evidence-ledger failure tests until branch coverage exceeds the configured 90% gate through meaningful cases.

2. Add the bounded source-backed corpus

The four scenarios need primary-source records with exact locators, retained snapshots or permitted archival references, hashes, authority type, effective interval, citation identity, review identity, and correction history. Customer overlays must be signed, versioned, tenant-bound, and unable to weaken the SMF safety floor.

If qualified legal review is unavailable, the evidence report will say so. Engineering review does not become legal review through wording.

3. Add cross-language vectors and OpenClaw

Python and TypeScript will consume the same canonical JSON, hash, predicate, operation, and decision vectors. Then the OpenClaw adapter will attempt the exact-current-call protocol against one pinned package version.

4. Add Hermes block-only controls

The Hermes profile will implement deterministic blocking, explicit unsupported approval, buffered output, startup attestation, and the bounded outbound-surface matrix.

5. Complete the Swarm scenario

The selector, claim validator, provenance report, and manual review will establish rule-identity traceability around the persona path.

6. Run the bypass matrix, packaging, and latency gates

The final candidate must cover unknown tools, missing facts, mapper exceptions, dynamic MCP, signature failures, rollback, revocation, expiry, changed approvals, replay, process restart, direct output, streaming, gateway delivery, cron, delegation, background work, adapter absence, and evidence-store outage.

We will build the wheel and source distribution, install the wheel into a clean environment, exercise the CLI and package data, and measure evaluator median, p95, and p99 latency against a predeclared budget.

7. Freeze one candidate and send that exact SHA to Aiona

Only after the complete suite passes will we commit one clean candidate. Every reviewer must inspect that SHA in an isolated clean clone and attest the SHA before and after review.

A finding creates a regression and a fix. The fix creates a new SHA. Every affected gate reruns, and every review lane reviews the new SHA again.

A timeout, partial report, dirty checkout, or SHA mismatch is no verdict.

Where we stand

We have crossed an important boundary. This is no longer an architecture-only experiment. The repository contains a deterministic control-plane core with strict operation models, bounded predicates, layered policy, reproducible dual projections, exact-action approval state, release verification, and minimized evidence storage.

We have not crossed the runtime boundary. No Hermes denial, OpenClaw held action, or Swarm provenance report has completed the full governed path. Coverage fails. The CLI fails. Four code defects remain open. No commit exists for final review.

Building in public means publishing those facts together:

34 tests pass.
88.91% does not satisfy the 90% gate.
The Python core is real.
The cross-runtime PoC has not passed.

The next milestone is not a launch announcement. It is an immutable candidate whose code, tests, runtime evidence, limitations, and independent review all resolve to the same SHA.

Even a technically passing PoC will not authorize a customer pilot, production legal reliance, broad thirteen-state coverage, Praxis equivalence or retirement, hosted distribution, billing, or a claim that Swarm citation checks eliminate hallucinations. Those decisions require new evidence and explicit authorization.

The experiment can also produce an honest negative result. If a pinned runtime cannot preserve final-parameter binding, close an outbound bypass, or prove at-most-once behavior, that runtime hypothesis fails within the tested profile. We will report the limitation instead of lowering the contract.

That is the work ahead: convert each green component into a regression-backed contract, cross the real runtime seams, freeze the evidence, and let Aiona review the implementation that actually exists.