Write the Assurance Contract — สัญญาที่ตรวจได้ ไม่ใช่คำโฆษณาWrite the Assurance Contract — Guarantees, Estimates, Evidence

🤔 If a customer complains tomorrow morning that Nong Kram explained their refund eligibility wrongly, what evidence can you pull up — and is there a line in your team's documents that said, in advance, whether a case like this is "guaranteed" or merely "estimated"?

In the previous post, Specify → Constrain → Generate → Verify, we wrote ai_core_operation() until Nong Kram had the full control loop: check before generating, authorise before any real effect, verify before releasing, and trace every path. But Section 6 of the paper warns bluntly that the four-verb workflow is a development workflow only — it orders activity, and establishes no property of the system^[1]. The running loop still tells nobody what it promises, under which assumptions, and who must answer when the promise is broken.

The answer this whole post gives is to write the assurance contract of the paper's Section 6: a document written one property at a time, eight elements per row, ruthlessly splitting what is structurally guaranteed from what is only estimated with error rates, defining the evidence, assigning an owner and a breach response, then making it machine-usable through a release manifest and a trace schema, with a conformance level honest about the mechanisms that really exist — and in the entire document, not one line claims that "the whole system is safe".

1. One Row per Property — Eight Elements, Two Claim Types

The heart of Section 6 is its structure: the contract never speaks about "the system" as one lump, but fixes, property by property, what is held, under which conditions, proven by what, and owned by whom^[1]. One property is one row, all eight elements filled in — leave one out, and the row instantly becomes unauditable.

• Assumptions — the implementation conditions that make the claim true, such as "every effect path passes the validator" — a fallen assumption voids the row, it does not merely weaken it
• Obligations — what the system must do on every execution, such as validating every tool call before it reaches the order store
• Guarantees or estimates — the claim's type, Guaranteed or Estimated, with its substance
• Evidence — the artifacts that discharge the row: tests, counters from traces, measurement reports, a mediation argument
• Owner — one name who can answer and order the fix. Not a team name
• Threshold — the number actually enforced at decision time, not a target on a slide
• Change conditions — the events that re-open the row for review, such as a model version change or a corpus snapshot change
• Breach handling — the route designed in advance for the day the row is no longer true

Guaranteed versus Estimated — the line the whole contract stands on

Contract claims come in two types, never to be swapped. A structural guarantee — a Guaranteed (G) row — is an invariant holding on every execution — but the paper writes the condition into the definition itself: "within stated implementation assumptions", namely complete mediation of the effect path and a correct validator^[1]. Guaranteed therefore never means "unconditionally absolute" — the day somebody opens an endpoint that bypasses the guard, the assumption falls and the guarantee is void at once.

A semantic estimate — an Estimated (E) row — is a measurement, not an enforcement, and it must always declare three things: the population measured, the operating threshold used for real decisions, and a non-zero error rate in both directions — letting bad output through (false accept) and holding good output back (false reject)^[1]. A sentence like "the system detects injection" with none of the three attached is not even a claim of type E — it is hope, set in type.

Walking the paper's Table 5, row by row

The paper fills in a model contract for six properties in Table 5 — I summarise each row's type, mechanism and evidence, then expand the two rows most often misread^[1].

The semantic quality row is the one whose numbers get over-read most often. The paper cites the RAGAS results as its example: faithfulness agrees with humans at roughly 0.95, answer relevance at roughly 0.78, context relevance at roughly 0.70 — measured on WikiEval, a 50-page set the RAGAS authors constructed themselves^[1][4]. These numbers are bound to that population, not universal properties — and the 0.70 means the scorer disagrees with humans on a substantial share of cases: usable as an advisory signal, never as the final judge in a human's place. This row is therefore E, forever.

The injection containment row forces two lines inside one row, because prompt injection poses two questions of different natures: "can the prohibited effect occur?" can be G, because the execution boundary is code that really closes the path; but "can we detect the attack?" is E and nothing more — the paper cites that a detector-based defence once held attack success near 8% under AgentDojo's original attacks, while adaptive attacks designed afterwards degraded the defences of that era^[1]. Writing this whole row as G is promising something no detector can deliver.

💡 My view: I make teams fill in the "type" column before every other column, and I ban the words "safe" and "trustworthy" from every cell. These two rules cut more than half of the wording arguments, because once G/E is pinned, each row writes itself according to its type — a G row automatically demands assumptions and a mediation argument, an E row a population and error rates.

2. Auditable Means Reconstructable from Retained Artifacts

A contract only means something if auditable has an operational definition, and the paper defines it narrowly and hard: auditable means the decision can be reconstructed from retained artifacts alone — not "we have logs", not "we have a dashboard"^[1]. The retention list, per request, has ten items, all of them:

1. The model's identity and version
2. The full decoding parameters, including the seed
3. The assembled context, verbatim, word for word
4. The version of the corpus snapshot used for retrieval
5. The retrieved passages, with per-passage provenance
6. The tool state before invocation, plus every call and its results
7. The identity and version of every evaluator that issued a verdict
8. The version of the policy in force
9. The ordered decision trace — each control rail's verdict and the routing that followed
10. The numeric threshold actually in force at decision time

The last item goes missing most often and costs the most. The paper's summary: scores without thresholds, or verdicts without context, yield a log, not an audit trail^[1]. Yesterday's faithfulness of 0.83 is worth nothing if you do not know whether the bar then stood at 0.80 or 0.85 — the same number is either "passed" or "should have been withheld".

The conformance profile — the contract in the version a machine reads

Section 6.1 makes the contract checkable by automated systems: a profile in the style of RFC 2119, using the keywords MUST / SHOULD / MAY^[3], plus two JSON Schemas — one for the release manifest, one for the route trace^[1]. The manifest identifies six things: the core+decoding policy, the context-template hash, the corpus snapshot+provenance policy, the tool-registry hash, the control profile+thresholds, and the evaluation-suite hash. The trace binds one request+principal to that manifest, records the ordered checks, violations and tool events, and always ends at one of four terminals: RELEASE / WITHHOLD / ESCALATE / FAIL-CLOSED.

{
  "trace_id": "kk-2026-09-08-000341",
  "manifest_hash": "sha256:9d2f44e1...",
  "principal": "line-user:U1a2b3c...",
  "checks": [
    {"rail": "input",              "verdict": "pass"},
    {"rail": "retrieval.provenance","verdict": "pass"},
    {"rail": "output.structure",   "verdict": "pass"},
    {"rail": "output.faithfulness","verdict": "pass",
     "score": 0.91, "threshold_in_force": 0.85}
  ],
  "tool_events": [],
  "terminal": "RELEASE"
}

Notice threshold_in_force — a good trace always retains the bar used to decide that particular event, attached to the score. This is item ten made concrete, and the line between a system that can answer an auditor and one that can only open a dashboard in hindsight.

The three conformance levels — pick the honest one, not the good-sounding one

The line I respect most is the paper's honesty toward itself: its specimen demonstrates selected mechanisms from Levels A and B, and states plainly that it does not claim Level C^[1] — demanding narrowness of claims from everyone else, then writing its own just as narrowly.

💡 My view: a level claimed low but true in every item is worth more than a level claimed high with asterisks. I have seen an "immutable archive" that was really a shared folder the whole team could delete — one false row makes every remaining row get read with distrust. Choose the level from tested mechanisms only; write what is missing as a plan.

3. The Seven Steps

These seven steps turn the paper's Section 6 into two files in your repo: one contract table and one manifest. Every step ends with real progress on Nong Kram, and by the end of step 7 you have the artifact section 4 shows in full.

Step 1 — List the Properties Your Application Must Hold

Start from your own application's effect boundary, not from generic risk lists: enumerate what leaves the system and touches the real world — every payload released to a user, and every external effect a tool creates — each one a candidate row. Write them in the language of your application, then check back against Table 5's six rows for anything missed.

For Nong Kram, walking the effect boundary yields six items, landing neatly on the paper's structure:

• Authorized effects — refund occurs only for requests the execution guard approves, within the declared cap
• Output structure — every answer must parse against the answer schema (text + the list of cited passages)
• Task correctness — answers about orders, shipping and the returns policy are correct against the shop's golden set
• Faithfulness — every policy claim points back to a real span in policy/ or catalog/
• Injection containment — instruction-bearing content arriving with product pages or customer messages must never become a prohibited effect
• Change control — the behaviour of every released version is pinned to one identifiable manifest

Step 2 — Split Each One into Guaranteed or Estimated, Ruthlessly

Apply the rule from section 1 to every item: who decides? Deterministic code every path passes through — G, with assumptions attached. A detector, scorer or judge — E, no exceptions. Teams want safety rows to read G because it sounds reassuring, but an honest E row protects the organisation more than a false G row, every time.

Done for real, the six items split into eight claim lines: injection divides in two — containing the effect is G at the guard, detecting it is E at the classifier — and semantic quality covers faithfulness and relevance with separate thresholds. More claim lines than properties is a good sign — we have stopped lumping.

Step 3 — Pick the Mechanism for Each Property

For each G row, pick a mechanism that closes the path, not one that asks the model nicely — text in the system prompt does not count as a G-row mechanism, because the model can choose not to follow it. For each E row, pick the measure you can genuinely calibrate against human labels from your own domain, not the most famous one.

Nong Kram wires all eight lines: effects use the execution guard from the previous post + the allow-list {refund} + schema and amount checks + an idempotency key; output structure uses post-hoc parsing + one repair round then reject; correctness uses a 120-case golden set (a tutorial number) + a check that cited order numbers exist; faithfulness mandates per-sentence citations + one evaluator; injection isolates untrusted content from the instruction channel + capability limits at the guard; change control is the manifest v2 of step 7.

Step 4 — Define the Evidence That Discharges Each Obligation

Good evidence answers one question: "if I had to prove this row is still true, which file would I hand over?" G rows: coverage tests, a mediation argument, real post-state. E rows: a measurement report declaring population, threshold, and error in both directions. Write the evidence file names into the contract — a row that cannot name its evidence is not finished.

One full example — the evidence for the effects row is four pieces:

• A test suite firing tool calls outside the allow-list and above the cap, confirming every one is rejected (coverage)
• A one-page mediation argument enumerating every path that can write the order store, showing each passes the guard
• Counters from real traces: unauthorized calls rejected, and escaped (which must be zero)
• A post-state oracle reading the order store after every refund against what was approved

Step 5 — Assign One Named Owner and a Failure Response per Row

Each row needs one named owner — someone who can answer and order the fix. A cell saying "the relevant teams" is an empty cell. Then write the response in advance, by row type: a breached G row = a fallen assumption — freeze the affected effects and open an incident at once. An E row crossing its threshold = the measurement doing its job — lower autonomy, raise sampling, review the bar. Binding duties to nameable roles is also the core of the GOVERN function in NIST's AI RMF^[2].

KramKraft is a small shop — owners may repeat, but they must be real names in real roles: the backend engineer holds all four G rows because they are code she maintains; the shop owner holds the two E rows, as the person who labels the golden set and decides escalations.

Step 6 — Set Thresholds and the Conditions That Re-open a Row

A threshold must be the number actually enforced — the same one as threshold_in_force in the trace — not a quarterly target. Re-open conditions cover at least three events: a model/version change (the whole golden set re-runs — old results are not portable across versions), a corpus snapshot change, and metric drift beyond its band. A row with no re-open condition rots quietly until the day of the incident.

One of Nong Kram's rows, in a shape a machine can begin to read (every number is a tutorial default):

# contract-nongkram.yaml — row 1 of 6
property: authorized-effects
claim: guaranteed          # only under the assumptions below
assumptions:
  - every-order-store-write-passes-execution-guard
  - guard-validator-passes-current-coverage-suite
mechanism: [allow-list, schema-check, amount-bounds, idempotency-key]
evidence:
  - tests/guard_coverage_test.py
  - docs/mediation-argument.md
  - traces/unauthorized-call-counter
  - tests/post_state_oracle.py
owner: "May (backend engineer)"
threshold:
  refund_max_thb: 3000
  approval: required-for-every-refund   # refunds are irreversible
reopen_when: [model-version-change, corpus-snapshot-change, tool-schema-change]
on_breach: freeze-refund-tool + incident-review

Step 7 — Make It Machine-Usable, Then Pick Your Honest Level

Convert the table into a conformance profile: write the requirements with MUST / SHOULD / MAY in the manner of RFC 2119^[3], declare the JSON Schemas for manifest and trace, and make CI reject any build whose manifest fails its schema. Then pick A / B / C by one rule: count only mechanisms that exist and have tests behind them. The level you get may be lower than the one you wanted to tell the executives — that is the correct value.

Nong Kram can claim B, because all five mechanisms genuinely exist: effects bound to an authenticated LINE user identity, an idempotency key rejecting duplicates, tool errors failing closed, every refund (irreversible) passing human approval, and a post-state oracle reading the real order store. It does not claim C — the reasons are in the next section, alongside the full manifest.

4. Nong Kram's Contract — Six Rows, Eight Elements, and Manifest v2

This post's artifact is two files: contract-nongkram.md, the six-row table with all eight elements filled in, and manifest-v2.json, upgrading the manifest from post #4 to carry the control profile and thresholds. Every number is a tutorial default — not the paper's numbers, and not universal recommendations.

Read this table and notice that not one cell says "safe" — every cell speaks of a mechanism, a file, a number, or a person's name, and every G row carries its assumptions inside itself. That is the difference between a document a committee can question further, and one that can only be nodded at.

Release manifest v2 — the contract in the version CI enforces

{
  "manifest": "kramkraft-release",
  "version": 2,
  "core": {
    "model": "M_v (pinned hosted LLM)",
    "decoding": { "temperature": 0.2, "top_p": 0.9, "seed": 20260908 }
  },
  "context_template_hash": "sha256:3f9c19ab...",
  "corpus": {
    "snapshot": "2026-09-01",
    "provenance_policy": "allow-list: policy/, catalog/"
  },
  "tool_registry_hash": "sha256:b41e77d0...",
  "control_profile": {
    "conformance_level": "B",
    "thresholds": {
      "task_correctness_per_slice": 0.90,
      "faithfulness_operating_point": 0.85,
      "injection_detector": 0.70,
      "output_repair_bound": 1
    },
    "effects": {
      "refund_max_thb": 3000,
      "approval": "required-for-every-refund",
      "authenticated_subject_binding": true,
      "idempotency": true,
      "tool_errors": "fail-closed"
    }
  },
  "evaluators": [
    { "id": "faithfulness-judge", "version": "1.3" }
  ],
  "policy_version": "policy-2026-09",
  "evaluation_suite_hash": "sha256:77aa02c4..."
}

Compared with v1, the addition is the entire control_profile block — the claimed level, the bars of the E rows, the bounds of the G rows. From now on, changing a threshold is a manifest change that passes gated promotion like a model change, not a quiet config edit. And why claim only B: three large pieces of C are still missing — full held-out and adaptive evaluation (post #9, Prove It, builds that harness), independent adjudication, and an immutable public archive. Claim what you have; write what is missing as a plan — the paper's own example.

5. Validation Check — Can Your Contract Actually Be Audited?

The principle behind this table is borrowed from the classic work on ML production readiness: readiness is measured by the tests and monitoring that actually exist, not by the team's feelings^[5]. Every question is therefore answered with a nameable artifact, following the paper's rule of answering with artifacts, never adjectives^[1]. Whichever row you fail is your next piece of work.

Question 1 should run first and is the harshest, because it tests the definition of auditable with nowhere to hide — many systems pass questions 2 through 7 on paper and then fail question 1 at passage provenance or at the threshold in force at decision time, the two pieces an ordinary dashboard never retains.

6. The Road Ahead

This post turned an unanswerable question — "is Nong Kram safe?" — into two auditable files: a six-row contract with all eight elements, and a manifest v2 that CI genuinely enforces. From here, every argument about risk points at a specific row — never floating in the air.

What it deliberately does not answer is the question of position: where on the request path should these mechanisms sit, which seam of the envelope makes each row true, how the three control classes — hard, soft, governance — differ and where they must never be blurred, and why the only rail that can hard-bound effects is the execution rail. That is the next post's entire job.