Five Tracks, One Release Gate — ปล่อยด้วยหลักฐาน ไม่ใช่วันที่Five Evaluation Tracks, One Release Gate — Release on Evidence, Not on a Date

🤔 If the model's benchmarks pass, but nobody ever wrote down which outcomes are "unacceptable" — is the system ready to release?

The previous post — #13 Five Rails and the Effect Guard — ended on the line that a proposal is not an effect. We put five rails around the system so the model may propose, a deterministic control authorizes the effect, and authoritative state proves what actually happened. The question left hanging is the natural next one: the rails are built — so how do we know they are good enough to put in front of real people, and which people first?^[1]

Chapter 9's one-line answer is that a release is neither a technical event nor a date in the calendar. It is a management decision about a complete sociotechnical system, and it has to rest on evidence — evidence that comes from several evaluation tracks running at once, not one score from one test suite. This post walks from those five tracks, through the six-rung release ladder and the ten-item dossier, to the release gate itself: an eight-row table that forces the verdict to be written down as Promote, Canary, Hold or Reject.

1. A Release Is a Management Decision, Not a Model Demo

Chapter 9 of the playbook opens with a single sentence that frames the whole chapter, and I think it is the sentence technology executives should tape to the side of their monitor.^[1]

"An AI release is a management decision about a complete sociotechnical system not a model demonstration or a software date." — a release is a decision taken by management about the whole sociotechnical system; it is not a model demonstration, and it is not a software delivery date.

The hardest-working word in that sentence is complete, because what we release has never once been a model. The playbook spells the list out: the release question is whether the entire system — model, data, retrieval, tools, permissions, interface, workflow and human oversight — has enough evidence to operate inside an agreed boundary.^[1] Notice how many items on that list no benchmark can measure at all: permissions are a question of entitlement, the interface is a question of whether a person can read the output in time, and the workflow is a question of headcount at the far end.

This is the model versus system distinction the series has been making since the first post, and it has one sharp practical consequence: an organisation that evaluates models learns how good its model is; an organisation that evaluates systems learns whether it can release — and neither answer substitutes for the other.

Evaluation begins before development, not after it

The next sentence in the chapter is the one I see violated most often in real projects. The playbook writes "Evaluation therefore begins before development" — evaluation starts before the build — and then lists what must be stated before the first line of code: the decision, the affected people, the baseline, the unacceptable outcomes, and the conditions that require stopping or escalation.^[1]

The logic is so obvious it is embarrassing that we still get it wrong: if you never declare which outcomes are unacceptable, you can never test whether they occurred. A test suite written after the system already works will always tend to ask the questions the system can already answer — not because the team is dishonest, but because our imagination has already been fenced in by the system sitting in front of us.

The way I recommend using this table is to fill it in before the kick-off meeting and carry it into the room as a draft. The cells you cannot fill are the project's real risks, not gaps in a document — and "unacceptable outcomes" is invariably the row that eats the most time, because most of what gets written there has nothing to do with model accuracy and everything to do with the actions the system can cause.

The five operating principles that govern the chapter

Chapter 9 closes with five operating principles.^[1] I am moving the first one down to section 3, because it is the heart of the release ladder; the other four belong here, so you can read the rest of the chapter with them in mind.

• Evaluate the system in context (#2) — "A model score is not workflow performance." A model score does not tell you whether the reviewer queue is overflowing, and it does not tell you how many times a customer had to call back.
• Match gates to impact and reversibility (#3) — "Stronger consequence demands stronger evidence." Consequence and reversibility are the two variables that set how strict the gate is, not the urgency of the delivery plan.
• Make every release observable and reversible (#4) — "A live system needs a service owner and kill path." The first half of that is observability; the second half is the way back. A system with no single named owner and no rehearsed way to switch it off is a system that cannot be released, however handsome the scores are.
• Treat near misses overrides and corrections as evidence (#5) — "Do not suppress the signals that teach the system." Every time a reviewer corrects the system's answer, that is the highest-quality data the organisation will ever get — and it disappears silently if there is nowhere to put it.

Read one at a time, all four sound like common sense. Notice, though, that each of them forces the same thing into existence: people, process and tooling that must exist before release day, not after it.

2. Five Evaluation Tracks — Fixed, Hidden, Adaptive, State and Fault, Live

The book splits evaluation into five tracks — Fixed, Hidden, Adaptive, State and fault, and Live monitoring — rather than one score from one suite.^[1] Figure 12 draws them as five tiles pointing down into a single gate, which then opens into four outcomes.

There are five tracks not because more numbers are better, but because each track answers a different question and none of them answers for another. With Fixed alone you learn only that the system has not gone backwards; you learn nothing about what it will do with something it has never seen.

The case types every track has to carry

Beyond the tracks, the playbook also fixes what the cases must cover: routine tasks, boundary cases, rare severe scenarios, Thai and other operating languages, affected groups, and downstream tool behaviour.^[1] It is a short list, but every word carries weight — especially the last two, which most teams do not test because they fall outside the model team's remit.

I will come back to the Thai-language item in a box of its own in section 5, because it is not a detail. It is a full evaluation track, and it is missing from a great many Thai organisations' test plans.

False accepts and false rejects of the semantic evaluator

The last requirement in that paragraph is the most technical and the most frequently skipped: report false accepts and false rejects for semantic evaluators.^[1] That is, when we use a model or a semantic rule to judge whether the system's answer "passes" or "fails", the judge itself can be wrong in two directions, and we need to know the rate of both.

Why the Adaptive track never closes

The third track — adaptive evaluation — is the only one of the five with no finish line, and the reason is not budget or team diligence. It is the nature of the problem itself.

On 9 June 2026 (updated 22 June 2026) NIST published a research summary of a mathematical proof that no finite set of guardrails is universally robust against adversarial prompts, supporting a continuous monitor-and-update posture rather than a one-time check^[2] — while the result itself specifies no cadence or set of controls for any particular operating context.

One sentence from that page sums it up better than any long explanation of mine: "you likely can't patch an AI system like an LLM and then expect to be OK forever"^[2] The proof is an information-theoretic result about the robustness of guardrails against adversarial prompts — it says nothing about release gates, nothing about five tracks, and nothing about Thai-language testing. The link from that result to a five-track evaluation suite is the playbook's synthesis, not NIST's conclusion. What the result supports directly is exactly two things: the Adaptive track is not a one-off job and Live monitoring is not an optional extra.

On the practical-guidance side, NIST AI 600-1 — the Generative AI Profile of the AI RMF — carries a suggested action to conduct adversarial testing "at a regular cadence" in order to map and measure GAI risks.^[3] Note the words suggested action, and note too that the document does not say how many weeks "regular" is — it states plainly that implementation will vary with the type of risk, the characteristics of the system, the lifecycle stage and the actors involved. Organisations must select and tailor the actions to their own use case and risk tolerance.

3. The Six-Rung Release Ladder, and "Production Is the Final Evaluation Environment"

The release ladder has six rungs — sandbox, offline evaluation, shadow mode, canary traffic, limited population, then wider deployment.^[1] The playbook does not present it as bureaucracy but as an instrument of exchange: each higher rung trades more risk for more knowledge, and the gate's job is to make that trade a deliberate decision rather than a by-product of schedule pressure.

One sentence governs the climb: higher consequence requires stronger evidence, smaller exposure, visible human control, and faster rollback.^[1] Notice that all four move together rather than trading against one another — many organisations read it as "plenty of evidence buys us a wide opening", when the sentence says a high-consequence case needs all four at once.

💡 My view: the chapter's first operating principle is the one I want on the wall of every room where somebody says "can we open it up a little more?" — Evidence before exposure: "Do not increase the affected population ahead of proof." It is a short line, but it changes the question in the room from "are we ready to go wide?" to "how many people does the evidence we hold right now actually cover?" — and the second question has a number for an answer.

When human review means something

The playbook attaches conditions to the words "human review" that I think a gate should check as carefully as it checks numbers: the reviewer must have time, competence, evidence, and the authority to disagree.^[1] The last of those fails most often in Thai organisations, because the reviewer is usually junior to the person who wants to release — and in practice an "authority to disagree" with no protection in the process is an authority in name only.

"Has time" is measurable too, and should be measured: if the reviewer queue grows faster than the review rate during canary, then increasing exposure in the next round is a decision to turn your reviewers into a rubber stamp. It is not a decision about the model.

Production is the final evaluation environment, not the end of evaluation

That sentence from the playbook^[1] matches a line in the MEASURE function of NIST AI RMF 1.0: "AI systems should be tested before their deployment and regularly while in operation"^[4] — tested before release, and tested continuously during operation. Both halves of that sentence carry equal weight; almost every organisation's budget gives nearly all of it to the first half.

Why the second half is necessary is in the playbook's very next line: supplier model changes, new user behaviour or policy updates can invalidate previous evidence.^[1] The first of those is the modern headache, because it happens while we do nothing at all. The model provider revises a version behind the scenes, and the evidence we signed off three months ago quietly expires without anyone being told.

4. The Ten-Item Release Dossier — the File That Speaks Instead of the Team

The release dossier the book specifies has ten items, from purpose and owners through to rollback and monitoring.^[1] The thought behind it is very simple: in an approval meeting, what should speak for the team is the file, not the team lead's confidence. A complete file makes the meeting shorter, not longer, because all the time goes to what is still unsettled rather than to explaining how the system works.

The sentence that follows in the playbook is the one I want read twice: "Model benchmarks can inform this package but cannot substitute for testing in the operating context" — model benchmarks can inform the package, but they cannot stand in for testing in the real operating context.^[1] The words inform and substitute are the whole boundary, and it is a boundary technology vendors have a natural incentive to blur.

How much of NIST's shape the playbook borrows — and what it does not

As of 5 September 2026, NIST's own page states that AI RMF 1.0 — released 26 January 2023 — is being revised under the White House AI Action Plan.^[4] So what the playbook borrows is the shape of the work, Govern, Map, Measure and Manage, rather than any one frozen text — and it should never be cited as though it were a binding requirement.

The framework document says of itself that it is voluntary, that it is not a certification, and that "Actions do not constitute a checklist, nor are they necessarily an ordered set of steps"^[4] — the suggested actions are neither a checklist nor an ordered sequence. The playbook accordingly writes its own role down plainly: it turns those principles into a release portfolio rather than claiming certification.^[1]

But the converse has to be said just as plainly: NIST does not require a release gate, does not require a canary, and sets no exposure cap of any kind. Both documents issue voluntary suggestions that each organisation must select and tailor for itself.^[3][4] The eight-row table in the next section therefore belongs to the playbook, not to NIST — and keeping that line clear matters enormously when you sit down with legal or with an auditor.

5. Artifact 7 — the Release Gate: Eight Rows, One Decision

We now have five evaluation tracks, six rungs and a ten-item file. What is still missing is the point where all of it converges and becomes a recorded verdict — the release gate — and the playbook hands it over as a ready-made table in Artifact 7.

The head of Artifact 7 sets the rules before the table begins.^[1]

• Purpose — make promotion a recorded decision against predeclared utility, risk, cost and operability thresholds for one manifest.
• Use when — moving from experiment to production, increasing exposure or authority, or changing any manifest component.
• Accountable owner — the release authority decides; property owners attest the evidence in their own area; independent challenge records dissent or exceptions.

That role structure is what keeps the table from becoming a ceremony, because the decider is not the same person as the evidence attester, and there is a third role whose whole job is to record disagreement — not to find a way to agree.

The copy-ready table, eight rows

Artifact 7 is an eight-row table — seven gate rows and a Decision row that records the exposure cap, the next review date and the rollback ID.^[1] It is meant to be copied and used immediately. The blanks are what the team has to fill in, and a blank that cannot be filled is itself the answer.

Seven words on Figure 12, seven rows in the table — two different vocabularies

There is a point of easy confusion here that I want to settle straight away. The gate bar in Figure 12 reads as seven words — Utility, Security, Effects, Trace, Operations, Economics, Recovery.^[1] The rows in the Artifact 7 table use a more detailed vocabulary that does not map word for word. Both describe the same thing at different altitudes: the figure explains what the gate weighs; the table is the form you have to fill in.

Note again that language is not a row of its own in the table. It sits inside the third row, in the words "by language" — and that is exactly why it drops out of test plans so easily.

The worked example — CX-REFUND-01

The playbook offers CX-REFUND-01 (a fictional case from the playbook) as a completed example: a bilingual refund assistant for a fictional company, Luma Commerce Thailand, which may propose issue_refund, and where an external guard may execute one eligible refund of no more than THB 2,000 after confirmation, with every other financial action going to a person.^[1]

Every value in that table is illustrative, not a universal threshold — the playbook prints that caveat itself.^[1] Do not take 94.6% or 5% and set them as your organisation's targets. These numbers have value as the shape of a decision, not as quantities.

Three things in the Decision row are worth noticing. First, the verdict is bound to a manifest identifier, not to a project name. Second, the next step up in exposure is tied in advance to a review time, rather than left to be re-argued on the day. Third, the Semantic/security row admits openly that text escapes remain, and the release still goes ahead — because what must not fail is a prohibited effect, which is a structural guarantee, not text quality, which is an estimate. This is a gate working the way a gate should work.

💡 My view: a table like this does not exist to make releasing harder. It exists to make releasing repeatable. Organisations without a gate do not ship faster — they ship with no record of who decided what on which evidence, and then have to reconstruct that record from memory when someone asks after the incident.

6. The Release Readiness Room — Six Agenda Items, and the Fifth Everyone Skips

The pre-release readiness room has six agenda items, and the fifth is the one people skip most often — rehearse detection, containment, rollback, notification and remedy for one failure.^[1] The playbook calls this a working session, not an approval meeting. The difference is in the output: an approval meeting ends with a resolution; a working session ends with a document that has been filled in.

Why the fifth item is what makes the session worth holding

The other five items are a review of documents; the fifth is an act, and it is the only one that proves what the file claims is true. I have watched a team with a complete and beautifully written rollback document discover during the drill that the person entitled to run the command had left the organisation. Not one character of the document was wrong, and the way back did not work.

The words in item five name different capabilities, and each should be timed separately: detection is how fast we know without waiting for a customer to call · containment is whether we can bound the effect before fixing the cause · rollback is how long it takes to return to a known build · notification is who has to know, within how many minutes, and who tells them · remedy is what the already-affected customer actually receives.

That last one is the item technical teams usually have not prepared, because it is not the system's job. But it is the part the affected person actually experiences, and it is the first thing a regulator asks about.

On the stop thresholds in item four, NIST AI 600-1 carries suggested actions that land exactly here: one to establish and regularly review the specific criteria that warrant deactivating a GAI system in line with the organisation's risk tolerance, and another to maintain procedures for escalating incidents to the organisational risk authority when those criteria are met.^[3] In the parent framework, AI RMF 1.0, the MANAGE function likewise speaks of mechanisms in place and responsibilities assigned to supersede, disengage or deactivate systems whose performance or outcomes are inconsistent with intended use.^[4] Put another way: the fail-safe state needs an owner, not just a button.

7. Metrics That Matter, and the Failure Patterns at the Gate

The playbook gives the metrics as a single paragraph and names no pass threshold for any of them,^[1] which I think is the right call, because the correct threshold depends on the consequence and reversibility of the particular task. I have laid them out in a table with a column I consider as important as the number itself — "how it gets misread" — and a Scorecard column saying which board-scorecard column each metric should surface in.

There is exactly one rule for reading this table, and it is the board scorecard's rule: read them together, never collapse them into one score. Faster work with more critical errors is not progress, and a falling cost per outcome that comes from removing human review is not efficiency. It is a transfer of risk onto the customer.

Failure patterns

The playbook lists the failure patterns as a single paragraph as well.^[1] I have pulled out only the ones that are about the gate itself; the rest belong to incident handling after release, which is what the next post is about.

• One average score — the easiest symptom to spot is a single large number in the middle of the approval slide. That gate is not working, because one average can never surface a structural failure or a slice that fell.
• Irrelevant benchmarks — a public suite measures what it measures. It does not measure your workflow, your data or your downstream tools.
• Testing only easy English cases — this one hurts particularly in a Thai context, because it makes the largest user group the group with the least evidence behind it.
• Releasing to everyone — jumping straight to the last rung turns every mistake into a full incident instead of a signal whose effects can still be contained.
• Treating vendor testing as sufficient — the supplier's test results are evidence about their product, not evidence about your system, which is assembled out of that product.
• No versioned evidence — evidence that cannot say which build it was measured on cannot be used for comparison, and cannot be traced back when something goes wrong.
• Ceremonial approval — a gate that has never once returned Hold or Reject is not a gate attached to a consistently excellent system. It is a gate that has never yet done any work.

That last one has a very easy test, and I recommend executives ask it in the meeting: over the past twelve months, what verdicts has our gate returned, and how many of them were not Promote? If the answer is "Promote every time", the next question is not about the quality of the system. It is about whether independent challenge in this organisation has any real authority.

8. The Road Ahead — Something Goes Wrong After Release. What Do You Do First?

If the chapter reduces to one picture, that picture is Figure 12: five tracks answering different questions, flowing into a single gate that weighs seven dimensions, and coming out as one of four verdicts — Promote, Canary, Hold and Reject.^[1] What makes the picture a management instrument rather than a technical diagram is the phrase "one of four" — because it forces the meeting's outcome to be something you can write down, rather than a shared feeling in the room.

There are three things I would do this week, and none of them needs a budget. One — take the eight-row table and fill it in for a system that is already released. The cells you cannot fill are the evidence you do not hold about the system your users are using right now. Two — time one real rollback drill and write the number into the file. Three — separate the Thai golden-utility figure from the pooled one and look at the gap between them. My guess is that the answer will make your next meeting considerably more interesting.

And there is one question this chapter deliberately leaves half-answered: what you do first when something actually goes wrong. The short answer, which the playbook already gives and the next post will open out in full, is preserve the evidence before you change the system — because rushing to fix before capturing the manifest and the traces destroys the only record that could tell you what caused it, and learning is complete only when the corrective control is verified and recurrence is monitored.^[1]