Evidence Before Change — จากเหตุการณ์ผิดพลาดสู่ระบบที่ดีขึ้นEvidence Before Change — From Incident to Improvement

🤔 A customer received the wrong eligibility explanation, taken from an expired promotion page still sitting in the source allow-list — is that the fault of the model, of the corpus, of the rails, or of the process?

The previous post — #14 Five Tracks, One Release Gate — ended at the release gate, the eight-row table that forces a release to become a decision you can write down: Promote, Canary, Hold or Reject. And I left it with a sentence every team nods at and almost no team rehearses: production is not the end of evaluation, it is the final evaluation condition.^[1] Which leaves the question you cannot dodge: when that final evaluation condition returns the result "it has happened", what do you do in the first minute?

Chapter 9's one-line answer is short and cuts against every engineer's instinct — preserve the evidence before changing the system. Rushing a patch into production before capturing the manifest and the traces destroys the only witness that can say what caused this, and because "learning is complete only when the corrective control or design change is verified and recurrence is monitored" — not when service comes back.^[1] This post walks from the three definitions people confuse most, through the eight-stage loop in Figure 13, through the nine-group trace schema that makes reconstruction actually possible, to a worksheet you can fill in during a real meeting — and it walks the CX-REFUND-01 incident through the whole loop, one stage at a time.

1. Do Not Settle for "the Model Hallucinated" — Incident, Near Miss and Hazard

In the incident reviews I have sat through, the sentence that appears most often on the summary slide is "the model hallucinated" — and it is the fastest way to close down learning I have ever seen. Once the root cause is written as a property of the model, all that remains to do is "change the prompt" or "wait for the next model", even though the same incident walked past rails, past retrieval, past checks, past an approval, and past human hands at several gates on its way out.

So the playbook closes Artifact 8 with a rule I want every team to print and pin on the war-room wall.^[1]

"Preserve the failed case before changing it. Do not reduce a system incident to 'the model hallucinated.' Ask which boundary admitted, trusted, authorized, released, or failed to observe the behavior."

The question in that last sentence is a real diagnostic instrument, not a rhetorical flourish, because those five verbs point at five different places in the system — admitted is the input rail or the source allow-list; trusted is the retrieval layer and the support scorer; authorized is the guard that stands in front of the effect; released is the output rail; and failed to observe is observability worn so thin that the signal never reaches a person. If your answer is "the model", you have not answered a single one of them.

Three words to separate in the first minute

Before the loop starts, you have to know which kind of event you are in, because the kind determines what the organisation owes and to whom. The playbook defines it in two sentences, compressed almost to an equation.^[1]

"An incident is a realized harm. A near miss or hazard reveals credible harm before full impact."

The observation that matters here, and that is usually read straight past, is that the playbook treats near miss and hazard as a single pair rather than as two categories. If you genuinely want a three-way split you have to borrow a definition from elsewhere, and the closest fit is the AI Incidents and Hazards Monitor (AIM) run by OECD.AI, which published its own definitions in May 2024: an AI incident is an event where the development, use or malfunction of an AI system leads to one of the listed harms, directly or indirectly, while an AI hazard is an event that "could plausibly lead to" the same categories of harm.^[3] It has to be said plainly that this set of definitions is what the OECD publishes for its own monitoring tool — not a standard binding anyone, and not a legal definition.

On the standards side, NIST AI 600-1 — the Generative AI Profile of the AI Risk Management Framework, published in July 2024^[2] — reproduces a definition of AI incident in Appendix A.1.8 rather than coining one, writing that an AI incident "can be defined as" an event or series of events where the development, use or malfunction of AI systems contributes to injury or harm to the health of a person or a group (psychological harms included), disruption to the management and operation of critical infrastructure, violations of human rights or a breach of obligations under law protecting fundamental, labour and intellectual property rights, or harm to property, communities or the environment.^[2] The thing I want you to notice is that this list never says "the system went down" — it is a list of harms to people, not a list of symptoms in a machine.

This difference is not a matter of vocabulary. In practice it settles two things at once — first, whom the organisation owes remedy and notification to, and second, what price the team pays for the lesson. A near miss is the same quality of evidence as an incident, obtained for free, and a great many organisations pay full price while the free version sits in front of them, purely because there is no field in which to record it.

2. Figure 13's Eight-Stage Loop — from Detect to Learn on the Axis "Evidence Before Change"

Figure 13 of the playbook draws the incident-learning loop as eight nodes turning around a single core, and that core is not a step — it is the principle that governs every step.

The loop in the figure has eight stages^[1] — Detect, Contain, Preserve, Diagnose, Remediate, Re-evaluate, Recover and Learn — turning around a dark core that reads EVIDENCE BEFORE CHANGE.

The sentence the playbook uses to order the response in the chapter prose reads like this, and I quote it whole because the second half is worth more than the first.^[1]

"Respond by detecting, containing, preserving evidence, scoping, diagnosing, remediating, reevaluating, recovering, and learning. Bound effects first, preserve the manifest and traces before modification, inspect authoritative external state, and add both the actual case and hidden neighboring cases to evaluation."

That second half is four instructions already ranked in order of priority — bound the effects first, then preserve the manifest and the traces before modifying anything, then inspect the authoritative external state, and finally add both the actual case and the hidden neighbouring cases to the evaluation set. Notice that the second instruction comes before diagnosis, not after, because most diagnosis is done by changing something and seeing whether the symptom disappears — which is unintentionally overwriting the witness.

The sentence that defines when it is over closes the incident paragraph in Chapter 9, and it is a good deal stricter than what most organisations run on.^[1]

"Learning is complete only when the corrective control or design change is verified and recurrence is monitored."

The words doing the hardest work are verified and monitored, because both are verbs that require evidence behind them, not states you can declare in a meeting. Writing "fixed" in a ticket is not verification. Having the measure written in a document is not verification either. And nobody complaining again within two days is not monitoring recurrence.

The standards side walks the same road. NIST AI 600-1 carries a suggested action to develop and update incident response and recovery plans and procedures for GenAI systems, covering newly encountered and unanticipated uses, and covering the whole value chain along with points of contact for downstream actors; and another to conduct an after-action assessment verifying that the response and recovery processes were in fact followed and were in fact effective.^[2] It must be said clearly that this document is voluntary rather than binding — "Suggested Actions" is NIST's own wording, and organisations have to select and tailor them to their own use cases and risk tolerance.

3. Artifact 6 — the Nine-Group Trace Schema and the Independent Reviewer's Seven-Part Test

The Preserve stage is impossible if there is nothing there to preserve, and that is why the playbook places Artifact 6 ahead of Artifact 8 — a reconstructable trace is not something you can find once trouble starts. It is something you design in advance, while nothing is happening at all. The playbook states the artifact's purpose in a single sentence, followed by a shorter and sharper one.^[1]

"Preserve enough evidence for a later authorized reviewer to reconstruct what the system knew, proposed, checked, did, released, and why it chose that route. Auditable means reconstructable, not safe."

That second sentence is the one I want compliance teams to read slowly, because having complete logs does not mean the system is safe. It means only that if something goes wrong, we will be able to find out what. Those two things differ by enough that an organisation which confuses them can invest in the wrong direction for a whole year.

The nine event groups to capture

The playbook's trace schema has nine groups of events^[1] — Identity, Release identity, Input and state, Context and retrieval, Checks, Generation, Effect, Release and Operations. I keep this nine-group structure in both the Thai and the English track, even though the book's Thai table compresses to eight rows by merging Release with Operations, because that merge is a page-layout compression rather than a different schema, and in practice Release and Operations answer plainly different questions.

Notice that these nine groups are not merely chronological — they are ordered by the questions a reviewer will ask. The place I see teams miss most often is groups 2 and 9: most teams capture input, output and latency in full, but capture the runtime-context manifest only partially, so when reconstruction time comes they cannot say which manifest the incident happened on, and everything reconstructed after that is guesswork.

The seven-part test

The playbook closes the schema table with a single sentence that works as a usable test — you do not have to wait for a reviewer to arrive to run it on yourself today. An independent reviewer must be able to do seven things^[1]: identify the release, rebuild the decision-time evidence, replay the hard checks, see the scores together with their thresholds, distinguish the proposal from the effect, verify the final state, and explain the route the system chose.

The item cut most often is the fourth — seeing a score without the threshold that was in force at that moment — because thresholds tend to live in configuration that can change at any time and is not written into the trace. Once the threshold is adjusted after the incident, the 0.91 sitting in the evidence becomes a number nobody can interpret. And the fifth — distinguishing the proposal from the effect — is the item that separates a system designed with proposal–effect separation from one that was not, because if the trace does not separate the two, there is no way to tell whether the guard did its job or the model happened to propose the right thing.

The vocabulary for saying what these things are structurally already has a common standard behind it. W3C's PROV-O supplies three starting-point classes^[4] — an entity is a thing acted upon (one retrieved document, one payload), an activity is something that occurs over a period of time and acts on entities (one rail check, one tool call), and an agent is whatever bears responsibility for that activity (the approver, the service owner, the system acting on someone's behalf). PROV-O has been a W3C Recommendation since 30 April 2013 and carried no obsoletion or superseding notice on the access date, but its scope is very clearly bounded: it supplies a vocabulary, not an audit policy and not a storage design. The nine-group schema above is that vocabulary instantiated in our domain — related, but not the same artefact and not interchangeable.

The playbook also splits ownership three ways, and it is a split I think Thai organisations can adopt immediately — the SRE or platform owner guarantees that capture is durable and nothing is lost; the data-protection team owns minimisation and retention; and the decision owner reviews the meaning of what is captured.^[1] This split fixes the classic failure where the trace is designed by engineers alone and ends up technically complete while answering not one policy question.

4. CX-REFUND-01 — Walking One Incident Through the Whole Loop

All the theory above only means something once it walks through a single real event. So the playbook uses CX-REFUND-01, the customer-service assistant of Luma Commerce Thailand (a fictional case from the playbook), as its running example from the opening chapters, and walks it through the entire loop in Artifact 8. Every number in this section is a composed value inside a fictional case — not a threshold, not an industry average, and never to be lifted into a metrics table as a target.

What happened

At a 25% rollout, a customer received an explanation stating they were eligible within 60 days — wording that came from an expired promotion page still sitting in the source allow-list. The model then proposed a refund of THB 2,400 on day 45 from the qualifying date, which exceeded the actual entitlement.^[1]

What happened next is the heart of the whole case, and the reason I am retelling it from #13 Five Rails and the Effect Guard at a new angle — the execution guard blocked the payment, but the output rail released the wrong explanation. Put another way: not one baht left the account, and yet the customer had already received a promise the organisation could not keep.

When the stop condition triggered, the team rolled back to manifest rc4 and routed every promotion-related case to human handling.^[1] Note the order: contain first, then reconstruct.

What the trace said, and the classification that followed

The trace confirmed that no financial effect had occurred, and located the problem at three points — a corpus dated 2026-09-01, an old and expired passage, a support scorer that false-accepted, and a payload that had already been released.^[1] On that evidence the team classified the event as a severe semantic policy escape plus a near miss, with the structural execution invariant still holding.

That one classification sentence carries two ideas this series has returned to throughout, and it is exactly where most teams fail to separate them — a structural guarantee is something provable deterministically, such as "no payment executes without approval and above the limit", while a semantic estimate is a probabilistic score of how well a piece of text matches policy. In this case the first held and the second leaked, and reporting either "the system worked correctly" or "the system failed" as a single verdict misleads the executive in both directions.

The table above is what the question "which boundary admitted, trusted, authorized, released or failed to observe" produces, and I would like you to compare it with what "the model hallucinated" produces — zero rows. A classification this granular is possible only because effect mediation was designed as something separate from text generation from the start, so the trace can tell a proposal apart from an effect.

Diagnose, correct and return to service

The diagnosis identified two causal layers, neither of them the model layer — the allow-list checked IDs but not effective dates, and the scorer checked entailment but not policy currency.^[1] The corrections therefore landed with different owners: the knowledge-base owner removed the stale page, made Owner, Effective date and Expiry metadata mandatory, and added a document-conflict check, while the CX Quality team added a time-validity check and a customer-facing correction to the explanation.

Then comes the stage I think real organisations skip most often — the failed case became a named regression case, CXGS-241, run alongside the related hidden, adaptive, bilingual and duplicate-state cases until all of them passed. Only then did an independent policy reviewer approve the new manifest rc5, restored at 5% with a 24-hour review cadence.^[1]

💡 My view: the order "regression before restore" is the line between an organisation that learns and one that merely repairs things. If the failed case still has no name and is still not in the suite that runs every time, the organisation is betting that whoever fixed it will not forget — and in a system with several manifest versions and several teams, that bet always loses.

The final stage is closure, and the playbook sets conditions longer than any ticket system enforces — update the contract, the manifest schema, the source-publication workflow and the runbook; carry the same freshness check across to the returns and warranty products; and close only once the customer has been remedied, recurrence has been monitored for seven days, and an evidence review has been signed off.^[1] The line "carry the freshness check across to the other products" is what turns a single incident into a reusable improvement — and it is what the metrics in section 7 are trying to measure.

5. Artifact 8 — the Eight-Stage Worksheet: Action, Evidence, Owner and Done-when

Artifact 8 is the working form of the loop in section 2. The playbook states its purpose, its trigger and its accountable owner in three lines^[1] — the purpose is to turn a failure into containment, evidence, verified correction and organisational memory; use it when a prohibited effect, a severe or repeated semantic escape, trace loss, a privacy event, a control bypass, a drift signal or a near miss occurs; and the accountable owner is the incident commander as coordinator, with the decision, engineering, security, privacy and people owners retaining their own domain obligations.

Artifact 8's stage names are not quite the node names in Figure 13 — here they are Detect and declare, Contain, Reconstruct, Classify, Diagnose, Correct, Verify and restore, Learn, which is also eight stages but a different set of words. I use Artifact 8's set throughout this worksheet and do not blend it with Figure 13's.

The playbook gives one column, "Record and decide". I expand it into four that work in a real meeting — Action, what to do; Evidence, what must exist before the action counts as done; Owner, who holds it; and Done-when, the condition that lets the stage close. That last column is what prevents the most popular disease of incident review: moving to the next stage while the previous one is still open.

The blank worksheet — print it and fill it in

The completed worksheet — CX-REFUND-01

The next table is the same fictional case as section 4, filled into the eight-stage worksheet. In the book, the case is presented compressed into five pairs (Detect/Contain, Reconstruct/Classify, Diagnose/Correct, Verify/Restore, Learn/Close); I unfold it back to eight rows so it lines up with the blank worksheet above, without adding any fact the playbook did not supply.

6. Near Misses, Overrides and Corrections Are Evidence, Not Embarrassments

Chapter 9 closes with five operating principles, and the fifth is the one I think organisations violate most often without noticing — because they do not violate it by decision, they violate it by having no field to fill in.

💡 My view: principle 5 of this chapter reads "Treat near misses overrides and corrections as evidence. Do not suppress the signals that teach the system."^[1] I would add that these three signals are suppressed almost entirely by our own measurement systems — a team measured on incident count will report fewer incidents, not commit fewer errors. And a frontline operator's override is the highest-quality data an organisation owns, because it is the moment a real expert tells us the system is wrong, before anyone has been harmed.

The chapter's other four principles form one frame with it and should always be read together^[1] — principle 1, evidence before exposure: do not increase the affected population ahead of proof · principle 2, evaluate the system in context: a model score is not workflow performance · principle 3, match gates to impact and reversibility: stronger consequence demands stronger evidence · principle 4, make every release observable and reversible: a live system needs a service owner and a kill path. Notice that the first four are about what you do before anything goes wrong, and only the fifth is about what you do after — and that is the one requiring culture rather than architecture.

Why near misses disappear from the system

From working with engineering and customer-service teams in Thailand, I keep seeing the same three patterns. One — there is no field to fill in: the ticket system has "incident" and "normal" but no "almost", so the person who caught it in time has nowhere to record it and goes back to work. Two — reporting has a cost and no return: whoever reports a near miss gets an extra meeting, whoever stays quiet does not. Three — an override is read as a failure of the model, so it is kept in the system's logs rather than in the organisation's evidence store for learning. All three are fixable by design, not by asking people to cooperate.

The standards side already has an item aimed exactly here. NIST AI 600-1 lists a suggested action to "Establish and maintain policies and procedures to record and track GAI system reported errors, near-misses, and negative impacts" — establish and maintain policies and procedures for recording and tracking a GenAI system's reported errors, near misses and negative impacts.^[2] The term "near-miss" appears in NIST's own text; it is not my gloss. And as before, it is a voluntary suggestion that organisations must select and tailor to context, not a requirement.

Looking outward — when your own data is still too thin

The statistical problem with learning from incidents is that a single organisation almost never has enough of them to see a pattern. By the time you meet the same semantic escape for the third time, two years may have passed. The way out the playbook points to is external scanning for patterns that happened to other people, and the source it names is the AI Incidents and Hazards Monitor (AIM) run by OECD.AI.

On 5 September 2026, the AIM listing with an unfiltered all-time date range showed roughly 17,392 entries labelled incidents & hazards.^[3] That number has to be read with three warnings attached, always. One — it is a count of events selected from news reporting: the system pulls articles via Event Registry, which processes over 150,000 articles a day, then uses language models to classify events and enrich them with metadata. Two — the page labels itself Beta, and the OECD states explicitly that it cannot guarantee and does not independently verify the accuracy of third-party information; inclusion is not an adjudication of fact or of liability. Three — the page carries no denominator at all: no count of deployed systems, no comparison baseline. So this number cannot be used to express a trend, a growth rate, or a per-system probability. Its one legitimate use is to confirm that an external corpus of this size exists to scan against.

What is interesting is that NIST AI 600-1 itself names the OECD AI incident monitor in its list of external information-sharing resources, alongside the AI incident database, AVID, CVE and NVD.^[2] That is the one place where two of this post's sources touch directly, and it is why I think external scanning should be a standing agenda item rather than something you do after trouble. The way I recommend running it is a quarterly cadence in which process owners read the patterns that hit products of the same type and ask one question: "if this happened to us, which of our rails would have caught it?" That question produces a recordable hazard immediately, without waiting for anyone to be harmed.

7. Metrics for the Learning Loop, and the Failure Patterns That Keep Coming Back

Chapter 9 gives its "metrics that matter" list as a single sentence covering both evaluation and release^[1] — quality-adjusted task success, critical-error rate, supported claims, security escapes, subgroup disparity, override and reversal, feedback latency, drift, cost per successful outcome, time to detect, contain and recover, near-miss reporting, recurrence, and releases with a tested rollback. The three I have set in bold are the ones belonging directly to the incident-learning loop; the rest belong to the release gate in post #14.

The table below is the set I actually use. The first four come from the playbook, and the last two are measures I add myself — they are not in Chapter 9's "metrics that matter" list, but they follow directly from Artifact 8's Verify and restore row and its Learn row. I mark the provenance in the last column so nobody mistakes them for something the playbook requires.

A caveat that has to be stated plainly: every number from the CX-REFUND-01 case in sections 4 and 5 is a composed value inside a fictional case, and none of them may be set as a target or a threshold in this table — not the 5% ceiling, not the 24-hour review cadence, not the seven-day recurrence monitor. What transfers is the structure of the decision, not the numbers.

Failure patterns

• Closing the incident when service comes back, without checking that the correction worked — the playbook names this pattern outright in Chapter 9's failure-pattern list.^[1] The observable symptom is a ticket closed on the same day service returned, with no verification document attached to it.
• Suppressing near misses — another pattern the playbook names itself. The symptom is reported incident counts falling in the quarter where performance pressure rose, a correlation that ought to make everyone uncomfortable.
• Writing "the model hallucinated" as the root cause — a direct contradiction of Artifact 8's closing rule. A root cause has to be the name of a boundary that admitted, trusted, authorized, released or failed to observe, not the name of a technology.
• Modifying the manifest before preserving it — a direct contradiction of Chapter 9's instruction to "preserve the manifest and traces before modification" and Artifact 8's rule "Preserve the failed case before changing it". This is the hardest pattern to fix, because it comes from an engineer's good instinct to minimise harm to users. The remedy is therefore not a warning but making evidence-freezing an automatic step that fires together with the rollback.

8. The Road Ahead — from One Incident's Evidence to a Single Evidence System

If this chapter had to be reduced to one sentence, it would be the core of Figure 13 — preserve the evidence before changing the system. And if it had to be reduced to actions you can take this week without waiting for budget, I would offer three. One — take the seven-part test in section 3 and run it against one incident you have already closed; the items you cannot answer are the fields your trace does not yet have. Two — open a near-miss field in whichever ticket system you use today, and watch how many entries arrive in the first two weeks. Three — check whether a rollback in your system freezes the evidence automatically; if it does not, that is the work to do before everything else on this list.

And there is one question this chapter deliberately leaves open. Every trace, every incident record and every worksheet we filled in through this post is not used only inside the team — it is the same body of evidence that several laws, standards and external reviewers will come and ask to see, each asking in its own language. The question is whether we build a separate evidence set for each of them, or design one and map it onto every duty. The playbook chooses the latter, and opens part four with the sentence that frames the whole part.^[1]

"Part four treats governance as an operating system rather than a committee, work redesign as a social choice rather than an afterthought, and the first 180 days as a proof of one complete learning loop rather than a competition to launch the most tools."