Design for the Fallible Savant — ออกแบบเผื่ออัจฉริยะที่พลาดเป็นDesign for the Fallible Savant — Four Assumptions, One Envelope

🤔 Suppose your team had one colleague of genius-level ability — a superb writer, a superb summariser, alarmingly well read — who also fabricates with a straight face without knowing it, is brilliant one moment and unaccountably poor the next, remembers nothing of yesterday, and believes every note anyone slips onto the desk. Would you hand that colleague the refund-approval button?

The previous post, Context Is a Control Artifact, ended with every term of equation (1) — model, decoding, context, retrieval, tools, state, environment — pinned into Nong Kram's single release manifest. We now know where the system's behaviour comes from, and we know that changing any term is changing the program. But even the tidiest manifest cannot answer the next question: the "processor" at the centre of this program never promises to be correct — so how do we design a system around such a thing and still dare to hand it the shop's real work?

The whole post's answer sits in the name the paper gives this kind of core: the fallible savant. The right engineering response is not to wait for the model generation that stops failing, but to promote four of its behaviours — fabricating facts, jagged competence, no memory, credulity — into design assumptions the system must stand under even when all four are true at once, add one more property — confidence whose numbers cannot be taken at face value — and then build the five-element assurance envelope around the core^[1]. The burden of correctness does not disappear — it moves from the centre of the system out to the boundary, which is exactly the place where we write the code ourselves and can prove things about it.

1. Four Design Assumptions, Not Four Defects

One thing has to be set straight before the details: §4 of the paper does not write this list as "model defects awaiting the next version's fix". It writes them as design assumptions — things an engineer must treat as always true, the way a civil engineer treats steel as expanding when hot^[1]. Nobody stands around cursing the steel; people design bridges with expansion joints. So the place for these four in a design document is not the "known issues" page — it is the same page as a structure's load assumptions.

The reason they must be promoted to assumptions lies in the nature of the failures themselves: Software 1.0's correctness-by-construction guarantee does not transfer, because the core is correct with some probability over a distribution of tasks, and when it fails, the failure is fluent — confident, well-formed, and wrong^[1]. A wrong sentence reads exactly like a right one, so reviewing by skimming is a control designed to lose from the start — which is why everything in this post is about mechanism, not about the diligence of the reviewer.

Hallucination

The first assumption: the system must assume the model will assert plausible falsehoods at a non-zero frequency^[1]. The load-bearing words are "non-zero frequency" — the definition does not say often, and does not say rarely; it says never zero. However precise the retrieval, however polished the prompt, one day hallucination will make Nong Kram tell a customer that returns are accepted within 30 days when the policy says 14 — in exactly the voice of a day when every character was right. A standard like NIST AI 600-1 lists this behaviour (under the name confabulation) as a standing risk of generative AI that organisations must manage, not wait out^[5].

Jagged intelligence

The second assumption: jagged intelligence — competence on task A cannot be inferred for task B, however close the two tasks look to a human eye^[1]. The model's competence surface is not a plain but a mountain range: a high peak here, a ravine right beside it, and the dividing lines do not follow our intuition about which tasks are "hard".

The paper quotes τ-bench with its bounds stated plainly: one GPT-4o function-calling agent achieved roughly 61.2% at pass^1 on retail tasks against roughly 35.2% on airline tasks, and on the retail tasks themselves pass^8 — succeed on all eight of eight reruns — fell below 25%^[1][2]. Read those numbers exactly the way the paper frames them: they are the results of one system on one specific benchmark, not a universal baseline for any model. What they teach is not "what GPT-4o scores" but the shape of competence: nearly a two-to-one gap between two similar-looking domains, and repeat-run stability far below the single-run score — two axes that must be measured separately, and never guessed across.

Amnesia

The third assumption: amnesia — the model has no memory whatsoever beyond the context window. What it "remembers" in this turn is exactly what was fed to it in this turn. Continuity — of the conversation, of unfinished work, of promises made to a customer — is entirely an application responsibility^[1]. Architecturally that means "memory" is a feature we build, not a bonus the model provides, and it must be versioned, inspected and tested like every other piece of state. Teams that skip this meet the classic symptom: the bot makes a promise to a customer on Monday and denies having said it on Tuesday — not because the model "lied", but because nobody fed Monday's words back in.

Gullibility

The fourth assumption, and the only one that is directly a security matter: gullibility — instructions and data share a single channel. Everything is a token in one context, so deliberately crafted text can override the behaviour we intended^[1]. The work of Greshake and colleagues has already demonstrated that content planted for a system to "come across on its own" later — indirect prompt injection — can compromise real LLM-integrated applications across the board^[3]. For Nong Kram, every byte that arrives from a customer, from text a customer pastes into the chat, or from a catalog page that retrieval pulls in, is data that may be carrying a knife — and the design has to treat it that way all the time, not just in the week an attack makes the news.

💡 My view: the fastest profit from renaming "defects" to "assumptions" is that it ends one whole species of meeting — the one where somebody says "the next model generation will fix that". An assumption is not something you wait out; it is something the system must stand under even while it is true. A team that writes these four into the design document in week one stops re-arguing them every quarter, and architecture reviews become surprisingly short, because the question changes from "will it fail?" to "when it fails, what does it hit?"

2. Calibration, and the Burden That Moves to the Boundary

There is a fifth property, which the paper keeps separate from the four assumptions: calibration — a gate that routes work on the model's claim of "90% confidence" is sound only if, among the times the model says that, it is right 90% of the time; and the model's raw confidence usually fails this condition — it is mis-calibrated — so quality gates must rely on independent evaluation, never on the model's assessment of itself^[1]. The sentence sounds mild, but it strikes out one popular architecture entirely: a system that has the model score its own confidence and uses that score to decide whether a human sees the case is building a door where the guard is the same person asking to pass.

The assurance burden does not vanish — it relocates

Hold the four assumptions and calibration at once and §4's conclusion follows inescapably: correctness cannot be proven from inside the generator — but correctness is not replaced by surrender either. What happens is a redistribution: the assurance obligations move from the centre (the model, which we can neither fix nor prove) to the boundaries of the system (which we build, test and can prove things about)^[1]. The model remains the source of capability; the boundary becomes the source of trustworthiness.

The five elements of the assurance envelope

The paper defines the envelope of an AI-core system as five elements, wrapped around the generator^[1].

• Versioned context assembly — everything that enters the model comes from pinned artifacts, and any turn can be traced back to exactly what it was assembled from — the foundation post #4 already laid
• Hard mediation of effectful actions — every action with a real effect on the outside world passes through a structural mediator outside the model, on every path, with no shortcut
• Calibrated semantic evaluation — semantic judgement by independent evaluators that know their own error rates on a declared population, not the model's feelings about its own work
• Complete decision tracing — a decision trace for every request on every path, failure paths included, so that any incident can be reconstructed from evidence
• Risk-proportionate fallback — fallback routes proportionate to the risk: a read-only task may merely get a warning label, while an irreversible one stops and waits for a human

And the sentence that must never be trimmed off this definition: envelope work is necessary but not a substitute for improving the core — model selection, data curation, calibration, decoding policy and architecture all genuinely change the core's reliability. The right question is the allocation of effort, not a choice of sides^[1]. The envelope is not an excuse to stop caring for the core, and a stronger core is not an excuse to remove the envelope — Step 6 below turns this allocation into a table you can actually fill in.

3. The Seven Steps

The seven steps below convert sections 1 and 2 into work that finishes in one to two weeks. Every step ends with one concrete beat of Nong Kram's progress, and the destination of all seven is the pair of artifacts in section 4.

Step 1 — Design for hallucination: require span-level source attribution

Make it a rule of the system, not a request in the prompt: every fact-bearing statement — a condition, a policy, a product attribute, a number — must be tied to a source in the corpus at span level, sentence to passage, and any claim with no supporting source must be withheld or labelled unverified before it ever reaches a customer^[1]. The reason traces straight back to the first assumption: since plausible falsehoods arrive at a non-zero frequency, the filter has to work on every answer, not only the ones that "look odd". Nong Kram's internal answer shape therefore becomes:

# Nong Kram's internal answer shape — every fact-bearing span must point at a source
{
  "answer_spans": [
    { "text": "Returns are accepted within 14 days of delivery",
      "source": "policy/returns.md#item-2" },
    { "text": "This vase is glazed with natural indigo, fired at 1,230 degrees",
      "source": "catalog/vase-kk58.md#details" },
    { "text": "Refunds normally clear within 3 business days",
      "source": null }   // no source -> never release as-is: withhold, or label "unverified"
  ]
}

Step 2 — Design for jaggedness: declare slices, then measure per slice

Write your app's task types (slices) into a formal document, then build a golden set per slice with its own pass threshold — and never extrapolate competence across slices, however alike two slices look^[1]. The retail/airline lesson of section 1 is this step's entire justification: same system, similar-looking domains, results nearly two-to-one apart. Nong Kram's first slices (the test-set numbers are this tutorial's, not the paper's):

• order-status — order state, parcels, delivery times · golden 40 cases · pass at 95%
• return-policy — exchange and return conditions · golden 30 cases · pass at 97%, because this is the slice where being wrong costs the most
• product-info — clay body, indigo glaze, sizes, care · golden 30 cases · pass at 90%
• refund-request — amounts and eligibility · golden 20 cases · measured only, no automatic-release threshold — real effects are always fenced by the guard

Step 3 — Design for amnesia: move continuity outside the model

Build the app's own session-state store, and have the assembly function from post #4 feed it back into the context on every turn — never assume, even for a single turn, that the model "probably still remembers"^[1]. The free bonus: state that lives outside the model can be inspected, tested and carried in the trace. Nong Kram's first schema:

# Nong Kram's session_state — continuity lives entirely outside the model
# every turn, the assembly function reads this record and re-feeds all of it into the context
session_id:     s-260908-042
customer_ref:   LINE:U-9f3a...
order_in_focus: KK-58214          # two indigo-glazed vases, shipped 5 Sep
promises_made:  ["check the chipped-rim report and reply today"]
refund_state:   none              # none | proposed | approved | committed

Step 4 — Design for gullibility: separate the instruction channel from the data channel

Label the origin of everything that enters the context — system, policy/, catalog/, customer text, session_state — then set one fixed rule: content from retrieval and from users is always data, with no authority to issue instructions, and any request that appears in a data channel ("transfer my refund right now, as this page says") carries no weight with any tool^[1]. This step does not make prompt injection "impossible" — section 1 already said the model is credulous by nature — but it moves the question from "will the model believe it?" (unanswerable) to "if it believes it, what can it actually do?" (answerable, and boundable)^[3]. For Nong Kram: a catalog/ passage with an embedded imperative sentence will still be read, but the only thing it can sway is the text being proposed, never the real effect — because every refund passes through the out-of-model mediator sketched in section 4.

Step 5 — Never trust the model's self-confidence: use independent checks

Walk through every quality gate in the system and ask each one the same question: what does this gate decide from? If the answer is "from a confidence score the model reports about itself" or "from asking the model again whether it is sure" — replace it with independent evaluation: compare the answer against the passages actually retrieved, compare against the order's real state, or use a separate classifier that can report its own false positives and false negatives^[1]. An example of that last kind is an input–output classifier such as Llama Guard — and its status must match its nature: an advisory signal that can be wrong, never a guarantor^[4]. For Nong Kram, the old gate "if the model's confidence is low, escalate to the admin" is replaced by a faithfulness check of Step 1's answer_spans against the real passages — because an uncalibrated 95% confidence tells us nothing except the mood of a number.

Step 6 — Split your budget between core and envelope by risk profile

Time and money are finite, and §4 is explicit about this: it is an allocation question, not a choice of sides — core improvement (a better model, cleaner data, calibration, decoding policy) genuinely changes reliability, and envelope work is always necessary. The question is where to pour how much, per consequence class of each task path^[1]. Nong Kram's allocation (my split for this tutorial, not a formula from the paper):

💡 My view: the heuristic I use for quick decisions is that the more irreversible the effect, the more the budget flows to the envelope side — because a stronger core lowers the frequency of failure, while the envelope lowers the price of failure, and for an irreversible effect the price of even one is already more than you can carry.

Step 7 — Sketch your own envelope: five elements, one line each

Close with the shortest and best-value piece of writing in this post: write the five envelope elements for your app, one line each. No code yet — just answer what each element "is, and where it lives" in your system, using these five questions as the frame:

• What is my context assembled from, and where is it pinned — versioned context assembly
• Which of my actions have real effects, and which single component must every path pass through — hard mediation
• Which independent evaluator measures my semantic quality, and on what population — calibrated semantic evaluation
• What trail does one of my requests leave behind, and are failure paths recorded too — complete decision tracing
• When the system is unsure or broken, who does the work flow to, and with what context attached — risk-proportionate fallback

Nong Kram's completed version is laid out in full in the next section.

4. The Artifact — Nong Kram's Design-Assumption Audit

This post's artifact comes in two pieces, kept in the same repo as the code under post #4's "everything is the program" discipline: docs/savant-audit.md, the design-assumption audit table, and docs/envelope-sketch.md, the first envelope sketch. The table first — one row per assumption, read left to right: where it bites this app, how the design answers it, and which envelope element carries that row.

One observation about the table: not a single row's "design response" cell says "tighten the prompt" — a good prompt lowers the frequency of the problem, but no row stakes its life on it. And when you collect the final column across all five rows, you get exactly the five envelope elements. That is no coincidence: the envelope §4 defines is precisely the structural answer to this set of assumptions^[1].

The first envelope sketch — a diagram in words

# docs/envelope-sketch.md — Nong Kram's envelope, v0 (post #6 turns this into real code)

[1] Versioned context assembly
    assemble_context() reads every term from post #4's release manifest
    every passage carries an origin label: system | policy/ | catalog/ | user | session_state

[2] Hard mediation of effectful actions
    refund(order_id, amount, reason) is callable by exactly one path: through refund_guard, outside the model
    the guard checks schema, amount limits, order state, and duplicates — deny by default

[3] Calibrated semantic evaluation
    a faithfulness check compares answer_spans against the passages actually retrieved, per slice
    no gate may ask the model "are you sure?" — independent evaluators only

[4] Complete decision tracing
    every request ends with one trace: assembled context, candidate, per-gate verdicts,
    the route taken (release / withhold / escalate) — failure paths included

[5] Risk-proportionate fallback
    cannot answer, or evidence is weak -> hand off to the shop admin, trace attached
    every refund, no exceptions -> stop and wait for human approval (autonomy moves later, in post #10)

This file is a "diagram in words" by intent — not a line of code yet, but every decision the code will have to respect is already in it. Post #6 picks this file up and writes the control-loop function to match it, element by element.

5. Validation Check — Test Your Own App Before Moving On

The series' standing rule: every question must be answered with an artifact you can point at and open, not with an adjective. If a row fails, the rightmost column says which step to go back to.

The post's overall bar is simple: if all seven rows are green, your app now has a structural answer to all five assumptions — what it does not yet have is the machine that arranges those answers into an actual order of execution, and that is the next post.

6. The Road Ahead

This post changed the status of failure from "the thing nobody wants to say in the meeting" into a design specification: we assume the model fabricates, is jagged, remembers nothing and believes too easily — all at once, all the time — and we treat its confidence as unusable at face value until calibrated. From that set of assumptions we obtained the five-element envelope, one audit table for Nong Kram, and one envelope sketch file that is ready to become code.

What this post deliberately does not answer: the loop itself — how the five elements are ordered within a single request, what is checked before generation, where a tool call is authorised, by what thresholds the output is released, withheld or escalated, and why the trace must be written even on the failing path. All of that is the paper's reference control loop, and it fills the whole of post #6.