Specify → Constrain → Generate → Verify — วงจรพัฒนายุค AI-CoreSpecify → Constrain → Generate → Verify — The AI-Core Function

🤔 Your team trusts its own code because the edit → compile → run → debug loop turns every day — so on the day the system's behaviour comes out of a model you cannot open up and edit, what loop do you turn to trust that behaviour?

The previous post, Design for the Fallible Savant, ended on the series' central bargain: we design the system on the assumption that its core is a fallible savant — it asserts plausible falsehoods, its competence is jagged, it remembers nothing outside the context, it is gullible, and its confidence does not track its accuracy — and the engineering answer is an assurance envelope around the core, not a prayer that the model stops failing. But that post was still the blueprint. Now we pour the concrete: what does that envelope look like as code, which function calls which, and what does a single KramKraft customer request pass through before the answer reaches their LINE screen?

The whole post's answer is the coding discipline of the paper's Section 5^[1]: the Specify → Constrain → Generate → Verify loop that runs in parallel with edit→compile→run→debug, and one reference control loop that makes the discipline concrete — check before generating and reject before the model, authorise before any real effect, verify before releasing, and write a trace on every path including the one where the code breaks. This is deliberately the most code-heavy post of the series, because we will write Nong Kram's ai_core_operation() for real, one chunk at a time, until the whole loop of roughly ninety lines is assembled — short enough to read over one cup of coffee, and that is the point.

1. Four Verbs — the Development Loop Running Beside edit → compile → run → debug

The paper states the discipline plainly: behaviour that comes from a frozen model plus runtime-assembled context has to be developed with a loop of its own — Specify the behaviour with examples and acceptance criteria, Constrain the space before and after the model, Generate with the model proposing inside a quarantined stage, Verify the result with checks and evals — and this loop does not replace edit→compile→run→debug but runs beside it, because the system's 1.0 code is still all there, still holds the control flow, the interfaces and every mediator that everything depends on, and still gets developed the 1.0 way^[1].

The rhythm-by-rhythm pairing in the table below is my own rendering for teaching — the paper speaks of a parallel loop without pairing the words one to one — but it makes visible where each rhythm we know by hand has a twin whose nature is different.

Debugging becomes evaluation

In the 1.0 loop, debugging ends when you can point at the wrong line. But post #2 of this series already laid out the 3.0 failure signature: the failure may not recur on a re-run, and diagnosis needs the full trace of that one invocation^[1]. So the paper redefines the whole job: reproduce the wrong behaviour first — pull the context, the manifest and the tool results from the failing request's trace — then add it as a case to the golden set and let that case drive a metric. The unit of debugging work is therefore not a patch that erases one symptom, but an evaluation case that pins that behaviour down for the life of the system — the moment the model changes version, this case runs again.

Concretely, for Nong Kram: suppose a customer asks for a refund on day 45 after purchase and Nong Kram answers "of course", even though the shop's policy (our tutorial's value) allows returns within 30 days. What the team does is not merely tweak the system prompt and re-ask until the answer looks right — it captures that request as case refund-012, with the correct answer and the citation that must be given, adds it to the golden set, and measures the refund task's pass rate across the whole slice, from today until every future re-pin of the model.

The artifact set that needs code-grade discipline expands

The direct consequence of this loop is that the list of things requiring versioning, review and regression protection no longer stops at source code. The paper enumerates it explicitly^[1]:

• Prompts — the text that declares intent and rules to the model; changing one sentence changes the program
• Context templates — the assembly skeleton that decides what the model sees and in what order
• Tool schemas — the contracts of the new era's system calls, which all validation anchors on
• Golden datasets — the golden set that is at once the behaviour spec and the regression barrier
• Release manifests — the record that pins every term of the program together per release
• Eval suites — the code of the Verify side, which must be reviewed as strictly as production code

This discipline did not appear from nowhere — ML engineering has had rubrics like the ML Test Score forcing data and models to be tested and versioned like production assets since 2017^[4]. What the 3.0 era adds is a whole new family of artifacts — prompts, templates, schemas, manifests — that determine behaviour while being neither code nor weights.

💡 My view: treat the golden set as the team's bug tracker — a behavioural bug that has not yet been converted into a golden case is not fixed, it has merely gone quiet, and it will come back quietly in the next model version at the moment nobody is watching.

The paper demotes its own loop — and that is the point to remember

Section 6 of the paper does what I wish every framework author did: it openly demotes its own loop. The four-verb cycle is a development workflow only — it orders the team's daily activity, but it establishes no property of the system, and it is not a "contract" the system gives anyone^[1]. One example shows why: a team that diligently performs all four verbs, but has one admin script calling refund() directly without the guard — that team has a complete process and zero guarantees.

2. The Reference Control Loop — Listing 1.1, Stage by Stage

The paper's whole Section 5 condenses into a single function — the paper calls it the "AI-core function" — and the structure of that function is the discipline itself: no path reaches the model without the prechecks, no real effect happens without the guard, nothing is released without passing the thresholds, and no exit exists that does not write a trace^[1]. This section walks Listing 1.1 stage by stage, so that every line's position explains itself.

The full skeleton, as pseudocode transcribed from the paper's listing — every principal function name is the paper's; the minor abbreviations are mine, to keep it readable on a narrow screen:

function ai_core_operation(request):
    trace = begin_trace(request, release_manifest)  # open the trace before any work
    state = initialise_state(request)
    try:
        loop:
            context = assemble_and_record(request, state, trace)
            pre = evaluate_pre_generation(request, context)
            if not pre.ok:                  # enforced, not merely logged
                return reject(request)      # exits before the model core
            proposal = model.generate(context)  # the only line that calls the model
            record_candidate(trace, proposal)
            if proposal.has_tool_calls:
                for call in proposal.tool_calls:
                    auth  = execution_guard.authorise_validate(
                                call, request, state, policy)
                    route = route_effect(call, auth, risk_policy)
                    if route == REJECT:
                        result = rejected(auth.reason)
                    elif route == NEEDS_APPROVAL:
                        result = wait_for_human_approval(call)
                    else:
                        result = sandbox.invoke_transactionally(call)
                    record_tool(trace, call, route, result)
                    state = incorporate(state, call, result)
                continue    # re-assemble the context, re-run every check
            break           # no tool call left: the candidate is the answer
        checks   = evaluate_output_dimensions(proposal, context)
        verdicts = apply_thresholds(checks)
        route    = output_route(checks, verdicts, risk_policy)
        if route != RELEASE:
            return withhold_repair_or_escalate(route, request)
        return execute_route(RELEASE, proposal)
    finally:
        log_trace(trace)    # every path, including failure

Before the model — open the trace, then check with the power to end the request

The first line opens the decision trace by binding the request to the release manifest while nothing has happened yet, because every event that follows must always be able to answer which version of the program it happened on — post #4 of this series laid down the principle: changing any term of equation (1) changes the program, so a trace that does not know the program's version is merely a log. Then assemble_and_record builds the context and records what was actually assembled, before the point where the paper raises its voice loudest in the whole listing: pre-generation checks must be enforced, not logged — a failing check must return out of the function before model.generate, not note the problem in a log and carry on^[1].

A side benefit teams tend to overlook: the reject path is the most deterministic, fastest and cheapest path in the system — no token cost, no variance. Programmable-rails tooling like NeMo Guardrails lets this check layer be declared as configuration^[2], but the substance of this post is not the tool — it is position and authority: the check must sit before the core, and it must have the power to end the request on its own.

The effect boundary — three exits for a tool call

If the candidate proposes a tool call, the loop never simply follows the proposal, because post #3 of this series already marked this seam: a tool call is a system call that arrives as a natural-language request, so authorisation must always be enforced outside the model^[1]. execution_guard.authorise_validate checks the call against the request, the state and the policy, and route_effect splits the outcome into three exits that are visibly separate code paths:

1. REJECT — the default for everything that cannot prove itself (deny by default); the tool result becomes a refusal the model gets to see on the next turn
2. NEEDS_APPROVAL — wait_for_human_approval: a call that is valid but large or irreversible must have a human decide before any real effect
3. EXECUTE — sandbox.invoke_transactionally: it happens entirely or not at all, with no half-state left to guess about

The result is recorded into the trace and incorporated into the state, and then the loop goes back to re-assemble the context and re-run every check — notice that the second turn inherits no accumulated trust from the first: the same prechecks run again on every turn, because a context that has just absorbed a tool result is a new context that has never been checked.

The output side — thresholds, the non-release paths, and the finally that matters most

When no tool calls remain, the final candidate enters the output gate: evaluate_output_dimensions measures per dimension, apply_thresholds compares against the declared bars, and output_route decides the path. If it is not RELEASE, the system chooses between withhold, repair and escalate — all three are designed paths with response patterns of their own, not exceptions floating up for middleware to handle. And it closes with the line I consider the most sacred in the listing: finally: log_trace — every path, including the one where the code breaks halfway, must always leave a trace behind^[1].

And this is the type division the loop makes visible all the way through, which will become the backbone of the contract in the next post: structural checks — schemas, allow-lists, parameter ranges, citation membership — enforce a structural guarantee, a bounded invariant on every run; semantic verdicts — faithfulness, relevance, tone — remain a semantic estimate that can always be wrong at some rate^[1]. The shape of the loop guarantees where things can happen — it never guarantees that the content is good.

3. The Seven Steps

The seven steps below turn the pseudocode into Nong Kram's real Python, one chunk at a time. Every chunk is the real thing that section 4 will assemble into three files. Every number you see — the 1,500-baht approval threshold, the 0.80 faithfulness bar, the 30-day return window — is a tutorial value I set so the example runs; none of them is a number the paper measured or recommends.

Step 1 — Specify: write golden examples and per-task acceptance criteria

Start by declaring the desired behaviour as cases, not as an essay: for each task, pick real examples covering both what should be done and what should be refused, with acceptance criteria a machine can judge pass/fail. The reason comes from the heart of the 3.0 era: the behaviour spec is intent plus exemplars, and the golden set is the one place that spec can be re-checked^[1]. For Nong Kram's refund task I start with the two head-of-line cases — one that must call the tool, and one that must refuse while citing policy:

# golden/refund.py — the golden set for the refund task (tutorial numbers)
GOLDEN_REFUND = [
    {"id": "refund-007",
     "request": "order KK-1042: mug arrived cracked, asks for a refund",
     "expect": {"tool": {"name": "refund", "order_id": "KK-1042"},
                "cite": ["policy/returns#damaged"]}},
    {"id": "refund-012",
     "request": "order KK-0980: changed mind on day 45, asks for a refund",
     "expect": {"tool": None,          # past the shop's 30-day return window
                "cite": ["policy/returns#window"]}},
]
ACCEPTANCE = {"tool_exact": 1.00,     # tool calls must match exactly
              "cite_only_retrieved": 1.00,
              "answer_pass_rate": 0.90}  # tutorial threshold, per slice

Step 2 — Constrain: declare schemas, allow-lists, parameter ranges and prechecks

Every constraint must be declared up front as structure, not pleaded for in the prompt — a sentence in the context is advice to the model, not a constraint on the system. This step is the behavioural loop's "compile" beat: what is malformed must fall before anything runs, and a failing precheck must have the power to end the request itself, before the core^[1]:

# kramkraft/constraints.py — declared up front, not pleaded for in the prompt
ALLOWED_TOOLS = {"refund"}            # Nong Kram's only effectful tool
TOOL_SCHEMA = {"refund": {
    "order_id": r"^KK-\d{4}$",
    "amount":   "int, 1..order_total_thb",
    "reason":   "str, 1..200 chars",
}}
MAX_CONTEXT_TOKENS = 6000

def evaluate_pre_generation(request, context):
    if request.channel not in {"web", "line"}:
        return Check(False, "unknown channel")
    if not context.passages:
        return Check(False, "no policy passage retrieved")
    if context.token_estimate > MAX_CONTEXT_TOKENS:
        return Check(False, "context overflow")
    return Check(True, "ok")

Step 3 — Generate: quarantine the model call as the one stage that touches nothing

The model call must be a fully quarantined stage: context in, a "proposal" out, and in between it touches nothing at all — no database reads, no network, no tools. The reason is blunt: every guarantee in the loop rests on the fact that every real effect must pass the mediator in the next step; if the model call can have side effects of its own, everything else is theatre. In Nong Kram's code, this doorway has exactly one door in the whole system:

# kramkraft/model_port.py — the one gateway in the system that talks to M_v
class ModelPort:
    def __init__(self, client):
        self._client = client         # frozen hosted model, version pinned

    def generate(self, context):
        raw = self._client.complete(context.prompt)
        return Proposal(text=raw.text,
                        tool_calls=parse_tool_calls(raw))
        # returns a proposal only — touches no store, network, or tool

Step 4 — Mediate: every effect passes the mediator before it happens, and the default is refusal

This is the step that carries the whole system's guarantee: every proposed tool call must be authorised and validated outside the model before any real effect, under the principle of complete mediation — no back-door path, no admin exception — and deny by default: a call that cannot prove every requirement is a rejected call, while a call that is valid but large or irreversible must stop and wait for a human^[1] — and for Nong Kram, a refund is irreversible from the second the payment processor accepts the instruction:

# in the loop: a proposal is not an effect until it passes the mediator
auth  = guard.authorise_validate(call, request)
route = route_effect(call, auth, risk_policy)
if route == "EXECUTE":
    result = sandbox.invoke_transactionally(call)   # all or nothing
elif route == "NEEDS_APPROVAL":
    result = wait_for_human_approval(call, request) # a human decides
else:
    result = ToolResult.rejected(auth.why)          # the default is refusal

def route_effect(call, auth, risk_policy):
    if not auth.ok:
        return "REJECT"
    if call.args["amount"] > risk_policy.approval_threshold_thb:
        return "NEEDS_APPROVAL"       # large and irreversible: needs a human
    return "EXECUTE"

Step 5 — Verify: structural checks are hard, semantic verdicts are estimates

The output gate has two kinds of check that must never share a line. The first kind can be decided deterministically: the payload must parse against the schema, and every citation must be a member of the passages actually retrieved — this is hard. The second kind is a question like "does the answer match the policy text", which only a fallible evaluator can answer. Model-family checkers like Llama Guard exist and are useful, but by nature they are classifiers with error rates of their own^[3] — and never route on the confidence the model reports about itself, because the previous post made it plain that its calibration cannot be trusted; use independent evaluation^[1]:

# the output-check chunk — hard lines kept separate from estimates
checks, payload = output_checks(proposal, context)  # structural: hard
verdicts = {"faithfulness": deps.judge(payload, context)}  # an estimate

def output_checks(proposal, context):
    payload = try_parse_json(proposal.text)
    schema_ok = payload is not None and \
        {"reply", "policy_refs"} <= set(payload)
    retrieved = {p.ref for p in context.passages}
    attribution_ok = schema_ok and bool(payload["policy_refs"]) and \
        set(payload["policy_refs"]) <= retrieved
    return {"schema": schema_ok, "attribution": attribution_ok}, payload

Step 6 — Route: make release / withhold / repair / escalate first-class code

The paths that do not release must have the same standing as the path that does: written as their own functions, with their own fallback answer and their own human queue — because designing the failure route into the system is what separates engineering from a demo. The routing rule mirrors step 5 exactly: a failed structural check is final, no exceptions; a low estimate means weak evidence, so it goes to a human to read, not to the bin:

def output_route(checks, verdicts, risk_policy):
    if not all(checks.values()):      # structural check failed: no exceptions
        return "REPAIR" if checks["schema"] is False else "WITHHOLD"
    if verdicts["faithfulness"] < risk_policy.faithfulness_min:
        return "ESCALATE"             # weak evidence: a human reads first
    return "RELEASE"

def execute_route(route, payload, request, deps):
    if route == "RELEASE":  return Reply.send(payload)
    if route == "REPAIR":   return retry_once_then_withhold(request, deps)
    if route == "WITHHOLD": return Reply.fallback(request)
    return deps.escalate(request)     # ESCALATE: into the human queue

Step 7 — Trace: write everything in finally, and let the release fail if the trace fails

The last step is the shortest and the least negotiable: the trace write lives in finally so that every path — release, refusal, escalation, or a crash mid-flight — always leaves evidence behind, and the relationship must be inverted the right way round: not "release, then try to record", but "no successful record means no release". This is fail-closed behaviour at the system's final seam^[1]:

def ai_core_operation(request, deps):
    trace = Trace(request, deps.manifest)   # opened before any work
    try:
        ...                                 # the whole loop from steps 2-6
    finally:
        trace.write(deps.trace_path)        # every path, incl. exceptions
# Trace.write deliberately raises when the write fails — an exception in
# finally replaces the pending return: nothing is released without its trace

4. The Full KramKraft Code — Three Files Wired to refund()

This post's artifact is the three files that assemble the seven chunks: kramkraft/guard.py, kramkraft/gate.py and kramkraft/loop.py — roughly ninety lines together, depending on no framework. The Reply, Auth, Check and ToolResult classes are straightforward dataclasses I leave to the reader. Start with the mediator that carries all of the guarantees:

# kramkraft/guard.py — the execution guard for the refund tool
# every number here is a tutorial value, not one the paper recommends
ALLOWED_TOOLS = {"refund"}

class ExecutionGuard:
    def __init__(self, order_store, refund_ledger):
        self.orders = order_store     # authoritative order state
        self.ledger = refund_ledger   # one refund per order, ever

    def authorise_validate(self, call, request):
        if call.name not in ALLOWED_TOOLS:
            return Auth(False, "tool not in allow-list")
        order = self.orders.get(call.args.get("order_id"))
        if order is None or order.customer_id != request.customer_id:
            return Auth(False, "no such order for this customer")
        amount = call.args.get("amount", 0)
        if not (0 < amount <= order.total_thb):
            return Auth(False, "amount outside 0..order_total")
        if self.ledger.has(call.args["order_id"]):
            return Auth(False, "duplicate refund for this order")
        return Auth(True, "within declared bounds")

def route_effect(call, auth, risk_policy):
    if not auth.ok:
        return "REJECT"               # deny by default
    if call.args["amount"] > risk_policy.approval_threshold_thb:
        return "NEEDS_APPROVAL"       # irreversible and large: a human decides
    return "EXECUTE"

Notice that the guard always reads real state from the order store — order ownership, the total, the refund history — and never trusts a single value narrated in the model's text. And the ledger is the plainest possible idempotency key: one order can be refunded once in its lifetime, no matter whether the model proposes again, the customer taps again, or the system retries on its own.

# kramkraft/gate.py — the output gate and the trace writer
import json

def try_parse_json(text):
    try:
        return json.loads(text)
    except ValueError:
        return None

def output_checks(proposal, context):
    payload = try_parse_json(proposal.text)
    schema_ok = payload is not None and \
        {"reply", "policy_refs"} <= set(payload)
    retrieved = {p.ref for p in context.passages}
    attribution_ok = schema_ok and bool(payload["policy_refs"]) and \
        set(payload["policy_refs"]) <= retrieved
    return {"schema": schema_ok, "attribution": attribution_ok}, payload

class Trace:
    def __init__(self, request, manifest):
        self.events = [{"begin": request.id, "manifest": manifest.version}]
    def record(self, kind, **data):
        self.events.append({kind: data})
    def write(self, path):            # a failed write raises: fail closed
        with open(path, "a", encoding="utf-8") as f:
            f.write(json.dumps(self.events, ensure_ascii=False) + "\n")

The line worth stopping at is attribution_ok: forcing every citation to be a member of the set of passages actually retrieved is a structural check — set membership is one hundred percent deterministic to decide — while the question of whether the answer's content actually matches those passages belongs to the judge in the main loop, which is an estimate. Those two lines stand on opposite sides of the dividing line the whole series stands on.

# kramkraft/loop.py — Nong Kram's AI-core function
MAX_TURNS = 4
RISK_POLICY = RiskPolicy(approval_threshold_thb=1500,  # tutorial values
                         faithfulness_min=0.80)

def ai_core_operation(request, deps):
    trace = Trace(request, deps.manifest)
    state = deps.init_state(request)
    proposal = context = None
    try:
        for _turn in range(MAX_TURNS):
            context = deps.assemble(request, state)
            trace.record("context", digest=context.digest)
            pre = evaluate_pre_generation(request, context)
            trace.record("precheck", ok=pre.ok, why=pre.why)
            if not pre.ok:
                return Reply.refuse(request)      # exits before the core
            proposal = deps.model.generate(context)
            trace.record("candidate", digest=proposal.digest)
            if not proposal.tool_calls:
                break                             # no effect proposals left
            for call in proposal.tool_calls:
                auth = deps.guard.authorise_validate(call, request)
                route = route_effect(call, auth, deps.risk_policy)
                trace.record("guard", tool=call.name, route=route,
                             why=auth.why)
                if route == "EXECUTE":
                    result = deps.sandbox.invoke_transactionally(call)
                elif route == "NEEDS_APPROVAL":
                    result = deps.wait_for_human_approval(call, request)
                else:
                    result = ToolResult.rejected(auth.why)
                trace.record("tool_result", ok=result.ok)
                state = state.incorporate(call, result)
        checks, payload = output_checks(proposal, context)
        verdicts = {"faithfulness": deps.judge(payload, context)}
        route = output_route(checks, verdicts, deps.risk_policy)
        trace.record("route", route=route, checks=checks,
                     verdicts=verdicts)
        return execute_route(route, payload, request, deps)
    finally:
        trace.write(deps.trace_path)              # every path

Three deliberate details: MAX_TURNS cuts the endless loop off structurally, without waiting for anyone to notice; every return — refuse, fallback, escalate or a genuine release — walks through the same finally; and execute_route returns a value, it does not send the message itself — the actual send happens after the function returns successfully, so if trace.write fails, the exception replaces the value about to be returned and nothing reaches the customer: the release path fails together with its trace, exactly as step 7 requires.

💡 My view: this loop's code is deliberately boring — no meta-programming, no framework, no abstraction beyond one layer — because boringness is a safety property. The path we are going to stand behind with "every real effect passes the mediator" has to be a path a reviewer can read with their eyes and see completely on one page. All of the cleverness has been pushed out into artifacts that can be versioned — the schema, the thresholds, the golden set — the one place it belongs.

5. Validation Check — Does Your Loop Enforce, or Merely Log?

The series' standing rule: every question is answered with an artifact you can point at, open, and attach to a review — not an adjective. This table is for your own app, not Nong Kram. A question you cannot answer does not mean the system is bad; it means the evidence does not exist yet — and question 1 matters most, because every guarantee row in the next post stands on it.

If you do only one of these this week, do question 1: open a terminal and grep for the call sites of every effectful tool in your system. Seeing with your own eyes how many paths there are — and there are usually more than you think, because migration scripts, internal endpoints and admin buttons never count themselves as "the AI system" — is the first piece of evidence the next post's contract will ask for.

6. The Road Ahead

This post turned the discipline of the paper's Section 5 into two tangible things: the Specify → Constrain → Generate → Verify loop that sets the team's daily rhythm, and the ai_core_operation() that turns that rhythm into structure — a precheck with the power to end a request before the model, a guard standing between proposal and effect, thresholds standing between candidate and customer, and a trace written on every path including the one that breaks. All of that is the machine — and Nong Kram now has all of it, in three files.

What this post deliberately does not answer is the question the risk committee will ask the moment they see the code: so what does all this machinery promise? Which rows are genuine guarantees, under which assumptions; which are only estimates with error rates in both directions; what evidence proves each one, who owns it, and what happens on breach — the answer has a name of its own: the assurance contract, and it is the entire subject of the next post.