Human in the Loop Is Not a Design — สี่เลนของกระบวนงานคนกับ AIHuman in the Loop Is Not a Design — The Four Lanes of a Human-AI Workflow

🤔 If the reviewer has two minutes per case, cannot see the sources, and cannot refuse, is that person a "control" or a designated blame-taker?

The previous post, #5 Decision Portfolio, left you with a decision inventory sorted along three dimensions and four moves to choose between — Scale, Contain, Redesign or Retire. This is the first post of the Redesign group, and the one that has to do actual work on the word "redesign", because being able to say which decision you intend to improve says nothing at all about what the workflow around that decision should look like, or where the people should stand inside it.

The one-line answer to the question above is that the person is a designated blame-taker — a human in the loop becomes a control only when that person has all four things: evidence, time, competence, and the authority to disagree. Remove any one of them and what you installed in the workflow is not a checkpoint but a ceremonial one, absorbing accountability when something goes wrong while preventing nothing. The rest of this post is how to design it so it really is a checkpoint: with four lanes, one seven-step working session, and metrics that can tell you whether the design works.

1. The Ceremonial Checkpoint — When "Someone Reviews It" Is Not Yet a Control

Chapter 4 of the playbook opens with what I consider the most expensive sentence in the book, because it demolishes the answer most teams have used to close down risk questions for two years running.^[1]

"Human in the loop is not a design. It becomes a control only when the person has evidence time competence and authority to disagree."

The argument the chapter makes on the page that follows spells the diagnosis out even more directly.^[1]

"Effective redesign begins with the decision and decomposes the job into tasks, handoffs, evidence, and consequences. A reviewer who lacks time, evidence, authority, or domain competence is a ceremonial control that absorbs blame without preventing failure."

I would like you to read the phrase ceremonial control slowly one more time, because it is not an insult. It is a diagnosis. In the governance documents of nearly every organisation whose papers I have read this year there is a line reading "a human reviews the output before it takes effect", and in nearly every case that line is all there is. Nothing follows about what the reviewer sees, how many minutes they get, what qualifies them, or where the case travels if they refuse. So the line does exactly one job: it moves accountability off the system and onto a person, without adding a single unit of capacity to prevent the error.

In design terms, the fix is to stop writing "a human reviews it" and start naming all four conditions in full, each with evidence you can point at showing that the condition exists in reality rather than in a policy. The table below is the form I use to interrogate a workflow in the first minute of every review.

Notice that the third column does not ask about intent. It asks for objects — a screen, a number, a register, a route, something you can point at. If the team answers that column with "we have a policy", that condition does not yet exist, and the checkpoint is still a ceremonial one.

This observation is four decades older than AI

I want to put some weight behind this by pointing out that it is not a problem that arrived with foundation models. Lisanne Bainbridge published an article titled "Ironies of automation" in the journal Automatica in November 1983 — 43 years before the year I am writing this — and it is still cited continuously, with Semantic Scholar recording 2,572 citations on the day I checked.^[2]

Two things have to be said straight. First, that article was written about control systems in industrial plants. It was not written about modern AI, and it did not "predict" this era's human-in-the-loop problem. Second, the article sits behind the publisher's subscription wall and its abstract has been withdrawn from the indexing databases, so I can confirm only its bibliographic identity, not its text — I therefore cite the title and the year and quote not one sentence from it. Connecting it to the reviewer problem in an AI workflow is my own synthesis, not the author's conclusion.

But the logic runs strikingly parallel. Safety engineering has known the pattern for a very long time: when an automated system performs the normal case better than a person almost all of the time, the role left to the person is "watching", which is the task humans do worst — and then, in the second the system fails, we expect someone who has not had their hands on the work for a long while to take over in the hardest situation there is. This is why the playbook refuses to let "a human reviews it" count as a design, and insists that design begins at the decision and then decomposes the job into tasks, handoffs, evidence, and consequences.

That phrase, "begins at the decision", picks up exactly where the previous post left off. The decision inventory you built in #5 is the starting point of the workflow — not a list of tools. And the next section is the reason why starting at the wrong point produces results that look excellent on a slide and move nothing in reality.

2. Task Automation vs Workflow Redesign — Ten Times Faster, Delivered Just as Slowly

The masterclass the playbook is drawn from separates these two terms in the 19:44–22:16 stretch, and the playbook expands on it in Appendix A §8.^[3] The substance is that automating a single task inside a badly designed workflow can produce a hugely impressive local number while changing the end-to-end result almost not at all. The example the playbook gives is a document that used to take two hours to draft and now takes two minutes — and still waits through five levels of approval exactly as before.

I meet this pattern often enough that I call it "the first-slide trap", because the number that goes from two hours to two minutes is real. It is measurable, it is provable, and it makes everyone in the room nod. The problem is that it measures the wrong unit. The unit the customer feels is the time from submitting their case to receiving an answer, not the time somebody spends typing a draft — and in most workflows the customer's waiting time does not sit inside the steps where people are working. It sits in the gaps between the steps.

The upper band of the figure is the old workflow: intake → draft → wait → review → wait → approve. The two highlighted boxes are both "wait" boxes, because that is where the time disappears. The numbers under the band read 18 minutes of hands-on work in total against 46 hours of waiting.

The lower band is the redesigned route: digital intake → bounded AI assessment → normal-case action → exception desk → outcome evidence. The three lines under that band summarise the intent of the whole design — routine cases flow continuously, people own ambiguity and consequence, and every outcome feeds back into learning.

The most important difference between the two bands is not the number of boxes. It is what each box is. In the upper band a box is a step of paperwork. In the lower band a box is a point where a decision is taken and evidence is produced. "Exception desk" is not a prettier name for review: it is a unit with declared competence, its own queue, the authority to refuse, and a route for handing off uncertainty — which is precisely the four conditions of section 1 turned into a real piece of organisational structure.

The playbook summarises this new route as six beats, which I use as a checklist every time I draw a workflow afresh — validate the input that comes in, create the prediction, apply the declared rules together with judgment, automate the ordinary cases, route exceptions to qualified people, and record the outcome back. If your new workflow is missing one of the beats, it is usually the last one — and that is the only beat that makes the system better on the next pass.

The last row of that table is the sentence the playbook states outright: performance and governance must be designed together.^[1] I agree with it without reservation, because redesigning a workflow without touching permissions and exception routes is simply raising the production rate of the good and the bad in the same proportion — and what comes back to you is a volume of rework that grows with the scale.

The name the playbook gives to all of this work is workflow redesign, and its central instrument is dividing the work into four lanes.

3. Four Coordinated Lanes — Who Holds What, and What They Must Never Hold

The central proposal of Chapter 4 is to stop asking "does a person do this task or does AI do it", because a binary question of that shape forces a wrong answer every time, and to place work instead into four coordinated lanes. The playbook names the four like this.^[1]

"Use four coordinated lanes. Purpose and judgment belongs to people who define outcomes, constraints, exceptions, and values that historical data cannot supply. Generation and analysis lets AI search, compare, draft, classify, or propose inside a declared boundary. Structural control uses deterministic software to enforce permissions, schemas, limits, source membership, transactions, and trace capture. Verification and recovery gives a qualified person or independent test the information and authority to reject, route uncertainty, approve consequential effects, and stop or reverse the workflow."

Before the detail, I want to be clear about where this frame comes from. The four lanes are the playbook's own synthesis. They do not come from the masterclass, they do not come from any international standard, and they do not come from ETDA or MIT. I cite them as a working scheme that holds up in practice, not as a requirement.

The value of the frame lies in the "must never hold" column more than in the "holds" column, because most of the failures I see do not come from one lane being unable to do its own job. They come from one lane being used in place of another — above all, from the fourth lane being put to work doing the third lane's job.

The third lane is the most neglected of the four, and it is the one that decides whether your workflow is a control or a ceremony. Human oversight should not be spent on work that software does more reliably, more cheaply, and without getting tired — checking whether the destination account number is on the approved list, whether the total exceeds the initiator's limit, whether the attachments are complete against the Schema. Every time you hand a person those three jobs, you are spending the scarcest resource in the organisation on the tasks with the highest human error rate, and you are eating the time they should be spending on the thing only they can do.

💡 My view: operating principle 3 of this chapter reads "Do not use attention to enforce what software can enforce. Permissions schemas and limits belong in hard controls."^[1] — I use that sentence as the first exam question for every workflow that comes to me for review. If the reviewer's checklist contains an item that could be written as an "if" condition, that item should never have been on the checklist. It belongs in code. And every item we move out of a person's field of vision and into code is time handed back to them for exceptions, which is the one job people do distinctly better than machines.

The other relationship to watch is the one between the second lane and the fourth. Proposals produced by AI carry one dangerous property: they always read smoothly, whether they are right or wrong. The consequence of a proposal therefore has to be assessed from what it is about to do, not from how credible its prose sounds — and the decision authority a system is granted has to track that level of consequence, not the model's confidence. That is the axis of the entire next post.

4. Allocating Work by Comparative Advantage and Consequence

A question follows immediately once the four lanes are laid out: in practice, which mode should any given task be placed in? The masterclass answers this in the 15:58–19:44 stretch, and the playbook paraphrases it in Appendix A §7.^[3] The answer begins by rejecting two positions at once — it does not start from a belief that "we should automate as much as possible", and it does not start from a belief that "every case must be approved by a person". It allocates work by comparative advantage and by consequence.

The phrase human–AI complementarity in the playbook means exactly this, and it is not a slogan about collaboration. It is an economic statement that each side should do what it is least disadvantaged at — and what settles how much error you can tolerate is the consequence when it is wrong, not the average capability of the tool.

The playbook adds one sentence after that table which I think the head of HR should read more urgently than the head of technology — people's roles shift from performing every step to reviewing, handling exceptions, framing judgment, communicating, and owning outcomes. Reskilling is therefore more than teaching people to write a Prompt: it is preparing them for a changed mix of work.^[1]

The evidence warns against a universal workflow

The playbook places three pieces of evidence here to stop readers picking a mode by feel. I bring them in for one paragraph only, and for one reason only: to say that you have to measure on your own real work, not to use them as targets. All three sets of numbers are the heart of the next post, not this one.

First: in one firm's deployment across 5,179 customer-support agents, issues resolved per hour rose 14% on average — 34% for novice and lower-skilled workers, and hardly at all for the most experienced. That is one firm, one workflow and one period, not a universal return, and the paper is still a working paper rather than a peer-reviewed article.^[4] Second: a preregistered experiment with 758 consultants found AI users completed 12.2% more tasks 25.1% faster inside the tested capability frontier — and were 19% less likely to be correct on one complex managerial task chosen to sit outside it. Simulated tasks, one population, one model generation, and that frontier moves.^[5] Third: a 2024 meta-analysis of 106 experiments (370 effect sizes) found human-plus-AI systems beat humans alone on average (g = 0.64) but lost to the better of human-alone or AI-alone (g = −0.23), with decision tasks the worst case. The literature ends in June 2023 and is highly heterogeneous, so the comparison has to be run on our own work.^[6]

Let me restate the condition on the third piece, because it is the one most often misused. Both values must always travel together. Carrying only −0.23 makes it read as though the research says people plus AI is worse than doing nothing, which it does not, and carrying only 0.64 makes it read as though the research certifies that pairing always improves things, which it also does not. The design conclusion the playbook draws from all three pieces is therefore a single sentence: measure all three arrangements — Human-only, AI-only, and the combination — on the same real task, and only then choose.^[1]

5. LannaBuild Engineering — Moving the Design Point Upstream

The playbook uses the case of LannaBuild Engineering (a fictional case from the playbook) as its example of moving the design point, and I like this case because it starts from a failure that matches section 1 almost exactly.

The old arrangement was to have AI write the entire bid, then hand a single senior engineer a long draft to read under deadline. This is the ceremonial checkpoint in its purest form. That engineer had full competence, but no time, no evidence of where each paragraph came from, and in practice no authority to refuse — because refusing meant missing the submission. So the organisation gained speed in drafting, and got invisible risk thrown in with it.

LannaBuild's redesign did not begin by making the review better. It began earlier than that — it began before the writing, which is what moving the design point upstream means. The table below transcribes the playbook's redesigned workflow and marks each step with a lane from section 3. The workflow itself is the playbook's; the lane and evidence columns are my own annotation.

The sixth row is the one I most want you to remember — "withhold, do not highlight". The difference looks tiny on paper, but it is the whole difference between the third lane and the fourth. Highlighting an unsupported statement and sending it to a person to judge adds load to the reviewer in proportion to what AI produces, which means the more you use it the worse it gets. Withholding means the statement is simply not in the draft at all. The reviewer's load therefore does not grow with volume, and the time they have is reserved for the genuine exceptions.

The result the playbook records is that AI reduced the document-assembly work while people moved toward strategy, exceptions, commitments, and learning.^[1] Let me repeat that this is a fictional case, and that the playbook gives it no figures at all — no bid value, no headcount, no win rate — and I am not going to invent any to fill the gap. What you can borrow is the sequence of the design, not a set of result numbers.

6. The Workflow Teardown and Rebuild Session — Seven Steps and the Four-Lane Marking Table

Before the session itself, I want to put this chapter's five operating principles up as the rules of the room, because the seven steps that follow become an empty form without them.^[1]

1. Decompose work at the decision level. Redesign tasks that create the outcome rather than automating a job title.
2. Place people where judgment changes the result. Reserve expertise for purpose, exceptions, commitments and ambiguity.
3. Do not use attention to enforce what software can enforce. Permissions, schemas and limits belong in hard controls. (the one I already quoted in section 3)
4. Design review as operating capacity. Specify competence, evidence, time, queue limit and authority to reject.
5. Preserve skill formation. Decide how novices build judgment if AI performs the routine cases that once trained them.

Principle 4 changes a team's thinking fastest. The words "operating capacity" mean that review has to be planned the way we plan a factory's production capacity — with a unit of measure, a ceiling, a queue, and a defined thing that happens when the queue overflows — rather than the way we plan "being careful". Principle 5 is the one whose bill arrives slowest and costs most, because the routine work AI takes over is the same work that used to train the next generation into judgment. Cut it away without building a new route and in five years there will be no qualified reviewers left at all.

The seven steps of Workflow teardown and rebuild

The Step column in the table below is the playbook's text. The Question, Output and Who columns are my own practical expansion, so that each step ends with something tangible and a named person against it, rather than ending in shared understanding.

The four-lane marking table

Steps 2 and 3 are the heart of what the playbook calls task decomposition, and steps 3 and 4 are where meeting rooms most often lose their way, because everyone agrees that Tasks should be assigned to lanes and then, when the assigning actually starts, it turns into an argument conducted on instinct. The table below is the form I use to force lane assignment to end in reasons and evidence. The example rows are taken from the LannaBuild case in section 5.

Who should be in the room while this table is filled in

This question matters more than it looks, because the table above can be filled in very fast by people who do not do the work — and the result will be beautiful and wrong. A 2023 MIT Sloan report drawing on more than 50 interviews puts worker voice in at problem definition and work-process design, not as "end-user" feedback after the design is set.^[7]

The reason the report gives lands exactly on the problem in step 2 — the people who do the work every day are the ones who know what sub-steps are really hidden inside each box on the map, and what an edge case actually looks like. That body of knowledge is practical knowledge that has never been written into any document. The report goes further and observes that having those people in the room is not enough, because tacit knowledge may never have been put into words before, so you need a way of asking that draws it out rather than merely an opportunity to speak.

The boundary has to be marked as well: this report studies the United States context in 2023. It does not discuss Thai organisations, it does not discuss Thai labour law, and it does not discuss Thai practice. The playbook itself notes that design guidance of this kind must be adapted to local representation structures, labour relations, culture, and law.^[1]

Step 7 is what gives the whole session its meaning. The output of a teardown is not a new policy, not an announcement, and not a prettier diagram. It is a bounded experiment with a named owner. The evidentiary reason is already in section 4 — when the meta-analysis says the effect of pairing people with AI is highly variable and heavily dependent on the type of task, the only thing that can tell you whether your design works is measuring it on your own work, not quoting an average from somebody else's. And measurement only happens when there is one person whose name is attached to the result.

7. Metrics That Matter, Failure Patterns, and the Thai Pathway

The playbook's metrics list for this chapter is longer than what I bring across, because it packs outcome metrics and economic metrics into a single paragraph. The table below is the subset that decides whether the design works, which is this post's question. I have added the Scorecard column from the series' six-column system, so you can see which view of the executive board each metric surfaces on.

There is one rule the playbook writes at the end of its metrics list that I treat as non-waivable — segment by experience level, because averages can hide both the beneficiary and the person losing practice.^[1] The reason is in the first piece of evidence in section 4, where the low-experience group and the highly expert group got completely different results from the same deployment.^[4] If you report a single average, you will see neither of the two things that matter most: who is really benefiting, and who is losing the chance to accumulate skill — which is precisely principle 5 of this chapter.

One further warning about using this table. Every number from the fictional case in section 5, and the 18 minutes and 46 hours in section 2, must never enter this table as a target or a baseline. Your baseline has to come from your own data, collected before the experiment begins.

Failure patterns

• Automating the old process unchanged, without touching its structure at all — you buy the speed of the wrong route instead of the speed of the right one
• Making one exhausted expert the universal reviewer — the scarcest person becomes the longest bottleneck, and review quality falls with every hour that passes in the day
• Rubber-stamp approval — the checkpoints are all still there, but nothing has been refused for months, which is a signal you can read out of the data without waiting for anyone to tell you
• Hiding uncertainty behind fluent prose — unsupported statements are written in the same register as supported ones, so the reviewer cannot separate them by reading alone
• Allowing tools before authorization — the system acts first and checks whether it was permitted afterwards, which turns a control into an after-the-fact report

The playbook names seven failure patterns. The two I did not break out as list items above are removing entry-level work without building a new learning path, which is the same matter as principle 5 in section 6, and measuring local speed while downstream rework rises, which is the same matter as the first-slide trap in section 2. I would suggest printing all seven and pinning them up in the workshop room, because every one of them reads as "our organisation would never do that" right up until you look at the real data.

8. The Road Ahead — From Arrows on a Whiteboard to Evidence

If you take three things from this post, I would like them to be these. One, the four-condition table in section 1, which separates a real checkpoint from a ceremonial one within ten minutes of a meeting. Two, the four-lane table in section 3, whose "must never hold" column is worth more than its first column. Three, the six-column marking table in section 6, which forces every lane assignment to end in a reason, a piece of evidence, and a destination for the handoff.

But I have to be straight about the limits of this post. What we have done throughout it is draw arrows on a whiteboard. We can now say who should do what, for what reason, and where it hands off — but we have not answered the two questions that follow immediately in a real meeting. The first is how much authority to act the system should be granted at each step, and what conditions would justify granting it more. The second is how we would know that the lane assignment we have just made is genuinely better than the old arrangement, rather than merely looking more systematic on a slide.

Both are questions of evidence, and the evidence that exists does not give everyone the same answer on every task. The three sets of numbers I passed over quickly in section 4 will be opened up in full, each with its own boundary, in the next post — alongside the playbook's four-step authority ladder, which ties a system's authority to the level of consequence instead of tying it to the model's confidence.