What the Evidence Says — บันไดอำนาจ และวิธีวัดคน+AI ให้ตรงงานจริงWhat the Evidence Actually Says — The Authority Ladder and Measuring Human + AI on the Real Task

🤔 If AI lifts novices by +34% while the most experienced barely change, should an organisation design one workflow for everybody?^[1]

The previous post, Human in the Loop Is Not a Design, closed on a structural proposal: a well-designed workflow separates the work into four coordinated lanes — purpose and judgment · generation and analysis · structural control · verification and recovery. What it left standing was the harder question — what does the evidence say about where the person belongs, and how would we know we had put them in the right place?

The short answer is that the evidence we have does not say AI "helps" or "does not help". It says the effect varies so widely that one workflow cannot be designed for everyone. So do not borrow another company's average as your decision. Measure three arms on your own real task — human-only, AI-only, and human plus AI — with the results reported separately by experience level. Then use that evidence to place the system's authority on a four-rung ladder — suggest, draft, act with approval, bounded autonomy — with one default red line that is never crossed.

1. The Evidence Warns Against a Universal Workflow

Chapter 4 of the playbook opens with a sentence that works as a test for any AI project: "Human in the loop is not a design. It becomes a control only when the person has evidence time competence and authority to disagree."^[2] — having a person in the loop is not yet a design; it becomes a control only when that person holds evidence, time, competence and the authority to disagree. Take away any one of the four and the reviewer is a ceremonial gate. The paragraph that follows is shorter and blunter still: "Evidence warns against a universal workflow."^[2] It sounds modest, but it is a very hard design constraint, because it means the workflow template a consultancy sells you, or the one head office sends down for the whole group, cannot be automatically right for your work.

One average cannot design work for two groups

A field study following 5,179 customer-support agents at a single firm used the staggered rollout of a conversational assistant to compare agents who already had the tool against those who did not yet have it, and reported issues resolved per hour rising by 14 percent on average — but the next line says the low-experience, lower-skilled group rose by 34 percent while the most experienced and highest-skilled group barely changed.^[1]

The 14 percent average is therefore not the result for "one worker". It is the sum of a population responding to the same tool in different ways, and it is the number of one company, one workflow, one period — not a universal productivity estimate. The design consequence is direct: design for the novice and you add steps the expert does not need until the work slows down; design for the expert and you leave the novice working without the scaffolding they need. And in both cases the pooled average will look good enough that nobody notices.

The first four operating principles of this chapter

• Decompose work at the decision level — redesign the Tasks that create the outcome, do not automate a whole job title. This is task decomposition
• Place people where judgment changes the result — reserve expertise for purpose, exceptions, commitments and ambiguity
• Do not use human attention to enforce what software can enforce — permissions, Schemas and limits belong in Hard Controls
• Design review as operating capacity — specify reviewer competence, evidence, time, queue limit and authority to reject, all five of them

The fifth is preserve skill formation, which I have lifted out into the whole of section 6, because it is the one organisations overlook most often.

2. Four Studies, Read Across Every Column

When someone brings research into a meeting, what goes missing almost every time are the columns that say "who was measured", "compared against what" and "when does this result stop applying". The table below holds the four studies the playbook cites in this chapter, opened out across all six columns — and the last column is the one I would like you to read before the results column.

What this table forbids

No arithmetic across studies. The four have different populations, tasks, model generations and outcome measures. Averaging them together, or saying "the research shows AI improves productivity by 14 to 40 percent", is statistically meaningless, because 14 percent is issues resolved per hour in a support centre while 40 percent is time per short writing task. And be especially careful with Dell'Acqua's quality figure — the published abstract says quality was "significantly improved" and gives no percentage.^[4] The "more than 40 percent higher quality" number circulating in several summary articles comes from the 2023 working-paper version; I cannot confirm it from the published text, so I do not use it.

The positive and negative halves of the meta-analysis are true at the same time, and they sit in different parts of the document. The value g = −0.23 (human plus AI worse than the better of the two alone) is in the abstract; g = 0.64 (human plus AI better than humans alone) is in the Results section.^[6] Citing only one half is how you make this paper say something it does not say. The negative half warns you not to assume that bolting a human to an AI is always best; the positive half warns you not to abandon the collaboration just because you saw a minus sign. And tasks that decide among a finite set of options are the hardest group of all, negative at g = −0.27, while content-creation tasks gain significantly more.

3. The Jagged Technology Frontier — Ragged, and Moving

Dell'Acqua and colleagues gave workflow designers their single most useful piece of vocabulary: the jagged technology frontier, a ragged boundary of capability — the term belongs to that paper, not to the playbook. What it means is that a model's capability does not line up along the axis of "difficulty" that humans feel. Two tasks a person would judge equally hard can sit on opposite sides of the boundary: on one the model does better than expected, on the other it fails confidently — and there is no way to guess from the outside which is which. Only testing tells you. The experiment was built to show exactly this. The 18 tasks were chosen to sit inside the frontier and the results came out well on all three measures, while one complex managerial task was chosen to sit outside it, and the group using AI was 19 percent less likely to reach a correct answer than the group without it.^[4] That is one task, not a set of them — but one task is enough, because the point is not "how much worse" but "same tool, same population, and the sign can flip".

💡 What makes the −19% figure more frightening than all the positive numbers put together is not its size. It is that the participants did not know they were working on a task outside the frontier. The task read like an ordinary task, the model's answer read like an ordinary answer, and the confidence of the language did not fall along with the accuracy. This is why "have a human check it" is not enough on its own — the person checking needs evidence about which side of the boundary this task sits on, not merely time to read.

All three experiments used a single model generation — GPT-4 for the consulting work, the ChatGPT of early 2023 for the writing tasks, and an assistant built on 2021-vintage data for the support centre. Every result is therefore a snapshot of that moment, not a description of what models can do today. The practical conclusion is that frontier-test results need an expiry date. The teams I work with write it down every time: which model generation was used, which test set, and when the test will be repeated — because changing model generation is changing the map, not updating a version number.

Four signals I use to check whether a task is approaching the ragged edge — these are my own qualitative criteria, not findings of the research — are: it depends on information that exists only inside someone's head · the right answer depends on organisational context that appears in no public document · it requires weighing conflicting goals with no declared weights · the error becomes detectable only more than one working cycle later. Two or more of those, and I will not let the task rise above the "draft" rung until there is evidence specific to that task itself.

4. The Authority Ladder — Authority Must Follow Consequence

Once you accept that AI's effect differs by person, by task and over time, the next question is how much authority to give the system. The playbook answers with a four-rung ladder that pairs decision authority with the control that has to sit beside it.

Read the figure left to right, one row at a time. The left side is the rung of authority we grant the system; the right side is the control that has to exist in reality before that rung is granted. The red wedge says that as you climb rung by rung, evidence and control have to rise with you rather than stay flat.

The first three columns are the book's, word for word. The last column is my own criterion — what the playbook guarantees is the pairing of rung with control; the question of how much evidence is enough belongs to each organisation.

The mistake I meet most often is people mixing this ladder with a different one. These four rungs are about authority against the consequence of one task — not an organisation's growth path, and not the five maturity levels — Explore, Assist, Manage, Integrate, Operate as AI-core — that we walked through in Five Maturity Levels. An organisation at a high maturity level will still have a great many tasks parked at "suggest" forever, because the consequence of those tasks does not permit anything higher. Climbing the whole portfolio at once because "we have grown up now" is precisely the mistake this ladder was drawn to prevent.

5. The Working Session — Measure Three Arms on the Real Task

The whole article converges here. The sentence the playbook uses to close its evidence paragraph is "The design lesson is to measure human-only, AI-only, and combined configurations for the actual task."^[2] — measure all three configurations on the real work. The load-bearing words are the actual task: not a benchmark, not a demo, and not the work of the company in the case study. Six metrics have to be collected for every arm, and all three arms must use identical definitions or the comparison means nothing: Cycle time from arrival to genuine completion · first-pass yield · severe error rate, counting only what reaches the recipient · unsupported-claim rate · correction minutes per item, broken out by Task · the lost-practice signal, meaning cases that used to train novices and no longer pass through their hands.

The three arms answer three different questions. The human-only arm answers "how good is the existing way, really", which most organisations have never measured. The AI-only arm answers "where is the model's ceiling on this task", which tells you whether you are inside or outside the boundary of capability. And the human-plus-AI arm answers "is the configuration we intend to run actually better than the other two" — a question the meta-analysis has already warned may come back as "no".^[6]

From evidence to a rung on the ladder

Once the three-arm results exist, placing the authority rung stops being an argument and becomes an exercise in reading a table. This table is mine, not the book's, and it deliberately contains no threshold numbers at all, because the thresholds have to come out of each organisation's own work.

The seven-step Workflow teardown and rebuild session

1. Define the outcome, the customer, the consequence and the baseline
2. Break the Workflow into decisions and evidence-producing tasks
3. Mark each Task as human, AI, deterministic system or shared, and give a reason
4. Design the handoffs, the context, the permissions, the tests and the uncertainty routes
5. Stress-test one normal case, one edge case, and one failure or attack
6. Estimate Cycle Time, reviewer capacity, cost and recovery effort
7. Select a bounded experiment and name the Workflow owner

The example the playbook uses is LannaBuild Engineering (a fictional case from the playbook), which used to instruct the AI to write the whole bid proposal and then hand a senior engineer a long draft to read under a deadline — a configuration in which the reviewer has neither time nor evidence, and is therefore a purely ceremonial gate. The redesign moved the point where AI works upstream: AI extracts the requirements and builds a Compliance Matrix whose sources can be traced back, deterministic systems check dates, totals, units and mandatory attachments, specialists write the technical exceptions themselves, AI assembles only the modules already approved, the commercial and legal owners approve price and commitments, text that makes a claim without support is withheld rather than highlighted for someone else to chase, and clarification requests and lost-bid reasons are gathered into the evaluation set. The case carries no numbers in the book and I will not supply any — the value lies in the shape of the division of labour, not in an invented result.

6. Preserve Skill Formation

💡 My view: the fifth operating principle of this chapter is the quietest and the most expensive one to forget — Preserve skill formation: "Decide how novices build judgment if AI performs the routine cases that once trained them."^[2] Preserve the path that builds skill; if AI takes the routine work away, you must be able to say where new people will accumulate judgment instead. I put the weight on the word decide in that sentence, because the playbook does not forbid AI from doing the routine cases. It says that if you are going to let it, someone has to decide the replacement learning path in the same document — not next year.

The reason this binds back to section 1 is the mechanism behind the 34 percent. The paper's authors offer the account that the tool spreads the practices of the best agents out to everyone else, but they label it themselves as suggestive evidence — indicative, not an established conclusion.^[1] So I do not say that AI "transfers expert knowledge" as though it were a confirmed fact. What can be said with confidence is that the novice group improved the most; why remains an open question.

But even with the mechanism unsettled, the organisational side effect is clear enough to design against. The work that used to be the training ground for new people — the easy cases, repeated until you begin to see what is out of place — is the same class of work AI does best and is always handed first. The result is that novices jump straight from no experience to reviewing the hardest work, skipping the stretch that used to build their instincts, and organisations usually do not notice for several years, until they find that nobody has grown into being a good reviewer. I always sum it up for teams in one sentence: routine work is not only a cost, it has a joint product, and that joint product is the next generation of people who know how to review.

Three things can be done immediately. First, put the lost-practice signal into the workflow's metrics from day one. Second, reserve a share of ordinary cases for new people to do themselves even though the system could do them, treating that as a declared cost of training rather than an inefficiency to be eliminated. Third, pair new people with reviewers on the cases where the system's answer is unclear, because the point at which the system is uncertain is the point where teaching works best.

7. Metrics That Matter

The playbook gives this chapter's metrics as a single paragraph. I have opened it out into a table and added the Scorecard column this series uses, so you can see which square of the organisational scorecard each metric answers to — and so the whole workflow does not end up measured on one dimension only, speed.

Failure patterns

The first seven are the playbook's list. The last three I have added from this article's own argument.

• Automating the old process unchanged — speeding up a route that was already the wrong one
• Making one exhausted expert the universal reviewer — a bottleneck that looks like a control
• Rubber-stamp approval — every signature present, no reading done
• Hiding uncertainty behind fluent prose — the tidier the text, the harder it is to check
• Allowing Tool calls before authorisation — the system does what nobody has permitted yet
• Removing entry-level work without building a new learning path — the whole of section 6
• Measuring local speed while downstream Rework rises — the speed is ours, the rework is somebody else's
• Designing one workflow for everyone (my own observation) — a single average makes you believe in a single kind of user
• Using benchmark scores in place of evidence from real work (my own observation) — especially an English benchmark against Thai-language work
• Reporting pooled results without segmenting by skill level (my own observation) — the surest way to see neither the benefit nor the damage

8. The Road Ahead

If the whole article had to reduce to one change of habit, I would pick this one: stop asking "can AI help with this work?" and start asking "on this work, what does human-only score, what does AI-only score, what does human plus AI score, and how do those three differ between novices and experienced staff?" The first question returns an opinion. The second returns evidence — and evidence is the only thing that can place authority on the ladder for a defensible reason.

Three things are doable next week. First, take one workflow that already uses a model, and write down on paper which of the four rungs it sits on today and what evidence supports that — the most common answer is "rung three with rung-one evidence". Second, design that task's three-arm measurement so that it fits on a single page, with a Thai-language arm in it. Third, check whether the default red line is enforced in software, or only by the team's shared understanding. The question this article leaves open is larger than any workflow: if AI takes over the work that used to train novices, an organisation has to decide where people will learn instead, and how the additional output is to be shared out between customer value, quality, learning, reduced workload, growth and financial return.