Decision Portfolio — เลิกเขียน Use-Case ListDecision Portfolio — Stop Writing Use-Case Lists

🤔 Two applications look like the same chat box. Why are their obligations from different worlds?

The previous post, Five Maturity Levels, ended on a sentence left deliberately hanging: the "right" maturity level for a workflow does not follow from how ambitious the organisation is, but from how much authority that decision hands to AI, whether the work still runs without the model, and how wide and how irreversible the damage is when it goes wrong. This post is the instrument that answers all three questions at once — and it is the instrument I have used instead of a use-case list for years.

The short answer to the whole article is this: stop building use-case lists and build a decision portfolio instead — classify every workflow on three dimensions that must never be collapsed into one score, then take one of four decisions, Scale, Contain, Redesign or Retire, for the next 90 days, and write down in advance what evidence would change that decision.

1. The Same Screen, Obligations from Different Worlds

Chapter 3 of AI Transformation as an Organizational Core opens with a sentence I think ought to be pinned outside every digital steering committee room: "A use case list tells where technology appears. A decision portfolio shows where authority dependency value and exposure accumulate." A list of use cases tells you where the technology surfaces; a decision portfolio tells you where authority, dependency, value and exposure pile up.

The difference is not rhetorical. The book makes it visible within two lines: two systems with much the same chat interface can create radically different obligations. A summarising tool that an analyst reviews every time is not equivalent to an agent that can actually issue a customer refund, even when both call the same model, through the same API, and appear on the same slide under the heading "AI for customer service".

They are not equivalent because what each system releases is different. The summariser releases text for one person to read, and that person carries the responsibility for it. The refund agent releases a real financial effect into the outside world with nobody standing in the way. The same model does not equalise permissions, does not equalise how many people are affected, and does not equalise the cost of putting things right.

How a use-case list happens, and what it hides

The pattern I have watched repeat is this: the organisation forms an AI working group, then circulates a form to every department asking them to "propose a use case where your team would like to apply AI". Before long a stack of entries comes back, sorted by the savings each department estimated for itself, and the meeting picks the top few for pilots.

The process looks orderly, but it selects wrongly, and it does so systematically, for three reasons. First, it ranks by numbers nobody has yet proved — "projected" savings are a promise, not evidence. Second, it is an internal competition between departments, so the unit that writes the best proposal wins, not the unit that owns the most valuable decision. Third, it cannot see the small decisions that happen thousands of times a day, because nobody thinks of those as a "use case" at all — even though their cumulative volume dwarfs executive decisions many times over.

The most important observation in that table sits in the last row: a use-case list ends in a list, while a decision portfolio ends in a decision. A list never expires on its own, which is exactly why pilots in so many organisations survive for years with nobody willing to close them — nobody ever agreed at the outset what the closing condition would be.

Classifying by context is not this book's private opinion

Before we reach the instrument, I want to pin down that the principle "classify by context, not by what the technology looks like" already has international backing. The OECD's Framework for the Classification of AI Systems (February 2022) sorts a system by context across five dimensions — People & Planet, Economic Context, Data & Input, AI Model and Task & Output — for policy use, by policymakers and regulators, and it does not return one aggregate risk number.^[1] That report also says of itself that an "actionable AI system risk methodology" was the next phase of work, not something the 2022 framework delivers.

NIST travels the same road. AI RMF 1.0 (26 January 2023) has four core functions — Govern, Map, Measure and Manage — where the Map function "establishes the context to frame risks related to an AI system" and ends in a first go/no-go decision about whether to design, develop or deploy the system at all.^[2] Let me be explicit about one thing: NIST states that this framework is "intended for voluntary use" — it is voluntary, not law.^[3] Real deployment in a Thai context still has to pass its own legal and sector-specific review regardless.

Put another way, both frameworks start from "context" every time, whereas a use-case list starts from "technology" every time. That is the ordering mistake this article is trying to correct. How the book compresses those two frameworks down to three dimensions is the subject of 3. Three dimensions that must be measured separately.

2. The Decision Inventory — Start from Decisions, Not from Technology

The step before the portfolio is building a decision inventory, which the book's appendix defines as "a structured register of recurring decisions, owners, inputs, timing, consequences, current performance, and candidate human–AI allocation". Notice that this definition contains no mention of a model, no mention of a use case, and not a single mention of technology.

The difference between the two kinds of search is easy to state. A technology-first search asks "where can we insert a model?" A decision-first search asks "which recurring choices determine value, cost, risk, customer experience or mission performance?" The first question returns a list of tools; the second returns a list of things the organisation can actually control.

Four properties of a good candidate

A good candidate decision usually carries four properties at once — high frequency, high value, data richness and available feedback.^[4] All four have to arrive together; you cannot take three out of four, because each does a different job:

• High frequency — volume is what makes a small improvement worth having, and what lets evidence accumulate fast enough to learn from. A decision taken once a year cannot be learned from within the life of the project.
• High value — if getting it right or wrong changes nothing, it is not worth the governance cost that follows.
• Data richness — the inputs have to be genuinely recorded somewhere, not lodged in one person's head and never written down.
• Available feedback — you have to learn, within a tolerable time, how the decision turned out. This is the property skipped most often, and it is the one that defines the horizontal axis of the 2×2 in section 4.

Once you have the list, record the anatomy of each decision in five parts: the prediction required, the judgment applied to interpret it, the action actually taken, the outcome observed, and the ethical or fairness constraints that sit over all of it.^[4] Splitting it five ways pays off immediately, because it shows where AI can help — usually the prediction — and which parts remain human.

The seven-column register

The register the book prescribes has seven fixed columns. My advice is to open a single spreadsheet for the whole organisation rather than letting each department keep its own, because half the value of this register is discovering that the same decision recurs in three departments and nobody knew.

The Current performance column is, in my experience, the column that kills the most projects, because many organisations discover they do not know the baseline of work they do every single day. What follows is that once the system has been live for a while, nobody can prove whether AI made anything better. If that column is empty, treat the workflow as not yet ready for the portfolio — go and measure the baseline first.

The other thing to watch is the last column. It records a candidate allocation, not a conclusion. At register time we decide nothing; we only record which splits are possible. The decision comes only after the three-dimension classification is done.

A note on evidence: the decision-inventory material in this section comes from the book's appendix, which states of itself that it is an editorial paraphrase of a talk rather than a word-for-word transcript. I have therefore written it as a summary rather than as quotation, and it should not be read as a controlled study.

3. Three Dimensions That Must Be Measured Separately

With the register in hand, the next step is classification, and the heart of the whole chapter is here — the book splits classification into three dimensions and insists they must never be collapsed into a single score, because each explains a different kind of control need.

Dimension 1 — decision authority: a four-rung ladder

Decision authority asks how far the AI may go. The answer has four fixed rungs, and it should be written against every row of the register.

The common mistake is to read the product name instead of the permissions. A system called a "copilot" may sit on the fourth rung if it can call tools that write to back-office systems on its own. Conversely, a system with a fierce name like "autonomous agent" may sit on the first rung if everything it proposes must pass a human first. Classify by what the system can actually release, not by the name on the slide.

Dimension 2 — indispensability: test it by removing the model

This is the dimension I consider most valuable, and almost nobody measures it. The question is: if we replace the declared model with a non-model fallback — the old rules, a lookup table, or a manual process — does performance on a predeclared task set fall below the threshold? If it does, the model is genuinely load-bearing. If it does not, we are paying for the model to buy convenience rather than capability.

The point to guard hardest is that business criticality is not the same as model indispensability. An invoicing system is unquestionably critical to the business, but if the part that uses the model is line categorisation, and the old rules achieve much the same accuracy, the model is not the load-bearing part of that system. The two live on different axes and must be recorded separately in the register.

The test is only credible if the conditions are declared before it runs — both the task set that will be used to measure and the threshold that counts as failing. Otherwise the test becomes an exercise in finding reasons for a decision already taken. How to design the test is in section 6.

Dimension 3 — consequence: six attributes

The third dimension asks what the output or action the system releases could cause, looking at six attributes, which likewise have to be recorded separately, because two workflows with "the same damage" may need entirely different controls.

• Severity — on one wrong call, how badly is the affected person harmed
• Reversibility — can it be undone, at what cost, and within what time
• Detectability — how do we know it was wrong, and how late do we find out? The error nobody sees is the most expensive error there is
• Scale — one person, a hundred people, or every customer who opened the site that day
• Permissions — what can the system touch? Read only, or write? Can it move money?
• Recovery cost — once it has happened, how many people, how many hours, and how much reputation does it take to get back to normal

The attribute I weight most heavily is detectability, because it multiplies the other five. A system that fails and is noticed at once can be handled. A system that fails and is noticed three months later has already accumulated the damage before anyone starts to fix it.

Why three dimensions, not one score

The distinction between authority and indispensability comes from my own r8 paper, which treats consequence as a separate dimension of its own.^[6] That document is unpublished and has no public URL, so I cite it as author synthesis rather than as peer-reviewed work.

The external frameworks the book invokes reinforce the principle rather than supply a formula. The OECD classification framework insists that classification must follow context,^[1] and NIST AI RMF prioritises risk treatment "based on impact, likelihood, and available resources or methods".^[2] One point has to be stated precisely: risk tolerance the framework deliberately does not prescribe — the organisation has to define that itself.^[2] None of these three sources supplies a universal portfolio number, and the book says outright that avoiding false precision is deliberate.

💡 My view: the second operating principle of this chapter is written the most briefly and costs the most — "Keep authority indispensability and consequence separate — each explains a different control need." I have sat in a room where a committee argued all afternoon about what risk score a workflow deserved, when laying the three dimensions on the table would have produced the answer in minutes: authority sits at "act with approval", indispensability has never been tested, and consequence is reversible within a day — so the next thing to do is run the removal test, not argue about a number.

4. Value × Learnability

The three dimensions in the previous section tell you what kind of control a workflow needs, but not which workflow to spend effort on first. So the book overlays a second diagram — a 2×2 on a different pair of axes: value against learnability, meaning how readily you can learn from the outcome.

The vertical axis is value, which is straightforward. The horizontal axis is learnability, which needs a word of explanation because people usually read it as "technical difficulty", which it is not. Learnability means how fast and how cleanly the real outcome comes back to us — a workflow whose result arrives within minutes and is unambiguously right or wrong has high learnability, while one whose real outcome shows up months or years later, tangled with countless other factors, has low learnability even if the model itself is trivial to build. Note that this axis is exactly the fourth property of a good candidate from section 2.

The size of each bubble in the figure is annual consequence, not the size of the project or the size of the budget. That detail genuinely changes the conversation, because a small decision taken a thousand times a day becomes a large bubble, while an executive decision taken twice a year stays a small one, however much money rides on each instance.

The quadrant I most want you to be careful with is the bottom right. The word selectively in its name is doing work. Low-value, easy-to-learn tasks are the ones teams love, because they finish fast and produce a handsome slide — but if each one needs its own data pipeline, its own evaluation harness and its own maintenance process, the total cost quietly overtakes the value. The criterion is: do it only when it reuses capability that already exists.

My favourite way to use this figure in practice is to have the team place the bubbles without mentioning technology at all. No model names, no providers, no products, until every bubble is on the board — because the moment a product name enters the room, the conversation drifts back to a use-case list within five minutes.

5. The Four Portfolio Moves

Once classified, leaders have exactly four moves. There is no fifth move called "carry on and see how it goes", which is nonetheless what happens in most organisations.^[7]

The third column is the one people skip most. The fifth operating principle of this chapter says that funding scale must always come with an exit — every proposal needs a fallback, a rollback and retirement conditions. I use a simple rule on the boards I sit on: a funding request without a retirement-conditions page is not admitted to the agenda. Not because we expect it to fail, but because writing the retirement conditions forces the proposer to say what success is from day one.

HarborLight Retail — four systems, four different decisions

The book uses HarborLight Retail (a fictional case from the playbook) as its single worked example across the chapter.^[7] This company called four systems "customer AI" on one slide — a label that makes everything look alike, when the obligations of the four are from different worlds.

I have deliberately left the third row incomplete, because the book describes the rebalancing system as advisory, manager-approved and reversible, but does not say which decision it received. Filling that cell myself would be inventing data inside a worked example, which is the thing I try hardest to avoid across this series — an honest gap is worth more than a cell that is full but fabricated.

The most interesting row is the second. The public product assistant has the lowest authority of all four, because all it can do is answer with text; it has no tool that creates a real effect at all. Yet it is the one that is operationally indispensable, because its text reaches large numbers of customers every day. Switch it off today and the page is blank. If the organisation used one composite risk score, this system would score low and be filed as "safe" — when it is in fact the system with the largest affected population in the portfolio.

The lesson of the case is compressed into a single sentence: projected savings did not decide the portfolio — because the obligations differed. If HarborLight had ranked its four systems by projected savings, the order it got would not have matched the safe order in a single row.

The five operating principles

I have already taken the second into section 3. The remaining four are:

• Classify the released decision — a chat surface can hide autonomous effects, so look at what the system can actually release, not at the screen the user sees.
• Test dependency with removal — do not fund a model as load-bearing when the fallback is adequate. This is the principle that saves the most budget in the chapter.
• Allocate assurance by exposure and recoverability — volume and irreversibility create different obligations. Two workflows at "the same risk" may need review budgets that differ several times over.
• Fund scale and exit together — every investment needs fallback, rollback and retirement conditions.

6. The Decision Portfolio Triage Workshop

All of the above stays theory unless there is a forum where the decision actually gets made. The book supplies a six-step working session. I run it in about half a day per round, with a set of workflows already filtered out of the register, and with the owner of every one of them in the same room rather than filling in a form afterwards.

The book asks you to decide Scale, Contain, Redesign or Retire for the next 90 days, then name the evidence that would change that decision. Please read the number 90 for what it is — it is the cadence the book prescribes for this session, not a figure measured by research, not an industry standard, and not a finding. Organisations with shorter business cycles may want a shorter round; organisations whose work has long cycles may need to stretch it. What matters is not the number but the fact that the decision has an expiry date.

Step six is what separates this workshop from an ordinary meeting. Writing down in advance "what evidence would change our minds" does two things at once — it forces us to be honest about what today's decision rests on, and it turns a future change of mind from "admitting we were wrong" into "doing what we agreed". That is a bigger difference in internal politics than most people expect.

Designing a removal test you can trust

Step two of the workshop stalls immediately if there is no removal-test result yet, because nobody can say which workflows genuinely cannot run without the model. The table below is the five things that must be declared in full before the test begins. My advice is to write them into a document and have the workflow owner sign it off before anything is run at all.

The test has three possible outcomes and all three are useful. First, performance falls clearly below threshold — the model genuinely is load-bearing, so invest in continuity: a backup provider, regression testing. Second, performance is not meaningfully different — the model is not the load-bearing part, so review what you are currently paying, immediately. And third, the result is borderline — which usually means the chosen task set does not yet reflect real work, so go back and fix the task set before concluding anything.

7. The Metrics That Matter

A portfolio without metrics becomes opinion within one quarter. The book lists twelve. I have added a Scorecard column, as this series does throughout, so you can see which cell of the organisational scorecard each metric belongs to, and so the whole portfolio is not measured on a single dimension called money.

The most neglected metric on this list is provider concentration. Every team assesses the risk of its own system separately and concludes that each one is acceptable, but nobody asks what the portfolio-level risk is when every indispensable workflow in the organisation sits on the same provider. This is a risk visible only from the portfolio view — there is no way to see it from inside a single project — and it is one of the strongest reasons a decision portfolio is worth building at all.

Failure patterns

• Ranking only by projected savings — an unproven number becomes the sort order for the whole portfolio, and the workflows with the heaviest obligations get pushed to the top because their numbers are the biggest.
• Assuming the label copilot implies low authority — classifying by product name instead of by real permissions and by what the system can release.
• Confusing business criticality with model indispensability — the two live on different axes, and merging them means paying for models where the fallback already works well.
• Using one risk score for incomparable harms — collapsing severity, reversibility and affected population into a single number, then comparing things that cannot be compared.
• Ignoring shared-model concentration across the portfolio — assessing every system one by one, but never assembling the picture of who the whole portfolio hangs on.
• Allowing pilots to persist without a scale-or-stop decision — the most expensive pattern of all, because a pilot carries the full maintenance cost while returning experimental value indefinitely.

The first five are classification errors, fixable with the instruments in section 3. The last is a management error, and it is fixable by one thing only: every workflow having a decision, and that decision having an expiry date.

8. The Road Ahead

If the whole chapter had to reduce to one change of practice, I would choose this: change the question in the meeting room from "where else can we use AI?" to "which recurring decision deserves a new decision this quarter, and is that decision Scale, Contain, Redesign or Retire?" The first question produces a list; the second produces change.

Three things can be done next week. One, open a single spreadsheet for the whole organisation and complete the seven-column register for the first ten entries. Two, take one workflow that already uses a model and design its removal test across all five fields before running it. And three, check whether your organisation's most recent AI funding request has a retirement-conditions page — if it does not, that is the first thing to fix.

The book opens Part two, Redesign the Organisation, with a sentence I think is the best bridge from this chapter to the next: "The unit of scale is not another pilot. It is a reusable organizational capability connected to an accountable value stream."^[7] Put another way: a classified portfolio is not yet a transformation. It is only a list of workflows we now decide about better. The transformation starts when we begin rebuilding the workflow around the decision.