Abstain & Route — ไม่มั่นใจให้หยุดได้Abstain, Route, and Fall Back — AI-Core + Jev in Practice

Part 5 of ten: Implement abstention and fallback without treating concentration as correctness or ignoring a human request.

Local review draft • Default code uses offline fixtures. Numbers are fictional scenarios, not measured Jev results. Faster performance and lower cost remain hypotheses.

Outcome and prerequisites

Start with the companion repository and its detailed README. The website ZIP remains an alternative offline snapshot; run the same Node commands from either project root. GitHub repository · README / Quickstart

Use the same Support Ticket Decision Lab throughout the series. Have Node 22.18+, a terminal and a writable directory. Run commands from the cloned ai-core-jev-tutorial root, or the extracted ai-core-jev-lab folder if using the website ZIP. Offline work needs no npm install. Allow roughly 20–40 minutes for experiments; the article reading-time label excludes hands-on work.

Download the complete code and tests (ZIP) · Project README

This chapter focuses on src/pipeline.mjs. Fixture outputs test the integration mechanics only. The author did not install the SDK or use an API key while preparing the lab. The separately documented live path has not been tested against the API.

How this boundary works

An abstention policy is executable product behavior. In this lab, a low Choice confidence can trigger a generative-baseline fixture, while a possible human request goes directly to review. These are different branches with different consequences. A fallback is another fallible opinion and another billable operation in a real deployment, not an exemption from the effect guard.

Choice confidence summarizes the concentration of an option distribution; it is not a measured fraction of correct decisions on your tickets. A value of 0.9 is not a warranty of 90 percent deployment accuracy. We measure calibration using the selected option probability against labels, and still treat that result as a property of a dataset and model version, not a universal constant.

Our 0.7 route threshold and 0.3 human-no boundary are teaching parameters. They have not been validated for a business or language population. Do not deploy them because a tutorial uses them. Predeclare acceptable quality and coverage, sweep only validation data, and keep the held-out test untouched until the policy is frozen.

Seven practical steps

Step 1 — Read the policy file as a release artifact

Read the policy file as a release artifact. Record its version before changing a threshold. routeConfidence applies to the Choice statistic; humanNoMax applies to a Noul probability. Keeping the two field names distinct prevents a generic confidence helper from applying the wrong meaning to both.

Step 2 — Run T-002

Run T-002. Its route fixture is ambiguous enough to fall below the route threshold, so hybrid calls the baseline once. Inspect the two attempts and their ordering. Do not discard the first call when the second call supplies the final route.

Step 3 — Run T-004

Run T-004. An explicit request for a human causes review before a fallback can rewrite the decision. This is a product obligation, not merely a classifier preference. The program must retain this condition even when another provider offers a confident alternative label.

Step 4 — Run T-006

Run T-006. Its Noul probability is near the uncertain middle. We do not convert that into a firm no and auto-route. The safe path here is needs_review. Write down how much such abstention costs your support workflow instead of hiding it as a model error or counting it as a free success.

Step 5 — Inspect T-011 against its label

Inspect T-011 against its label. It deliberately contains a high-confidence wrong Jev-shaped prediction. The fixture teaches a limitation, not a measured Jev failure rate. Keep such counterexamples in your real evaluation set so a high score cannot become a substitute for correctness.

Step 6 — Run the validation-only sweep

Run the validation-only sweep. For each threshold read selective accuracy, coverage and fallback rate together. If every candidate fails your predeclared constraints, the answer is not to choose the cheapest loser; revise the questions, dataset or architecture and start a new validation cycle.

Step 7 — Freeze one policy before the next chapter

Freeze one policy before the next chapter. Keep a copy with the report and the question version. The test split is for evaluating that frozen candidate once, not for selecting the threshold after seeing which number gives the nicest table.

Files and commands

Companion code on GitHub: ai-core-jev-tutorial · README. Use the companion repository as the default working copy. These file links pin the tested revision. src/pipeline.mjs · config/policy.json

Compare this file with the source in the ZIP before editing. The listing is the implementation used by the lab, not pseudocode. Model output is input to validation and routing code; it grants no permission to bypass approval.

src/pipeline.mjs

import {validateTicket,parseJev,parseBaseline,probability} from './contracts.mjs';
export function validatePolicy(policy) {
  probability(policy.routeConfidence,'policy routeConfidence');probability(policy.humanNoMax,'policy humanNoMax');
  if(policy.humanNoMax>=0.5||typeof policy.fallbackOnLowConfidence!=='boolean')throw new Error('invalid abstention policy');return policy;
}
export async function decide(input,{mode='hybrid',providers,policy}) {
  const ticket=validateTicket(input);validatePolicy(policy);
  if(!['baseline','jev','hybrid'].includes(mode))throw new Error('invalid mode');
  const attempts=[];let decision=null,source=null,reason='not_evaluated';
  async function obtain(kind,parse) {
    try {const result=await providers[kind](ticket);attempts.push(result.attempt);source=kind;return parse(result.value);}
    catch(error) {
      if(attempts.at(-1)?.provider!==kind)attempts.push({provider:kind,mode:'error',usage:null,wallMs:null,scenarioLatencyMs:null,httpAttempts:null,costComplete:false});
      reason=`${kind}_failed_validation_or_call`;return null;
    }
  }
  if(mode==='baseline')decision=await obtain('baseline',parseBaseline);
  else {
    decision=await obtain('jev',parseJev);
    if(decision&&decision.wantsHuman>policy.humanNoMax)reason='human_request_or_uncertainty';
    else if(!decision||decision.confidence<policy.routeConfidence) {
      const humanProbability=decision?.wantsHuman;
      if(mode==='hybrid'&&policy.fallbackOnLowConfidence) {
        const fallback=await obtain('baseline',parseBaseline);
        decision=fallback?{...fallback,wantsHuman:Math.max(fallback.wantsHuman,humanProbability??0)}:null;
      } else reason='low_route_confidence';
    }
  }
  const auto=!!decision&&decision.wantsHuman<=policy.humanNoMax&&(source==='baseline'||decision.confidence>=policy.routeConfidence);
  if(auto)reason=source==='baseline'?'validated_baseline_proposal':'validated_jev_proposal';
  else if(decision?.wantsHuman>policy.humanNoMax)reason='human_request_or_uncertainty';
  return {ticketId:ticket.id,status:auto?'proposed':'needs_review',reason,source,decision,attempts,action:auto?{kind:'route_ticket',ticketId:ticket.id,queue:decision.route}:null};
}

Run these offline commands without an SDK installation or API key:

node src/cli.mjs demo --mode hybrid --ticket T-002
node src/cli.mjs demo --mode hybrid --ticket T-004
node src/cli.mjs demo --mode hybrid --ticket T-006
node src/cli.mjs demo --mode jev --ticket T-011
node src/cli.mjs sweep

Verify the expected behavior

T-002 uses two attempts, T-004 and T-006 need review, and T-011 can be confidently wrong. Those three observations are more instructive than a single average. They distinguish routing confidence, product obligations and semantic correctness.

Read the named tests and refusal paths, not just the exit code or final label. Unknown values must remain unknown, and every report must distinguish simulation inputs from real measurements.

Troubleshooting

If increasing the threshold seems to improve everything, inspect missing attempts and unresolved reviews. If a human-request case reaches a proposal after fallback, the fallback has erased a constraint. If test-set results guided the chosen threshold, treat that test set as validation and obtain a new held-out set.

When previewing with Python http.server, use the .html continuation links below when needed. Extensionless series chips rely on GitHub Pages resolution; that documented preview limitation is separate from the Node example.

Change one thing, then test

Choose a hypothetical coverage floor and a selective-accuracy floor before examining the sweep. Explain which candidates fail and why. If none qualify, report that outcome. This exercise practices a decision process; it does not turn the tiny fixture population into evidence for a production threshold.

Make changes in a disposable copy, retain the reference files for comparison, and restore them before the next chapter. If you later use a real provider, create a new experiment record rather than overwriting the fixture transcript.

Next checkpoint

Before moving on, identify the contract file, the policy file, what was observed in a run, and what remains untested against a real model. Keeping those four things separate matters more than making the demo look effortless.

← Previous chapter · Next chapter →

References and evidence

Key takeaways