Full Task Cost — วัดทั้งเวลาและต้นทุนงานMeasure Latency and Full Task Cost — AI-Core + Jev in Practice

Part 9 of ten: Calculate p50/p95 and task cost with fallback, review and rework; distinguish fictional scenarios from measured results.

Local review draft • Default code uses offline fixtures. Numbers are fictional scenarios, not measured Jev results. Faster performance and lower cost remain hypotheses.

Outcome and prerequisites

Start with the companion repository and its detailed README. The website ZIP remains an alternative offline snapshot; run the same Node commands from either project root. GitHub repository · README / Quickstart

Use the same Support Ticket Decision Lab throughout the series. Have Node 22.18+, a terminal and a writable directory. Run commands from the cloned ai-core-jev-tutorial root, or the extracted ai-core-jev-lab folder if using the website ZIP. Offline work needs no npm install. Allow roughly 20–40 minutes for experiments; the article reading-time label excludes hands-on work.

Download the complete code and tests (ZIP) · Project README

This chapter focuses on src/metrics.mjs. Fixture outputs test the integration mechanics only. The author did not install the SDK or use an API key while preparing the lab. The separately documented live path has not been tested against the API.

How this boundary works

A lower token bill is not automatically a lower operating cost. Count every inference attempt, fallback, retry, infrastructure operation, human review and known failure or rework. Divide by a declared successful-task denominator. Our downloadable report uses correct automatic routing tasks; unresolved reviews are not counted as successful completions, so do not label that denominator end-to-end ticket resolution.

As checked on 23 October 2026, the official model page lists Jev input at USD 0.042 per million tokens and output tokens free. Recheck before spending. The baseline rates, infrastructure amount, reviewer wage, review duration and rework charge in prices.example.json are fictional assumptions. They are editable scenario inputs, not current market quotes or measurements.

Latency also needs a clock definition. The fixture report sums fictional per-attempt milliseconds, including sequential fallback, and computes nearest-rank p50 and p95. harnessWallMs measures the local evaluator itself and is not inference latency. For live measurements record end-to-end wall time, concurrency, warm-up, input sizes, cache policy, retry behavior and sample counts.

Seven practical steps

Step 1 — Open prices.example.json before calculating

Open prices.example.json before calculating. Read its status and note fields as part of the data contract. Check the per-million unit: multiplying a token count by a per-million price without dividing by one million introduces a million-fold error.

Step 2 — Run the metrics tests

Run the metrics tests. They verify nearest-rank percentiles, the million-token conversion, unknown usage and the zero-success denominator. A division-by-zero result should be null or explicitly undefined, never a zero-cost claim suggesting the system completed work for free.

Step 3 — Run validation to see a fully specified cost scenario

Run validation to see a fully specified cost scenario. Read knownInferenceUSD separately from modeledOverheadUSD. In this artificial workload, review assumptions can dominate token prices. That is a lesson about the accounting model, not evidence that this is what a real support organization pays.

Step 4 — Run the test split and inspect T-036

Run the test split and inspect T-036. Its Jev usage is deliberately null. The harness retains known inference components but marks the full modeled total and cost per successful task unknown. Replacing null with zero would make the hybrid look cheaper precisely when evidence is missing.

Step 5 — Compare p50 with p95 and fallback rate

Compare p50 with p95 and fallback rate. A fast common path can coexist with a slow tail when a second model is called. Hold concurrency and workload fixed for real measurements, and report the full distribution or enough quantiles to prevent a median-only success story.

Step 6 — Apply the equal-quality gate before ranking costs

Apply the equal-quality gate before ranking costs. The fixture hybrid intentionally does not dominate every metric, so it cannot justify a savings claim. For a real claim, require acceptable accuracy, coverage and human-request handling first, then compare complete cost at that operating point.

Step 7 — Write an honest result statement

Write an honest result statement. Say whether the run was simulated or live, which assumptions were used, which costs remain unknown and whether the quality gate passed. If you have not measured live Jev, the conclusion is that the harness is ready to test the hypothesis, not that Jev saved a stated percentage.

Files and commands

Companion code on GitHub: ai-core-jev-tutorial · README. Use the companion repository as the default working copy. These file links pin the tested revision. src/metrics.mjs · config/prices.example.json

Compare this file with the source in the ZIP before editing. The listing is the implementation used by the lab, not pseudocode. Model output is input to validation and routing code; it grants no permission to bypass approval.

src/metrics.mjs

export function percentile(values,p) {if(!values.length)return null;const s=[...values].sort((a,b)=>a-b);return s[Math.max(0,Math.ceil(p*s.length)-1)];}
export function money(attempt,prices) {
  if(!attempt.usage||!attempt.costComplete)return null;const p=attempt.provider==='jev'?'jev':'baseline';
  return (attempt.usage.input*prices[p+'InputPerMillionUSD']+attempt.usage.output*prices[p+'OutputPerMillionUSD'])/1e6;
}
export function summarize(rows,prices) {
  const n=rows.length,auto=rows.filter(r=>r.result.status==='proposed');
  const correct=r=>r.result.decision?.route===r.ticket.goldRoute,success=r=>correct(r)&&!r.ticket.goldHuman;
  const successful=auto.filter(success).length,valid=rows.filter(r=>r.result.decision),human=rows.filter(r=>r.ticket.goldHuman);
  const costs=rows.flatMap(r=>r.result.attempts).map(a=>money(a,prices));
  const reviews=n-auto.length,wrongAuto=auto.length-successful;
  const knownInferenceUSD=costs.reduce((s,c)=>s+(c??0),0);
  const modeledOverheadUSD=n*prices.infrastructurePerTaskUSD+reviews*prices.assumedReviewSeconds/3600*prices.reviewHourlyUSD+wrongAuto*prices.reworkPerWrongAutoTaskUSD;
  const complete=costs.every(c=>c!==null);
  const latencies=rows.map(r=>{const xs=r.result.attempts.map(a=>a.scenarioLatencyMs);return xs.every(Number.isFinite)?xs.reduce((a,b)=>a+b,0):null;});
  const calibration=rows.filter(r=>r.result.decision?.probabilities),bins=[];
  for(let k=0;k<5;k++) {
    const members=calibration.filter(r=>Math.min(4,Math.floor(r.result.decision.probabilities[r.result.decision.route]*5))===k);
    if(members.length)bins.push({n:members.length,meanTopProbability:members.reduce((s,r)=>s+r.result.decision.probabilities[r.result.decision.route],0)/members.length,accuracy:members.filter(correct).length/members.length});
  }
  const ece=calibration.length?bins.reduce((s,b)=>s+b.n/calibration.length*Math.abs(b.meanTopProbability-b.accuracy),0):null;
  const ratio=(a,b)=>b?a/b:null;
  return {evidence:'OFFLINE FIXTURE SIMULATION — NOT MEASURED JEV PERFORMANCE OR SAVINGS',n,
    rawRouteAccuracy:ratio(valid.filter(correct).length,valid.length),coverage:ratio(auto.length,n),selectiveAccuracy:ratio(successful,auto.length),abstentionRate:ratio(reviews,n),
    humanRequestMissRate:ratio(human.filter(r=>r.result.status==='proposed').length,human.length),fallbackRate:ratio(rows.filter(r=>r.result.attempts.some(a=>a.provider==='baseline')&&r.result.attempts.some(a=>a.provider==='jev')).length,n),
    successfulAutomatedTasks:successful,unresolvedReviewTasks:reviews,
    latency:{label:'fictional per-attempt scenario milliseconds, summed for sequential fallback',p50:percentile(latencies.filter(Number.isFinite),0.5),p95:percentile(latencies.filter(Number.isFinite),0.95)},
    calibration:{metric:'5-bin ECE of selected-option probability; not Choice confidence',n:calibration.length,ece,bins,note:'Synthetic cases only; no production calibration claim.'},
    cost:{currency:'USD',knownInferenceUSD,modeledOverheadUSD,allAttemptsKnown:complete,modeledTotalUSD:complete?knownInferenceUSD+modeledOverheadUSD:null,
      costPerSuccessfulTaskUSD:complete&&successful?(knownInferenceUSD+modeledOverheadUSD)/successful:null,
      denominator:'correct automatic routing tasks that did not ignore a human request; reviews unresolved',note:'All usage/latencies/review/rework inputs are scenario fixtures. Unknown usage or hidden retries makes total unknown.'}};
}

Run these offline commands without an SDK installation or API key:

node --test --test-reporter=spec test/metrics.test.mjs
node src/cli.mjs evaluate --split validation --save
node src/cli.mjs evaluate --split test --save
node src/cli.mjs demo --mode jev --ticket T-036

Verify the expected behavior

Validation cost fields should be finite under the complete fictional fixture inputs. On test, Jev and hybrid include an unknown-usage attempt, so full cost fields are null. A smaller simulated p50 is not a live speed measurement, and no savings percentage should be reported from these outputs.

Read the named tests and refusal paths, not just the exit code or final label. Unknown values must remain unknown, and every report must distinguish simulation inputs from real measurements.

Troubleshooting

If a report says zero dollars for missing usage, inspect null handling. If p95 improves after dropping errors, restore the failed requests to the population. If review time is not measured, keep it labeled as an assumption and show sensitivity rather than presenting one invented wage/time pair as operating truth.

When previewing with Python http.server, use the .html continuation links below when needed. Extensionless series chips rely on GitHub Pages resolution; that documented preview limitation is separate from the Node example.

Change one thing, then test

Change assumedReviewSeconds only in a disposable prices file and rerun the validation scenario. Explain why a small inference-price change can be outweighed by review effort. Do not fill T-036 usage with a guessed number to manufacture a complete total; unknown evidence remains unknown until it is collected.

Make changes in a disposable copy, retain the reference files for comparison, and restore them before the next chapter. If you later use a real provider, create a new experiment record rather than overwriting the fixture transcript.

Next checkpoint

Before moving on, identify the contract file, the policy file, what was observed in a run, and what remains untested against a real model. Keeping those four things separate matters more than making the demo look effortless.

← Previous chapter · Next chapter →

References and evidence

Key takeaways