Fair Evaluation — แยกชุดวัดก่อนเทียบCompare Without Test-Set Leakage — AI-Core + Jev in Practice

Part 6 of ten: Separate validation, test and attack cases; prevent label leakage and report quality together with coverage.

Local review draft • Default code uses offline fixtures. Numbers are fictional scenarios, not measured Jev results. Faster performance and lower cost remain hypotheses.

Outcome and prerequisites

Start with the companion repository and its detailed README. The website ZIP remains an alternative offline snapshot; run the same Node commands from either project root. GitHub repository · README / Quickstart

Use the same Support Ticket Decision Lab throughout the series. Have Node 22.18+, a terminal and a writable directory. Run commands from the cloned ai-core-jev-tutorial root, or the extracted ai-core-jev-lab folder if using the website ZIP. Offline work needs no npm install. Allow roughly 20–40 minutes for experiments; the article reading-time label excludes hands-on work.

Download the complete code and tests (ZIP) · Project README

This chapter focuses on src/metrics.mjs. Fixture outputs test the integration mechanics only. The author did not install the SDK or use an API key while preparing the lab. The separately documented live path has not been tested against the API.

How this boundary works

A comparison needs equal inputs, equal success definitions and a documented handling of unresolved work. The harness runs baseline, Jev-only and hybrid over the same split and uses the same labels and policy definition. It does not compare a classifier on easy tickets with a generator on all tickets. Keep that equality when replacing the fixtures with recorded real-provider results.

Our forty synthetic tickets are partitioned into twelve validation cases, twenty-four test cases and four attack cases. These names teach the evaluation protocol; they do not create a statistically valid blind study. The canned predictions were authored alongside the cases. No reported fixture accuracy estimates Jev generalization, Thai-language reliability or performance on your customer distribution.

For a real evaluation, gather permitted, representative tickets, define an annotation policy, resolve disagreements and separate related conversations before splitting. Tune questions and thresholds only on validation data. Freeze the candidate, then evaluate the held-out set once. If you change the candidate after reading the test result, obtain another untouched test set rather than continuing to call the same cases held out.

Seven practical steps

Step 1 — Inspect the split fields

Inspect the split fields. Verify the 12/24/4 partition and note the language field. The small Thai subset demonstrates that Unicode text flows through the program; it is not large enough to support a claim about Thai model quality. Keep that claim out of the report.

Step 2 — Read validateTicket before the provider call

Read validateTicket before the provider call. It returns only id, text and language, leaving goldRoute and goldHuman behind. The pipeline test checks the actual object delivered to the provider. Data separation should be an executable check rather than a comment saying do not leak labels.

Step 3 — Run validation first and save its JSON

Run validation first and save its JSON. Read both coverage and selective accuracy for every mode. rawRouteAccuracy covers valid final route decisions, whereas selectiveAccuracy evaluates correct automatic tasks that also respect human requests. The two metrics answer different questions and must not share an ambiguous accuracy label.

Step 4 — Predeclare a quality gate for the real workload

Predeclare a quality gate for the real workload. Decide what selective accuracy, coverage and human-request miss rate are acceptable before choosing a threshold. This tutorial supplies no production values. The point is to reject an experiment that cannot meet the product requirement, not to optimize whatever scalar looks best.

Step 5 — Run the test split only after freezing the demonstration policy

Run the test split only after freezing the demonstration policy. Saving out/test.json gives you a reproducible fixture transcript, not a publishable model benchmark. Compare sample counts, malformed-response handling and unresolved-review counts before considering any fictional latency or price field.

Step 6 — Read the calibration bins with their sample counts

Read the calibration bins with their sample counts. ECE uses the selected-option probability, not the Choice confidence statistic. A small ECE on a tiny synthetic sample is not a deployment guarantee. Keep bin counts, model ID, question version and language slices next to the number.

Step 7 — Audit the decision to continue

Audit the decision to continue. If the hybrid fails equal-quality or coverage constraints, stop treating its lower simulated inference bill as a win. Write the failed criterion into the experiment record and plan a new candidate. Reporting a negative result is a useful engineering outcome.

Files and commands

Companion code on GitHub: ai-core-jev-tutorial · README. Use the companion repository as the default working copy. These file links pin the tested revision. src/metrics.mjs · src/report.mjs · src/cli.mjs

Compare this file with the source in the ZIP before editing. The listing is the implementation used by the lab, not pseudocode. Model output is input to validation and routing code; it grants no permission to bypass approval.

src/metrics.mjs

export function percentile(values,p) {if(!values.length)return null;const s=[...values].sort((a,b)=>a-b);return s[Math.max(0,Math.ceil(p*s.length)-1)];}
export function money(attempt,prices) {
  if(!attempt.usage||!attempt.costComplete)return null;const p=attempt.provider==='jev'?'jev':'baseline';
  return (attempt.usage.input*prices[p+'InputPerMillionUSD']+attempt.usage.output*prices[p+'OutputPerMillionUSD'])/1e6;
}
export function summarize(rows,prices) {
  const n=rows.length,auto=rows.filter(r=>r.result.status==='proposed');
  const correct=r=>r.result.decision?.route===r.ticket.goldRoute,success=r=>correct(r)&&!r.ticket.goldHuman;
  const successful=auto.filter(success).length,valid=rows.filter(r=>r.result.decision),human=rows.filter(r=>r.ticket.goldHuman);
  const costs=rows.flatMap(r=>r.result.attempts).map(a=>money(a,prices));
  const reviews=n-auto.length,wrongAuto=auto.length-successful;
  const knownInferenceUSD=costs.reduce((s,c)=>s+(c??0),0);
  const modeledOverheadUSD=n*prices.infrastructurePerTaskUSD+reviews*prices.assumedReviewSeconds/3600*prices.reviewHourlyUSD+wrongAuto*prices.reworkPerWrongAutoTaskUSD;
  const complete=costs.every(c=>c!==null);
  const latencies=rows.map(r=>{const xs=r.result.attempts.map(a=>a.scenarioLatencyMs);return xs.every(Number.isFinite)?xs.reduce((a,b)=>a+b,0):null;});
  const calibration=rows.filter(r=>r.result.decision?.probabilities),bins=[];
  for(let k=0;k<5;k++) {
    const members=calibration.filter(r=>Math.min(4,Math.floor(r.result.decision.probabilities[r.result.decision.route]*5))===k);
    if(members.length)bins.push({n:members.length,meanTopProbability:members.reduce((s,r)=>s+r.result.decision.probabilities[r.result.decision.route],0)/members.length,accuracy:members.filter(correct).length/members.length});
  }
  const ece=calibration.length?bins.reduce((s,b)=>s+b.n/calibration.length*Math.abs(b.meanTopProbability-b.accuracy),0):null;
  const ratio=(a,b)=>b?a/b:null;
  return {evidence:'OFFLINE FIXTURE SIMULATION — NOT MEASURED JEV PERFORMANCE OR SAVINGS',n,
    rawRouteAccuracy:ratio(valid.filter(correct).length,valid.length),coverage:ratio(auto.length,n),selectiveAccuracy:ratio(successful,auto.length),abstentionRate:ratio(reviews,n),
    humanRequestMissRate:ratio(human.filter(r=>r.result.status==='proposed').length,human.length),fallbackRate:ratio(rows.filter(r=>r.result.attempts.some(a=>a.provider==='baseline')&&r.result.attempts.some(a=>a.provider==='jev')).length,n),
    successfulAutomatedTasks:successful,unresolvedReviewTasks:reviews,
    latency:{label:'fictional per-attempt scenario milliseconds, summed for sequential fallback',p50:percentile(latencies.filter(Number.isFinite),0.5),p95:percentile(latencies.filter(Number.isFinite),0.95)},
    calibration:{metric:'5-bin ECE of selected-option probability; not Choice confidence',n:calibration.length,ece,bins,note:'Synthetic cases only; no production calibration claim.'},
    cost:{currency:'USD',knownInferenceUSD,modeledOverheadUSD,allAttemptsKnown:complete,modeledTotalUSD:complete?knownInferenceUSD+modeledOverheadUSD:null,
      costPerSuccessfulTaskUSD:complete&&successful?(knownInferenceUSD+modeledOverheadUSD)/successful:null,
      denominator:'correct automatic routing tasks that did not ignore a human request; reviews unresolved',note:'All usage/latencies/review/rework inputs are scenario fixtures. Unknown usage or hidden retries makes total unknown.'}};
}

Run these offline commands without an SDK installation or API key:

node --test --test-reporter=spec test/pipeline.test.mjs
node src/cli.mjs evaluate --split validation --save
node src/cli.mjs evaluate --split test --save

Verify the expected behavior

Each mode must evaluate n=24 for the test split. Review tasks remain unresolved, and calibration fields may be null when a provider supplies no probabilities. If a method has a smaller denominator, explain the refusal or failure path rather than silently dropping the cases.

Read the named tests and refusal paths, not just the exit code or final label. Unknown values must remain unknown, and every report must distinguish simulation inputs from real measurements.

Saved reports retain a run ID, timestamp, release/model/SDK/question/dataset identifiers and SHA-256 hashes of the question, policy, price, dataset, fixtures and implementation. Each mode includes every ticket, gold labels attached after inference, decisions, attempts, refusal reasons and action proposals, plus separate Thai and English summaries. Inspect cases and languageSlices in out/test.json; never interpret the tiny slices as deployment-quality estimates.

Audit record implementation: src/report.mjs · src/cli.mjs

Troubleshooting

If a new question was selected by test accuracy, the test set has become development data. If near-duplicate conversations cross partitions, split by conversation or customer group before measuring again. If reviewers disagree about a label, record the ambiguity rather than declaring whichever label benefits the hybrid to be correct.

When previewing with Python http.server, use the .html continuation links below when needed. Extensionless series chips rely on GitHub Pages resolution; that documented preview limitation is separate from the Node example.

Change one thing, then test

Write an annotation note for a ticket that mixes a refund request with a broken account page. Decide how the primary queue is labeled and whether the customer explicitly requested a human. Have another reviewer challenge the rule before collecting real model outputs. Record ambiguity instead of silently fixing labels to match predictions.

Make changes in a disposable copy, retain the reference files for comparison, and restore them before the next chapter. If you later use a real provider, create a new experiment record rather than overwriting the fixture transcript.

Next checkpoint

Before moving on, identify the contract file, the policy file, what was observed in a run, and what remains untested against a real model. Keeping those four things separate matters more than making the demo look effortless.

← Previous chapter · Next chapter →

References and evidence

Key takeaways