Baseline — วาง baseline ที่ตรวจได้Establish a Measurable Baseline — AI-Core + Jev in Practice

Part 2 of ten: Freeze the success definition and JSON baseline; retain attempts and unresolved work before comparing the Jev path.

Local review draft • Default code uses offline fixtures. Numbers are fictional scenarios, not measured Jev results. Faster performance and lower cost remain hypotheses.

Outcome and prerequisites

Start with the companion repository and its detailed README. The website ZIP remains an alternative offline snapshot; run the same Node commands from either project root. GitHub repository · README / Quickstart

Use the same Support Ticket Decision Lab throughout the series. Have Node 22.18+, a terminal and a writable directory. Run commands from the cloned ai-core-jev-tutorial root, or the extracted ai-core-jev-lab folder if using the website ZIP. Offline work needs no npm install. Allow roughly 20–40 minutes for experiments; the article reading-time label excludes hands-on work.

Download the complete code and tests (ZIP) · Project README

This chapter focuses on src/contracts.mjs. Fixture outputs test the integration mechanics only. The author did not install the SDK or use an API key while preparing the lab. The separately documented live path has not been tested against the API.

How this boundary works

A baseline is a frozen procedure plus evidence, not a weaker model selected to make the new system look good. Here it is an offline stand-in for a generative classifier that returns a JSON string. In a real experiment, use your current production-quality generative model, prompt, repair rules and complete billing trace. We do not connect a live generative provider in this download.

Define success before reading comparative numbers. A successful automatic task routes to the correct queue and does not ignore a request for a human. A task sent for review is unresolved in this harness, not a magically correct answer. We report coverage and selective accuracy together because refusing more cases can make the remaining cases look better.

The baseline parser protects structure only. A valid billing label can still be semantically wrong. Separate invalid JSON, invalid label and valid-but-wrong classification in your diagnosis. Otherwise a parser improvement can be mistaken for a reasoning improvement, and a retry that fixes formatting can disappear from the cost calculation.

Seven practical steps

Step 1 — Write the acceptance rule

Write the acceptance rule. Use the three queue labels and the explicit-human-request condition in the lab. Keep the same rule for baseline, Jev-only and hybrid. Do not let a later model change the goal from correct routing to merely returning parseable JSON.

Step 2 — Read parseBaseline in contracts.mjs

Read parseBaseline in contracts.mjs. It accepts a closed route set and a boolean wantsHuman. The internal normalized object uses a numeric 0 or 1 for that boolean so the shared policy can read it; this conversion is application code, not an extra Jev response field.

Step 3 — Run a clear baseline case

Run a clear baseline case. T-001 produces a billing proposal with one baseline attempt. Read its artificial input/output token counts and keep the fictional-evidence label attached. This shows the accounting shape without pretending to measure any named generative service.

Step 4 — Inspect a deliberately wrong fixture

Inspect a deliberately wrong fixture. T-005 is a dashboard-loading problem but the canned baseline answer chooses another queue. The schema still passes. Compare the ticket text with the evaluation label separately; never send the label to the provider to repair its prediction.

Step 5 — Test the parser failure path

Test the parser failure path. The contract test passes malformed JSON and an out-of-set route to the parser. Both must fail before a proposal can reach an effect guard. In a live baseline, log a repair attempt as a second inference attempt rather than overwriting the first response.

Step 6 — Run the validation split once

Run the validation split once. Save its report as your pre-change baseline artifact. The twelve synthetic validation tickets are enough to exercise the harness but too small and artificial to estimate deployment accuracy. Keep the held-out test split out of threshold selection.

Step 7 — Review coverage before cost

Review coverage before cost. An apparently cheap method that sends everything to review has not completed the same workload. Record an acceptable selective-accuracy floor and coverage floor for your future real experiment before comparing prices or choosing the winning configuration.

Files and commands

Companion code on GitHub: ai-core-jev-tutorial · README. Use the companion repository as the default working copy. These file links pin the tested revision. src/contracts.mjs · test/contracts.test.mjs

Compare this file with the source in the ZIP before editing. The listing is the implementation used by the lab, not pseudocode. Model output is input to validation and routing code; it grants no permission to bypass approval.

src/contracts.mjs

export const QUEUES = Object.freeze(['billing', 'technical', 'other']);
export function probability(value, name) {
  if (typeof value !== 'number' || !Number.isFinite(value) || value < 0 || value > 1)
    throw new Error(`invalid ${name}: expected a finite number in [0,1]`);
  return value;
}
export function validateTicket(value) {
  if (!value || typeof value.id !== 'string' || !/^T-\d{3}$/.test(value.id)) throw new Error('invalid ticket id');
  if (typeof value.text !== 'string' || !value.text.trim() || value.text.length > 8000) throw new Error('invalid ticket text');
  return Object.freeze({id:value.id, text:value.text, language:value.language ?? 'en'});
}
export function parseJev(response) {
  if (!response || response.model !== 'jev-1.13.0') throw new Error('unexpected or missing pinned model');
  const route = response.answers?.route, human = response.answers?.wantsHuman;
  if (route?.type !== 'choice' || !QUEUES.includes(route.choice)) throw new Error('invalid route label/type');
  const confidence = probability(route.confidence, 'choice confidence');
  const probabilities = route.probabilities;
  if (!probabilities || Object.keys(probabilities).sort().join() !== [...QUEUES].sort().join()) throw new Error('incomplete option distribution');
  const values = QUEUES.map(k => probability(probabilities[k], `probability ${k}`));
  if (Math.abs(values.reduce((a,b)=>a+b,0)-1) > 1e-5) throw new Error('probabilities do not sum to one');
  if (probabilities[route.choice] + 1e-8 < Math.max(...values)) throw new Error('selected choice is not maximal');
  const expected=(Math.max(...values)-1/QUEUES.length)/(1-1/QUEUES.length);
  if(Math.abs(confidence-expected)>1e-5)throw new Error('inconsistent Choice confidence');
  if (human?.type !== 'noul') throw new Error('missing Noul answer');
  return Object.freeze({route:route.choice, confidence, probabilities:{...probabilities}, wantsHuman:probability(human.noul,'Noul')});
}
export function parseBaseline(raw) {
  const value=JSON.parse(raw);
  if (!QUEUES.includes(value.route) || typeof value.wantsHuman !== 'boolean') throw new Error('invalid baseline schema');
  return Object.freeze({route:value.route,wantsHuman:value.wantsHuman?1:0,confidence:null,probabilities:null});
}
export function tokenUsage(response) {
  const input=response?.usage?.input_tokens, output=response?.usage?.output_tokens;
  const valid=x=>Number.isSafeInteger(x)&&x>=0;
  return valid(input)&&valid(output) ? {input,output} : null;
}

Run these offline commands without an SDK installation or API key:

node src/cli.mjs demo --mode baseline --ticket T-001
node src/cli.mjs demo --mode baseline --ticket T-005
node --test --test-reporter=spec test/contracts.test.mjs
node src/cli.mjs evaluate --split validation --save

Verify the expected behavior

T-005 should demonstrate a valid schema with a wrong canned route, not a parser exception. The report must expose coverage and unresolvedReviewTasks. No result in this chapter establishes the relative quality of Jev and a real generative model.

Read the named tests and refusal paths, not just the exit code or final label. Unknown values must remain unknown, and every report must distinguish simulation inputs from real measurements.

Troubleshooting

If all three pipelines report the same results, check which provider each attempt names. If malformed JSON silently becomes other, remove that default: it hides a failure. If the denominator changes between runs, stop and restore one success definition before interpreting any difference.

When previewing with Python http.server, use the .html continuation links below when needed. Extensionless series chips rely on GitHub Pages resolution; that documented preview limitation is separate from the Node example.

Change one thing, then test

In a disposable copy, make the T-001 baseline string malformed. Confirm that parser tests or the baseline demonstration refuse it rather than choosing a default route. Restore the fixture before comparing modes. A formatting failure and a valid wrong answer should remain separate findings in your notebook.

Make changes in a disposable copy, retain the reference files for comparison, and restore them before the next chapter. If you later use a real provider, create a new experiment record rather than overwriting the fixture transcript.

Next checkpoint

Before moving on, identify the contract file, the policy file, what was observed in a run, and what remains untested against a real model. Keeping those four things separate matters more than making the demo look effortless.

← Previous chapter · Next chapter →

References and evidence

Key takeaways