Validate Responses — ตรวจคำตอบก่อนเชื่อValidate Every Response — AI-Core + Jev in Practice

Part 4 of ten: Validate labels, distributions, confidence and Noul at runtime; reject malformed responses and preserve unknown usage as null.

Local review draft • Default code uses offline fixtures. Numbers are fictional scenarios, not measured Jev results. Faster performance and lower cost remain hypotheses.

Outcome and prerequisites

Start with the companion repository and its detailed README. The website ZIP remains an alternative offline snapshot; run the same Node commands from either project root. GitHub repository · README / Quickstart

Use the same Support Ticket Decision Lab throughout the series. Have Node 22.18+, a terminal and a writable directory. Run commands from the cloned ai-core-jev-tutorial root, or the extracted ai-core-jev-lab folder if using the website ZIP. Offline work needs no npm install. Allow roughly 20–40 minutes for experiments; the article reading-time label excludes hands-on work.

Download the complete code and tests (ZIP) · Project README

This chapter focuses on src/contracts.mjs. Fixture outputs test the integration mechanics only. The author did not install the SDK or use an API key while preparing the lab. The separately documented live path has not been tested against the API.

How this boundary works

Types at a client boundary do not replace runtime validation. A fixture can be corrupted, a network response can be incomplete, and a future adapter may return the wrong model or field names. Our contract requires the pinned model, the expected answer types, all route probabilities, a normalized distribution, a maximal selected option and a finite Noul.

Validation establishes bounded structural facts. It can prove that admin is not an allowed queue and that NaN is not a usable probability. It cannot prove that a billing classification is correct for a particular sentence. Keep that distinction visible in both tests and explanations, otherwise green parser tests become a misleading claim about semantic safety.

The live billing boundary is also a schema boundary. usage.input_tokens or usage.output_tokens may be absent or null in an integration. tokenUsage returns null for unknown usage instead of an attractive zero-dollar estimate. A complete final response also does not reveal every internally retried HTTP attempt, which is why the live wrapper does not claim a complete bill.

Seven practical steps

Step 1 — Read the probability function first

Read the probability function first. Reject strings, null, infinities, NaN and values outside zero to one. Avoid coercion such as Number(value): converting a missing field into zero can turn uncertainty into apparently strong evidence for the negative class.

Step 2 — Validate the complete route distribution

Validate the complete route distribution. All three expected keys must be present, values must sum to one within a small numerical tolerance, and the chosen label must be maximal. This catches incomplete transport and inconsistent adapters; it is not a semantic truth test.

Step 3 — Check confidence consistency

Check confidence consistency. The documented Choice formula derives confidence from the top option probability and option count. The validator checks that relationship, so a response cannot pair a nearly flat distribution with confidence one and accidentally bypass the abstention policy.

Step 4 — Validate Noul on its own terms

Validate Noul on its own terms. Require a noul answer type and a finite noul value. Do not require a confidence field that the API does not document for Noul. Our tests explicitly cover the valid case in which Noul has no such field.

Step 5 — Exercise a corrupted fixture

Exercise a corrupted fixture. T-040 contains the unsupported admin label. Jev-only mode must produce needs_review with no action. Hybrid mode may ask the separate baseline provider for a new proposal, but it never blesses the invalid Jev answer as if validation had succeeded.

Step 6 — Exercise missing usage separately from a bad decision

Exercise missing usage separately from a bad decision. T-036 contains a structurally valid decision and null usage. It can still be proposed under the policy, but its full cost remains unknown. Availability of a route and availability of a reliable accounting record are two different properties.

Step 7 — Run and read the named tests

Run and read the named tests. Add a new malformed-response case before extending the adapter. A useful test starts at a failure you can describe and asserts the refusal route; it does not simply copy the implementation expression and congratulate the same expression for evaluating identically.

Files and commands

Companion code on GitHub: ai-core-jev-tutorial · README. Use the companion repository as the default working copy. These file links pin the tested revision. src/contracts.mjs · test/contracts.test.mjs

Compare this file with the source in the ZIP before editing. The listing is the implementation used by the lab, not pseudocode. Model output is input to validation and routing code; it grants no permission to bypass approval.

src/contracts.mjs

export const QUEUES = Object.freeze(['billing', 'technical', 'other']);
export function probability(value, name) {
  if (typeof value !== 'number' || !Number.isFinite(value) || value < 0 || value > 1)
    throw new Error(`invalid ${name}: expected a finite number in [0,1]`);
  return value;
}
export function validateTicket(value) {
  if (!value || typeof value.id !== 'string' || !/^T-\d{3}$/.test(value.id)) throw new Error('invalid ticket id');
  if (typeof value.text !== 'string' || !value.text.trim() || value.text.length > 8000) throw new Error('invalid ticket text');
  return Object.freeze({id:value.id, text:value.text, language:value.language ?? 'en'});
}
export function parseJev(response) {
  if (!response || response.model !== 'jev-1.13.0') throw new Error('unexpected or missing pinned model');
  const route = response.answers?.route, human = response.answers?.wantsHuman;
  if (route?.type !== 'choice' || !QUEUES.includes(route.choice)) throw new Error('invalid route label/type');
  const confidence = probability(route.confidence, 'choice confidence');
  const probabilities = route.probabilities;
  if (!probabilities || Object.keys(probabilities).sort().join() !== [...QUEUES].sort().join()) throw new Error('incomplete option distribution');
  const values = QUEUES.map(k => probability(probabilities[k], `probability ${k}`));
  if (Math.abs(values.reduce((a,b)=>a+b,0)-1) > 1e-5) throw new Error('probabilities do not sum to one');
  if (probabilities[route.choice] + 1e-8 < Math.max(...values)) throw new Error('selected choice is not maximal');
  const expected=(Math.max(...values)-1/QUEUES.length)/(1-1/QUEUES.length);
  if(Math.abs(confidence-expected)>1e-5)throw new Error('inconsistent Choice confidence');
  if (human?.type !== 'noul') throw new Error('missing Noul answer');
  return Object.freeze({route:route.choice, confidence, probabilities:{...probabilities}, wantsHuman:probability(human.noul,'Noul')});
}
export function parseBaseline(raw) {
  const value=JSON.parse(raw);
  if (!QUEUES.includes(value.route) || typeof value.wantsHuman !== 'boolean') throw new Error('invalid baseline schema');
  return Object.freeze({route:value.route,wantsHuman:value.wantsHuman?1:0,confidence:null,probabilities:null});
}
export function tokenUsage(response) {
  const input=response?.usage?.input_tokens, output=response?.usage?.output_tokens;
  const valid=x=>Number.isSafeInteger(x)&&x>=0;
  return valid(input)&&valid(output) ? {input,output} : null;
}

Run these offline commands without an SDK installation or API key:

node --test --test-reporter=spec test/contracts.test.mjs
node src/cli.mjs demo --mode jev --ticket T-040
node src/cli.mjs demo --mode jev --ticket T-036

Verify the expected behavior

The corrupted admin response must not produce an action. Unknown usage must remain null, while a genuine known zero stays a numeric zero. Test the two cases together to stop a convenience default from quietly breaking cost reports.

Read the named tests and refusal paths, not just the exit code or final label. Unknown values must remain unknown, and every report must distinguish simulation inputs from real measurements.

Troubleshooting

If the validator rejects all real responses, compare the actual SDK response shape with the pinned official documentation before weakening checks. If confidence and probability disagree, inspect option count and adapter transformations. Do not add fallback aliases for invented field names just to make a demo run.

When previewing with Python http.server, use the .html continuation links below when needed. Extensionless series chips rely on GitHub Pages resolution; that documented preview limitation is separate from the Node example.

Change one thing, then test

Add a contract test that deletes confidence from Choice, then one that replaces a Noul value with the string zero. Both should fail validation. Keep a valid Noul without separate confidence as a passing counterexample, so tightening validation does not accidentally reject the documented response shape.

Make changes in a disposable copy, retain the reference files for comparison, and restore them before the next chapter. If you later use a real provider, create a new experiment record rather than overwriting the fixture transcript.

Next checkpoint

Before moving on, identify the contract file, the policy file, what was observed in a run, and what remains untested against a real model. Keeping those four things separate matters more than making the demo look effortless.

← Previous chapter · Next chapter →

References and evidence

Key takeaways