Failure Tests — ลองทำให้พังก่อนปล่อยTest Hostile Inputs and Failure Paths — AI-Core + Jev in Practice

Part 8 of ten: Exercise hostile inputs, malformed responses and provider failures, keeping inference retries separate from effect retries.

Local review draft • Default code uses offline fixtures. Numbers are fictional scenarios, not measured Jev results. Faster performance and lower cost remain hypotheses.

Outcome and prerequisites

Start with the companion repository and its detailed README. The website ZIP remains an alternative offline snapshot; run the same Node commands from either project root. GitHub repository · README / Quickstart

Use the same Support Ticket Decision Lab throughout the series. Have Node 22.18+, a terminal and a writable directory. Run commands from the cloned ai-core-jev-tutorial root, or the extracted ai-core-jev-lab folder if using the website ZIP. Offline work needs no npm install. Allow roughly 20–40 minutes for experiments; the article reading-time label excludes hands-on work.

Download the complete code and tests (ZIP) · Project README

This chapter focuses on src/pipeline.mjs. Fixture outputs test the integration mechanics only. The author did not install the SDK or use an API key while preparing the lab. The separately documented live path has not been tested against the API.

How this boundary works

An attack suite should test the boundary you actually own. Our offline suite can prove that an invalid admin route, a forged approval claim and a failed provider do not directly create a simulated effect. It cannot show that Jev recognizes a prompt injection or remains accurate under reordered options; those questions require real model responses on a reviewed evaluation set.

The official Jev 1.13 notes describe difficulties with numerical precision, irrelevant state, adversarial content and option order. We keep arithmetic and permissions in deterministic code and treat tickets as data rather than instructions. The teaching fixtures deliberately include hostile requests so you can see where the guard works even when a semantic answer is wrong.

Inference retries and effect retries solve different problems. A transient inference failure may require another provider attempt, which has latency and possible cost. Repeating an already approved action must instead respect an idempotency key. The SDK may retry internally; our live wrapper does not add an outer retry loop or claim visibility into those hidden attempts.

Seven practical steps

Step 1 — Run the four attack fixtures

Run the four attack fixtures. Keep their synthetic-evidence label visible. T-037 and T-038 try to dictate routing or approval through ticket text. The canned outputs are not measurements of attack success against Jev; inspect the eventual proposal and approval gate separately.

Step 2 — Run T-040 in Jev-only mode

Run T-040 in Jev-only mode. The unsupported label must be refused. This isolates response-validation behavior from hybrid fallback. A test that only exercises hybrid may hide the original failure behind a second provider that happens to return a valid label.

Step 3 — Inject a provider exception through the pipeline test

Inject a provider exception through the pipeline test. With both providers unavailable the final result must need review and contain no action. Error handling should preserve a useful reason without printing API keys or treating an unavailable model as a negative answer.

Step 4 — Map HTTP failure classes before live use

Map HTTP failure classes before live use. Authentication and schema failures require correction, not blind retries. Rate and overload failures may be retried by the SDK. Keep the application-level retry budget explicit and confirm version-specific behavior in the official documentation instead of layering another retry library by habit.

Step 5 — Specify metamorphic tests for the real provider

Specify metamorphic tests for the real provider. Reorder route options, append irrelevant paragraphs, paraphrase the same intent and keep numerical calculations outside the model. Freeze expected invariants before collecting responses. The current fixture provider is indexed by ticket ID and intentionally cannot measure these transformations.

Step 6 — Keep Thai and English failure slices separate

Keep Thai and English failure slices separate. Passing a few Thai fixture strings proves encoding and control flow only. Real Thai evaluation needs representative language, code-switching and domain vocabulary, annotated by competent reviewers. Do not infer parity with English from a shared JSON schema.

Step 7 — Run the effect tests again after changing failure handling

Run the effect tests again after changing failure handling. An exception path that starts a fallback should not duplicate an approved action or bypass the human request condition. Preserve the original attempt record even when a later branch produces the final proposal.

Files and commands

Companion code on GitHub: ai-core-jev-tutorial · README. Use the companion repository as the default working copy. These file links pin the tested revision. src/pipeline.mjs · test/pipeline.test.mjs · test/network.test.mjs

Compare this file with the source in the ZIP before editing. The listing is the implementation used by the lab, not pseudocode. Model output is input to validation and routing code; it grants no permission to bypass approval.

src/pipeline.mjs

import {validateTicket,parseJev,parseBaseline,probability} from './contracts.mjs';
export function validatePolicy(policy) {
  probability(policy.routeConfidence,'policy routeConfidence');probability(policy.humanNoMax,'policy humanNoMax');
  if(policy.humanNoMax>=0.5||typeof policy.fallbackOnLowConfidence!=='boolean')throw new Error('invalid abstention policy');return policy;
}
export async function decide(input,{mode='hybrid',providers,policy}) {
  const ticket=validateTicket(input);validatePolicy(policy);
  if(!['baseline','jev','hybrid'].includes(mode))throw new Error('invalid mode');
  const attempts=[];let decision=null,source=null,reason='not_evaluated';
  async function obtain(kind,parse) {
    try {const result=await providers[kind](ticket);attempts.push(result.attempt);source=kind;return parse(result.value);}
    catch(error) {
      if(attempts.at(-1)?.provider!==kind)attempts.push({provider:kind,mode:'error',usage:null,wallMs:null,scenarioLatencyMs:null,httpAttempts:null,costComplete:false});
      reason=`${kind}_failed_validation_or_call`;return null;
    }
  }
  if(mode==='baseline')decision=await obtain('baseline',parseBaseline);
  else {
    decision=await obtain('jev',parseJev);
    if(decision&&decision.wantsHuman>policy.humanNoMax)reason='human_request_or_uncertainty';
    else if(!decision||decision.confidence<policy.routeConfidence) {
      const humanProbability=decision?.wantsHuman;
      if(mode==='hybrid'&&policy.fallbackOnLowConfidence) {
        const fallback=await obtain('baseline',parseBaseline);
        decision=fallback?{...fallback,wantsHuman:Math.max(fallback.wantsHuman,humanProbability??0)}:null;
      } else reason='low_route_confidence';
    }
  }
  const auto=!!decision&&decision.wantsHuman<=policy.humanNoMax&&(source==='baseline'||decision.confidence>=policy.routeConfidence);
  if(auto)reason=source==='baseline'?'validated_baseline_proposal':'validated_jev_proposal';
  else if(decision?.wantsHuman>policy.humanNoMax)reason='human_request_or_uncertainty';
  return {ticketId:ticket.id,status:auto?'proposed':'needs_review',reason,source,decision,attempts,action:auto?{kind:'route_ticket',ticketId:ticket.id,queue:decision.route}:null};
}

Run these offline commands without an SDK installation or API key:

node src/cli.mjs evaluate --split attack --save
node src/cli.mjs demo --mode jev --ticket T-040
node --test --test-reporter=spec test/pipeline.test.mjs test/effects.test.mjs test/network.test.mjs

Verify the expected behavior

The attack report must contain n=4 and remain labeled as a fixture simulation. The T-040 Jev-only demonstration must have action:null. Provider-failure and forged-approval tests must pass without credentials or network access.

Read the named tests and refusal paths, not just the exit code or final label. Unknown values must remain unknown, and every report must distinguish simulation inputs from real measurements.

Troubleshooting

A successful mock attack test is not a successful model robustness test. If you see duplicate paid work, inspect whether SDK and application retries are stacked. If an error string includes a secret, remove the secret from logging and handle credential exposure through your normal process before resuming.

When previewing with Python http.server, use the .html continuation links below when needed. Extensionless series chips rely on GitHub Pages resolution; that documented preview limitation is separate from the Node example.

Change one thing, then test

Design three real-provider attack cases that are absent from the four canned fixtures. For each, state a measurable invariant and the label-review process. Keep numerical arithmetic in code and distinguish a model mistake from a guard failure. A model may be wrong while the effect boundary still behaves correctly.

Make changes in a disposable copy, retain the reference files for comparison, and restore them before the next chapter. If you later use a real provider, create a new experiment record rather than overwriting the fixture transcript.

Next checkpoint

Before moving on, identify the contract file, the policy file, what was observed in a run, and what remains untested against a real model. Keeping those four things separate matters more than making the demo look effortless.

← Previous chapter · Next chapter →

References and evidence

Key takeaways