Evaluation — ทดสอบการโจมตีและต้นทุนรวมEvaluate Adversaries and Full Cost

Part 11 of twelve. Keep held-out evaluation separate from tuning and report false accepts, language slices and complete task cost.

Default decisions and draft text are authored mocks, not live Jev results. Use the tutorial with the matching companion code, tests and exercise linked to a fixed GitHub version.

Outcome and prerequisites

Use Node22.18+ and a terminal. Clone the GitHub companion using its README, or download the complete ZIP, extract it and enter ai-core-functional-lab. No npm install, API key or network inference is needed. The reading-time label excludes about25–45 minutes of hands-on work. Keep one reference copy and make exercises in another.

Download companion ZIP · README / lesson map

This part focuses on src/evaluation.mjs. Expected gate states describe content and continuation; no state sends a message or changes a real account.

Old functions and the new boundary

Unit tests establish implementation behavior; evaluation asks whether accepted artifacts meet an independent reference criterion. Keep validation for policy design, held-out cases for a frozen candidate and attacks for stress testing. These public authored fixtures demonstrate that separation, not a statistically valid blind study.

Acceptance coverage alone rewards a permissive system. Report accepted precision and false accepts with review/reject counts and language slices. A deliberately flawed held-out fixture passes every runtime semantic check, making the gap between self-checking and independent evaluation visible.

Cost per accepted result and cost per correct accepted result have different denominators. Include every inference attempt, failed retry, generation, fallback infrastructure, review time and rework. Unit prices, completeness and the acceptance rule must stay visible; missing usage must never become zero.

Seven practical steps

Step 1

Run tune with validation only and record the teaching thresholds before opening held-out results. The CLI refuses a heldout tuning split. Small canned examples cannot estimate production-optimal thresholds.

Step 2

Run the held-out evaluation once for the frozen reference. Keep out/heldout.json rather than only copying a summary. Its context includes release IDs and exact source/data hashes.

Step 3

Inspect H010 after evaluation. The authored judges missed an unsupported emailing claim; it is a false accept against the fixture reference label. This teaches a control limitation and is not a measured Jev1.14 error rate.

Step 4

Read Thai and English slices and their case counts. Equal-looking fixture outputs do not demonstrate language parity. A real study needs independently annotated examples, uncertainty estimates and enough cases in every relevant slice.

Step 5

Run the attack split and examine every case history. Missing adversarial failures from an average can make a system look cheap and accurate. Keep refusal and zero external-effect assertions separate from semantic detection claims.

Step 6

Run cost for V001 and V010. The first has complete fictional usage; the second deliberately lacks usage despite an accepted artifact. Full held-out cost is also unknown because one case lacks usage.

Step 7

Inspect p50/p95 labels before comparing timings. Scenario milliseconds are authored inputs and harnessWallMs is actual local code runtime. Neither is live Jev latency; incomplete timing data should not be silently dropped from a speed claim.

Files and runnable commands

Open the exact source and read its dependencies before editing. This is runnable reference code, not a fictional AI-Core SDK. The source links point to a fixed version of the public GitHub companion.

src/evaluation.mjs · Matching tests · This lesson’s exercise

import {readFile} from 'node:fs/promises';
import {createHash,randomUUID} from 'node:crypto';
import {performance} from 'node:perf_hooks';
import {runPipeline} from './pipeline.mjs';
import {fixtureProvider} from './fixtures.mjs';
const load=async p=>JSON.parse(await readFile(new URL('../'+p,import.meta.url),'utf8'));
export function percentile(xs,p){if(!xs.length)return null;const s=[...xs].sort((a,b)=>a-b);return s[Math.max(0,Math.ceil(p*s.length)-1)]}
export function summarize(rows,prices) {
  for(const k of ['jevInputPerMillionUSD','jevOutputPerMillionUSD','generatorInputPerMillionUSD','generatorOutputPerMillionUSD','infrastructurePerTaskUSD','reviewSeconds','reviewHourlyUSD','falseAcceptReworkUSD'])
    if(typeof prices[k]!=='number'||!Number.isFinite(prices[k])||prices[k]<0)throw new Error('invalid price '+k);
  const n=rows.length,accepted=rows.filter(r=>r.result.state==='accept');
  const correct=accepted.filter(r=>r.case.acceptable),wrong=accepted.length-correct.length;
  const reviews=rows.filter(r=>r.result.state==='human_review').length;
  const costs=rows.flatMap(r=>r.result.attempts.map(a=>{
    if(!a.usage)return null;
    const prefix=a.stage==='generate'?'generator':'jev';
    return (a.usage.input*prices[prefix+'InputPerMillionUSD']+a.usage.output*prices[prefix+'OutputPerMillionUSD'])/1e6;
  }));
  const knownInferenceUSD=costs.reduce((s,c)=>s+(c??0),0),costComplete=costs.every(c=>c!==null);
  const modeledOverheadUSD=n*prices.infrastructurePerTaskUSD+reviews*prices.reviewSeconds/3600*prices.reviewHourlyUSD+wrong*prices.falseAcceptReworkUSD;
  const times=rows.map(r=>{const xs=r.result.attempts.map(a=>a.scenarioMs);return xs.every(Number.isFinite)?xs.reduce((s,x)=>s+x,0):null});
  const latencyComplete=times.every(Number.isFinite),total=costComplete?knownInferenceUSD+modeledOverheadUSD:null;
  const ratio=(a,b)=>b?a/b:null;
  return {n,accepted:accepted.length,acceptedCorrect:correct.length,falseAccepts:wrong,humanReview:reviews,
    coverage:ratio(accepted.length,n),acceptedPrecision:ratio(correct.length,accepted.length),
    states:Object.fromEntries(['accept','reject','revise','human_review'].map(s=>[s,rows.filter(r=>r.result.state===s).length])),
    transportRetries:rows.reduce((s,r)=>s+r.result.attempts.filter(a=>a.transportAttempt>1).length,0),
    semanticRevisions:rows.reduce((s,r)=>s+r.result.history.filter(g=>g.state==='revise').length,0),
    latency:{evidence:'FICTIONAL scenario milliseconds, not measured inference latency',complete:latencyComplete,p50:latencyComplete?percentile(times,.5):null,p95:latencyComplete?percentile(times,.95):null},
    cost:{evidence:'FICTIONAL usage/generator/review/rework assumptions; Jev unit price dated2026-10-24',complete:costComplete,knownInferenceUSD,modeledOverheadUSD,totalUSD:total,
      perAcceptedUSD:total!==null&&accepted.length?total/accepted.length:null,
      perCorrectAcceptedUSD:total!==null&&correct.length?total/correct.length:null,
      denominator:'accepted artifacts versus fixture-reference acceptable artifacts (authored teaching labels), not completed support resolutions'}};
}
export async function context() {
  const paths=['config/release.json','config/policy.json','config/prices.json','data/cases.json','data/mock.json','data/facts.json',
    ...['contracts','input','policy','questions','semantic','generation','output','quality','budget','fixtures','pipeline','live','evaluation','cli'].map(n=>'src/'+n+'.mjs')];
  const hashes={};for(const p of paths)hashes[p]=createHash('sha256').update(await readFile(new URL('../'+p,import.meta.url))).digest('hex');
  return {runId:randomUUID(),at:new Date().toISOString(),node:process.version,release:await load('config/release.json'),sourceSHA256:hashes};
}
export async function evaluate(split,{policy,facts,prices,cases,signal}={}) {
  if(!['validation','heldout','attack'].includes(split))throw new Error('invalid split');
  const start=performance.now(),rows=[];
  for(const item of cases.filter(c=>c.split===split)) {
    const provider=await fixtureProvider(item.id);
    rows.push({case:structuredClone(item),result:await runPipeline(item.input,{provider,policy,facts,signal})});
  }
  return {evidence:'OFFLINE AUTHORED FIXTURES, NOT LIVE JEV PERFORMANCE',split,context:await context(),policy:{...policy},priceAssumptions:{...prices},trustedFacts:structuredClone(facts),
    summary:summarize(rows,prices),languageSlices:Object.fromEntries([...new Set(rows.map(r=>r.case.input.language))].map(l=>[l,summarize(rows.filter(r=>r.case.input.language===l),prices)])),
    cases:rows,harnessWallMs:performance.now()-start};
}
export function releaseGate(report) {
  if(report?.split!=='heldout')return {state:'reject',reasons:['release_requires_heldout_split']};
  const invalid=releaseMetricErrors(report.summary);
  if(invalid.length)return {state:'reject',reasons:['invalid_release_metrics',...invalid]};
  if(report.summary.falseAccepts)return {state:'reject',reasons:['heldout_false_accepts']};
  if(!report.summary.cost.complete)return {state:'human_review',reasons:['unknown_total_cost']};
  if(report.summary.coverage===null||report.summary.coverage<.5)return {state:'revise',reasons:['coverage_below_teaching_gate']};
  return {state:'accept',reasons:['synthetic_release_checks_only_not_production_approval']};
}

// Validate evidence before comparisons: undefined and NaN must never open a gate.
// This checks metric consistency, not report authenticity or production readiness.
export function releaseMetricErrors(s) {
  const record=x=>x!==null&&typeof x==='object'&&!Array.isArray(x);
  if(!record(s))return ['summary_required'];
  const errors=[],add=x=>errors.push(x);
  const count=x=>Number.isSafeInteger(x)&&x>=0;
  const money=x=>typeof x==='number'&&Number.isFinite(x)&&x>=0;
  const close=(a,b)=>Math.abs(a-b)<=1e-9*Math.max(1,Math.abs(a),Math.abs(b));
  const counts=['n','accepted','acceptedCorrect','falseAccepts','humanReview','transportRetries','semanticRevisions'];
  for(const k of counts)if(!count(s[k]))add(k+'_invalid_count');
  if(!count(s.n)||s.n===0)add('nonempty_evaluation_required');
  if(errors.length)return errors;
  const states=['accept','reject','revise','human_review'];
  if(!record(s.states))add('states_required');
  else {
    if(Object.keys(s.states).some(k=>!states.includes(k)))add('unknown_state');
    for(const k of states)if(!count(s.states[k]))add('states_'+k+'_invalid_count');
    if(states.every(k=>count(s.states[k]))&&states.reduce((n,k)=>n+s.states[k],0)!==s.n)add('state_total_mismatch');
    if(s.states.accept!==s.accepted||s.states.human_review!==s.humanReview)add('state_count_mismatch');
  }
  if(['accepted','acceptedCorrect','falseAccepts','humanReview'].some(k=>count(s[k])&&s[k]>s.n))add('count_exceeds_evaluation');
  if(count(s.acceptedCorrect)&&count(s.falseAccepts)&&s.acceptedCorrect+s.falseAccepts!==s.accepted)add('accepted_count_mismatch');
  if(typeof s.coverage!=='number'||!Number.isFinite(s.coverage)||s.coverage<0||s.coverage>1)add('coverage_invalid');
  else if(s.n>0&&!close(s.coverage,s.accepted/s.n))add('coverage_count_mismatch');
  if(s.accepted===0) {if(s.acceptedPrecision!==null)add('empty_precision_must_be_null')}
  else if(typeof s.acceptedPrecision!=='number'||!Number.isFinite(s.acceptedPrecision)||s.acceptedPrecision<0||s.acceptedPrecision>1)add('precision_invalid');
  else if(!close(s.acceptedPrecision,s.acceptedCorrect/s.accepted))add('precision_count_mismatch');
  if(!record(s.latency)||typeof s.latency.complete!=='boolean')add('latency_required');
  else if(s.latency.complete) {
    if(!money(s.latency.p50)||!money(s.latency.p95)||s.latency.p50>s.latency.p95)add('latency_percentiles_invalid');
  } else if(s.latency.p50!==null||s.latency.p95!==null)add('unknown_latency_must_be_null');
  const c=s.cost;
  if(!record(c)||typeof c.complete!=='boolean')add('cost_required');
  else {
    const componentsValid=money(c.knownInferenceUSD)&&money(c.modeledOverheadUSD);
    if(!componentsValid)add('cost_components_invalid');
    if(c.complete) {
      if(!money(c.totalUSD)||!componentsValid||!close(c.totalUSD,c.knownInferenceUSD+c.modeledOverheadUSD))add('cost_total_invalid');
      for(const [k,n] of [['perAcceptedUSD',s.accepted],['perCorrectAcceptedUSD',s.acceptedCorrect]]) {
        if(n===0) {if(c[k]!==null)add(k+'_must_be_null')}
        else if(!money(c[k])||!money(c.totalUSD)||!close(c[k],c.totalUSD/n))add(k+'_invalid');
      }
    } else if(['totalUSD','perAcceptedUSD','perCorrectAcceptedUSD'].some(k=>c[k]!==null))add('unknown_cost_must_be_null');
  }
  return errors;
}

Offline commands:

node src/cli.mjs tune --split validation
node src/cli.mjs evaluate --split heldout --save
node src/cli.mjs evaluate --split attack --save
node src/cli.mjs cost --case V001
node src/cli.mjs cost --case V010
node --test --test-reporter=spec test/evaluation.test.mjs

Expected output and verification

The reference held-out run has12 cases,12 accepts and one deliberate false accept; total cost is null due to missing usage. Single V001 modeled cost per correct accepted artifact is approximately USD0.00037024, while V010 is null. These are fictional scenario calculations only.

Inspect reasons, history and every attempt rather than only the final label. Unknown usage must remain unknown. A green test demonstrates the specified code path, not universal semantic safety.

A failure to understand

Optimizing cost per accept can reward accepting wrong answers. The false-accept example makes that failure concrete. Compare costs only after equal acceptable quality and coverage constraints are satisfied on appropriate data.

Exercise in a disposable copy

Add an independently labeled case to a new validation dataset version, not the frozen held-out set. Define what constitutes a false accept and which cost inputs you would measure. Do not replace null usage with a guessed zero.

Follow the matching exercise file, retain the original and restore the reference before the next chapter. Do not use held-out results to choose a threshold.

Troubleshooting

If totals are null, inspect completeness and failed attempts before changing arithmetic. If every artifact accepts, inspect the independent reference labels; runtime gate agreement can still hide factual errors.

Next checkpoint

Before continuing, name the property that code checks exactly, the judgment that may be wrong, the state that stops work and the evidence needed before release.

← Previous lesson · Next lesson →

Sources and version notes

Key takeaways