Shadow Release — ตรึงรุ่นแล้วลองแบบเงาFreeze a Release and Run in Shadow Mode — AI-Core + Jev in Practice

Part 10 of ten: Pin model, SDK, questions, policy and data; prepare shadow mode and rollback while keeping live calls explicitly optional and unverified.

Local review draft • Default code uses offline fixtures. Numbers are fictional scenarios, not measured Jev results. Faster performance and lower cost remain hypotheses.

Outcome and prerequisites

Start with the companion repository and its detailed README. The website ZIP remains an alternative offline snapshot; run the same Node commands from either project root. GitHub repository · README / Quickstart

Use the same Support Ticket Decision Lab throughout the series. Have Node 22.18+, a terminal and a writable directory. Run commands from the cloned ai-core-jev-tutorial root, or the extracted ai-core-jev-lab folder if using the website ZIP. Offline work needs no npm install. Allow roughly 20–40 minutes for experiments; the article reading-time label excludes hands-on work.

Download the complete code and tests (ZIP) · Project README

This chapter focuses on src/providers.mjs. Fixture outputs test the integration mechanics only. The author did not install the SDK or use an API key while preparing the lab. The separately documented live path has not been tested against the API.

How this boundary works

A release is more than a model name. Pin the runtime, SDK, questions, label definitions, policy, dataset and approval/effect code together. The lab records a model and exact SDK version but intentionally does not install dependencies or supply a transitive lockfile. Before a real run, create and review the lockfile in your own environment and record actual versions.

Shadow mode means observing a candidate without giving it an external effect path. The optional live demonstration calls Jev for one synthetic ticket, validates the response and returns a proposal or review result. It does not call a real generative fallback, run bulk paid evaluation, send customer data or execute a support action. That limited scope is deliberate.

The live path has not been executed by the author. Documentation verification is not network verification, account access is not assumed, and an API key is not evidence that a use case is approved. Keep credentials server-side and out of HTML, screenshots, commits and downloaded example files. Recheck official model availability and pricing before opting into a paid request.

Seven practical steps

Step 1 — Open config/release.json and compare every version with the code

Open config/release.json and compare every version with the code. Check that the policy version matches policy.json and the model ID matches the live client. A copied manifest with stale fields is worse than a missing one because it gives false confidence in reproducibility.

Step 2 — Run the full offline suite and doctor from a clean extraction

Run the full offline suite and doctor from a clean extraction. The complete solution must work without an SDK installation, an API key or a network connection. Keep this as the first release gate even if you later add a paid provider: new integration work should not break the learning path.

Step 3 — Save validation, test and attack reports with their evidence labels intact

Save validation, test and attack reports with their evidence labels intact. Store the frozen policy next to the outputs. Do not paste fictional fixture numbers into a product performance claim. A useful release record distinguishes what was executed from what was merely specified.

Step 4 — Review the optional installation instructions without running them as part of the offline course

Review the optional installation instructions without running them as part of the offline course. If you choose live testing on your own account, install the pinned SDK server-side, review its dependency lockfile and provide the key through your shell environment. Never put the key into a tutorial page or browser JavaScript.

Step 5 — Make only an explicit one-ticket live call after reviewing its cost and data scope

Make only an explicit one-ticket live call after reviewing its cost and data scope. The command is shown separately below and was not executed for this draft. A missing key, unavailable SDK or failed call must not create an action. Confirm the returned model and retain a redacted trace for review.

Step 6 — Plan the real shadow experiment before granting authority

Plan the real shadow experiment before granting authority. Use permitted representative inputs, a frozen baseline and identical quality/coverage gates. Record actual end-to-end latency and complete cost evidence, including hidden retry uncertainty and manual outcomes. Do not graduate from shadow merely because the endpoint returns typed JSON.

Step 7 — Prepare rollback and stop conditions

Prepare rollback and stop conditions. Define who can disable the candidate, which release artifact restores the baseline and which increase in errors, human-request misses or unknown costs stops the trial. Authority should grow only after evidence justifies a reviewed change in scope.

Files and commands

Companion code on GitHub: ai-core-jev-tutorial · README. Use the companion repository as the default working copy. These file links pin the tested revision. src/providers.mjs · config/release.json · test/report.test.mjs

Compare this file with the source in the ZIP before editing. The listing is the implementation used by the lab, not pseudocode. Model output is input to validation and routing code; it grants no permission to bypass approval.

src/providers.mjs

import {readFile} from 'node:fs/promises';
import {performance} from 'node:perf_hooks';
import {requestShape,ROUTES,ROUTE_QUESTION,HUMAN_QUESTION} from './questions.mjs';
import {tokenUsage} from './contracts.mjs';
export async function fixtureProviders() {
  const fixtures=JSON.parse(await readFile(new URL('../data/predictions.mock.json',import.meta.url),'utf8'));
  async function call(kind,ticket) {
    const f=fixtures[ticket.id]?.[kind];if(!f)throw new Error('fixture not found');
    return {value:structuredClone(f.value),attempt:{provider:kind,mode:'MOCK / FICTIONAL SCENARIO',usage:f.usage,scenarioLatencyMs:f.scenarioLatencyMs,wallMs:null,httpAttempts:1,costComplete:true}};
  }
  return {jev:t=>call('jev',t),baseline:t=>call('baseline',t)};
}
export async function liveJev(ticket,{live=false}={}) {
  if(!live)throw new Error('Live API requires explicit --live');
  if(!process.env.TYPESAFE_API_KEY)throw new Error('TYPESAFE_API_KEY is not set');
  // Offline mode never resolves the SDK or reads a secret.
  const {choice,noul,TypeSafeClient}=await import('@typesafe-ai/sdk');
  const client=new TypeSafeClient({defaultModel:'jev-1.13.0'});
  const started=performance.now();
  const value=await client.systemOne({state:requestShape(ticket).state,questions:{route:choice(ROUTE_QUESTION,{...ROUTES}),wantsHuman:noul(HUMAN_QUESTION)}});
  return {value,attempt:{provider:'jev',mode:'LIVE / NOT VERIFIED BY THE AUTHOR',usage:tokenUsage(value),wallMs:performance.now()-started,scenarioLatencyMs:null,httpAttempts:null,costComplete:false}};
  // SDK retries may occur internally. No outer retry loop is added. Final
  // response usage alone cannot establish the total bill across failed attempts.
}

Run these offline commands without an SDK installation or API key:

node src/cli.mjs doctor
node --test --test-reporter=spec
node src/cli.mjs evaluate --split validation --save
node src/cli.mjs evaluate --split test --save
node src/cli.mjs evaluate --split attack --save

Optional live path only: it can incur API charges on your account. These commands were not run or live-verified for this draft. Never put a key in the website.

npm install --ignore-scripts @typesafe-ai/[email protected]
# Set TYPESAFE_API_KEY privately in your server-side shell first.
node src/cli.mjs demo --ticket T-001 --live

Verify the expected behavior

All offline commands must pass without node_modules. The live command below is a documented, untested option and is not part of the paid-call-free acceptance suite. A successful shadow observation still authorizes no external help-desk action.

Read the named tests and refusal paths, not just the exit code or final label. Unknown values must remain unknown, and every report must distinguish simulation inputs from real measurements.

Troubleshooting

If the model alias moved, keep the pinned model until re-evaluation. If a live bill lacks failed-attempt usage, label total cost unknown rather than extrapolating zero. If a rollout cannot be stopped without asking the candidate model, implement an application-owned stop control before expanding the trial.

When previewing with Python http.server, use the .html continuation links below when needed. Extensionless series chips rely on GitHub Pages resolution; that documented preview limitation is separate from the Node example.

Change one thing, then test

Draft a go/no-go record with the exact release identifiers, quality constraints, cost completeness, rollback owner and unresolved issues. The correct conclusion may be remain in shadow. Do not authorize an external action merely to make the final tutorial look like a deployment success.

Make changes in a disposable copy, retain the reference files for comparison, and restore them before the next chapter. If you later use a real provider, create a new experiment record rather than overwriting the fixture transcript.

Next checkpoint

Before moving on, identify the contract file, the policy file, what was observed in a run, and what remains untested against a real model. Keeping those four things separate matters more than making the demo look effortless.

← Previous chapter · Return to the series index

References and evidence

Key takeaways