Evaluation — วัดผลโดยไม่สับสน fixture กับหลักฐานMeasure Outcomes Without Confusing Fixtures with Evidence

1. The engineering problem

A fixture answers exactly what a test author told it to answer. Seven fixture cases agreeing with seven expected outcomes demonstrate that the wiring follows those scripted conditions. They do not demonstrate seven correct Jev predictions. The companion therefore reports mode=scripted_architecture_fixtures, agreements and cases, while model_accuracy, live_latency_ms and live_cost remain null. Missing evidence must stay missing rather than becoming a reassuring zero.

Create separate development, validation and held-out sets before tuning questions or thresholds. Split by repository, issue family or time when near-duplicate tasks could leak between sets. Use validation to choose thresholds and freeze them before testing. Label route fit, attempted policy override, candidate correctness and explanation quality independently. Preserve adjudication rules and disagreements. AgentBench shows why agent evaluation needs multiple environments and failure analysis [1]; our small case register is only a starting architecture test.

2. The boundary to design

Report acceptance coverage and error among accepted cases together. Coverage is accepted divided by all eligible requests. Accepted error is incorrect accepted divided by accepted; it is undefined when none are accepted. Review rate and rejection rate complete the picture. A system that reviews every case may have no bad automatic actions but provides no automated coverage. Inspect Thai and English separately, and sample reviewed cases to estimate unnecessary escalation. Distribution-derived confidence needs local calibration checks; calibration research explains why confidence and accuracy can diverge [2].

3. Read the implementation

Implementation excerpt from the companion. Run the complete lesson below with its imports and fixtures.

from workbench.evaluation import evaluate

report = evaluate()
print(report["agreements"], report["cases"])
print(report["model_accuracy"])  # None: fixtures are not live evidence

Measure the entire task path: decision calls, generation, repairs, fallback, human review and rework. Cost per successful task divides complete cost by tasks meeting the predeclared success contract. Keep provider token usage, pricing version, elapsed-time measurement and human-cost assumptions explicit. A p95 from a few toy requests is unstable, and a zero-failure sample does not prove zero risk. Compare against a deterministic baseline and the existing human workflow under the same task definition. Publish live results only after collecting them; this course claims tested architecture behavior, not model superiority.

4. Compare the outcomes

The following output comes from scripted fixtures that verify architecture behavior. It is not a measurement of live Jev accuracy, latency, or cost.

{
  "mode": "scripted_architecture_fixtures",
  "cases": 7,
  "agreements": 7,
  "model_accuracy": null,
  "live_latency_ms": null,
  "live_cost": null,
  "rows": [
    {
      "case": "valid",
      "expected": "accepted_draft",
      "actual": "accepted_draft"
    },
    {
      "case": "permission",
      "expected": "reject",
      "actual": "reject"
    },
    {
      "case": "schema",
      "expected": "reject",
      "actual": "reject"
    },
    {
      "case": "injection_fixture",
      "expected": "review",
      "actual": "review"
    },
    {
      "case": "path",
      "expected": "reject",
      "actual": "reject"
    },
    {
      "case": "arithmetic",
      "expected": "review",
      "actual": "review"
    },
    {
      "case": "quality",
      "expected": "review",
      "actual": "review"
    }
  ]
}

5. Run, break, and explain

Part 14 code · Implementation · Tests: test_fixtures_do_not_claim_model_accuracy · Exercise

git clone https://github.com/Anirach/ai-core-engineering-workbench.git
cd ai-core-engineering-workbench
git checkout 18cdfcd4156d3d02acb8badf3b3a0a823113e0ed
python3 -m workbench.lesson 14
python3 -m unittest tests.test_boundaries.BoundaryTests.test_fixtures_do_not_claim_model_accuracy -v

Your turn

Design a held-out set with language and task slices. Write the coverage and accepted-error formulas, including the zero-accepted case.

Keep three artifacts: the input you tried, the resulting state and reason, and an explanation of which boundary owns that result. Do not change an expected result merely to make a test pass.

6. Sources and limits