Behavior — ทดสอบผลกับคำตอบที่เป็นอิสระTest Behavior Against an Independent Oracle

1. The engineering problem

The candidate price - percent passes our restricted-language validator. For price=1999 and percent=25 it returns 1974, while the declared requirement expects 1499. This is the cleanest reason to keep output shape separate from output behavior. A parser cannot know the business requirement. A model's explanation of its own code is also not an independent oracle. Put exact arithmetic in trusted tests and reserve semantic evaluation for questions that actually need interpretation.

The companion interprets the validated return expression with two integer inputs. It never executes the source as Python. Six maintained examples cover normal discounting, zero percent, full discount, fractional-cent rounding and zero price. behavior_check returns revise for a permitted expression whose result differs from an expected value. Division by zero and bounded-arithmetic failures reject. The interpreter has a small node and value budget so an admitted expression cannot request arbitrary computation.

2. The boundary to design

Example tests establish behavior only at their chosen points. The test suite adds a monotonicity property: as discount percent increases from 0 through 100 for selected prices, the result must stay nonnegative and never increase. This catches a different class of mistakes, but still does not prove equivalence for every integer. Property selection is a specification activity. A malicious or overfit candidate can agree on visible examples while failing elsewhere. Keep held-out cases and use stronger methods when the domain or consequence demands them.

3. Read the implementation

Implementation excerpt from the companion. Run the complete lesson below with its imports and fixtures.

def behavior_check(candidate):
    valid = output_check(candidate)
    if valid.state != "pass":
        return valid
    expression = ast.parse(candidate["source"]).body[0].body[0].value
    try:
        for price, percent, expected in CASES:
            if interpret(expression, {"price": price, "percent": percent}) != expected:
                return Gate("revise", "behavior_mismatch")
    except (ArithmeticError, ValueError, KeyError, RecursionError):
        return Gate("reject", "arithmetic_failure")
    return Gate("pass", "behavior_examples_pass")

Specify the function's input domain separately: integer price >= 0 and integer percent in [0,100], with host validation before invocation in an actual service. The lab's trusted examples already satisfy that domain. Do not read the interpreted function as a complete pricing API. TypeSafe's documented numeric limitations reinforce the division of labor [1], but no model-specific result is needed to justify exact tests. The observable outcome here is decisive: shape=pass, behavior=revise; after replacing the formula with price * (100 - percent) // 100, behavior=pass.

4. Compare the outcomes

The following output comes from scripted fixtures that verify architecture behavior. It is not a measurement of live Jev accuracy, latency, or cost.

{
  "shape": "pass",
  "behavior": "revise",
  "fixed": "pass",
  "trusted_examples": 6
}

5. Run, break, and explain

Part 11 code · Implementation · Tests: test_shape_does_not_imply_behavior · Exercise

git clone https://github.com/Anirach/ai-core-engineering-workbench.git
cd ai-core-engineering-workbench
git checkout 18cdfcd4156d3d02acb8badf3b3a0a823113e0ed
python3 -m workbench.lesson 11
python3 -m unittest tests.test_boundaries.BoundaryTests.test_shape_does_not_imply_behavior -v

Your turn

Create a formula that passes one visible example but fails a boundary case. Add a property that catches it without copying its implementation.

Keep three artifacts: the input you tried, the resulting state and reason, and an explanation of which boundary owns that result. Do not change an expected result merely to make a test pass.

6. Sources and limits