Runtime Validation

Unit tests assert on mocked inputs and isolated functions. Runtime validation asserts on the actual running system — logs, real HTTP responses, and cross-layer data. A suite can be 100% green while the product is broken, because the mocks never hit the code path that fails. This skill is the discipline that closes that gap.

Why this exists (real forge incident, not hypothetical)

hooks/scripts/smoke-test-reminder.sh in this plugin carries the origin verbatim:

On 2026-02-06, an agent spent an entire day writing 2326 mock unit tests while the actual product was broken. A duplicate placeholder route intercepted all real requests. Nobody caught it because nobody started the actual server.

That is the whole thesis: a passing mock suite is not evidence the running system works. The three techniques below are how you get that evidence.

The three techniques

| Technique | What it observes | Catches | |-----------|------------------|---------| | Log monitoring | app logs during a test/server run | validation errors, stack traces, swallowed exceptions the assertion never saw | | Contract validation | real HTTP responses vs their schema | schema drift, missing/renamed fields, type mismatches between layers | | Invariant checking | derived data values | math/domain violations (density > 1.0, negative counts, % > 100) |

Runtime validation composes with your existing tests — it does not replace them. Run the unit suite for logic, then observe the running system for everything the mocks hid.

1. Log monitoring during test/server runs

The cheapest, highest-yield technique: watch the app's own logs for error signatures while you exercise it, and fail the run if any appear — even when every assertion passed.

# Tail logs while a test or smoke run executes; fail on known error signatures.
: > logs/app.log                      # truncate so you only see this run
npx vitest run                        # or: cargo test / pytest / your smoke script
if grep -Eiq 'validation error|panicked at|Input should be|type=\w+_error|Unhandled' logs/app.log; then
  echo "RUNTIME ERRORS in logs despite green suite:" >&2
  grep -Ei 'validation error|panicked at|Input should be|type=\w+_error|Unhandled' logs/app.log >&2
  exit 1
fi

In-test framing (pytest example — the pattern is identical in vitest with a global setup/teardown):

@pytest.fixture
def log_guard(tmp_path):
    log = tmp_path / "app.log"
    patterns = [r"validation error for \w+", r"Input should be .* than", r"type=\w+_error"]
    yield log
    hits = [l for l in log.read_text().splitlines() if any(re.search(p, l) for p in patterns)]
    assert not hits, f"Runtime errors in logs (suite passed anyway): {hits}"

2. Contract validation (response vs schema)

Assert the real response shape against the schema its consumers expect. The status code being 200 says nothing about the body being valid — parse it back through the model.

# pytest + Pydantic
def test_graph_overview_contract(client):
    resp = client.get("/api/graph/overview")
    assert resp.status_code == 200
    GraphOverviewResponse.model_validate(resp.json())  # raises on drift/missing fields

// vitest + Zod (this repo's stacks are JS/vitest and Rust — prefer these over Python here)
const resp = await fetch("/api/graph/overview").then(r => r.json());
GraphOverviewResponse.parse(resp); // throws ZodError on any contract mismatch

The value is the round-trip: response leaves one layer typed, arrives at another, and you re-validate at the boundary instead of trusting that both sides agree.

3. Invariant checking (worked example)

Unit tests often assert a bound but not both. The classic miss:

def test_density():
    density = clustering.calculate_density(nodes, edges)
    assert density > 0          # passes — but never checks the UPPER bound

A green test, a real bug: calculate_density returned 4.5. Runtime validation catches it because the schema (or an explicit invariant) rejects the value the moment it appears:

validation error for ClusterMetadata
  density  Input should be less than or equal to 1  [input_value=4.5]

Encode invariants once and check them against real produced data:

INVARIANTS = [
    ("density",  lambda d: 0.0 <= d["density"] <= 1.0),
    ("percent",  lambda d: 0.0 <= d["percent"] <= 100.0),
    ("count",    lambda d: d["count"] >= 0),
]
for cluster in service.compute_clusters():
    for name, ok in INVARIANTS:
        assert ok(cluster.dict()), f"invariant {name} violated: {cluster.dict()}"

(The density > 1.0 case is an illustrative pattern, not a claimed forge audit finding — use it as the shape of the bug class, not as a citation.)

Wiring into forge

• forge_run_tests (governance-mcp) auto-detects the runner in order vitest → jest → pytest (servers/governance-mcp/tools.mjs:getTestResults). It returns pass/fail counts — use it for the green baseline, then layer log/contract checks on top; a green forge_run_tests is necessary, not sufficient.
• smoke-test-reminder.sh (Stop hook) fires when >3 *.test.* files OR any api-server/server/ file changed in the session, pointing you at ./scripts/smoke-test.sh. Treat that reminder as the trigger to run the log-monitoring step above.
• CRUCIBLE Gate 9 (two-box live-proof) — the ASIF ship gate that a provider is proven on the consuming machine with a live run, not a mocked unit test. Runtime validation is how you satisfy it: start the real thing, exercise it, watch the logs.

Gotchas

• A 200 status is not a valid body. Frameworks return 200 with a malformed or partially serialized payload constantly. Always parse the body back through its schema; never let the status code stand in for contract validity.
• Truncate the log before each run. Grepping a long-lived app.log matches last week's errors and fails a clean run (or, worse, a rotated log hides this run's error). : > log first, or capture only the run's time window.
• set -o pipefail + git diff in a non-git dir exits non-zero and can abort your validation wrapper before it runs — the exact class of bug behind the v3.3.1 forge hook incident (see plugin memory). Guard with require_git / git rev-parse before piping git.
• grep -c counts lines, not matches — two errors on one line count as one. For a pass/fail gate that's fine; for metrics use grep -o … | wc -l.
• Assert BOTH bounds. assert x > 0 is the single most common invariant miss. Range checks need a floor and a ceiling; the ceiling is where the real bug hides.
• Mocks can't validate contracts. If the test mocks the response, it validates your mock, not the server. Contract validation only means something against a real (or realistic integration) response.
• pytest detection depends on .venv/bin/pytest or which pytest in forge_run_tests; a global-but-unlinked pytest, or a differently-named venv, reports "No test runner detected" rather than erroring. Confirm the runner was actually found before trusting a "0 failures".
• Runtime validation flakes on shared/dirty state. Logs and live responses carry state from prior runs; isolate (fresh log, fresh DB/fixture, fresh server) or you'll chase ghosts.

Quick start

1. Run the existing suite green first (npx vitest run, cargo test, or pytest).
2. Re-run the same flow with the server actually started and app.log truncated.
3. grep -Ei 'validation error|panicked|Input should be|Unhandled' logs/app.log — any hit is a bug the green suite missed. Fix root cause; do not suppress the log line.
4. For each externally-facing response, parse it back through its schema at the boundary.
5. For each derived numeric, assert floor and ceiling.

Related skills

• crucible-audit — audits whether the test suite itself catches bugs (hollow assertions, silent failures). Runtime validation observes the running system; CRUCIBLE audits the tests.
• verify-governance — surfaces test-vs-reality mismatches as a governance concern.
• testing-strategy — unit/integration/E2E structure (the layers this skill observes).