Conformance

Why it is done this way

Neither harness interprets a test expectation. The reference implementation’s live response is the oracle, which means a mistake in the extractor shows up as a case both servers answer the same way rather than as a false failure. A harness that scored itself against a suite’s written expectations would be measuring the extractor as much as the server.

Agreement is reported at four strictness levels rather than one, because a single systemic gap — one header never emitted — would otherwise sink every case and hide the hundreds that differ in nothing else.

Both numbers carry their provenance. Each harness builds its own candidate, because which features it was built with is part of what is being measured and cannot be read off the binary, and writes a record of the reference version, the features, the commit, and whether the reference was replayed or a recording reused. The generators that put these figures on the site read that record and refuse a run that cannot account for itself — a run measured with the wrong binary produces a number that looks exactly like a good one.

What is not measured

Both harnesses run one configuration each, and say so. The PostgREST harness measures compatibility mode, so the default mount under /api is not covered by it. The Hasura harness authenticates with the admin secret, as Hasura’s own suite does, so role selection by a bearer token is implemented and unmeasured. Each FINDINGS.md records the gaps a case cannot reach, alongside the faults found in the harness itself.