Hasura conformance
How closely the GraphQL dialects agree, measured by replaying Hasura’s own test corpus against both servers and diffing the live responses.
| Compared on | All cases468 cases | Reads (query)271 cases | Writes (mutation)197 cases |
|---|---|---|---|
| HTTP status | 100%468/468 | 100%271/271 | 100%197/197 |
| …and the same outcome | 99.6%466/468 | 99.6%270/271 | 99.5%196/197 |
| …and the same data | 97.4%456/468 | 95.9%260/271 | 99.5%196/197 |
| Whole body, wording included | 96.6%452/468 | 95.6%259/271 | 98%193/197 |
Measured 2026-08-29 against hasura/graphql-engine:v2.50.1, on Postrust built with admin-ui,compat-key-order, over 468 cases in 59 groups. Every figure is recomputed from the run’s per-case output; nothing on this page is typed by hand.
What is compared
Agreement is reported at four strictness levels, because one systemic gap would otherwise sink every case and hide the hundreds that differ in nothing else.
The third is the one that matters, and the one that needs explaining. Two servers agree about data when they return the same rows — and they also agree when both refuse. Counting only the first would score a case where Hasura itself raises an error as a failure of this server to match it, which is backwards. Of the 456 cases that level counts, 325 agree about data and 131 agree because both servers refuse.
The strictest level compares the whole body, errors included — wording and all. Bodies are compared as parsed JSON.
How it is measured
The harness extracts cases from Hasura’s own test corpus and replays each one over HTTP against both hasura/graphql-engine and Postrust, on identically loaded fixture databases, then diffs the live responses. Hasura is the oracle: no expectation written in the corpus is ever interpreted, so a mistake in the extractor shows up as a case both servers answer the same way rather than as a false failure.
What Hasura keeps in metadata rather than in the database — relationship names, what each role may do — is converted from each group’s own metadata and given to the candidate. This measures a configured server, which is what migrating involves, not a bare one.
scripts/hasura-conformance/conformance.sh
node scripts/gen-hasura-conformance.mjs scripts/hasura-conformance/.work/diff.jsonThe harness builds its own candidate and records what it built beside the results. The generator reads that record rather than its own arguments and refuses to publish a run that cannot account for itself — a run measured with the wrong binary produces a number that looks exactly like a good one.
Where the two disagree on purpose
Some cases fail because neither answer is wrong, and some because matching Hasura would mean matching something worth not matching. They are listed so nobody later “fixes” one without deciding to.
An unsecured server trusts no header
Hasura with no admin secret treats every caller as an administrator, which also lets any caller name its own role and its own identity. Here, with no secret configured, x-hasura-* headers carry no weight and session variables come only from a verified token. A policy reading a value the caller chose is not a policy, and the failure is silent — the query succeeds, against the wrong rows. It costs nothing measured: every case in the corpus that names a role sends the secret beside it.
Relationship names are derived, not configured
Every relationship in the corpus is named by a metadata command a human wrote; here they come from foreign keys. Where a fixture chose something other than the convention, the field is not there under that name. This is structural to reflecting a database instead of configuring one.
Two databases naming different constraints
pg_dump restores constraints in a different order than the fixture created them, so PostgreSQL reports a different constraint name for the same violation. Two cases, proven against one PostgreSQL — neither server is answering wrongly.
Introspection belongs to an administrator
The corpus expects __schema to be refused for a role that has it disabled. A v2.50.1 reference does not refuse it: asked with the admin secret beside X-Hasura-Role, it answers from that role's own restricted schema, so the permissions apply and the introspection rule does not. This server was built to the corpus's text first and measured against the reference second, which is the right way round to find that.
Where the remaining disagreement lives
By corpus group, on the same-data level, worst first. Groups that agree completely are not listed.
| Group | Agreement |
|---|---|
graphql_query/computed_fields | 90.9% (10/11) |
Known gaps
The largest is introspection, and it is not reachable from here: async-graphql builds its own registry and keeps it private, so the directives it installs, the types it adds, and the order it lists them in cannot be changed from outside the library. Every large introspection case needs at least one of those.
Beside it: _stream subscriptions, the cursor-based half of the subscription surface; a function returning a single row as a root field; and generated descriptions, which are this server’s wording rather than Hasura’s. Actions and Apollo federation are subsystems rather than gaps.
The run history — including the runs whose numbers are not publishable, and why — is in scripts/hasura-conformance/FINDINGS.md, which also records four faults found in the harness itself. One of them invalidated eleven runs.