Conformance suite

Every backend must pass the same cases: the suite is the oracle that keeps the seven targets identical. Expected JSON and bytes are computed independently of every backend, by hand or by the small generators in conformance/scripts/, never by running a backend.

1. Layout

conformance/
  cases/<area>/schema.yfl      entry schema (it may import other .yfl files in the directory)
  cases/<area>/cases.json      the cases (§2), generated by scripts/areas/<area>.ts
  cases/<area>/yaffle.json     optional: extern implementations per target (§5)
  externs/<target>/            the conformance externs for each target (§5)
  scripts/                     case generators, their byte/JSON library (lib.ts), gen-cases.ts
  src/                         runner, case loader, canonical-JSON comparison, reports, CLI
  test/conformance.test.ts     vitest entry: every case on every available backend
  test/*.test.ts               tests of the runner, the generators and the externs

Each backend provides an adapter at packages/compiler/src/backends/<target>/conformance.ts exporting adapter: ConformanceAdapter. The contract is conformance/src/adapter.ts:

export interface ConformanceAdapter {
  target: string; // "ts", "rust", …
  capabilities?: { views?: boolean }; // views: also run as "<target>:view" (§7)
  available(): Promise<boolean>; // is the toolchain usable here?
  run(input: AdapterRunInput): Promise<CaseResult[]>;
}
export interface AdapterRunInput {
  caseDir: string; // absolute
  program: IrProgram; // the case directory's schema.yfl, compiled by the frontend
  cases: Case[]; // normalized by the runner (below)
  workDir: string; // scratch directory for this target and case directory
  externs: Record<string, string>; // extern name → module for this target (§5)
  mode?: "parse" | "view"; // §7
}
export interface CaseResult {
  name: string;
  ok: boolean; // the adapter's own verdict
  message?: string;
  skipped?: string; // the target can't run this case (unsupported feature): reason
  json?: unknown; // actual canonical JSON (parse cases): always return it
  bytes?: string; // actual output hex (serialize, or the round trip of a parse case)
  error?: { phase; code; path?; offset?; message? }; // the error raised, if any
}

Adapters should be thin. They return json, bytes and error, and the runner re-checks them with its own comparison, so every backend gets the same diffs. bytes may be left out when it would be huge (large files); then only ok counts for the round trip. An adapter that throws an error whose message contains “not implemented” skips the whole case directory for its target; any other throw fails every case of the directory.

What adapters receive (Case, normalized by the runner): bytes is canonical hex (no whitespace) or { file: "<absolute path>" }; slices are already cut into a scratch file; expect and description are removed.

2. cases.json

{
  "description": "…",                   // optional
  "generatedBy": "conformance/scripts/areas/arrays.ts",   // optional
  "cases": [
    {
      "name": "basic",
      "root": "Table",                  // exported struct
      "args": { "version": 144 },       // root parameters (canonical JSON), optional
      "bytes": "0300000001020304",      // hex (spaces and _ allowed), or { "file": "input.bin" }
      "json": { "count": 3, "entries": [ … ] },   // expected parse result (canonical JSON)
      "roundtrip": true,                // default true (see below)
      "strict": false                   // parse in strict mode
    },
    {
      "name": "bad magic",
      "root": "Table",
      "bytes": "ffff",
      "error": { "phase": "parse", "code": "CHECK", "path": ["magic"] }
    },
    {
      "name": "write derives count",
      "root": "Table",
      "input": { "entries": [ … ] },    // serialize-only case: input JSON (count omitted)
      "bytes": "…"                      // expected output
    },
    {
      "name": "large sample",
      "root": "Archive",
      "bytes": { "file": "sample.bin", "offset": 768, "length": 4160320 },
      "expect": [ { "pointer": "/count", "equals": 6 }, { "pointer": "/names", "length": 598 } ]
    }
  ]
}

• Parse cases (bytes + json and/or expect): toJson(parse(bytes)) must deep-equal json (§3), and every expect check must hold on it.
• Round trips of parse cases:
  • true / "json" (the default): serialize(fromJson(json)) must equal bytes, where json is the actual canonical JSON (equal to the expected one when the case has it);
  • "value": serialize(parse(bytes)) must equal bytes in the same process, so codec reuse applies (IR.md §3.6). Used for codecs whose re-encoding differs from the original bytes;
  • false: no round trip (non-canonical input, e.g. non-zero padding or a NaN payload).
• Serialize cases (input + bytes): serialize(fromJson(input)) must equal bytes.
• Error cases: the phase and code must match, and the path too if given.
• bytes: hex, or { "file", "offset"?, "length"? }. A file is relative to the case directory. offset/length cut a slice.
• expect (runner-side checks on the actual JSON, for files too big to spell out): a list of { "pointer": "<JSON Pointer>", "equals"?: <canonical JSON>, "length"?: n, "byteLength"?: n, "startsWith"?: "<prefix>" }. length is an array’s or string’s length, byteLength a hex string’s byte count.

The case loader validates every cases.json (unknown keys, duplicate names, bad hex, missing files, error codes, mutually exclusive fields); an invalid directory fails all its cases. cases.json is never edited by hand: change the generator and run node conformance/scripts/gen-cases.ts.

Rules the cases pin beyond §3 and §4:

• Bytes after the root struct are ignored by parse.
• Computed fields are not checked on read (a wrong crc32 or size parses; the rebuild recomputes it), except constants.
• A count larger than the remaining data is EOF, whatever its magnitude.
• @ref resolves by identity first (the input object itself, e.g. restored by $ref), then by equal encoding (the first placed value wins).
• On write, fields of an untaken conditional branch are ignored; a missing field of the taken branch is INPUT.

Known gaps:

• The suite has no negative compile cases (diagnostics are covered by the frontend’s own tests) and no surgical-write or async-source cases (view mode covers reads and no-op commits).

3. Canonical JSON

This is the JSON form of values, used by toJson/fromJson in every backend and by the cases.

Floats: all NaNs read as "NaN"; writing "NaN" gives the canonical quiet NaN (f16 7e00, f32 7fc00000, f64 7ff8000000000000). A number written to a narrower float rounds to nearest, ties to even. Key order is significant: the runner reports fields out of declaration order.

fromJson accepts this form. Fields that are absent from the input type are ignored if present, so fromJson(toJson(x)) works. $ref restores object identity; a dangling $ref is an INPUT error.

Root arguments (args): explicit parameters by name, bound parameters without $, and endian as "le" or "be" (default "le").

4. Errors

Error objects expose code, path, offset (absolute, or -1 on write) and message. path lists field names and array indices from the root to the innermost field (or element) being read or written when the error happened. Blocks, switches and bits add no segment (their fields are flat); pointers are transparent. Examples: ["entries", 2, "x"], ["ifd", "count"]. A case leaves the path out where it is not determined (e.g. DERIVE between two fields, reserved bits, a switch without a match).

5. Externs in cases

A case directory whose schema declares externs has a yaffle.json mapping each extern to a host implementation per target, with paths relative to that yaffle.json:

{
  "sources": ["schema.yfl"],
  "targets": {
    "ts": { "out": "…", "externs": { "rle": "../../externs/ts/rle.ts" } },
    "rust": { "out": "…", "externs": { "rle": "../../externs/rust/rle.rs#RLE" } }
  }
}

The runner resolves them and passes the adapter its own target’s map (AdapterRunInput.externs): file paths (starting with ./, ../ or /) become absolute and keep an optional #ITEM suffix naming the item in that file; anything else (e.g. a Rust path like crate::hash::gt_hash) is passed through unchanged. A mapped file that doesn’t exist makes the directory invalid. The adapter makes generated code use them; each file in conformance/externs/<target>/ documents its target’s host shape.

TS: the module exports a value named exactly like the extern, in the shapes of IR.md §5 (export const rle = { decode, encode }, export function sum8(…), export const VarInt = { size, read, write }). Codec arguments arrive in ctx.args (u8 → number, u64 → bigint, u8[n] → Uint8Array).

Rust (the host interface of packages/runtime-rust): the file becomes a module of the generated crate and can use yaffle_runtime. The item is the extern’s name, or the #ITEM of the mapping. Dependencies other than the runtime are listed in yaffle.json under targets.rust.dependencies.

conformance/externs/rust/stub/yaffle_runtime.rs mirrors that interface so the extern files can be unit-tested without the runtime (conformance/scripts/test-rust-externs.sh).

Errors returned by a codec are CODEC errors.

The conformance externs (conformance/externs/<target>/, one implementation per target):

6. Running

The CLI’s --filter and --targets set the first two.

7. View mode

Adapters that declare capabilities: { views: true } are also run as a second target named <target>:view (for example ts:view). In view mode the runner sends the same case directories with mode: "view" and skips serialize-only cases. For every other case, the adapter must:

1. open the target’s lazy view over the case bytes (the bytes, or a random-access source over the file for { file } cases);
2. produce json by reading every field through the view (materialize, then canonical JSON), not by calling parse;
3. produce bytes by committing the view with no edits, which must reproduce the input exactly (the round-trip check), unless roundtrip is false;
4. report errors raised while reading through the view (same code and path as parse mode).

The runner judges view results exactly like parse results. View mode exists because views have their own code paths (lazy offsets, decoded regions, streams) that parse-mode cases never exercise.