Rulebearing Playground Coverage Schemas GitHub

ADR-0075: The coverage tabs' rows are kept in a tracked ledger with their evidence, and each row's status is computed from the gates' results

Context

ADR-0009 makes the two coverage tabs the ledger the conformance gates report against: a row may not say Parity until the pinned suite says so. Until now the proof of each row was recorded by hand in the plans, row by row with the test that proves it (plan 0002's ledger of 91 rows, plan 0003's of 13). Plan 0004 asks for that to be live: the nightly writes each row's status from the gates' results, and the docs site renders the tables, so a row whose evidence is missing shows as unproven whatever the tab says.

Four facts shape the design:

Decision

A tracked ledger per tab. conformance/ledger/dependency-cruiser-18.2.0.json and conformance/ledger/archunitnet-0.13.4.json hold, per ## section of the tab, its column headers and each row's cells exactly as exported, the row's claim (the cell under the tab's status column: Rulebearing in the dependency-cruiser tab, Status in the ArchUnitNET tab; ArchUnitNET's selector table has no status column, and its lead defines every listed method as Parity, so a method row's claim is Parity and the (none) row's is Addition), the sha256 of the tab's text, and the row's evidence. The ledger has no status field.

Command What it does
coverage_status.py rows Derives every row from the local tabs and keeps each row's evidence, keyed by section and first cell. --check exits 1 when a ledger differs from its tab. Run on the maintainer's machine, where the tabs are
coverage_status.py check Every evidence reference names something the tree holds: a known results name, a ported case file, a crate, a test target and a function of that name. The pytest suite runs it over the committed ledgers on every pull request
coverage_status.py tests Runs each Rust test target the ledger names, with every feature, and records each test's verdict from libtest's own lines (test <path> ... ok)
coverage_status.py jobs Records a CI run's job conclusions from gh run view --json jobs
coverage_status.py write Computes each row's status and writes site/data/coverage.json

One results shape. Every producer writes target/conformance/<name>.json as { results, ok, passed, total, items: { <item>: "pass" | "fail" } }, keys sorted:

Results Written by Items
layer1 extract_fixtures.rs each replayed test/extract case
layer2 run-layer-2.mjs (a full run in gate mode) each test/validate and test/graph-utl spec
layer3 run-layer-3.mjs (a full run) each test/report spec, and oracle:<output type> for each reporter compared with upstream's (compared at least once, never differing)
layer5 read from run-layer-5.sh's folders each oracle repository whose zeroDiff is yes; mutations:mutations and mutations:affected, written only when both comparisons held
gate2-archunitnet, gate2-netarchtest gate2_common::report each ported/<file>.yaml with at least one case and none differing
tests coverage_status.py tests <crate>/<target>::<test path>, and <crate>/<target> when the whole target passed
jobs coverage_status.py jobs each job of the run, with the run's id
unported-archunitnet, unported-netarchtest read from the tracked unported.json each reason with an entry (the proof of a row that stays upstream)

The status rule. A row is proven when it names evidence and every piece holds: the results file exists and records the item as pass, or, for a whole results file, records every item pass and at least one. Anything else is unproven, a missing file included. The site shows the tab's claim only for a proven row; an unproven row shows "unproven", with the tab's text beside it for reference. Each piece of evidence links to its source (the upstream spec at the pinned tag, the case file, the test file) and, when it holds, to the run that proved it.

The nightly writes it, by pull request. A conformance job in nightly-testbeds.yml runs gate 1's layers 1 to 4 (run.sh), gate 2, and coverage_status.py tests, and uploads target/conformance/*.json. The summary job adds layer 5's folders (the existing layer5 artifact) and the job conclusions of the latest ci.yml run on main, writes site/data/coverage.json with the night's run id, and opens or updates a pull request from testbeds/coverage-tables, as the scale table's step does from testbeds/scale-table. main takes the file only through review, so a status can change only through a reviewed diff of a file the script wrote.

The site renders the data. site/src/tables.ts renders coverage/index.html, coverage/dependency-cruiser.html, coverage/archunitnet.html and coverage/scale.html from site/data/coverage.json and testbeds/results/scale.json at build time; docs.yml rebuilds when either changes.

Evidence beyond the upstream suites is labelled. A piece of evidence from gate 1 or gate 2 is shown as the upstream spec or case file; a Rust test is shown as <crate>/<target>: <test>. A row proven only by Rulebearing's tests is still proven, as plans 0002 and 0003 accepted, and the reader can see that it is.

Consequences

Alternatives considered

Option Why not
Read the tabs in CI They are git-ignored (ADR-0039), so CI and the docs build have no copy.
Keep the row list inside the plans Also git-ignored, and the plans record a row's proof once, at a date; they cannot be recomputed.
A status column in the ledger, set by the nightly It would be a second place a status can be typed. Computing it at write time leaves nothing to edit.
Parse the gates' console output Brittle against wording changes, and the gates did not print per-spec verdicts. A results file per gate costs a few lines each.
libtest's JSON output (--format json) Unstable: it needs -Z unstable-options, which stable Rust rejects. libtest's pretty lines are stable and enough.
Download the gates' results from the last ci.yml run on main The night would depend on the last push having passed every job, and CI's test job records no per-test verdict. Rerunning the gates gives one run id for everything they prove. CI-job evidence is used only for the two rows a nightly runner cannot build (the Node addon on three platforms, the .NET adapters).
Accept only upstream evidence About a third of the rows have no upstream case (additions, the .NET and Python halves, options upstream tests only through a transpiler). They would show as unproven forever, which says nothing true about them.