The ByJTT Gauntlet

What it is

The ByJTT Gauntlet is the critical review framework that emerged from the first experiments and now defines the review policy for new work. It is Experiment 004 — the meta-experiment: adopted, not built; it is the process itself. Under the current framework, a RELEASE PASS requires deterministic evidence to be green and every required critic to independently score ≥ 7/10.

The gate

• Evidence-first. No review from memory or description. Deterministic checks (build, tests, lint, attribution) run before any model — an objective failure overrides every model score.
• Free-first. Reviewers are resolved from a cost-aware catalog: the cheapest cross-family pair that satisfies the level. Paid escalation requires an explicit justification.
• Fail-closed. Four outcomes — PASS, REVISE, FAIL, BLOCKED. Anything other than PASS is non-zero. A reviewer that cannot run is BLOCKED, never a quality verdict.

The mechanics

1. Deterministic evidence first. The harness runs objective checks against the build (syntax, tests, attribution, licenses, mobile layout, self-containment) and records the pass count (e.g. 17/17, 10/10). These are not opinions; they are checks a machine can run. A failure here short-circuits the gate and overrides all model scores.
2. Three gate levels. ROUTINE (evidence only, zero model calls) for trivial changes; SIGNIFICANT (evidence + one independent different-family reviewer) for features and non-ship decisions; RELEASE (evidence + two independent different-family critics with an adversary mandate) for anything that ships.
3. Free-first model resolution. Reviewers come from a maintained cost tier catalog (free → cheap → mid → premium). The harness picks the cheapest cross-family pair that satisfies the level, checks runtime availability, and records BLOCKED when a reviewer is unavailable. Paid escalation requires --escalate plus a documented reason.
4. Blind, diverse critics. Reviewers see the artifact and the rubric — never builder claims or session logs. The two RELEASE critics must come from different model families (enforced by family-divergence logic, not an allowlist). Builder-family reviewers are flagged as degraded; unknown builder families are never assumed.
5. Deterministic rubric. Each critic scores ten dimensions out of 10: correctness, fidelity, twist, ux, mobile, accessibility, performance, security, docs, originality. The gate averages numerators only within each individual review (no denominator inflation).
6. Every required critic ≥ 7/10. PASS requires green evidence and each reviewer required by the selected level independently scoring at least 7/10. One strong review cannot average away a weaker required review. A quality gap is REVISE.
7. Delegation failures are BLOCKED. Zero tokens, timeout, empty output, missing model, quota — the reviewer did not run, and that is never described as a successful review. It is archived and the gate exits non-zero.
8. Full transparency. Every run archives critiques/<slug>.critique.{json,md}: models, families, scores, verdicts, evidence steps, raw outputs, delegation errors. The process is auditable and backed by a no-model node:test suite.
9. Self-improving. The framework is version-controlled, open-sourced (MIT), and updated from lessons learned — the rubric is not sacred, the evidence is.

Why it exists

The first experiments taught us that self-review is unreliable: a builder reviewing its own work argues itself into passing. The Gauntlet makes the review blind, cross-family, and deterministic — the same insight as the Gauntlet Loop pattern from agentic development, applied to our own pipeline. The critic is separate from the builder, the reference is real, and the human stays the brake.

Status

Version 2.1.1 is the current framework definition. Its deterministic self-test suite is 34/34 PASS on the recovered source, including the RELEASE-review regression tests. The framework itself is not yet RELEASE-certified: the final live two-family RELEASE re-review remains BLOCKED until the required independent reviewer infrastructure is available. Earlier timeline experiments also predate parts of the current v2.1.1 gate semantics, so their historical review scores should not be read as proof that they passed today's RELEASE policy.

The framework is open-sourced in the byjtt-experiments repo — copy framework/critic.sh + models.json + PROTOCOL.md and run the gate on your own results. See the lessons learned for how it has already changed.