Environments an agent cannot bluff its way through.
Coding, long-horizon agentic work, computer use, terminal, tool and API use, browser tasks — delivered as Docker episodes behind a Gymnasium-compatible env.py.
Production-grade business software as the substrate, some of it on Stackhouse, our own open-source backend. Grading is state-based and adversarial by design: a no-op earns nothing, and we demonstrate that before delivery.
We own every layer under the reward signal.
Application, database, task authoring, verifier. Vendors who assemble environments from someone else’s software cannot make the guarantees below, and cannot fix a grader that leaks.
Real business software, not fixtures — with the schema depth, referential constraints and seeded history an agent can genuinely damage.
A calibrated spread of difficulty and workflow class, each task carrying a proven gold solution and an equivalence-tested alternate.
State-based, multi-assertion, nonce-defended and canary-hashed — held outside the container and proven to fail closed.
Our open-source Rust backend, the substrate under a number of environments — what makes deterministic reset and untouched-collection hashing possible.
The market is split between benchmarks too shallow to be informative and benchmarks too soft to be trusted.
The shallow end is a CRUD demo the model has already memorized. The soft end is more expensive, because it looks rigorous while it inflates your numbers — seed data that already satisfies the criteria, expected values that never move between episodes, graders that reveal their answer to anything that polls them. Teams discover this after the training run, not before it. We build for the space in between: deep enough that competence is required, graded strictly enough that competence is the only thing that pays.
Our build standard
Five stages. Nothing advances on assertion — each is proven against a live container before the next begins.
Every task ships with its attack surface documented
Not a policy statement. Per task, in writing: the exploit, the assertion that closes it, and the transcript showing the exploit earning nothing. This is the document your eval team asks for on the second call.
law_firm_software — the same QC discipline, a second vertical
Two shipped environments now run the same pipeline. This one is a Stackhouse-backed law-firm practice-management app — matters, clients, billing & trust accounting, e-discovery holds and productions, grant management — and it's where the QC battery grew two gates, where the Phase 4 hacker-fixer loop ran task-by-task for the first time in any of our shipped suites, and where every task carries a genuine dependent-stage chain.
The first 6 are the same baseline/gold/idempotency gates the vet-clinic-ops methodology runs. Two are new here: an independently-authored second gold solution, and a budget-fit check against the measured call count of the winning run.
| # | Gate | Expected |
|---|---|---|
| 1 | No-op baseline (verifier on pristine state) | FAIL |
| 2 | 10-random-action baseline (real API traffic) | FAIL |
| 3 | Lazy-hardcode baseline (plausible static answer, stale nonce) | FAIL |
| 4 | gold.py + verify | PASS |
| 5 | Verify re-run on unchanged state (idempotent) | PASS |
| 6 | Second gold run + verify (re-run idempotency) | PASS |
| 7 | gold_alt.py — independent second solution | PASS |
| 8 | max_steps budget fit (measured calls ≤ budget) | PASS |
Result: 26 / 26 tasks pass all 8 gates.
Adversarial layer — Phase 4, dependency-break attacks, dual-path verification
Run individually against every task, live, against the real image: a canary/scope-violation attack (mutate a collection outside the task's declared blast radius after a correct gold run → must FAIL), a partial-completion attack (kill the gold script mid-run → must FAIL), a retry/flake attack (verifier ×5 on identical state → 5/5 identical PASS), and metadata-inference/evaluation-tampering checks (failure detail capped to one reason; grading files root-owned chmod 400 in-image; the agent's SQL role confirmed rolsuper = false live, closing pg_read_file / COPY ... TO PROGRAM as ways to read the verifier off disk).
Unique to this suite. For every task, a naive gold script that gets stage 1 right but breaks the specific link into stage 2 — a stale local copy, a field-confusion swap, a wrong threshold or comparison direction — was run live and confirmed to FAIL with a plausible wrong value, not a crash. 25 / 25 tasks, plus task 026's two hazard-targeted variants.
Every aggregate a verifier checks is computed two independent ways — a raw-row Python filter and a separate SQL query — and must agree before it's ever compared to the agent's own report.
- task 004's exact-set-equality check wrongly rejected correct solutions against intentionally-messy seed data
- task 007's unbounded blast radius measured 7,881 API calls before being rescoped
- a suite-wide canary audit found 19 tasks whose out-of-scope hashing missed read-source collections — 024 and 025 initially failed the canary attack, fixed and re-verified through the full battery
A single-container episode can't run a real background clock, so multi-period horizon is simulated by seeding ~13 months of dated history up front, and grading whether the agent handles the interior story correctly — not just the final reconciliation. Every one of the 26 tasks carries a dependent stage chain: stage 2 reads state stage 1 actually mutated or wrote, never an independent recompute from the seed. A wrong or skipped stage 1 doesn't error — it silently produces a plausible wrong stage-2 number, which is exactly what each dependency-break attack above proves per task.
True multi-session checkpointing is declared future work, not faked with a bigger step budget. Policies like TRUST-REC-01 and BILL-GEN-01 live as data in firm_policies / firm_controls collections the agent has to discover and read — the task instruction names the policy and the report contract, never the formula or the stage dependency. The v1 task instructions leaked the derivation recipe directly; they were rewritten for exactly that reason.
- 0Capture opening control totals and the close's billing candidates.
- 1Trust reconciliation per firm policy TRUST-REC-01 — correct every client's stored trust balance to its sub-ledger value.
- 2Status-blind, boundary-delimited trust-applied backfill across every invoice, regardless of status.
- 3Bill policy BILL-GEN-01's top-N candidates and flip exactly the entries billed.
- 4Point-in-time AR aging reconstructed at each of the 12 calendar month-ends preceding the episode month — an invoice that didn't exist at a month-end must not appear in that month-end's buckets (~2,600 of 3,000 invoices in-window).
- 5The close-pack rollup.
- 6Tie-out proving the WIP drop equals the billed value and the AR delta includes the new invoices.
The rollup numbers are only correct if every earlier stage was.
Horizon design corrections caught during QC
- task 002's initial stage-2 threshold was always false against real data — replaced with a genuinely-varying exposure ratio
- task 018's planned violation flag was vacuous (every real collection was clean) — replaced with a span-based ranking
- tasks 023/024's tie-break relied on JSONB key order Postgres doesn't preserve — fixed in code and in instruction wording
Chains that can't produce a wrong answer aren't chains — this suite's QC treats a vacuous dependency as a bug, not a pass.
par_steps is the measured call count of the gold solution's final passing run; max_steps is ceil(par_steps × 1.5) (floor 20) — the budget-fit gate above means no task ships with a budget a real agent can't fit inside. The median task is ~115 calls; the long pole is the 1,987-call time-entry-to-invoice conversion. Tasks 004, 013 and 025 sit in the 250–400 call band.
| Task | Horizon tier | Measured (gold) | max_steps |
|---|---|---|---|
| 007_time_entry_invoice_conversion | 2 | 1987 | 3044 |
| 001_trust_ledger_reconciliation | 2 | 1012 | 1547 |
| 015_client_lifetime_value_ar_aging_report | 2 | 527 | 794 |
| 021_invoice_aging_dispute_risk_report | 2 | 512 | 670 |
| 026_firmwide_month_end_close_pack | 6 | — | — |
007's original design — invoice every matter with approved time — measured 7,881 calls at 50k-row scale and was deliberately rescoped to a top-15 slice found by full-firm ranking, because an unbounded blast radius isn't a reasonable single-episode mission. Scale is engineered here, not just endured.
Difficulty here is author-asserted, not measured — no calibration run (frontier pool × trials × per-task pass rate) has been executed against this suite yet, and there are no model eval scores for law_firm_software. The QC evidence above proves the grading is honest; it does not prove the tasks are hard, and we're not claiming otherwise.
Breadth, held privately
Our library spans the domains below — each a working application with its own schema, constraints and seeded history. The veterinary clinic and law-firm environments are our public references, both with full QC methodology and task design shown on this page. Inventory, roadmap and commissioned work beyond those two are disclosed under NDA, on a call.
v2 is hard enough to separate every model tested
8 live eval runs, 7 models, on the v2 (hardmode) vet-clinic-ops environment ↗ — 25 tasks per run.
| Model | Reward | Size | Source |
|---|---|---|---|
| gpt-oss-120b | 0.000 | 120B total · 5.1B active | OpenAI open-weight release |
| deepseek-v4-flash | 0.120 | 284B total · 13B active | DeepSeek open-weight release |
| glm-5.3-flash | 0.800 | 320B total · 18B active | Zhipu open-weight release, MIT |
| grok-4.6 | 0.800 | 1.5T total · dense | third-party estimates, not published by xAI |
| qwen3.8-max | 0.080 | 2.4T total · 95B active | Alibaba's own announcement |
The spread is now 0.000 to 1.000 — v1's tightly clustered 0.760–0.980 band is gone, and v2 actually discriminates capability instead of bunching every model near the ceiling. Parameter count still doesn't predict score inside that spread: grok-4.6 (1.5T, confirmed largest) and glm-5.3-flash (320B) tie at 0.800 despite a ~4.7x size gap, while qwen3.8-max (2.4T, the second-largest model on the board) scores lower than deepseek-v4-flash (284B) — 0.080 vs 0.120 — despite being ~8.5x larger.
Cost tracked even less with score: grok-4.6's run cost $36.53 against glm-5.3-flash's $3.21 for an identical 0.800.
gpt-5.6-sol: closest available successor to the model formerly listed as gpt-5.6-luna — not the same model id. gemini-3.8-flash: two independent runs were executed (0.040, then 0.000 on rerun); the rerun is shown here as canonical.
v1 results (100-task suite, since superseded)
Parameter count doesn't predict the score
Nine live eval runs, eight models, on our vet-clinic-ops environment ↗. Only five models have a parameter count on the public record — the rest is undisclosed by every lab, not just estimated here. Plotted against those five, reward doesn't climb with scale.
| Model | Reward | Size | Source |
|---|---|---|---|
| gpt-oss-120b | 0.760 | 120B total · 5.1B active | OpenAI open-weight release |
| deepseek-v4-flash | 0.760 | 284B total · 13B active | DeepSeek open-weight release |
| glm-5.3-flash | 0.880 | 320B total · 18B active | Zhipu open-weight release, MIT |
| grok-4.6 | 0.980 | 1.5T total · dense | third-party estimates, not published by xAI |
| qwen3.8-max | 0.800 | 2.4T total · 95B active | Alibaba's own announcement |
The largest confirmed model on the board — Qwen3.8-Max at 2.4T total parameters — scores 0.800, below the 320B glm-5.3-flash at 0.880. The best score among size-disclosed models is grok-4.6 at an estimated 1.5T (0.980 on the full 100-example run), a figure xAI has not itself confirmed. And the two highest scores overall — gpt-5.6-luna at 1.000 and gemini-3.7-flash at 0.960 — belong to models neither lab has disclosed a size for at all.
A number of environments run on Stackhouse
Our open-source backend-as-a-service in Rust — Axum, Postgres, Qdrant — covering auth, document storage and query, vector search, realtime, serverless functions, and billing. Where a vertical sits on it, we own the substrate, which is how an episode resets deterministically and how we hash a collection the agent wasn’t supposed to touch. Stackhouse is a distinct project with its own site and audience.
What arrives in your training loop
License from the library, or commission against your own taxonomy and failure modes. Either way, delivery is a package your team can drop into an existing harness the same week.
One container per episode. Deterministic reset in seconds, no shared state between rollouts.
Gymnasium-compatible and OpenEnv-shaped — into your rollout harness without a wrapper.
Difficulty-labelled across the workflow taxonomy, each task with a gold solution and an equivalence-tested alternate.
State-based, idempotent, held outside the container, proven to fail closed.
Baseline runs, determinism proof, and the attack runs that earned nothing.
Per-task documentation of the reward-hacking surface and the assertion closing each hole.