Environments an agent cannot bluff its way through.
Coding, long-horizon agentic work, computer use, terminal, tool and API use, browser tasks — delivered as Docker episodes behind a Gymnasium-compatible env.py.
Production-grade business software as the substrate, some of it on Stackhouse, our own open-source backend. Grading is state-based and adversarial by design: a no-op earns nothing, and we demonstrate that before delivery.
We own every layer under the reward signal.
Application, database, task authoring, verifier. Vendors who assemble environments from someone else’s software cannot make the guarantees below, and cannot fix a grader that leaks.
Real business software, not fixtures — with the schema depth, referential constraints and seeded history an agent can genuinely damage.
A calibrated spread of difficulty and workflow class, each task carrying a proven gold solution and an equivalence-tested alternate.
State-based, multi-assertion, nonce-defended and canary-hashed — held outside the container and proven to fail closed.
Our open-source Rust backend, the substrate under a number of environments — what makes deterministic reset and untouched-collection hashing possible.
The market is split between benchmarks too shallow to be informative and benchmarks too soft to be trusted.
The shallow end is a CRUD demo the model has already memorized. The soft end is more expensive, because it looks rigorous while it inflates your numbers — seed data that already satisfies the criteria, expected values that never move between episodes, graders that reveal their answer to anything that polls them. Teams discover this after the training run, not before it. We build for the space in between: deep enough that competence is required, graded strictly enough that competence is the only thing that pays.
Our build standard
Five stages. Nothing advances on assertion — each is proven against a live container before the next begins.
Every task ships with its attack surface documented
Not a policy statement. Per task, in writing: the exploit, the assertion that closes it, and the transcript showing the exploit earning nothing. This is the document your eval team asks for on the second call.
Breadth, held privately
Our library spans the domains below — each a working application with its own schema, constraints and seeded history. The veterinary clinic environment is our public reference. Inventory, roadmap and commissioned work are disclosed under NDA, on a call.
v2 is hard enough to separate every model tested
8 live eval runs, 7 models, on the v2 (hardmode) vet-clinic-ops environment ↗ — 25 tasks per run.
| Model | Reward | Size | Source |
|---|---|---|---|
| gpt-oss-120b | 0.000 | 120B total · 5.1B active | OpenAI open-weight release |
| deepseek-v4-flash | 0.120 | 284B total · 13B active | DeepSeek open-weight release |
| glm-5.3-flash | 0.800 | 320B total · 18B active | Zhipu open-weight release, MIT |
| grok-4.6 | 0.800 | 1.5T total · dense | third-party estimates, not published by xAI |
| qwen3.8-max | 0.080 | 2.4T total · 95B active | Alibaba's own announcement |
The spread is now 0.000 to 1.000 — v1's tightly clustered 0.760–0.980 band is gone, and v2 actually discriminates capability instead of bunching every model near the ceiling. Parameter count still doesn't predict score inside that spread: grok-4.6 (1.5T, confirmed largest) and glm-5.3-flash (320B) tie at 0.800 despite a ~4.7x size gap, while qwen3.8-max (2.4T, the second-largest model on the board) scores lower than deepseek-v4-flash (284B) — 0.080 vs 0.120 — despite being ~8.5x larger.
Cost tracked even less with score: grok-4.6's run cost $36.53 against glm-5.3-flash's $3.21 for an identical 0.800.
gpt-5.6-sol: closest available successor to the model formerly listed as gpt-5.6-luna — not the same model id. gemini-3.8-flash: two independent runs were executed (0.040, then 0.000 on rerun); the rerun is shown here as canonical.
v1 results (100-task suite, since superseded)
Parameter count doesn't predict the score
Nine live eval runs, eight models, on our vet-clinic-ops environment ↗. Only five models have a parameter count on the public record — the rest is undisclosed by every lab, not just estimated here. Plotted against those five, reward doesn't climb with scale.
| Model | Reward | Size | Source |
|---|---|---|---|
| gpt-oss-120b | 0.760 | 120B total · 5.1B active | OpenAI open-weight release |
| deepseek-v4-flash | 0.760 | 284B total · 13B active | DeepSeek open-weight release |
| glm-5.3-flash | 0.880 | 320B total · 18B active | Zhipu open-weight release, MIT |
| grok-4.6 | 0.980 | 1.5T total · dense | third-party estimates, not published by xAI |
| qwen3.8-max | 0.800 | 2.4T total · 95B active | Alibaba's own announcement |
The largest confirmed model on the board — Qwen3.8-Max at 2.4T total parameters — scores 0.800, below the 320B glm-5.3-flash at 0.880. The best score among size-disclosed models is grok-4.6 at an estimated 1.5T (0.980 on the full 100-example run), a figure xAI has not itself confirmed. And the two highest scores overall — gpt-5.6-luna at 1.000 and gemini-3.7-flash at 0.960 — belong to models neither lab has disclosed a size for at all.
A number of environments run on Stackhouse
Our open-source backend-as-a-service in Rust — Axum, Postgres, Qdrant — covering auth, document storage and query, vector search, realtime, serverless functions, and billing. Where a vertical sits on it, we own the substrate, which is how an episode resets deterministically and how we hash a collection the agent wasn’t supposed to touch. Stackhouse is a distinct project with its own site and audience.
What arrives in your training loop
License from the library, or commission against your own taxonomy and failure modes. Either way, delivery is a package your team can drop into an existing harness the same week.
One container per episode. Deterministic reset in seconds, no shared state between rollouts.
Gymnasium-compatible and OpenEnv-shaped — into your rollout harness without a wrapper.
Difficulty-labelled across the workflow taxonomy, each task with a gold solution and an equivalence-tested alternate.
State-based, idempotent, held outside the container, proven to fail closed.
Baseline runs, determinism proof, and the attack runs that earned nothing.
Per-task documentation of the reward-hacking surface and the assertion closing each hole.