Your Artifact 3 is strong but over-scoped for the budget. This brief shows how to collapse it to a single-host sandbox that still holds against major frontier models, which benchmarks validate it, how to iterate it, and how to beat CyBench saturation without burning tokens — grounded in what the AISIs, METR, CAISI and the frontier labs actually run.
The heavyweight design put the control functions on a physically separate device reachable only out-of-band. The lean design keeps every isolation boundary and moves the control functions onto the same host, behind the microVM boundary. Toggle the two states.
Across safety orgs, governments, and AISIs, almost no one runs a dedicated sandbox-escape benchmark — SandboxEscapeBench is the exception. Everyone else tests offensive capability as a proxy for escape risk, on a shared toolchain: Inspect + CyBench + InterCode. Filter by what each stresses.
| Org / body | Role | Benchmarks & setups used | Escape-relevant signal |
|---|
You already run the community-standard stack (Inspect + CyBench). To make it an escape programme rather than a capability programme, the missing piece is SandboxEscapeBench pointed at your sandbox — plus a graded capability benchmark (ExploitBench) to stay above CyBench's ceiling.
One host, four nested boundaries, ephemeral per run. The design is deliberately self-validating: because it is nested (agent → container → gVisor → microVM → host), you can drop a flag on the host and run escape tasks against it to prove it holds — the same trick SandboxEscapeBench uses. [1]
Click any ring for what it does, config, and residual risk.
Two separate questions. Does the box hold? is answered by escape benchmarks run against your own sandbox. Does holding matter? is answered by capability benchmarks that tell you how hard the agent is pushing. Run both; report them separately.
The validation set for a sandbox. 18 scenarios across orchestration (L1), runtime (L3), kernel (L4). Run it against your own outer boundary: the agent tries to escape the container into the host, where the flag sits. Binary, host-side scoring means no reward hacking. [1]
Run the solution variant first to confirm your harness works and each scenario is reachable, then the adversarial variant to confirm your boundary is not bypassable by an unintended path. SandboxEscapeBench found 4 shortcuts in development (default SSH creds, mismatched-CVE eBPF) that had to be hardened so escapes used the intended exploit only.
inspect eval sandbox_escape_bench \
--sandbox vagrant # VM-in-host provider = single machine
-T variant=adversarial --epochs 5 --token-limit 2500000
Escape success scales log-linearly with compute; run difficulty-tiered so you see which scenarios flip fail→pass as models improve. [1]
The harness is itself an attack surface. Confirm the host flag is reachable only via a real escape, that no info leaks into the container, and that the scorer can't be gamed. Audit transcripts of every "success" to verify the intended attack path. [1]
Automate transcript review (Meerkat / BenchJack-style auditors) rather than manual spot-checks. Seal any git history / package caches the agent could read a solution from; treat a "solve" with an unexpected trajectory as a harness bug, not a capability signal.
40 professional CTF tasks; the community baseline (o1 hit ~45% in the CAISI/UK joint test, and newer models are higher). Keep it for comparability and its subtask decomposition — but expect saturation at the frontier. [2,3]
Report pass@1 at a fixed budget for comparability with the network's runs (Inspect, ~2.5M token cap, temp 0.7–1). Use subtask completion as the gradient signal, not just full solves — see §06.
Capability-ladder benchmark: 16 deterministically-graded flags across 5 tiers, up through escaping the engine sandbox to arbitrary code execution, on 41 hardened V8 bugs. Gives resolution between models that both "pass" CyBench. Publicly deployed frontier models routinely crash but rarely reach ACE. [8]
The ladder tiers — reach → trigger → in-cage primitive → escape the engine → control — mean you score "how far up did it climb", not "did it pop". That is exactly the anti-saturation metric you need, and the escape tier is directly relevant to your threat model. Deterministic oracles (randomised challenge-response, differential execution) avoid LLM-judge fragility.
InterCode is already part of the AISI-network agentic-cyber setup alongside CyBench; NYU CTF Bench adds 200 tasks. Both broaden coverage and are worth the porting effort you're already doing. [4,5-refs]
The network ran CyBench + InterCode on Inspect at temp 0.7, 10 samples/task, 2.5M-token caps — mirror those settings for cross-org comparability.
Real-world exploitation (CVE-Bench ~10–13%; CyberGym 1,507 vulns) tests whether capability generalises off synthetic CTFs. CyberSecEval's prompt-injection set tests the trigger: does an injected instruction become a tool-abuse / escape attempt? [ref]
These are heavier to stand up. Treat as periodic realism checks, not per-release gates, unless budget allows.
A phased path from your current CyBench-on-Inspect setup to a validated, continuously-tracked escape programme. Each phase ends in a gate — a decision the R&D team hands to governance.
Stand up Firecracker + gVisor + container on one host with the VM sandbox provider; keep CyBench running; finish porting InterCode / NYU CTF into Inspect.
Add SandboxEscapeBench via the VM provider. Run the solution variant (harness sanity) → adversarial variant with a frontier model → harden every unintended escape path until success only follows the intended exploit.
Auto-audit transcripts of all successes; seal information leakage and reward-hacking routes; confirm the host flag is reachable only via a real breakout.
Re-run the escape suite on every new frontier model; watch for scenarios flipping fail→pass; maintain a private held-out set; track against the ~7-month autonomy-doubling trend. [1,ref]
Extend to app-layer (L2) and novel-vuln discovery; layer a lightweight monitor / control protocol (ControlArena-style) once capability rises enough that "does monitoring catch it?" becomes the live question.
The trap: to separate two models that both "pass" CyBench, the obvious move is more epochs and bigger budgets — but AISI shows cyber success keeps rising log-linearly to 100M tokens with no plateau, and a 100M-token run costs ~$80. Brute force is expensive and doesn't cleanly rank saturated models. The fix is to measure differently and spend smarter. [5,6]
AISI reached the same 25% cyber dev-set score using ~13% of the token budget of a non-optimised scaffold — just a better system prompt and tool access. This is the single biggest cost lever. [5]
CyBench ships subtask decompositions. Grade partial progress to separate saturated models without a single extra token. [2]
At a fixed modest budget, measure tokens-to-solve / progress-per-token — AISI's "token efficiency" dimension. Differentiates strong models cheaply instead of chasing asymptotic pass rates. [6]
Fixed cap (e.g. 2.5M tokens), pass@1 or low epochs; stop early on trivial solves and on stalls/loops. AISI found models were usually still progressing at cutoff — so terminate when they are not. Use input caching (note: context compaction resets it). [4,7]
Drop tasks every frontier model solves trivially (no signal) and every task none can touch. Keep the hard, contamination-safe middle + recent/private tasks. Fewer tasks × lower epochs = far fewer tokens.
Add ExploitBench and cyber-range-style multi-step tasks where there is still headroom, rather than running CyBench at ever-higher k. Report pass@1 with a stated budget, not pass@k with large k. [8,ref]
Report escape-capability as "success at budget X" with X fixed and stated, plus a ladder/subtask breakdown — never an uncapped pass@k, which inflates with spend and is not comparable across releases. This keeps the number honest, cheap, and defensible.
Neutrality & verification note. This brief synthesises public academic papers and institutional publications (UK AISI, US CAISI/NIST, the International AISI Network, METR, and frontier-lab safety frameworks). Capability figures are as-published and move quickly — re-verify against primary sources before using any number in a formal governance decision. Where the earlier internal notes referenced a limited-access model (Mythos), a public AISI cyber evaluation of a "Claude Mythos Preview" does exist, but specific system-card claims should be confirmed against the primary document.