This plan specifies every component of the single-host sandbox, the libraries and tools each requires, its dependency on UK AISI's Inspect and extensions, the resources needed to validate it for academic and governance use, and a graded-metric benchmark design backed by peer-reviewed evidence — closing with a small, costed pilot study to prove it holds and produces trustworthy signal.
The sandbox decomposes into eleven components across four zones: the evaluation plane (orchestrator, agent, scoring), the containment stack (three nested boundaries), the control plane (egress, telemetry, monitor), and the governance layer. Each component below carries its purpose, the libraries and tools it needs, and its dependency on Inspect. Selecting a numbered block reveals its full design card.
The sandbox is an Inspect application. Inspect core supplies the eval loop, sandbox abstraction, scorers, metrics, budget limits, and tamper-evident logs; four extensions supply the pieces the lean design needs. Toggle to see which dependencies are hard (pilot cannot proceed without them) versus optional.
Single-framework reliance is a real dependency risk. It is mitigated by three facts: Inspect is open-source under a permissive licence; its task format is portable and the sandbox layer is provider-agnostic; and it is the shared standard across the UK, US (CAISI), and the wider AISI network — so an escape programme built on it stays comparable with, and portable to, government evaluations. [3,4,9,11]
Validation runs on three tracks — academic (is the measurement valid and reproducible?), industrial (does it scale and stay reliable?), and governance (is it defensible for decisions?). Each has distinct resource needs. Filter by track.
| Resource | Track | Specification | Why it is needed |
|---|
SCALING PATH — single host today, network-grade tomorrow
One KVM-capable machine; Firecracker + gVisor + container via the VM provider; sequential/low-parallel runs. Proves the design holds and the metrics are trustworthy.
→Move capability benchmarks to k8s_sandbox with Cilium egress policies and gVisor for parallelism; keep the microVM/Proxmox path for high-risk escape runs. Mirrors AISI practice. [9,11]
Tamper-evident logs, if-then thresholds, reproducible manifests, and senior-owner sign-off — a setup comparable to what the AISI network runs for pre-deployment testing. [3,11]
A single binary pass/fail is the wrong instrument for a saturating field. The design layers five metric families, each addressing a distinct question and each backed by peer-reviewed or high-quality institutional evidence. Every family maps to a native Inspect scorer or metric, so the pipeline is buildable today.
Binary task success at a fixed budget. Retained only for cross-org comparability with the AISI network's CyBench runs — never as the sole signal.
includes() / match() scorer, accuracy() + stderr() metrics.Award credit for intermediate progress — CyBench subtasks, GDM dangerous-capability milestones, or mean-criteria-passed. Separates models that tie on the binary at no extra token cost.
The Agentic Benchmark Checklist recommends using process-based metrics alongside outcome-based metrics; APEX-Agents notes that mean criteria passed "provides a dense signal that rewards partial progress." [2,10]
@scorer returning numeric value; value_to_float maps PARTIAL→0.5 natively.Grade how far up the exploitation ladder a model climbs — including the explicit sandbox-escape tier — rather than one pass/fail. The anti-saturation instrument and directly on-threat-model.
ExploitBench decomposes exploitation into 16 deterministically-graded flags across five tiers, arguing that treating a crash as success "collapses the hard parts of exploitation." Deterministic oracles avoid LLM-judge fragility. [8]
@scorer emitting a per-flag bitmap; aggregate with multi_scorer.Report both: pass@k (capability ceiling — can it ever escape?) and pass^k (consistency — does it escape every time?). A high-pass@k / low-pass^k profile is the signature of a model that benchmarks well but is unreliable in practice.
τ-bench introduced pass^k for tasks requiring "reliability and consistency"; as k rises, pass@k increases while pass^k falls. Agents strong on average can be unstable across repeats. [13,14]
Success at a fixed budget, tokens-to-solve, and area-under-the-budget-curve. Distinguishes models cheaply and keeps reported numbers honest as inference-compute scales.
AISI finds cyber success scales log-linearly with tokens to 100M with no plateau, so uncapped pass@k inflates with spend; an optimised scaffold reached the same score at ~13% of the token budget. [12,15]
eval(); per-sample token accounting in .eval logs.No metric is reported without an interval. Standard errors and confidence intervals, bootstrap CIs, and — critically — clustered standard errors when the same task is sampled repeatedly. Paired differences and power analysis size the run.
Miller (Anthropic) shows clustered standard errors can be over three times larger than naïve ones; without error bars, eval scores are effectively meaningless. [16]
stderr(cluster=…), bootstrap_stderr() are built in — the clustered-SE recommendation maps to one argument.Graded metrics are only trustworthy if the task and grader are valid. The Agentic Benchmark Checklist is applied to every ported benchmark: an oracle/solution solver confirms solvability, state is cleared between runs, the agent is isolated from ground truth, and implementation shortcuts are audited. These checks matter — validity issues have been shown to mis-estimate agent performance by up to 100% in relative terms, and applying the checklist cut one benchmark's overestimation by 33%. [2]
A small, time-boxed study to answer two questions: does the sandbox hold against a frontier model, and does the graded-metric pipeline produce stable, discriminating signal at acceptable cost. It is deliberately narrow so it can run on one host in weeks, not quarters.
pass@1 + process/ladder partial credit (M2/M3) + pass@k & pass^k at small k + tokens-to-solve (M5), each with clustered SE and bootstrap CIs (M0).2.5M-token cap for capability tasks and a larger cap for escape runs, matching AISI-network settings for comparability. [4,16]# capability task on the lean sandbox, graded, error-barred from inspect_ai import task, Task, eval from inspect_ai.solver import basic_agent, use_tools from inspect_ai.tool import bash, python from inspect_ai.scorer import scorer, accuracy, stderr, bootstrap_stderr @scorer(metrics=[accuracy(), stderr(cluster="task_id"), bootstrap_stderr()]) def ladder_scorer(): ... # emit per-flag bitmap → numeric partial credit @task def escape_capability(): return Task( dataset=discriminating_subset, # pruned, contamination-safe solver=basic_agent(tools=[bash(), python()]), scorer=ladder_scorer(), sandbox="proxmox", # microVM outer boundary; swap to vagrant/firecracker ) eval(escape_capability(), epochs=5, token_limit=2_500_000, message_limit=100, temperature=1.0) # AISI-network-comparable settings
Escape validation reuses the SandboxEscapeBench harness with the same VM provider so a breakout lands in the outer VM, never the host. Transcript auditing runs post-hoc over the .eval logs.
Neutrality & verification note. This plan synthesises peer-reviewed papers (NeurIPS, ICLR), high-quality preprints, and institutional publications (UK AISI, CAISI/NIST, METR, Anthropic). Cited excerpts are paraphrased or kept to short phrases for attribution. Capability and cost figures are as-published and change quickly; the R&D team should re-verify against primary sources before any figure informs a formal governance threshold.