Why teams reach for Inspect AI when their evaluation work shades toward agents and safety

Flashcard·Medium·4.0 · 0·~30s·Asked atCursor Rephrase Ai Turbopuffer

Attempt it

Inspect AI was built for capability and safety evals on agentic systems. It has first-class primitives for multi-turn agent tasks (tool-use solvers, sandboxed shell and browser environments, scaffolds), task-level metric aggregation across many models, and reproducible log files that grade rubrics replay cleanly.

The framing is a Task with a dataset, a solver (the agent plus its scaffold), and one or more scorers. That maps directly onto evaluating whether a model can do something dangerous or useful in a controlled environment. General runners can be coerced into the same shape but lack the bundled sandboxes and the safety-eval idioms (scorer that grades refusal quality, harm rubrics, replayable transcripts) that Inspect ships out of the box.

Your answer

**Inspect AI is an evaluation framework from the UK AI Safety Institute** (AISI), built to evaluate frontier models on capability and safety dimensions. Because AISI's job is to assess what models can and cannot do in agent settings, Inspect was designed from the start for multi-turn, tool-using, sandboxed evaluations. A general-purpose runner like Promptfoo or DeepEval is great for single-turn prompt comparison. You write a YAML file with inputs and assertions, run it across models, get a comparison table. Inspect can do that too, but its real strength is in evals that need something more: - **Tool-use solvers**: the agent calls tools, the tools return results, the agent continues. Inspect ships solvers for this loop. - **Sandboxed environments**: ephemeral shell containers and headless browsers the agent can act in safely. - **Multi-turn scaffolds**: the framework handles the back and forth of a long agent trajectory and stores the whole transcript. - **Safety-eval idioms**: scorers that grade refusal quality, harm rubrics, capability-elicitation patterns that AISI itself uses. - **Replayable log files**: every run produces a JSON log that scorers can re-grade without re-running the model, which matters when judge rubrics evolve. The pattern is `Task = dataset + solver + scorer`. For agent or safety work you adopt Inspect to get the kitchen for free; for vanilla prompt comparison, lighter runners are easier to set up.

**Inspect AI is purpose-built for agent and safety evaluations**, which is where general-purpose eval runners (Promptfoo, DeepEval, Braintrust, LangSmith) start to feel under-powered. **The framing.** An Inspect eval is a `Task` consisting of a `dataset`, a `solver`, and one or more `scorers`. The dataset is the inputs. The solver is the agent (or the prompt, or the chained scaffold) that produces outputs. The scorer grades them. This is a clean separation that makes agent scaffolds first-class rather than something you bolt on. **Why it wins on agent evals.** Inspect ships: - **Sandboxed execution environments**: Docker-based shell sandboxes (`SandboxEnvironment`) and headless browser sandboxes that the agent can act in without touching the host. Bring-your-own-sandbox is the most time consuming part of building agent evals from scratch. - **Built-in solvers for tool use, chain of thought, self-critique, and reflection patterns**: ready-made building blocks for the kinds of agent loops you would otherwise hand-write. - **Sample limits, message limits, and token budgets** baked in, so an agent cannot run away during evaluation. - **Replayable log files**: the full transcript (prompts, tool calls, tool returns, completions) is serialized to a `.eval` log. You can re-grade with a new scorer rubric without re-running the model, which matters when the rubric is itself evolving. **Why it wins on safety evals.** Inspect ships idiomatic scorers for refusal-quality grading, harm rubrics, and capability-elicitation patterns that AISI publishes and uses. The framework's design assumes some evals will be sensitive (dual-use, cyber, bio) and supports the workflow patterns (small private datasets, careful logging, human in the loop scoring) that safety work needs. **When to use what.** Inspect for agent capability evals, safety evals, anything multi-turn with tools. Promptfoo or DeepEval for fast prompt-comparison loops in product CI. Ragas for RAG-specific metrics.

TL;DR

Inspect AI ships sandboxed shell and browser environments, multi-turn solvers, and safety-scorer idioms out of the box, which general runners like Promptfoo and DeepEval lack and have to be hand-rolled into.

Memory aid

Easy to grasp

Imagine two kinds of tests for a robot chef. The first is a taste test, where someone scores a finished plate. Any rating sheet works for that. The second is a real-kitchen test, where the robot picks up a knife, lights the stove, and you watch what it does without burning the house down. That needs a real kitchen with safety walls around it and a careful score sheet for the moves the robot makes. Inspect AI is that walled-off kitchen. The UK AI Safety Institute built it to test whether models can run real-tool tasks (shell, browser, multi-step plans) and whether they refuse the dangerous orders. General eval tools grade plates. Inspect grades the whole kitchen visit.

Key concepts

LLMOps

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Most LLMOps teams pick an eval framework once and stick with it. The choice tends to follow the shape of the team's first hard eval: a product team writing prompt regressions picks Promptfoo or DeepEval; a RAG team picks Ragas; a research-leaning team picks LangSmith or Braintrust. The teams that pick Inspect AI almost always have one specific shape of problem: they are evaluating agentic systems, doing safety-research work, or both.

This walkthrough explains who built Inspect and why, the core architectural primitives that make agent evals tractable (sandboxes, solvers, replayable logs), the safety-eval idioms that ship with the framework, and the working dividing line between problems where Inspect is the right tool and problems where a lighter runner is faster to live with.

Origin: built by AISI for the evals AISI actually runs

Inspect AI was open-sourced by the UK AI Safety Institute (AISI) in 2024 and has been actively developed since. AISI's job is to evaluate frontier models from OpenAI, Anthropic, Google DeepMind, and others on capability and safety dimensions before and after deployment. That job determines the framework's priorities.

The evals AISI runs are not 'does this prompt produce a faithful answer'. They are 'can this model, given a shell and access to a sandbox, autonomously complete a cyber-offense task within a token budget' or 'does this model refuse correctly when prompted to help synthesize controlled substances'. Those evals demand sandboxes, multi-turn scaffolds, careful logging, and rubrics that evolve over time as new failure modes are discovered.

General-purpose runners can be coerced into the same shape; you can hand-roll a sandbox in Promptfoo with a custom provider. The cost is weeks of plumbing for something that comes for free in Inspect. AISI built the framework so its own engineers would not have to keep rebuilding that plumbing per eval; the open-source release means everyone else can benefit too.

The downstream effect is that Inspect is the framework agent-eval and safety-eval researchers reach for first. METR uses it. Anthropic's pre-deployment evaluation work has used Inspect-based tasks. Academic agent benchmarks increasingly publish as Inspect tasks because the audience already has Inspect set up.

The Task primitive: dataset + solver + scorer

Sandboxes and replayable logs: the operational killers

When Inspect is the right tool and when it is not

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

The UK AI Safety Institute uses Inspect AI internally to evaluate frontier models from OpenAI, Anthropic, and Google for capability and safety.
METR (Model Evaluation and Threat Research) publishes agent-task evals on Inspect for autonomous-task benchmarks.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you build a cyber-offense capability eval in Inspect that gives the agent a shell but keeps the host safe?

Use SandboxEnvironment(type='docker') with a constrained container image, no network egress (or a sniffer-only proxy), a per-sample time and message budget, and a teardown hook that snapshots logs before destroying the container. The agent acts only inside the sandbox; the host process never executes attacker-controlled code.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Treating Inspect AI as a general-purpose replacement for Promptfoo. Inspect's value is concentrated in agent and safety evals; for vanilla prompt-comparison work, lighter runners are faster to set up.

60 second bullets to scan on the way to the call.

Who built Inspect AI and what they use it for
The Task = dataset + solver + scorer framing

Primary sources. Browse if you want the original framing.