Benchmarking Five Sandbox Providers with a Devin-Style Agent Build

We contracted an independent engineering team to benchmark five sandbox providers on a single question: which one is easiest for a coding agent to build against? Here is what they reported.

The test: let a builder agent build a Devin clone with this spec:

• A React web app for managing multiple agent sessions.
• Users log in with GitHub OAuth, pick a repo, and start a session with a prompt.
• Each session gets its own sandbox with the repo cloned and gh authenticated.
• A coding-agent loop runs inside: it acts on prompts, writes files, works on a fresh branch, and runs tests.
• Users can stop a session by pausing its sandbox, then resume it from that snapshot.
• Users can fork a session: the new agent gets a copy of the parent's sandbox filesystem, including uncommitted work.

We ran the same spec against five providers: box, E2B, Daytona, exe, and Islo. Each builder agent got an identical prompt; the only provider-specific input was a link to that provider's docs. The agent had to figure out the rest: read the docs, learn the API or SDK, and build the whole app in one run.

We ran the test three times per provider and made sure no run failed on auth or API-key issues. The builder model was Claude Opus 4.7, and every step was traced with Laminar.

Results

TL;DR:

• All 15 runs produced a working Devin clone. Only box got pause, resume, and fork working in all three of its runs.
• E2B builds were the cheapest and fastest, but had to fake pause, resume, and fork with fallback code.
• Builds cost between $6.19 (E2B) and $33.32 (exe) in builder-agent tokens.

Here are the full results, averaged over each provider's three runs:

Where box wins

• box was the only provider where pause, resume, and fork worked in every run.
• box is also cheaper to run: a 4 vCPU / 8 GB sandbox costs about $0.036/hr, vs roughly $0.33/hr for the closest alternatives and $0.6/hr for Islo.

Where others do better

• E2B builds were cheaper and faster: $6.38 and 21 minutes on average, vs $8.77 and 28 minutes for box. The tradeoff: its clones had to improvise the pause/resume/fork lifecycle.
• Islo averaged the fewest errors per run of any provider.

Sandbox runtime pricing

As of June 2026, normalized to a 4 vCPU / 8 GB sandbox-hour:

Methodology

• Each provider's lane started from an empty repo with only the spec, credentials, and that provider's docs link.
• Claude Opus 4.7 ran fully unattended through the Claude Agent SDK, with no human hints mid-run.
• We ran several cold cycles and report three clean runs per provider, 15 builds total.
• All cost, token, and timing numbers come from Laminar traces at the API boundary.

What we measured was integration difficulty: how easily the agent could read the docs, learn the API or SDK, build the app, and get the sandbox lifecycle working.

One caveat: time and cost here are the builder agent's. They don't include sandbox runtime bills, which depend on the app's usage and each provider's pricing.

How agents build a Devin clone

We also read through the builder agents' transcripts to understand how they actually work with sandbox provider APIs. Here is what we found.

1. Steps to build a Devin clone

Across providers, agents followed a surprisingly consistent order:

1. Read the local files, then the docs. The repo only contains the prompt and .env, so the agent reads those, then follows the docs link.
2. Install the SDK, if there is one, and check the package shape.
3. Test the SDK or CLI. Inspect types, run small commands, learn the real method names before committing to an implementation.
4. Test the live provider before trusting it. Not just one API-key check: agents made a batch of live calls first. In one box run, the agent created a throwaway box, prompted it, polled its events, and deleted it before writing a single source file.
5. Plan, scaffold, then wire the provider. Once the provider surface looks real, the agent writes a plan, builds the app skeleton, and connects the sandbox lifecycle, fixing bugs along the way.

2. How an agent learns to use the sandbox provider

Agents learn a provider two ways: reading the docs, and poking the live API. Here is how much of each they did per provider:

Discovery depends on docs quality. The box agent read 9 docs pages on average and found the lifecycle endpoints directly. The Islo agent read 14 and still got stuck on auth, because the key was not a bearer token.

E2B needed almost no reading: 2 docs pages on average, then 25.7 live calls to validate the SDK. Faster, but this agent also hit the most errors.

exe took the most discovery effort: no SDK, so every call went through the raw API.

Agents read docs structurally. They look for an index like llms.txt, then jump straight to endpoint-level pages like api-reference/sandboxes/create-sandbox.md. If those are missing, the agent has to crawl, guess, and test the API live.

None of the builds hit context compaction. Every run fit in a single context window (the largest was exe's 291-turn run), so docs read in the first two minutes were still in context 50 minutes later.

3. What's the slowest step?

We reconstructed a timeline for all 15 builds from their tool-call timestamps:

Discovery is 5–16% of the run. For the rest of the time, the agent does the build and validate loop.

Docker builds vary the most, even with caches cleared before every run. The variance comes from the agent, not the provider: how many rebuilds it needed, and whether it built everything in one pass or incrementally.

The two slowest runs, explained:

• An exe run took 53 minutes because the builder had to debug the sandbox's first-boot flow. It spent many turns polling the in-VM agent server, then rewrote it to remove a fragile pip install startup dependency.
• A Daytona run took 37 minutes because stale sandboxes had filled the org's 30 GiB disk quota. The builder had to delete old sandboxes through the raw API, then lower the requested disk, CPU, and memory to fit.

4. The hardest features: pause, resume, and fork

A real pause/resume stops the sandbox, snapshots its filesystem, and later resumes that same sandbox. When a provider doesn't offer this, or doesn't make it discoverable, the builder fakes it: the app marks the agent stopped while the VM keeps running. Since every provider bills by active time, that fallback gets expensive.

Only 8 of 15 builds got true pause/resume. box and Islo did it in every run, Daytona in one of three. One E2B build resumed the same sandbox; the other two hit a 404 on resume or fell back to a fresh sandbox. No exe build paused the VM at all.

A real fork copies the whole filesystem, including uncommitted work. 8 of 15 builds failed to do this and instead created a fresh sandbox and re-cloned the parent's branch. Per provider:

• box has a simple native fork, and every build used it: forked sandbox, branch, history, and parent lineage.
• E2B had no native snapshot fork, so all three builds rebuilt the child from a new sandbox.
• Daytona exposed _experimental_fork, but one build still kept a fresh-sandbox fallback.
• exe has cp as its fork primitive, but one build got a 403 from it.
• Islo forked correctly once; the other two runs re-cloned into a new sandbox.

Conclusions

Every provider let the agent create and stop a sandbox. The split showed up in the lifecycle: not everyone offers pause, resume, and fork as real primitives, and fallback code means re-cloning work or keeping machines warm, which costs money at scale.

box was the best overall balance here: it completed the full lifecycle in every build and had the lowest runtime price in the comparison at $0.036/hr.

Pick the sandbox that preserves state cleanly, not just the one with the fastest quickstart. For a Devin-like product, that was box.

Curious why box uses full VMs instead of containers in the first place? Read Containers vs VMs: why your agent sandbox should be a real machine.