Benchmarking Five Sandbox Providers with a Devin-Style Agent Build

Here is what the benchmark team reported:

To do this, we let a builder agent build a Devin clone with this spec:

• A React web app where users can create and manage multiple agent sessions.
• Users log in with GitHub OAuth, select one repository, and start an agent session by giving a prompt.
• Each agent session runs in its own isolated sandbox, with the selected repo cloned and gh CLI authenticated inside it.
• A Claude-compatible coding-agent loop runs inside the sandbox, acts on prompts, writes files, creates a fresh branch, and runs verification tests.
• Users should be able to stop a session by stopping or pausing its sandbox, then resume it later from that stopped sandbox snapshot.
• Users should be able to fork the conversation, creating a new agent from the selected agent's context and branch state, and forking the sandbox when the provider supports it.

And we compared box with 4 popular sandbox options on the market:

• box
• E2B
• Daytona
• exe.dev
• Islo

Each builder instance got the same spec to implement but was given links to a different provider's docs. The agent had to figure out the rest by reading their docs, learning their API/SDK surfaces and then one-shotting a full-blown Devin clone.

We ran this test 3 times per provider, making sure none of them failed because of auth or API key related issues. We used Claude Opus 4.7 and observed all steps with Laminar.

Results

TL;DR.

• All 15 runs one-shotted a working Devin clone but 4/15 (27%) runs failed to build features to pause,resume and fork a sandox, which was included in the feature specification.
• Builder agents using box and Islo were the only ones to complete the full feature spec.
• E2B and Daytona builds used less tokens and were faster, but their devin agents had to build fallback behaviour for the sandbox pause/resume/fork features.
• The build that used the least tokens was E2B at $6.19; the one that used the most was exe.dev at $33.32 / 53 minutes. (These are builder agent's token costs, not the cost for the agent running inside the sandbox or the sandbox itself.)

Here are the full results, averaged over each provider's three runs:

Where box wins:

• Every provider got a full-stack Devin clone running. But box and Islo were the only ones to correctly implement pause, resume and fork for the sandbox.
• Box is also cheaper to run. The price for a 4 vCPU / 8 GB RAM sandbox on box is about $0.036/hr, while the closest alternatives here are roughly $0.33/hr and Islo is $0.6/hr.

Where others do better:

• E2B averaged a lower token cost and build time at $6.97 per build (vs $8.77 for box), and 21 minutes (vs 28 min for box). This is the one time token cost of actually building this Devin clone. Not the cost of running sandboxes on it. The big tradeoff is that pause/resume/fork is where it had to improvise.
• Islo's reruns averaged less errors than all other providers runs.

Sandbox runtime pricing

As of June, 2026, normalized to a 4 vCPU / 8 GB sandbox-hour:

Benchmark Methodology

How we actually ran it:

• We gave Claude Code the same Devin-clone product spec five ways: one lane for Box, E2B, Daytona, exe.dev and Islo.
• Each lane started from an empty repo with only the prompt/spec and credentials. The only provider-specific input was the provider name and docs link.
• Claude Opus 4.7 was held constant. The builder ran fully unattended through the Claude Agent SDK, with no human hints mid-run.
• We ran several cold cycles and report three clean runs per provider, 15 builds total.
• Every run was traced with Laminar at the API boundary. The cost, token and timing numbers in this report come from those traces.

What we measured was integration difficulty: how easily a builder agent could read the docs, discover the API or SDK, build the app, and get the real sandbox lifecycle to round-trip.

Two caveats:

• Time and cost are builder-agent time and builder-agent token cost. They measure how long the builder model took, and how many tokens it burned, to build the integration. They don't include sandbox runtime bills, because that depends on the app's usage and sandbox provider pricing.

Insights: How do Agents build a Devin-clone

We looked through the builder agent's transcripts to understand how agents work with Sandbox provider APIs.

Here are some interesting insights we found:

1. Steps to build a Devin clone

The first thing we wanted to extract was the general path a builder agent takes when you ask it to build a Devin-like clone from an empty repo.

Across different providers, the order in which the agent worked was surprisingly consistent:

1. Read the local files, then read the docs. The agent starts with the prompt and .env which are the only files in the repo, then follows the sandbox provider docs link. Before the first source write, the agent read about 9 docs pages on average. (More on this next.)
2. Install the provider surface. If there is an SDK, the agent installs it and checks the package shape.
3. Test the SDK or CLI. The agent inspects types, runs small manual commands, checks help output when it exists, and tries to learn the real method names before committing to an implementation.
4. Test the live provider before trusting it. This was not usually one API-key check. In the raw HTTP and CLI-heavy runs, the agent made a small batch of live calls before writing product code: auth, list, help, create, inspect, prompt or exec, events, and cleanup. In the clean Box run, it hit /me, /limits, /boxes, created a throwaway box, prompted it, checked events a few ways, polled events, and deleted the box before the first source file was written.
5. Make a plan, scaffold, then wire the provider. Once the provider surface looks real, the agent writes a plan, starts the app skeleton, and then connects the sandbox lifecycle into the product, fixing any bugs along the way.

2. How an agent learns to use the sandbox provider

Agents do two things to learn how to integrate with a sandbox provider: by reading the docs or reverse-engineering the live API.

The agent does this discovery work early, then tests the provider with a script before integration.

This graph shows the average number of docs pages the agent read for each provider compared to the average number of live provider smoke-calls it made:

Discovery depends on docs quality. The agent working with box read 9 docs pages on average and found the lifecycle endpoints directly. Meanwhile, the agent using Islo read 14 pages on average and still got stuck on API authentication because the key was not a bearer token.

For E2B, the agent barely had to read before coding: 2 docs pages on average, followed by 25.7 live calls to validate the SDK. Even though this was faster, this agent encountered the most errors when building.

For exe.dev, discovery took more effort because of the lack of an SDK and the fact that every call had to be made via an API.

• Agents read docs structurally. They look for an index like llms.txt, then jump into endpoint-level .md pages like api-reference/sandboxes/create-sandbox.md, cli/sandbox-commands.md, or docs/cli-cp.md. If those pages are missing, the agent has to crawl, guess, and test the API live.

None of the builds hit context compaction. The entire build loop fits inside a single context window for every provider run, (largest run was exe.dev's 291-turn run). Docs the agent read in the first two minutes were still in context during final validation 50 minutes later, with no forgetting and no re-fetching.

3. What's the slowest step?

We reconstructed a timeline for all 15 builds from their tool-call timestamps. This is what we found:

Discovery is 5–16% of the run. For the rest of the time, the agent does the build and validate loop.

• Docker builds vary in timing even though we cleared all docker caches before every run. This has nothing to do with the sandbox used but seems to depend on how many times the agent had to rebuild and on whether the agent one shotted the whole thing or built components incrementally.

Some provider-specific post-mortems for slow build times:

• One of exe.dev's runs took 53.1 mins because the builder had to debug the first-boot flow inside the exe.dev sandbox VM. It spent many turns polling the in-VM agent server at :9000/ping, checking Shelley conversations and setup logs, then rewrote the agent server to avoid a fragile pip install anthropic startup dependency.
• One of Daytona's runs took 36.6 mins because stale Daytona sandboxes had filled the org's 30 GiB disk quota. The builder had to inspect and delete old provider sandboxes through the raw API, lower the requested disk from 8 GiB to 5 GiB, then lower CPU and memory after hitting the next quota wall.

4. What are hardest features to implement

Implementing sandbox state lifecycle is the hardest feature to implement.

A real pause/resume should pause or stop the provider sandbox, keep its filesystem and working state saved as a snapshot, then resume that same sandbox later.

Some providers either don't offer sandbox pausing, or don't make it discoverable enough for an agent. In those cases, the builder agent implements a fallback resume that marks the agent stopped/idle in the database (and thus, the app UI) but leaves the VM running.

Considering all sandbox providers bill by time active, this could significantly increase the cost of running such a Devin clone.

Only 9/15 builds have true pause/resume behavior that preserves the same sandbox. Box, Daytona and Islo have native sandbox lifecycle commands the agents could use. Agents building on E2B and exe.dev used fallbacks: E2B either killed and recreated the sandbox, hit a 404 while reconnecting to a paused sandbox, or documented a fresh-sandbox-with-same-branch path. exe.dev either controlled an in-VM endpoint, marked the agent stopped while keeping the VM alive, or only got Stop/Resume DB transitions after restart was denied.

A real fork should copy the entire OS image and filesystem state including uncommitted work.

Nearly half the runs (7/15) failed to build a real fork implementation and instead built a fallback that built a fresh sandbox from scratch and re-cloned the parent's branch.

Box had the cleanest implementation because it had a clear, simple native fork feature. Every build created a forked sandbox with branch, history and parent lineage.

E2B lacked a native sandbox-snapshot fork in these builds, so all three rebuilt the child from a new sandbox, the parent's branch and copied app history.

Daytona exposed _experimental_fork, but one build still kept a fresh-sandbox fallback as a backup.

exe.dev had cp as the VM fork primitive, but one build could not use it because cp returned Error 403.

Islo implemented it correctly once but its other two runs created a new sandbox or re-cloned while copying transcript and branch context.