Efficient LLM Agents with REPL Harnesses: Executable Composition and Validated Reuse

TL;DR

An LLM agent never touches the world directly. It emits text; a harness turns that text into actions. The language the harness lets the agent speak in — a JSON function call, an MCP endpoint request, a shell command, or a line of executable Python — is the action substrate. This blog asks a deliberately narrow, causal question:

With the model, task set, primitive operations, and budgets all held fixed, how much does only changing the action substrate change success and cost?

The answer is: conditionally, a lot — but not universally. Two mechanisms explain the wins, and the blog is unusually careful to also map where they vanish:

1. Executable composition — when tasks contain dependent computation, a stateful REPL (or a "batch function-call DAG") stays flat at ~2 model round-trips while plain function-calling/CLI add ~1 round-trip per dependency step.
2. Validated reusable artifacts — when later queries share policy structure, promoting a privately-validated executable helper lets the agent reuse it instead of replaying the full spec each time, saving turns and tokens.

And the boundary: on stateful "observe-before-commit" API tasks (BFCL) the discrete interface wins; on broad shell-native Terminal-Bench tasks the substrates are at parity; and a no-Python "Batch-FC" ablation recovers most of the turn savings without the success gap — proving the effect is executable composition, not "code beats calls."

1. Why This Question Matters

Modern agent leaderboards quietly confound three things: the model, the scaffold (planning loop, memory, retries), and the interface/tools through which actions are executed. When "Agent A beats Agent B," you usually can't tell whether it was a better model, a smarter loop, or just a richer tool surface.

This blog isolates one slice of that — the model-visible action substrate — and treats it as a controlled experimental variable. The framing is a credit-assignment problem: not "is code-action good?" (that's established prior art — CodeAct [3], OpenHands [4], Voyager-style skill accumulation [5], MCP-as-code [6], and executable tool blocks) but which specific interface affordance buys which specific gain.

For Mind Lab, this distinction matters because agents that learn from real experience must repeatedly convert model outputs into reliable actions. If model quality and harness expressivity are confounded, it becomes difficult to tell whether an agent has learned a better policy or merely received a more efficient way to execute one.

2. The Substrate Ladder

The blog organizes interfaces by expressivity, exposing the same underlying primitives through each rung:

function call  →  protocol endpoint (MCP)  →  shell command (CLI)  →  host-language expression (REPL)
   (FC)                (MCP)                       (CLI)                 (code-as-action)

• FC — one JSON {tool, args} per turn.
• MCP — the same capabilities as JSON-RPC endpoints (modeled here as endpoint capability provision).
• CLI — one shell command per turn.
• REPL — the action is the evaluation of a Python expression: compose primitives, bind intermediate results to variables, define helpers, and reuse them on later turns — without round-tripping a tool schema.
• Batch-FC — a crucial falsifier rung: a discrete substrate that may submit a dependency graph of function calls in a single action, but with no host language. It tests whether the REPL's advantage comes from Python or from executable composition.

Figure 1: The action substrate ladder exposes the same primitives through increasingly expressive interfaces, with Batch-FC isolating executable composition from the host language.

The core mechanism, stated crisply:

An action substrate determines how much of an agent's computation can be executed rather than re-described. A discrete-call substrate forces every intermediate value back through the model as text — re-read, re-typed, and re-marshaled on the next turn — whereas an expression substrate keeps that computation inside an executable state the model never has to restate.

A Worked Example: Why Depth Costs Round-Trips

Task: Total net revenue of paid NA orders, where net = amount − refund − loyalty_discount.

Function-calling has to bounce each intermediate value back through the model as text:

turn 1  assistant: get_orders()                         → [ …big JSON of orders… ]
turn 2  assistant: (re-reads that JSON) filter_paid_na(orders) → [ …subset… ]
turn 3  assistant: (re-reads subset) compute_net(each)   → [ …numbers… ]
turn 4  assistant: sum(those)                            → 48213

Every arrow is a model round-trip: the orders list is re-read, the subset re-typed, the partial sums re-marshaled.

REPL keeps the whole chain inside one executable state:

# turn 1 — one action block
orders = [o for o in get_orders()
          if o["status"] == "paid" and o["region"] == "NA"]
net = sum(o["amount"] - o["refund"] - round(o["amount"] * o["tier"] * 0.015)
          for o in orders)
submit(net)                                              # 48213

One round-trip. The intermediates (orders, net) live in the namespace; the model never restates them.

Batch-FC achieves the same turn profile by submitting the 4 calls as one dependency graph — which is exactly how the blog proves the win is composition, not host language.

Figure 2: Function-calling returns each intermediate value through the model, while a REPL keeps the dependent computation inside one executable state.

3. Substrate Surgery: The Method

The experimental protocol is a counterfactual "substrate surgery": hold the model, task distribution, primitive implementations, and declared budgets fixed, and re-express identical capabilities across substrates. Fairness rules that matter:

• One action per model turn, for every substrate (a REPL turn can't secretly batch many commands the others can't).
• Same primitives behind every rung; prompts/parsers differ only in the action-format block.
• A shared "completion-discipline" instruction on every substrate, treated as a fairness control, not a result.
• Falsifiers (Batch-FC, noisy-intermediate, sham/trivial promotion) and boundary tasks are first-class.

4. What We Found

Experiment 1 — ComposeBench: Flat vs. Linear Round-Trips

What it is: The domain is a synthetic sensor network with 7 deterministic pure-function tools (read, calibrate, scale, combine, difference, aggregate, threshold), so ground truth is closed-form and exact. Two axes are swept: chain depth d∈{1..5} (a chain of d dependent calls, output of step $k$ feeding k+1) and fan-in width w∈{3,5,7,9} (read+calibrate $w$ sensors, then aggregate+scale — forcing many intermediates to be carried forward), 12 tasks each. Depth-1 is the deliberate falsifier (a single call with nothing to compose — the REPL should not win there). Four substrates are implemented over the same tools; the key one is Batch-FC, a no-host-language rung that submits a {id, tool, args} dependency graph in one action (args can reference earlier steps' outputs) — it isolates "collapse a dependent chain into one action" from "have Python." Turn budget 20 (removes the tight-budget confound); a noise condition clutters observations with distractor fields.

Result: As dependency depth grows, REPL and Batch-FC hold a ~2-turn profile, while FC ≈ 3.34 and CLI ≈ 3.77 turns and climbing (~+1 round-trip per dependency step). Because Batch-FC (no Python) also stays flat, the flat profile is localized to executable composition, not the host language.

Figure 3: ComposeBench shows flat turn growth for REPL and Batch-FC as dependency depth increases, localizing the gain to executable composition.

Experiment 2 — FlowBench-Substrate: Success and Matched-Correct Cost

What it is: a 300-task closed-world business-data analytics grid — a synthetic, fully deterministic e-commerce-operations domain (orders/refunds/customers/policies) with internal-label ground truth, organized by composition depth 1–6 so tasks range from a single lookup to deeply-chained metrics. It moves the ComposeBench question onto realistic business analytics under a generous budget, and adds a real code-action comparator, smolagents (Hugging Face's CodeAgent).

Our REPL harness achieves the highest success rate in this comparison.

Figure 4: FlowBench-Substrate compares success rates across REPL, Batch-FC, function-calling, CLI, and the smolagents CodeAgent.

On matched-correct cells (tasks both substrates solve), discrete substrates pay dramatically more: FC ≈ 6.5× turns / 8.2× input tokens, CLI ≈ 6.5× / 10.5× vs the REPL. Batch-FC narrows the turn gap (≈2.05× / 2.34×) but not the success gap — again separating "compose in one action" from "have a full programming language for filtering/arithmetic/final-answer logic."

Figure 5: On tasks solved by both compared substrates, discrete interfaces require substantially more turns and input tokens than the REPL.

External Checks A–D

• A. M3ToolEval (82 tasks, CodeAct's own suite [3], fixed 10-turn protocol). REPL reproduces the CodeAct pattern: 0.85 success in 4.6 turns / 17.5K input tokens, vs FC 0.68 (6.6 turns / 27.7K) and CLI 0.69 (6.6 / 26.6K) — a +16-point gap, largest in compositional domains (trade calculation, travel planning), and absent on retrieval-bound web browsing. Honest caveat: a 25-turn audit flips 33/49 previously cap-failed FC/CLI cases to passing — so the success gap is a turn-cap-sensitive round-trip effect, not a matched-budget capability win.
• B. API-Bank (250 scorable level-1/2 tasks [7], 8-turn cap, free-running goal completion). REPL leads: strict goal completion 56.8% vs FC 44.0% / CLI 43.2% (paired +6.0 points, 95% CI [+2.0,+10.0] vs the per-task better discrete arm); valid-composition 65.2 vs 50.4/50.8 (+7.2 [+2.8,+12.0]). But at a real cost premium: REPL averages 482 output tokens / 2.07 calls vs 268/1.45 (FC) and 313/1.84 (CLI). This is a composition-accuracy win, not a cost win.
• C. BFCL v4 multi-turn base (200 stateful tasks, 600 rollouts, official checker [8], matched 12-call/user-turn capacity) — the clean negative result. With per-turn call capacity equalized, REPL trails: REPL 49.5% vs FC 54.0% vs CLI 45.0%, i.e. REPL is −7.5 points below the better discrete arm. On stateful observe-before-commit API tasks, issuing many calls inside one code block is a liability — the agent needs to see each call's result before the next state change. BFCL narrows the positive API-Bank result rather than extending it.
• D. Terminal-Bench — On the full 89-task Terminal-Bench-2.0 [9], REPL and CLI are at parity (REPL 0.46 vs CLI 0.43). CLI has the lower process cost, while REPL/FC/MCP carry high cap pressure. In the controlled fixed-model run summaries, at parity success the REPL spends roughly 2× the input tokens of CLI — on the order of ~700K vs ~340K per task in the run, at similar turn counts (~18) — because independent shell tasks give it little to compose, so each turn mostly re-reads a growing context instead of composing new work. This is the cleanest illustration of the central message: the REPL's efficiency advantage is conditional on compositional or reuse structure, not a property of the shell at large.

5. Validated Reuse Across Tasks

Executable composition reduces coordination cost within a task. Validated reuse extends the same idea across tasks: once an executable helper has been privately tested against a stable policy pattern, later queries can call that helper instead of replaying the entire specification through the model. The artifact becomes a compact carrier of previously validated behavior.

This mechanism is also conditional. Reuse helps only when later tasks share genuine structure, and promotion must follow validation; otherwise an incorrect helper turns a local error into a persistent one. The useful abstraction is therefore not unrestricted self-modifying code, but a controlled lifecycle of compose, validate, promote, reuse, and audit.

Conclusion

The result is not that REPL harnesses universally outperform discrete tools. Their advantage appears when a task contains dependent computation that can stay inside an executable state, or when validated behavior can be reused across related tasks. When work is stateful, requires observation before each commitment, or is already naturally expressed as independent shell steps, that advantage can narrow, disappear, or reverse.

For agents that learn from real experience, this points to a practical systems principle: the harness should expose enough executable structure to reduce re-description, while preserving validation and observation boundaries where the environment demands them. Better agent learning depends not only on the model and its data, but also on assigning credit correctly to the action substrate that turns reasoning into experience.

References

[1] ComposeBench (MindLab Research et al, 2026)

[2] FlowBench-Substrate (MindLab Research et al, 2026)

[3] Executable Code Actions Elicit Better LLM Agents (Wang et al, 2024)

[4] OpenHands (OpenHands et al, 2026)

[5] Voyager: An Open-Ended Embodied Agent with Large Language Models (Wang et al, 2023)

[6] Model Context Protocol (Model Context Protocol et al, 2026)

[7] API-Bank (AlibabaResearch et al, 2023)

[8] Berkeley Function Calling Leaderboard (Berkeley Gorilla et al, 2026)

[9] Terminal-Bench (Harbor Framework et al, 2026)

Author

Mind Lab

Core Contributors

Bo Wu, Di Zhang, Jun Gao, Murphy Zhuang, Kieran Liu, Andrew Chen, Pony Ma

Team

Asher Cai, Song Cao, Andrew Chen, Kaijie Chen, Cleon Cheng, Steven Chiang, Kaixuan Fan, Jun Gao, Pyke Han, Nolan Ho, Mutian Hong, Charles Huang, Fancy Kong, Andrew Lei, Lucian Li, Ray Li, Fan Lin, Kieran Liu, Logan Liu, Neo Liu, Xiang Liu, Yuxin Lu, Pony Ma, Cole Qiao, Vince Qu, Vincent Wang, Bo Wu, Chengdong Xu, Rio Yang, Regis Ye, Yihang Zeng, Di Zhang, Jiahao Zheng, Adrian Zhou, Yuhua Zhou, Murphy Zhuang and Mindverse Team

Names are listed alphabetically within team.

Citation

Please cite this work using the BibTeX citation:

@misc{wu2026replharnesses,
  author = {Bo Wu and Di Zhang and Jun Gao and Murphy Zhuang and Kieran Liu and Andrew Chen and Pony Ma and {Mind Lab}},
  title = {Efficient LLM Agents with REPL Harnesses: Executable Composition and Validated Reuse},
  year = {2026},
  howpublished = {Mind Lab: A Lab for Experiential Intelligence},
  note = {https://macaron.im/mindlab/research/efficient-llm-agents-with-repl-harnesses-executable-composition-and-validated-reuse}
}