Introducing LoopBench Optimizer: an evaluator-gated loop that makes your code faster — and proves it

Today I'm open sourcing LoopBench Optimizer: an autonomous optimization harness powered by coding LLMs. You point it at a repo and a file, and it evolves that code to run faster — while proving, every step of the way, that the change is both correct and measurably better.

You can run LoopBench on your laptop. There's no cloud service to set up and no privileged access to grant. For inference it talks to any OpenAI-compatible endpoint, so you can use Groq, Google Gemini, OpenAI, or a local model — including free tiers. Every candidate is executed in a locked-down Docker sandbox (--network=none, read-only mount), so untrusted, LLM-generated code never touches your machine or the network.

The whole run is bounded: LoopBench stops on a token, dollar, runtime, or iteration budget, so an experiment can't quietly run up a bill.

Architecture

At its core, LoopBench runs an LLM in a closed feedback loop and treats the LLM as an architect — it rewrites the logic, and the system turns that into a guaranteed-valid patch. Every candidate is created, isolated, tested, scored, and recorded before the loop decides what to try next.

The cycle each generation:

• Map — a repo context mapper builds a token-budgeted, LLM-ready map of the repository so the model sees the relevant code, not the whole tree.
• Generate — the LLM proposes an improvement as a full-file rewrite or Aider-style search/replace blocks, auto-routed by file size. It never hand-writes diffs.
• Apply — the change is materialized in a fresh, disposable git worktree, and a guaranteed-valid unified diff is computed with Python's difflib — so "corrupt patch" failures are impossible.
• Test — the candidate runs in a Docker sandbox (network-off, read-only). Correctness comes from your test suite; speed comes from a printed marker or a regex metric.
• Record — every attempt, patch, metric, and failure is written to a SQLite audit trail.
• Select — an auto search strategy picks the next baseline: it starts greedy, and only when it plateaus does it escalate (restart, then diversify), before stopping.

The key design decision: LLMs are brilliant at logic and terrible at being diff engines. Asking them for byte-exact unified diffs produces corrupt patches 20–30% of the time. LoopBench sidesteps that entirely — the model rewrites logic, difflib produces the patch, and the loop becomes unbreakable.

Running LoopBench on real code

LoopBench has been run against real, external repositories, not just toy fixtures.

Pointed at a classic O(n²) bubble sort using a free model (Groq's llama-3.3-70b), it took the sort from 443 ms to 0.18 ms — over 2,000× faster — by autonomously replacing the whole algorithm with an O(n log n) sort, with every correctness test still green inside the sandbox. On real numeric code, the same loop found a +38% NumPy vectorization of a Python loop.

The result I care about most is a negative one. Pointed at a Vyper smart contract that had no real optimization headroom, LoopBench reported +0.00% and kept the original. It refuses to ship a change it can't prove is better. That's the whole philosophy: a scientist, not a cheerleader. If a candidate can't demonstrate it's faster and still correct, it never ships — full stop.

Where it works, and where it doesn't

LoopBench works best when three things are true: the target has a measurable metric (latency, gas, throughput, memory), a correctness gate (tests, or an exit code), and a pip-installable toolchain. That covers Python and Python-hosted ecosystems like Vyper cleanly.

It's not a fit for code with no measurable signal — UI polish, prose, or logic with no tests to anchor correctness. And result quality tracks model quality: a stronger model finds more aggressive, correct rewrites; a weaker one plays it safe. Because the loop only keeps verified wins, a poor fit shows up honestly as a small or zero improvement rather than a broken patch.

Defining what "better" means

You tell LoopBench what to optimize with a small evaluator — your test file is the benchmark. It asserts correctness and prints a speed marker (LOOPBENCH_SPEED_MS), or you extract a custom metric (gas, throughput, memory) with a regex and optimize that instead. There's no separate scoring engine to learn; if your tests pass and your number improves, the patch is kept.

Do I need a frontier model?

No. LoopBench is provider-agnostic and works with off-the-shelf and free models — the 2,000× bubble-sort result above was produced by a free Groq model. Stronger models simply find bolder rewrites. You set LLM_API_BASE and LLM_MODEL and the loop does the rest.

Getting started

Install, add your key, and scaffold a job:

pip install -e .
cp .env.example .env        # add any OpenAI-compatible key (Groq, Gemini, OpenAI)

# Optimize a file that already has a timing test:
loopbench run --target . --target-file src/hotpath.py --metric latency

# Or scaffold a job to optimize someone else's repo:
loopbench init --job my_job                 # creates loopbench.yaml + test_target.py
loopbench run --config my_job/loopbench.yaml

Docker Desktop must be running — that's where the verification happens. The full 5-minute quick start is in the README.

Feedback welcome

LoopBench Optimizer is built on the excellent OpenEvolve project and is still early. Issues, ideas, and contributions on GitHub are welcome.