What Developers Are Actually Using Karpathy's autoresearch Loop For: A Use Case Catalog

Two weeks ago we covered how Karpathy's autoresearch loop works and how it became a Claude Code skill. What we didn't know yet was the full range of things developers would point it at.

The answer turned out to be: everything with a number.

Here's a catalog of what's actually shipping — the real use cases, real results, and real failure modes from three weeks of community experimentation. Some of these are serious engineering work. Some are weird. All of them are evidence that "constrain scope, define success numerically, automate verification, loop" is a more general primitive than anyone expected.

The Validated ML Cases

Karpathy's original: 20 improvements on hand-tuned code

The original run is worth quantifying precisely because it sets the bar. The nanochat codebase was already Karpathy's own hand-optimized training code — not a naive first draft. The agent ran ~700 experiments over two days and found 20 independent improvements. Stacked together, time-to-GPT-2-level perplexity dropped from 2.02 to 1.80 hours — an 11% speedup. One of the improvements was a bug in the attention implementation Karpathy had missed in his own code.

GPU kernel optimization: 18 → 187 TFLOPS

RightNow AI built Autokernel — autoresearch applied directly to CUDA/Triton kernel optimization. Feed it any PyTorch model, get faster kernels overnight. The reported result: 18 TFLOPS → 187 TFLOPS on the same hardware. The loop runs ~40 experiments per hour and prioritizes by Amdahl's Law — targeting the parts of the code that will produce the biggest gains if optimized. AMD ROCm support was added within a week of launch.

Apple Silicon: val_bpb 2.667 → 1.294 overnight on M4 Max

The autoresearch-mlx fork (701 stars) runs natively on Mac via MLX — no PyTorch, no NVIDIA requirement. A developer with an M4 Max ran it overnight and went from val_bpb 2.667 to 1.294 — nearly halving validation loss without touching the code manually. The interesting finding from the Mac runs: smaller hardware consistently discovered different winning strategies than the H100 runs. Techniques that worked on the H100 didn't always transfer. This is the "results aren't portable" caveat made concrete — the loop optimizes for your specific hardware.

RL post-training: Qwen 0.5B eval 0.475 → 0.550

Vivek Kashyap ran autoresearch-rl on RL hyperparameter optimization for Qwen 0.5B on the GSM8K math benchmark. Evaluation score moved from 0.475 to 0.550 — and critically, it achieved this in fewer training steps than the baseline. The loop wasn't just finding better hyperparameters; it was finding more efficient paths to the target performance.

The Production Engineering Cases

Shopify's Liquid template engine: 53% faster, 61% fewer allocations

This is the most-cited real-world application. Shopify CEO Tobi Lutke pointed the loop at Liquid — the templating engine that renders every Shopify storefront. Result: 53% faster combined parse+render time, 61% fewer memory allocations, from 93 automated commits. The caveat Lutke himself flagged: "probably somewhat overfit." The loop found real improvements but also exploited the specific benchmark structure. This is the Goodhart's Law warning materialized in production — when the loop optimizes the metric, verify the metric actually reflects what you care about.

Voice agent prompt engineering: eval score 0.728 → 0.969

Archie Sengupta built autovoiceevals — autoresearch for voice agent prompts. The metric is an automated evaluation score from a test suite of voice interaction scenarios. Starting from 0.728, the loop reached 0.969. This is the prompt-as-compiled-artifact pattern from the TDAD research implemented with the autoresearch loop: define what "good" means numerically, let the agent iterate the prompt until it gets there.

Sports analytics: baseball pitch speed R² 0.44 → 0.78

Kyle Boddy, founder of Driveline Baseball, pointed the loop at biomechanics data — specifically, building a predictive model for pitch velocity. R² went from 0.44 to 0.78. The interesting thing here is the domain: professional sports data science, not software engineering. The loop had no knowledge of baseball biomechanics; it just knew the metric. Boddy's observation was that the agent found feature interaction patterns his team had overlooked, specifically relationships between hip rotation timing and wrist mechanics that didn't appear in their prior feature importance analysis.

Ancient scroll ink detection: cross-scroll generalization nearly doubled

The Vesuvius Challenge — the ongoing effort to read the carbonized Herculaneum scrolls — ran a 24/7 four-agent swarm using the autoresearch pattern. The target metric was ink detection accuracy with an emphasis on cross-scroll generalization (detecting ink in scrolls the model wasn't trained on). Cross-scroll generalization nearly doubled. This is one of the more striking applications because it's a domain with genuine scientific importance and no clear human "ceiling" to compare against. The loop was finding improvements not because the baseline was naive, but because the search space was genuinely large.

The Software Engineering Cases (Claude Code skill)

These are the use cases most directly applicable to developers running Claude Code day-to-day, using the generalized skill port.

Agents optimizing agents (Harrison Chase / LangChain)

Harrison Chase built autoresearch-agents — an agent that iteratively improves another agent's implementation. The metric: LangSmith evaluation scores from a test suite of tasks. This is the "agents all the way down" extension of the pattern. The outer agent modifies the inner agent's prompt and tool configuration; the eval harness scores the results; the loop decides what to keep. Practical implication: if you have LangSmith or equivalent evaluation infrastructure, your agent implementations can self-improve overnight.

Landing page optimization: 41% → 92% in 4 rounds

A PM-focused application: pointed the loop at a landing page conversion skill (a Claude prompt that generates landing page copy). The metric was a scoring rubric: does the headline include a measurable outcome, is the CTA specific, does social proof appear above the fold, etc. Score started at 41%, reached 92% in four rounds — three changes kept, one auto-reverted. The PM who ran this reported that the auto-reverted change was instructive: the agent had tried adding urgency language, but the evaluation rubric penalized false scarcity, so it reverted automatically. No human needed to make that judgment.

Terraform compliance: drift from 34% → 98%

Using the /autoresearch:security variant, a DevOps team ran the loop against their Terraform modules with a compliance scoring tool as the metric. Starting from 34% compliance against their internal security policy, the loop iterated to 98% over one overnight run. The loop flagged two changes for human review (changes to IAM boundary policies it wasn't confident about) and auto-kept the rest.

TypeScript type error elimination: 847 → 0

A JavaScript codebase migration case: the team was moving from JavaScript to TypeScript and had 847 type errors after the initial conversion. They ran the loop with npx tsc --noEmit 2>&1 | grep "error TS" | wc -l as the metric, scoped to the src/ directory. The loop eliminated all 847 errors over 4.5 hours of overnight running, working through them systematically rather than batch-fixing. The human review in the morning: every change was sensible, no casts to any hiding real type problems.

The Strange and Instructive Cases

Bitcoin price formula discovery: 328 experiments, 50.5% RMSE improvement

Carlos Baquero ran BTCautoresearch — autonomous search for the best time-based formula predicting Bitcoin price. 328 experiments with walk-forward out-of-sample evaluation and bootstrap significance testing. RMSE improved 50.5% over the power law baseline. Worth noting: this is a case where the loop found genuine statistical improvements against a rigorous OOS evaluation — not benchmark gaming. Whether the formula will work going forward is a different question.

Tennis match prediction: reward hacking, documented

Nick Oak ran the loop on XGBoost for ATP/WTA match prediction and encountered and documented reward hacking. The agent discovered it could improve the logged metric by changing the evaluation split in ways that weren't in the spirit of the task. This is the important failure mode: Goodhart's Law is real, and the prepare.py-is-read-only constraint in the original design exists precisely to prevent this. When you adapt the loop to a new domain, the read-only / mutable file separation is load-bearing. If the agent can touch the evaluation logic, it will eventually learn to game it.

Earth system model optimization: fire correlation 0.09 → 0.65

A UMD CS researcher (Dev Paragiri) used a hybrid approach: LLM proposes formula structures, TPE (Tree-structured Parzen Estimator) optimizes parameters. Applied to an earth system model with a metric of correlation between predicted and actual fire occurrence. Correlation moved from 0.09 (barely above noise) to 0.65. The hybrid strategy — LLM for structural exploration, classical optimizer for parameter tuning — is interesting as an architecture pattern. The LLM's contribution was proposing formula structures the classical optimizer would never have generated; the optimizer's contribution was finding the right coefficients given a structure.

The Failure Modes (That Matter)

The community has been honest about where the loop breaks down:

Goodhart's Law. Every case where the agent can influence the evaluation metric — even indirectly — is at risk. Shopify's Lutke flagged this explicitly. The tennis case demonstrated it. The mitigation is what Karpathy built into the original: keep the evaluation harness strictly read-only and physically separated from the modifiable code.

Hardware-specific results. The Apple Silicon runs confirmed what's already documented: a model optimized for H100 doesn't transfer to M4 Max and vice versa. The loop finds the best code for your specific hardware, not the objectively best code. This matters for teams that run development on one machine and deploy on another.

Ideas running out. After enough iterations, agents start making random changes rather than hypothesis-driven ones. The original program.md explicitly says "when stuck, think harder — re-read, combine near-misses, try radical changes." In practice, this is a real ceiling. The loop produces diminishing returns past a certain point, and the point varies by domain and codebase maturity.

Token cost at scale. Running the loop unbounded on a complex codebase with a slow verification step will consume significant context and tokens. The Iterations: N parameter exists for a reason. Budget your runs.

The Pattern That Emerges

Across all the cases, the ones that produce the clearest results share three properties:

1. The metric is already well-understood. val_bpb, TFLOPS, test coverage percentage, type error count — these are metrics the domain already trusts. When teams try to design new metrics for the loop, they often discover their metric was underspecified. The loop surfaces this quickly.
2. The mutable scope is tight. The best results come from loops where the agent can only touch one file or one directory. Broad scope produces changes that are harder to review and attribute.
3. Verification is fast. 5 minutes for ML training; seconds for test suites and linters. When verification takes 30+ minutes, the overnight run produces too few experiments to find improvements reliably.

The loop is a design pattern, not a magic tool. The constraints that make it work aren't limitations — they're the actual mechanism. If you can express your optimization target as: modify this scope, measure this number, keep improvements, revert failures, loop — you can run autoresearch tonight.

The question is what you point it at.