Karpathy's AutoResearch loop uses three files: train.py (the only file the agent may edit), prepare.py (the evaluator, off-limits to the agent), and program.md (instructions). Because the agent cannot modify the scoring code, it cannot cheat by making the test easier. It must actually improve the model to pass. Run for two days against a model Karpathy had already hand-tuned over two decades, the loop ran 700 experiments and found 20 improvements he had missed, including a subtle attention-scaling fix. Shopify's CEO ran it overnight on an internal model and woke up to a 19% quality gain in a model half the previous size. The generalizable rule: any autonomous loop needs a verifier the loop cannot rewrite. Give an agent write access to its own grading criteria and it optimizes the grade, not the goal. That is the same failure mode as a student who can edit the answer key, or a CI pipeline where a pull request can modify its own tests. ## Cross-Domain Connections - [[A Hill-Climbing Loop Rewrites the Agent's Own Prompt From Run Traces]]: sibling, the rewrite loop is only safe if it can touch the agent's prompt but never the verifier scoring its output. - [[Bilevel Loops Beat Single Loops Through Architecture Not Model Intelligence]]: sibling, once the base loop is verifier-safe, a second loop watching it can multiply the gains further. - [[A Loop Earns Its Cost Only When Four Conditions Hold]]: sibling, automated verification is one of the four gating conditions for building a loop at all. ## Source - [[Loop Engineering: The Karpathy Method - and the workflow that just made it 5x better]] — codila (X), 2026-07-04 — https://x.com/0xcodila/status/2072329149520232639/?s=12&rw_tt_thread=True