Code as policy

Sam's reframe: the code is the policy, and the LLM is the optimiser that rewrites it after each failure. His intuition: one failure teaches code far more than it teaches PPO, because you can watch the failure mode and patch it directly.

Trade-offs: code learns a lot from each failure, but only from failures the LLM can explain, and each attempt is expensive (minutes and dollars). PPO learns little from each failure, but its tries are nearly free, and it absorbs failures nobody can explain (contact, slip, high-dimensional limbs). Code is readable and editable; a policy is opaque. Related work: Code as Policies, Eureka (the LLM writes the reward, PPO does the rest), FunSearch/AlphaEvolve (an LLM as the optimiser over programs). Possible hybrid: code for structure, a small learned policy for the messy bits.