Why gyms work

Same author, different job. Direct control requires already holding the fine-grained dynamics, at control-loop speed. Writing a gym only requires knowing what success looks like and how to set up a search: checking a solution is easier than generating one. The gym turns coarse knowledge (goals, rewards, rough physics) into a process that collects the missing information from experience. The author also borrows the knowledge built into the physics engine, and the learned policy runs a fast feedback loop the LLM can't. Deutsch framing: the gym is the criticism machine.

Correction (Sam): direct control isn't conjecture without criticism. A bike-riding agent knows when it falls. The real difference is how much criticism there is, how detailed it is, and whether it sticks. Direct control: sparse, late, whole-episode feedback ('fell over'), no way to tell which moment was wrong, a few slow real-world tries, and lessons that live only in the context window. Gym: millions of cheap tries, detailed rewards the author designed, and corrections baked into weights.

Is it just feedback latency? Partly. Three factors: latency/volume (how fast and how often feedback arrives), resolution (whether feedback points to the moment that went wrong), retention (whether the lesson persists in weights vs context).