Compare Methods at the Same Simulated Wall-Clock
Pick a set of methods and drag the master bar. Every tile advances on the same simulated clock, so you can see who's still thinking when others have already acted.
Each tile plays one episode of Crafter, a 2D open-world survival game with 22 achievements (collect wood, make a pickaxe, defeat a zombie…). The ★ badge on each method is how many of the 22 that run unlocked; the reward curve above tracks them accumulating over time.
How can we compare these fairly? SpeedRunner executes its skills as programs — deterministic sequences of actions that run without an LLM call per step. OPO and the other baselines still call the LLM at every step. To put them on one clock, we charge a fixed simulated time per action: 5 s per LLM call and 0.5 s per env step(real env steps are ~1.8 ms; we slow them so each primitive is visible, and real LLM round-trips average ~3 s but vary widely). A method running a program races through its env steps; a method thinking at every step pays 5 s each time. So when one tile is still “thinking” while another has already acted, you're seeing the cost difference — not an artifact of the visualization.
methods
# of training rollouts
speed
00:00
/ 18:28
209 LLM calls
performance
achievements unlocked / 22
Speedrunner0·0 steps
ASI0·0 steps
Voyager0·0 steps
OPO (text manual)0·0 steps
ReAct (no library)0·0 steps
Speedrunner★ 8/22codebook
press ▶ to start the run
0 LLM·0 env·0/22 ach
8 fns · 9,406 tok
OPO (text manual)★ 5/22text manual
- Reliable opener: get 2-3 wood, then place_table, then craft immediately. If you want both wood tools, gather enough wood before placing the table; do not assume 2 wood covers table + both crafts. - Front-tile rule: `do()` only works on the tile directly ahead. If it changes nothing, use at most 1 reposition move, then re-observe before retrying. - Stop rule: if 2 consecutive `do()` calls on the same target still change nothing, abandon that target and switch to a new nearby one. - Anti-idle rule: never chain `noop()`. If uncertain, call `get_current_observation()`; if tired, use `sleep()` instead of noop. - Survival override: the moment food or drink is not full, pivot immediately to the nearest safe source. Water first when drink drops; cows are the best early food if directly ahead or reachable in 1-2 moves. - Hard danger rule: if health drops at all, stop crafting/gathering plans and avoid hostiles/lava. Do not keep working near zombies/skeletons while injured or when they block the route. - Hostile handling: only attack zombies/skeletons when directly ahead and you can afford misses. If a hostile is adjacent but not in front, reposition away first. Avoid skeletons unless there is no safer route. - Movement discipline: use at most 1-2 moves to align with a target. If it is still not directly ahead, re-observe instead of wandering. - Crafting rule: craft only when the table is directly in front and ingredients are already in inventory. After each craft, verify inventory changed; if not, reposition or choose a different resource. - Resource priorities: wood -> table -> wood tools -> water/cow whenever any status drops -> stone/coal/iron only when safe and nearby. Prefer the closest visible high-value safe resource. - Success pattern: a compact opening works best—collect nearby wood, place table, craft right away, then switch to survival resources as soon as any stat starts dropping. - Failure pattern: long idle/noop stretches caused food/drink collapse and deaths. When status drops, act immediately; do not wait for perfect conditions.
0 LLM·0 env·0/22 ach
manual · 519 tok