Reinforcement learning · Sim-to-real

Teaching an AI to play a game that kills you instantly

One button, instant death, and a game that runs at 60 fps. The interesting problem was not the agent — it was refusing to train inside the thing I was training for.

Geometry Dash is one button. You jump, or you do not. Touch anything and you die instantly and restart from the beginning. That combination — trivial action space, zero tolerance, precise timing — makes it a much better reinforcement learning testbed than it has any right to be.

The catch is that the game runs at 60 frames per second, and an agent that learns inside it learns at 60 frames per second. Prior work does exactly that, and the speed of the game becomes the speed of research.

Decouple training from the thing you are training for

So I did not train in the game. I wrote a headless simulator of its physics — all seven gamemodes, slopes, gravity and speed portals — where step() is a pure function of state, action and level. It runs at about 2,500× real time, measured at 147k–163k steps per second on a single CPU core.

The retail game then becomes the evaluation target rather than the training loop. A C++ mod hooks the game’s update call, streams state out and injects input back over a binary socket. Agents observe structured state — player kinematics plus a look-ahead occupancy grid — not pixels, which is dramatically more sample-efficient and means the sim and the real game can hand the policy identical inputs.

The finding I did not expect

I ran a controlled comparison — three algorithms, five seeds, two levels, identical observations and rewards. DQN, PPO, and a genetic algorithm.

PPO, the modern default, got stuck. In four of five seeds it settled into a local optimum where it simply refused the first risky jump, because attempting it and failing cost reward. DQN and the genetic algorithm both sidestepped that trap and cleared the levels.

A five-fold increase in the entropy coefficient is what got PPO out. A single-algorithm project would have concluded the task was hard.

That is the argument for bake-offs in miniature. If I had only run PPO, the honest write-up would have been “RL struggles on this task”, and it would have been wrong. The problem was not the task. It was one hyper-parameter interacting with a reward structure that punishes exploration.

Crossing the gap to the real game

A policy trained only in the simulator, driving the retail game, reaches about 11% before dying at an obstacle where the sim and the real physics disagree. That number is the sim-to-real gap made concrete.

The standard fix is domain randomization: vary the physics constants every episode so the policy learns to tolerate a range instead of memorising a point. Trained with and without ±18% randomization across five seeds, then evaluated on held-out physics perturbations, it widens the tolerance band by 1.60× on jump strength and 1.67× on horizontal speed — and degrades gracefully where the point-estimate policy falls off a cliff.

Being precise about the full clear

The project does clear a complete official level. It is important to say exactly how, because the honest version is less impressive than the headline and more useful than a vague one.

The full clear is learning from demonstration — a recorded human run, replayed deterministically on the real game. It is not autonomous RL solving the level end to end. A policy trained by behaviour cloning on that demonstration predicts the human’s jumps at 90.7% validation accuracy, with a jump F1 of 0.82, and a checkpoint search on the real game clears the full cube section at about 35%.

Those are three different claims with three different strengths, and collapsing them into “an AI beat Geometry Dash” would be the kind of sentence that falls apart in the first follow-up question.

What transfers to work that is not a game

Build the fast simulator. Compare more than one algorithm before concluding anything about the problem. Randomize what you are uncertain about. Test the integration layer against a mock so it is not the thing that breaks at 2am. And separate what the system did from what you would like to say it did.