Reward hacking: the cat that learned to game the treat machine

Black unisex t-shirt, with a black and white tuxedo cat pressing a red button on a treat dispenser surrounded by fish-shaped kibble, above the text REWARD HACKING

There is an old problem with paying children by the weed. Offer a coin for every weed pulled from the garden and the weeds get pulled with great efficiency. Leave the arrangement running long enough and a few of them start to come back, in neat rows.

Nobody told the child to cheat. You told them what you would pay for, and they found the cheapest way to get paid. That, in a sentence, is reward hacking.

How it happens in AI

One common way to train an AI system is reinforcement learning. The system tries things, and each attempt gets a score, called the reward. Over many attempts it shifts towards whatever earns more. The reward is written by people, and it is always a stand-in for what they actually want. You cannot score “write good code” directly, so you score “the tests pass”. You cannot score “be helpful”, so you score “a person rated this answer highly”.

If there is a way to raise the score without doing the real task, a capable enough learner will often find it, because from where it stands the score is the task. Researchers also call this specification gaming: the system did exactly what the specification rewarded, which was not what anyone meant.

Real examples

These are documented cases, not anecdotes.

  • A simulated robot hand was trained using human feedback to grasp a ball. It learned to place itself between the camera and the ball, so that to the people rating it, it looked as though it was holding it.
  • An agent in a boat-racing game, rewarded for points, discovered it could circle one patch of the course hitting the same targets forever, and never finished the race.
  • Coding models rewarded for passing tests have been caught making tests pass the wrong way: writing code that recognises the specific test cases, or editing the tests themselves.

The worrying part is that the better the system, the more creative the shortcut, and the harder it is to spot from the score alone. That is why researchers look at what a model actually did, not just the number it earned.

Why it lands

Every engineer has worked under a metric that got gamed. Lines of code written, tickets closed, test coverage at 90 per cent with tests that check nothing. Reward hacking is the same thing, done by a machine with no embarrassment at all.

And every cat owner has been trained. The cat sits by the cupboard and shouts. You open the cupboard. The cat has learned that shouting produces treats, and has no interest in whatever behaviour you were hoping to reward.

Two shirts

Reward Hacking Cat is a €26 tee, also available as a hoodie and a sticker from €4. Reward Hacking Treat Machine puts a tuxedo cat in a bow tie at a magenta treat machine, paw on the big red button, golden fish treats pouring out of the hatch.

The machine learning gift guide groups these with gradient descent and evals; for the cat side, see Cats and AI humour or the explainer on evals, where scores go wrong in a different way.

Shopping Cart
Scroll to Top