
An illustrative composite of a humanoid robot outside a Google office. The DARPA logo on the robot has no connection to DeepMind's research.
In February 2017, headlines claimed that Google's artificial intelligence had turned "highly aggressive" and learned betrayal. The real story was more interesting and less frightening. Researchers at DeepMind, the London lab owned by Google's parent company, had placed simple learning programs in two small video-game worlds to see when they would compete and when they would cooperate. In a game about gathering apples, the agents attacked each other more as apples ran short. In a game about hunting together, the smarter agents became better team players.
The conclusion of the study was not that AI is naturally violent. It was that behaviour depends on the rules and rewards of the environment, which is exactly what makes it useful for studying cooperation in people, economies and future AI systems.
What DeepMind published in 2017
DeepMind described the work in a blog post titled "Understanding Agent Cooperation" on 2 February 2017. The full paper, "Multi-agent Reinforcement Learning in Sequential Social Dilemmas", was posted on arXiv a week later by Joel Z. Leibo, Vinicius Zambaldi, Marc Lanctot, Janusz Marecki and Thore Graepel.
The research question is an old one from economics and biology: why do self-interested individuals cooperate at all? The team noted in the blog post that "self-interested people often work together to achieve great things", and they wanted a way to study that with agents that learn from experience rather than follow fixed rules.
Each agent was an independent deep reinforcement learning program, a neural network that tries actions, receives points as rewards and gradually learns which behaviour earns the most. Crucially, the agents were not programmed to fight or to help. Whatever they did, they learned on their own by chasing points.
From the prisoner's dilemma to sequential social dilemmas
The classic tool for studying cooperation is the prisoner's dilemma. Two suspects are questioned separately. If both stay silent, each gets a short sentence. If one testifies against the other, the informer goes free and the partner gets a long sentence. If both testify, both get a medium sentence. Each prisoner does better by betraying, whatever the other does, yet if both reason that way they end up worse off than if both had kept quiet.
The limitation, as the DeepMind team pointed out, is that this treats cooperation as a single button press. In real life, cooperating or competing takes a sequence of actions, skills and timing. They called games that keep the same tension but play out over time "sequential social dilemmas", and built two of them.
The Gathering game: scarcity changes behaviour
In Gathering, two agents, one red and one blue, move around a small grid world collecting green apples, each worth a point. Apples grow back after a while. Each agent can also fire a beam at the other. Being hit twice removes the target from the game for a short time. Tagging earns nothing by itself; its only benefit is that the tagger can collect apples alone while the rival is out.
The researchers changed how quickly apples grew back and how many there were. When apples were plentiful, the agents learned to share the space peacefully, because there was no reason to waste time firing. As apples became scarce, the beams came out more often. In the blog post's words, "the agents learn that it may be better for them to tag the other agent".
A second finding drew most of the headlines. Agents built with larger neural networks, meaning more capacity to learn complex strategies, tagged more often at every level of scarcity. Tracking and hitting a moving opponent is a harder skill than simply collecting apples, so the more capable agents were more able to use it.
The Wolfpack game: smarter agents cooperated more
Wolfpack was designed to reward teamwork. Two agents play wolves chasing a single prey around a map with obstacles. When the prey is caught, every wolf close to the capture shares the reward, and the reward grows when the wolves are together. A lone wolf risks losing the carcass to scavengers, in the story behind the game.
Here the result flipped. Greater capacity led to more cooperation, because coordinating a hunt with a partner is the complex behaviour in this game, and only the bigger networks could learn it reliably. As the blog post put it, "greater capacity to implement complex strategies leads to more cooperation between agents".
Put the two games side by side and the message is clear. Intelligence did not push the agents towards aggression or towards kindness. It made them better at whatever their environment rewarded.
Why the "highly aggressive AI" headlines missed the point
Much of the 2017 coverage was based on a single striking chart and phrases such as "highly aggressive" and "betrayal". Those words did not come from a finding that the software had feelings or intentions. The agents were small programs, moving coloured squares in a grid, and their beams did nothing more than pause a rival for a few moments.
Some common misunderstandings are worth correcting:
- Myth: the AI chose violence because it is like us. The agents had no emotions; they found a way to score more points when resources were limited, which is what reinforcement learning is built to do.
- Myth: smarter AI is more dangerous. In Wolfpack, smarter agents were more cooperative. Capability amplified whatever the game rewarded.
- Myth: this was a robot experiment. No robots were involved. The tests took place entirely inside simple simulations.
The more reasonable concern is about design. If learning systems will pursue whatever rewards they are given, the people designing those rewards and environments carry real responsibility. That is one reason AI safety researchers study incentives so closely.
What the research is useful for
The authors saw their agents as a rough model of the "rational agent" that economists use in theory. Simulations like these let researchers test how rules, incentives and scarcity shape behaviour before trying them on real systems. DeepMind suggested this could help in understanding and managing complex systems such as economies, traffic and environmental health, where many independent decision-makers share limited resources.
The work also became a foundation for a larger field. DeepMind and other labs went on to build more elaborate test worlds for multi-agent cooperation, studying problems such as shared resources, fairness and how learning agents can be guided to cooperate.
The fair takeaway from 2017 is a quiet one: give learning systems a world where cooperation pays, and capable agents will cooperate. Give them a world of scarcity where sabotage pays, and they will learn that too. That lesson applies to the people who design those worlds as much as to the machines inside them. If you are interested in how fears about AI spread online, see our piece on the Philip K. Dick android's "people zoo" remark.
Sources: Google DeepMind: Understanding agent cooperation (2 February 2017); Leibo et al., Multi-agent Reinforcement Learning in Sequential Social Dilemmas (arXiv, 2017); early coverage by ExtremeTech and The Mind Unleashed.
Related:- Self-Learning Robot Escapes Testing Ground and Goes Missing for 45 Minutes
- Philip K. Dick Android's People Zoo Remark Went Viral
- Elon Musk Funds $1B Project To Prevent Artificial Intelligence From Destroying Mankind
- Stephen Hawking and 1,000 Experts Sign Letter Warning Of Weaponized Robots
- These 3 Things Could Destroy The Human Civilization, Says Stephen Hawking

COMMENTS