Toward Evolutionary Autocurricula: Emergent Sociality from Inclusive Rewards

The competitive and cooperative forces of natural selection have driven the evolu-tion of intelligence for many millions of years, eventually culminating in nature’svast biodiversity and the complexity of our human minds. In this paper, we presenta novel multi-agent reinforcement learning framework, inspired by the process ofevolution. We assign a genotype to each agent, and propose an inclusive rewardthat optimizes for the fitness of an agent’s genes. Since an agent’s genetic materialcan be present in other agents as well, our inclusive reward also takes geneticallyrelated individuals into account. We study the effect of inclusion on the resultingsocial dynamics in two network games with prisoner’s dilemmas, and find that ourresults follow well-established principles from biology. Furthermore, we lay thefoundation for future work in a more open-ended 3D environment, where agentshave to ensure the survival of their genes in a natural world with limited resources.We hypothesize the emergence of an arms race of strategies, where each new strat-egy will be a gradual improvement in response to an earlier adaptation from otheragents, effectively creating a multi-agent autocurriculum similar to biological evo-lution. Our evolutionary rewards provide a novel social dimension that features anon-stationary spectrum of cooperation due to the finite environmental resourcesand changing population distribution. It has the potential to create increasinglyadvanced strategies, where agents learn to balance cooperative and competitiveincentives in a more complex and dynamic setup than previous works, whereagents were often confined to predefined team setups that did not entail the so-cial intricacies that biological evolution has. We argue this could be an importantcontribution towards creating advanced, general and socially intelligent agents.

Conference paper