DeepMind found an AI learning technique also works in human brains

Ooh, a reward

Tetra Images/Getty Images

Developments in artificial intelligence often draw inspiration from how humans think, but now AI has turned the tables to teach us about how brains learn.

Will Dabney at tech firmDeepMind in London and his colleagues have found that a recent development in machine learning called distributional reinforcement learning also provides a new explanation for how the reward pathways in the brain work. These pathways govern our response to pleasurable events and are mediated by neurons that release the brain chemical dopamine.

“Dopamine in the brain is a type of surprise signal,” says Dabney. “When things turn out better than expected, more dopamine gets released.”

Advertisement

It was previously thought that these dopamine neurons all responded identically. “Kind of like a choir but where everyone’s singing the exact same note,” says Dabney.

But the team found that individual dopamine neurons actually seem to vary – each is tuned to a different level of optimism or pessimism.

Read more: DeepMind AI beats humans at deciphering damaged ancient Greek tablets

“They all end up signalling at different levels of surprise,” says Dabney. “More like a choir all singing different notes, harmonising together.”

The finding drew inspiration from a process known as distributional reinforcement learning, which is one of the techniquesAI has used to master games such asGoand Starcraft II .

At its simplest, reinforcement learning is the idea that a reward reinforces the behaviour that led to its acquisition. It requires an understanding about how a current action leads to a future reward. For example, a dog may learn the command “sit” because it is rewarded with a treat when it does so.

Previously, models of reinforcement learning in both AI and neuroscience focused on learning to predict an “average” future reward. “But this doesn’t reflect reality as we experience it,” says Dabney.

“When someone plays the lottery, for example, they expect to win or they expect to lose, but they don’t expect this halfway average outcome that doesn’t necessarily really occur,” he says.

When the future is uncertain, the possible outcomes can instead be represented as a probability distribution: some are positive, others negative. AIs that use distributional reinforcement learning algorithms are able to predict the full spectrum of possible rewards.

Read more: It’s too soon to tell if DeepMind’s medical AI will save any lives

To test whether the brain’s dopamine reward pathways also work via a distribution, the team recorded responses from individual dopamine neurons in mice. The mice were trained to perform a task and were given rewards of varying and unpredictable sizes.

The researchers found that different dopamine cells showed reliably different levels of surprise.

“Associating rewards to certain stimuli or actions is of critical importance for survival,” says Raul Vicente at University of Tartu, Estonia. “The brain cannot afford to throw away any valuable information about rewards.”

“At large scale, the study is in line with the current view that to operate efficiently the brain has to represent not only the average value of a variable but how often a variable takes different values,” says Vicente. “It is a nice example of how computational algorithms can guide us in what to look for in neural responses.”

However, adds Vicente, more research is needed to demonstrate whether the results apply to other species or regions of the brain.

Journal reference: Nature , 10.1038/s41586-019-1924-6