There is also a Transformer in the brain! The same mechanism as the 'hippocampus'

I can't create it, I don't understand —— Feiman

## If wanting to create artificial intelligence, we must first understand the human brain cause How intelligent.

With the birth of neural networks and their subsequent brilliant development, researchers have been looking for biological explanations and biological progress for neural networks. It is also inspiring AI researchers to develop new models.

But researchers in the field of artificial intelligence actually have a more ambitious pursuit: Use AI models to help understand the brain .

Recent research has found that although the most popular Transformer model was developed without any assistance from biological knowledge , but its structure is very similar to the hippocampal structure of the human brain.

##Paper link: https://arxiv.org/pdf/2112.04035.pdf

After the researchers equipped the Transformer with recursive position encoding, they found that the model could accurately replicate the spatial representation

of the hippocampal formation.

However, the author also said that he is not surprised by this result, because Transformer is closely related to the current hippocampal model

in neuroscience, the most obvious one is

Place cells(place cells) and Grid cells(grid cells). Moreover, it was found through experiments that the Transformer model has a huge performance improvement compared to the model provided by the neuroscience version.

This work combines artificial neural networks with computational brain networks to provide new understanding of the interaction between the hippocampus and cerebral cortex, And hints at how cortical areas perform a wider range of complex tasks beyond current neuroscientific models, such as language understanding.

Transformer simulates the hippocampus? It is still difficult for humans to understand their own brains. For example, studying how the brain organizes and accesses spatial information to solve "where we are, what's around the corner, and how to get there" is still a daunting task. challenges.

The entire process may involve calling entire memory networks and stored spatial data from tens of billions of neurons, each connected to thousands of other neurons.

While neuroscientists have identified several key elements, such as grid cells, neurons that map location, how to go deeper remains unknown: researchers cannot move Divide and study slices of human gray matter to see how location-based memories of images, sounds and smells flow and connect to each other.

Artificial intelligence models provide another way to understand the human brain. Over the years, neuroscientists have used various types of neural networks to simulate the firing of neurons in the brain.

Recent research shows that the hippocampus (a brain structure crucial for memory) is basically similar to the Transformer model.

The researchers used the new model to track spatial information in a way that is similar to the inner workings of the brain, and achieved some remarkable results.

James Whittington, a cognitive neuroscientist from the University of Oxford and Stanford University, said that when we know that these brain models are equivalent to Transformer, it means The new model will perform better and be easier to train.

As seen in the work of Whittington and others, Transformer can greatly improve the ability of neural network models to imitate the various calculations performed by grid cells and other parts of the brain.

Whittington says such models could advance our understanding of how artificial neural networks work and, more likely, how computations are performed in the brain.

David Ha, a computer scientist at Google Brain who is mainly engaged in Transformer model research, said that we are not trying to recreate a new brain, but can we create a mechanism to do what the brain can do? thing?

Transformer was first proposed five years ago as a new model for artificial intelligence to process natural language. It was also the secret weapon of "star models" such as BERT and GPT-3. . These models can generate convincing song lyrics, compose Shakespearean sonnets, or do some human customer service work.

The core mechanism of Transformer is self-attention, in which each input (such as a word, a pixel, a number in a sequence) is always connected to all other inputs, and other Common neural networks simply connect inputs to certain inputs.

Although Transformer was designed specifically for natural language tasks, subsequent research has also proven that Transformer also performs well in other tasks, such as classifying images, and now The brain is modeled.

In 2020, a team led by Sepp Hochreiter, a computer scientist at Johann Kepler University in Linz, Austria (the first author of the LSTM paper), used a Transformer to repurpose a powerful, long-term The existing memory retrieval model Hopfield network.

These networks, first proposed 40 years ago by Princeton physicist John Hopfield, follow a general rule: Neurons that are active at the same time establish strong connections with each other.

Hochreiter and his collaborators noted that researchers are always looking for better models of memory retrieval, and they saw how a new class of Hopfield networks retrieves memories and how Transformers perform attention. connection between forces.

These new Hopfield networks, developed by Hopfield and Dmitry Krotov of the MIT-IBM Watson Artificial Intelligence Laboratory, feature more efficient connections than standard Hopfield networks , more memories can be stored and retrieved.

Paper link: https://papers.nips.cc/paper/2016/hash/eaae339c4d89fc102edd9dbdb6a28915-Abstract.html

Hochreiter’s team upgraded these networks by adding a rule similar to the attention mechanism in Transformer.

In 2022, this new paper further adjusted Hochreiter's method and modified the Transformer so that it no longer treats memory as a linear sequence, but like a string in a sentence words, encoding them as coordinates in a high-dimensional space.

The researchers say this "twist" further improves the model's performance on neuroscience tasks. The experimental results also showed that the model is mathematically equivalent to the model of grid cell firing patterns that neuroscientists see in fMRI scans.

Grid cells have this exciting, beautiful, regular structure with striking patterns, says Caswell Barry, a neuroscientist at University College London. Too likely to appear randomly.

This new work shows how exactly the Transformer replicates those patterns observed in the hippocampus.

They also realized that the Transformer model can figure out where it is based on its previous state and how it moved, and in a key way way into the traditional grid cell model.

Other recent work also suggests that Transformers can advance our understanding of other brain functions.

Last year, computational neuroscientist Martin Schrimpf of MIT analyzed 43 different neural network models to understand their effects on human neural activity measurements reported by fMRI and electrocorticography. How predictable are the results.

He found that Transformer is currently the leading and most advanced neural network and can predict almost all changes found in imaging.

David Ha and Yujin Tang, also a computer scientist, recently designed a model that can deliberately input large amounts of data to the Transformer in a random and disorderly manner, simulating how the human body transmits data to the brain. Transmitting sensory observations. It turns out that Transformer can successfully process disordered information flows just like our brains.

##Paper link: https://arxiv.org/abs/2111.14377

Yujin Tang said that neural networks are hardwired and can only receive specific inputs. But in real life, data sets often change rapidly, and most AIs don’t have any way to adjust. In the future we would like to try an architecture that can adapt quickly.

The above is the detailed content of There is also a Transformer in the brain! The same mechanism as the 'hippocampus'. For more information, please follow other related articles on the PHP Chinese website!