How Does Your Brain Combine The Five Senses?


Using information from all the senses is vital to construct a robust, rich representation of our surroundings. But with the wealth of multisensory information constantly bombarding us, how does our brain know which signals go together and thus need to be combined?

And how does it integrate such related signals?

A computational model that explains multisensory integration in humans utilizing a surprisingly simple processing unit has been proposed by scientists from the Cluster of Excellence Cognitive Interaction Technology (CITEC) at Bielefeld University and the Max Planck Institute for Biological Cybernetics.

A sudden explosion, cracking sounds and flashing lights. In a blink of an eye, you realize that sounds and lights belong together, you look down and see firecrackers on the sidewalk.

The human brain is surprisingly efficient at processing multisensory information.

Correspondence Between The Senses

However, we still do not know how it solves the seemingly simple task of deciding whether sound and light belong together or not.

“Figuring out a correspondence between the senses is by no means a trivial problem,"

says Dr. Cesare Parise, who works at CITEC in the research group Cognitive Neurosciences. Parise, who is also active at the Max-Planck Institute for Biological Cybernetics, is the lead author of the new study, together with Professor Dr. Marc Ernst.

“Despite originating from the same physical events, visual and auditory information are processed in largely independent neural pathways, and yet, with no apparent effort, we can instantly tell which signals belong together. Such a task would be challenging, even for the most advanced robots."

To understand how humans combine visual and auditory information, volunteers agreed to a perception experiment in which they observed random sequences of clicks and flashes. After each sequence, they had to report whether sound and light perceptually be-longed together, and which signal appeared first.

Statistical analyses revealed that human responses were systematically determined by the similarity (i.e., correlation) of the temporal sequences of the clicks and flashes.

“This is a very important finding,” says Prof. March Ernst, “not just because it shows that the brain uses the temporal correlation of sound and light to detect whether or not they are physically related, but also because it opens an even more intriguing question: how does the brain detect correlation across the senses?"

Multisensory Correlation Detector

To answer this question, Parise and Ernst used computational modeling and computer simulations, and identified an elementary neural mechanism that could closely replicate human perception.

Such a mechanism, called the Multisensory Correlation Detector, monitors the senses and looks for similarity (correlation) across visual and auditory signals.

If the stimuli have a similar temporal structure, the brain concludes that they belong together, and integrates the stimuli. Remarkably enough, this mechanism is surprisingly similar to the motion detectors found in the insect brain.

“This is exciting because it shows that the brain systematically exploits general-purpose processing strategies, which can be implemented across very different domains of perception where the correlation between signals is a key feature, such as the perception of visual motion, 3-D perception using binocular disparities, binaural hearing, and now multisensory processing.

Furthermore, such correlation mechanisms can be found in very different animal species, from insects to vertebrates, including humans,"

says Prof. Marc Ernst, who has just accepted a new position at Ulm University.

To further test the generalizability of this model, Parise and Ernst ran additional computer simulations, where they used the Multisensory Correlation Detector model to replicate several previous findings on the temporal and the spatial aspects of multisensory perception.

Without further changes, the same model proved capable of replicating human perception in all simulated studies, and displayed the same temporal and spatial constraints of multisensory perception found in humans.

Dr. Cesare Parise, who has just accepted a new position as research scientist at Oculus VR, says:

“Over the last decade we have discovered that the brain integrates multisensory information in a statistically optimal fashion. However, the nature of the underlying neural mechanisms has so far defied proper scientific explanation. This study marks a milestone in our understanding of human perception, as it provides for the first time a general mechanism capable of explaining a large variety of findings in multisensory perception.

This result has strong application potential. A deep understanding of multisensory processing opens new clinical perspectives for neurological syndromes that are associated with multisensory impairments, such as Autism Spectrum Disorder and Dyslexia. Moreover, our computational model could be easily implemented for use in robots and artificial perception."

Cesare V. Parise et al.
Correlation detection as a general mechanism for multisensory integration
Nature Communications (2016). DOI: 10.1038/NCOMMS11543