Multisensory Integration

In sensory perception, multisensory integration is the rule, not the exception. In audition, we don’t just hear with our ears, we use our eyes to locate the apparent sources of sounds in the cinema where we “hear” the voices coming from the actors’ mouths on the screen, although the sounds are coming from the sides of the theater. This is known as the ventriloquism effect. Similarly, retronasal odors detected by olfactory receptors in the nose are experienced as tastes in the mouth. The sensations get relocated to the mouth because oral sensations of chewing or swallowing capture our attention, making us think these olfactory experiences are occurring in the same place.

Other surprising collaborations among the senses are due to cross-modal effects, whereby stimulation of one sense boosts activity in another. Looking at someone’s lips across a crowded room can improve our ability to hear what they are saying, and the smell of vanilla can make a liquid we sip “taste” sweeter and less sour. This is why we say vanilla is sweet-smelling, although sweet is a taste and pure vanilla is not sweet at all. Industrial manufacturers know about these effects and exploit them. Certain aromas in shampoos, for example, can make the hair “feel” softer; red-colored drinks “taste” sweet, whereas drinks with a light green color “taste” sour. In many of these interactions vision will dominate, but not in every case.

[...]

The more science taps into these hidden channels, the more it becomes clear that our brains are tuned to detect a shockingly small fraction of the surrounding reality. Our sensorium is enough for us to get by in our ecosystem, but it does not approximate the larger picture.

Barry C. Smith describes how our senses collaborate, our hearing with our sight to read lips and our sense of smell with taste to enhance one another.