Have you ever noticed that, in difficult listening conditions, you can often better understand the person talking to you if you can see the talker? This is a common occurrence and is typically referred to as lipreading or speechreading. Speechreading benefits all sighted people, including those with good hearing and those with profound hearing losses, because of the relationship between lip movements and the speech signals received by our ears. Speechreading can be extremely useful for persons with substantial hearing losses, as it has been shown to greatly improve speech understanding. To develop this skill, people possess a neural system capable of combining information received by the ears and the eyes.
Most of the time, the way the brain combines information from our ears and eyes is helpful in understanding what is happening around us. Sometimes, however, this ability of our brain to combine this information can be used to play tricks on us. Auditory illusions induced by conflicting signals received by the eyes and ears further illustrate the powerful interactions between the auditory and visual systems. Ventriloquism, in which a voice is heard as coming from a wooden dummy’s mouth, represents one of these convincing illusions.
This course will review the effects of visual information on auditory sensation, with special emphasis on the particular aspects of sound and visual images that are useful for communication. Students will learn the neural mechanisms that underlie the combination of sight and hearing and how illusions, such as ventriloquism, are generated. Multi-modal neural representation in hearing and sighted people will be presented. The impact of deafness and blindness on the typical or normal neural representations of sound and visual images in the brain will also be discussed. The course will also include presentations on the nature of the benefits of speechreading for the deaf and hard of hearing.