A Ride Down Multisensory Integration Lane

This morning I experienced a variation of the ventriloquist effect which I’ll explain soon, but first, a bit of background to the story. I’ve started riding to UQ this year, and I’m so into riding now that I just bought a new, shiny road bike, and go for long rides around the Brisbane River before getting to uni. An advantage of my road bike is that the gears shift very smoothly and easily, which makes it easier to adjust my pedaling to go really fast, or to go up hills. Every time I press the gear levers with my hands, I feel the change of the bike chain move from one cog to another in my feet, which is usually accompanied by a metal “clicking” noise. So, every time I change gears, I feel something in my feet, I hear a “click”, and the feeling and the sound always happen at exactly the same time.

On my ride this morning I got to a part of the bike path next to a road works area and I needed to change gears. I pressed the gear levers with my hands and felt the familiar gear-changing feeling in my feet. Instead of hearing a subtle “clicking” noise, however, my changing gears seemed to make an extremely loud BANG noise that sounded like a blacksmith hammering an anvil. I panicked! Not only was this a brand new, expensive bike that I might have just broken, but also I couldn’t work out what on earth would have made my gears make such a loud noise. A few seconds after I inspected my chain, gears and pedals to find that everything seemed normal, I realized the experience was just an illusion! I had experienced a variation of the ventriloquist effect.

The ventriloquist effect occurs, for example, when you watch a ventriloquist perform live, and the actor’s voice really seems to be coming from the dummy’s mouth. If you’ve never seen a live ventriloquist performance (like most of the population), then think of every time you have been to the cinemas - the same ventriloquist effect happens: the voices coming out of the speakers around the cinema seem actually to be coming from the on-screen characters’ mouths, right?

The reality of the situation is that the location of the sound source, the cinema speaker for example, is far away from the location of the vision, the image of the person talking. Despite this physical separation of vision and sound, your brain assumes they are a single event because they are happening at exactly the same time and combines them. In the literature, this is referred to as multisensory integration (because information from multiple senses is being integrated, duh). And it makes perfect sense: if there is a flash of light and a crack of sound that occur at exactly the same time, it’s not unreasonable to infer the two are related. Multisensory integration can occur for different combinations of different senses, like vision and sound, vision and touch, and touch and sound, and usually makes it easier for us to perceive and respond to things around us. It’s important to know that the “more clear” sense tends to pull the perceived location of the “less clear” sense closer to it. So when we are at the cinema, the image is so big and clear, whereas we don’t know where the speakers are, so we experience the sound as being pulled towards the vision; we perceive vision and sound as one.

Back to my bike ride! What I finally worked out had happened was that a workman working next to the bike path had dropped a metal rod onto concrete. It was entirely coincidental that the sound of the rod hitting the ground occurred at exactly the same time I felt the gears changing in my feet. Because multisensory integration occurs automatically given the right conditions, my own brain tricked me into thinking the noise I heard came from the same thing that I felt in my feet! Because I didn’t know where the sound was coming from, but I knew exactly where my feet were, the sound got pulled towards the physical feeling. I experienced a ventriloquist effect of the tactile (touch) and auditory (sound) senses. An important distinction to point out is that my experience was “instant”, and made me perceive the noise and feeling as one thing, rather than being an afterthought where I cognitively decided that the two things were related.

When I worked out that I’d had a rather unique multisensory integration experience, I felt a nerdy buzz that allowed me to maintain a constant 30km/h speed on my bike for the rest of my trip! But this experience was especially cool for me because I’ve been working on several multisensory projects for almost two years now. Professor Penelope Sanderson, Matt Thompson and I are currently finishing revisions of a paper summarizing an experiment where people wore a mini computer monitor over one of their eyes while walking around, an experiment that I conducted for my honours thesis in 2008. We showed that multisensory integration of vision and sound seems to occur less when people are walking because of the flow of background visual information that happens when you move your head around. Over last Christmas, I was working with Dr Ada Kritikos on a project investigating the role of proprioception (the sense of where your limbs are in space) in the integration of touch and vision. The project with Ada has spawned a few different follow up experiments, so I can’t write about those paradigms explicitly. Suffice to say the experiments involve fake dummy hands and strangely shaped tools!