Tuesday, November 3, 2009

A Ride Down Multisensory Integration Lane

This morning I experienced a variation of the ventriloquist effect which I’ll explain soon, but first, a bit of background to the story. I’ve started riding to UQ this year, and I’m so into riding now that I just bought a new, shiny road bike, and go for long rides around the Brisbane River before getting to uni. An advantage of my road bike is that the gears shift very smoothly and easily, which makes it easier to adjust my pedaling to go really fast, or to go up hills. Every time I press the gear levers with my hands, I feel the change of the bike chain move from one cog to another in my feet, which is usually accompanied by a metal “clicking” noise. So, every time I change gears, I feel something in my feet, I hear a “click”, and the feeling and the sound always happen at exactly the same time.

On my ride this morning I got to a part of the bike path next to a road works area and I needed to change gears. I pressed the gear levers with my hands and felt the familiar gear-changing feeling in my feet. Instead of hearing a subtle “clicking” noise, however, my changing gears seemed to make an extremely loud BANG noise that sounded like a blacksmith hammering an anvil. I panicked! Not only was this a brand new, expensive bike that I might have just broken, but also I couldn’t work out what on earth would have made my gears make such a loud noise. A few seconds after I inspected my chain, gears and pedals to find that everything seemed normal, I realized the experience was just an illusion! I had experienced a variation of the ventriloquist effect.

The ventriloquist effect occurs, for example, when you watch a ventriloquist perform live, and the actor’s voice really seems to be coming from the dummy’s mouth. If you’ve never seen a live ventriloquist performance (like most of the population), then think of every time you have been to the cinemas - the same ventriloquist effect happens: the voices coming out of the speakers around the cinema seem actually to be coming from the on-screen characters’ mouths, right?

The reality of the situation is that the location of the sound source, the cinema speaker for example, is far away from the location of the vision, the image of the person talking. Despite this physical separation of vision and sound, your brain assumes they are a single event because they are happening at exactly the same time and combines them. In the literature, this is referred to as multisensory integration (because information from multiple senses is being integrated, duh). And it makes perfect sense: if there is a flash of light and a crack of sound that occur at exactly the same time, it’s not unreasonable to infer the two are related. Multisensory integration can occur for different combinations of different senses, like vision and sound, vision and touch, and touch and sound, and usually makes it easier for us to perceive and respond to things around us. It’s important to know that the “more clear” sense tends to pull the perceived location of the “less clear” sense closer to it. So when we are at the cinema, the image is so big and clear, whereas we don’t know where the speakers are, so we experience the sound as being pulled towards the vision; we perceive vision and sound as one.

Back to my bike ride! What I finally worked out had happened was that a workman working next to the bike path had dropped a metal rod onto concrete. It was entirely coincidental that the sound of the rod hitting the ground occurred at exactly the same time I felt the gears changing in my feet. Because multisensory integration occurs automatically given the right conditions, my own brain tricked me into thinking the noise I heard came from the same thing that I felt in my feet! Because I didn’t know where the sound was coming from, but I knew exactly where my feet were, the sound got pulled towards the physical feeling. I experienced a ventriloquist effect of the tactile (touch) and auditory (sound) senses. An important distinction to point out is that my experience was “instant”, and made me perceive the noise and feeling as one thing, rather than being an afterthought where I cognitively decided that the two things were related.

When I worked out that I’d had a rather unique multisensory integration experience, I felt a nerdy buzz that allowed me to maintain a constant 30km/h speed on my bike for the rest of my trip! But this experience was especially cool for me because I’ve been working on several multisensory projects for almost two years now. Professor Penelope Sanderson, Matt Thompson and I are currently finishing revisions of a paper summarizing an experiment where people wore a mini computer monitor over one of their eyes while walking around, an experiment that I conducted for my honours thesis in 2008. We showed that multisensory integration of vision and sound seems to occur less when people are walking because of the flow of background visual information that happens when you move your head around. Over last Christmas, I was working with Dr Ada Kritikos on a project investigating the role of proprioception (the sense of where your limbs are in space) in the integration of touch and vision. The project with Ada has spawned a few different follow up experiments, so I can’t write about those paradigms explicitly. Suffice to say the experiments involve fake dummy hands and strangely shaped tools!


  1. Just as a quick note on this post, as we get permission to link to publications we'll be sure to post them up with the relevant stories!

  2. I think you should pay more attention to the relationship between the illusion you experienced and the ventriloquist effect. The ventriloquist effect involves misattributing sounds to the location of a visual stimulus. As you say, the ventriloquist effect is a multisensory effect. If your "gear-bang" illusion is genuinely multisensory, you might be misattributing the bang (what was this bang?) to the location of the tactile stimulus ("familiar gear-chagining feeling in my feet"). Has a tactile ventriloquist effect been demonstrated? I'm too lazy to investigate.

    I think your illusion was unimodal - auditory mislocalisation - arising via expectations/task set. The clue is that you had to look around to see where the sound was coming from. Imagine the scene. You're thinking about your bike, digging its ride, thinking how great it is to have a road bike, not like those other chumps on mountain bikes. Suddenly, you hear a "bang!". Initially, you interpret the sound as coming from the gears, etc. You temporarily fear for the bike's health. You look down, expecting the worst. When your gears etc. are intact, you substitute your original erroneous interpretation with another. The ventriloquist effect cannot be dispelled so easily.

  3. Hi Phineas, you might have missed this part:
    "An important distinction to point out is that my experience was “instant”, and made me perceive the noise and feeling as one thing, rather than being an afterthought where I cognitively decided that the two things were related."

    That is, the sound genuinely seemed to have come from the gears; I heard the sound at my gears.

    Auditory-tactile integration is well established. For Example:
    Sequences of auditory beeps and tactile taps were simultaneously presented and participants were instructed to focus on one of these modalities and to ignore the other. We tested whether (i) the two sensory channels bias one another and (ii) the interaction depends on the relative reliability of the channels. Audition biased tactile perception and touch biased auditory perception. Lowering the reliability of the auditory channel (i.e. the intensity of the beeps) decreased the effect of audition on touch and increased the effect of touch on audition. These results show that simultaneous auditory and tactile stimuli tend to be automatically integrated in a reliability-dependent manner.

    Please let me know if the link doesn't work.

    Regardless of whether or not you're as convinced by the story as I was by the illusion, I hope it's a nice illustration of multisensory integration.