1. © 1998 Peter B.L. Meijer On cross-modal sensory streams... The vOICe by Peter B.L. Meijer The vOICe by Peter B.L. Meijer Philips Research Laboratories Peter.B.L.Meijer@philips.com Invited Panel Presentation at SIGGRAPH '98, Orlando, Florida

2. © 1998 Peter B.L. Meijer The vOICe: “Seeing with your Ears” Basic facts: Blind people cannot see... but usually they can hearBlind people cannot see... but usually they can hear Multimedia computing allows audio-visual transformationsMultimedia computing allows audio-visual transformations Basic facts: Blind people cannot see... but usually they can hearBlind people cannot see... but usually they can hear Multimedia computing allows audio-visual transformationsMultimedia computing allows audio-visual transformations

3. © 1998 Peter B.L. Meijer The vOICe: “Seeing with your Ears” Basic facts: Blind people cannot see... but usually they can hearBlind people cannot see... but usually they can hear Multimedia computing allows audio-visual transformationsMultimedia computing allows audio-visual transformations Basic facts: Blind people cannot see... but usually they can hearBlind people cannot see... but usually they can hear Multimedia computing allows audio-visual transformationsMultimedia computing allows audio-visual transformations VideoAudio Mental Video VideoAudio The vOICe Human brain So translate video to audio!?

4. © 1998 Peter B.L. Meijer The vOICe implementations HardwareprototypeHardwareprototype Java applet Windows95/98/NTWindows95/98/NT http://www.seeingwithsound.com/

5. © 1998 Peter B.L. Meijer The vOICe image-to-sound mapping Sum of sinusoids sinusoids Frequency Time Amplitude

6. © 1998 Peter B.L. Meijer The vOICe image-to-sound mapping Frequenc y Time Spectrographic synthesis

7. © 1998 Peter B.L. Meijer Could this really work? Check: map sounds back to images Could this really work? Check: map sounds back to images The vOICe image-to-sound mapping Yes! (at least technically) TheoryTheory

8. © 1998 Peter B.L. Meijer The vOICe image reconstruction “Soundscape”“Soundscape” The vOICe SpectrogramSpectrogram

9. © 1998 Peter B.L. Meijer The vOICe image reconstruction “Soundscape”

10. © 1998 Peter B.L. Meijer The vOICe and human speech Pronouncing one image per second Pronouncing one image per second

11. © 1998 Peter B.L. Meijer The vOICe mental imagery? Representation may dramatically affect recognition! Representation may dramatically affect recognition!

12. © 1998 Peter B.L. Meijer The vOICe limitations Issues: 3Frequency-Time uncertainty Physiology / Perception:Physiology / Perception: 3just-noticeable difference (JND) 3critical bands (human cochlea) Issues: 3Frequency-Time uncertainty Physiology / Perception:Physiology / Perception: 3just-noticeable difference (JND) 3critical bands (human cochlea)

13. © 1998 Peter B.L. Meijer The vOICe limitations Issues: 3Frequency-Time uncertainty Physiology / Perception:Physiology / Perception: 3just-noticeable difference (JND) 3critical bands (human cochlea) ?auditory streaming/segregation ?neural pathways & bandwidth ?neural processing & plasticity Issues: 3Frequency-Time uncertainty Physiology / Perception:Physiology / Perception: 3just-noticeable difference (JND) 3critical bands (human cochlea) ?auditory streaming/segregation ?neural pathways & bandwidth ?neural processing & plasticity

14. © 1998 Peter B.L. Meijer The vOICe limitations Issues: 3Frequency-Time uncertainty Physiology / Perception:Physiology / Perception: 3just-noticeable difference (JND) 3critical bands (human cochlea) ?auditory streaming/segregation ?neural pathways & bandwidth ?neural processing & plasticity Issues: 3Frequency-Time uncertainty Physiology / Perception:Physiology / Perception: 3just-noticeable difference (JND) 3critical bands (human cochlea) ?auditory streaming/segregation ?neural pathways & bandwidth ?neural processing & plasticity Psychology / Education:Psychology / Education: ?sounds (too) unpleasant ?minimum required results ?acceptable training effort ?motivation for learning ?training programs Psychology / Education:Psychology / Education: ?sounds (too) unpleasant ?minimum required results ?acceptable training effort ?motivation for learning ?training programs

15. © 1998 Peter B.L. Meijer The vOICe approach 3needs more research 3is cheap technology 3is non-invasive The vOICe approach 3needs more research 3is cheap technology 3is non-invasive Conclusions

16. © 1998 Peter B.L. Meijer The vOICe approach 3needs more research 3is cheap technology 3is non-invasive 3is available Now! The vOICe approach 3needs more research 3is cheap technology 3is non-invasive 3is available Now! Conclusions http://www.seeingwithsound.com/