Changing qualia, acquiring new qualia Class 4 Changing qaulia between sight and sound Re-wired ferret Soundscape New qualia – magnetic north

Re-wiring the brain: Seeing the sounds? Ferret

Re-wiring the Cortex removing ascending auditory projections causes retinal fibers to innervate the MGN.

Orientation selectivity in A1 Orientation selectivity in the re-wired ferret A1 is similar to normal V1. In the MGN there is little selectivity, it arises in A1

Optical imaging of orientation maps

Orientation maps in re-wired A1 Orientation maps, with pinwheels Black = Vertical

Re-wiring half the brain (left hemisphere) The right visual field goes to the auditory cortex (left hemisphere)

Visual Response with the Auditory Cortex

Re-wiring half the brain (left hemisphere) The right visual field goes to the auditory cortex (left hemisphere)

Input to the auditory cortex only Lesion to the LGN as well, vision can only go to the auditory cortex

First the LGN was removed Second – the auditory cortex was removed

Seeing with your ears: Humans

Vision Technology for the Totally Blind

See with your Ears! The vOICe vision technology for the totally blind offers the experience of live camera views through sophisticated image-to- sound renderings. In theory this could lead to synthetic vision with truly visual sensations ("qualia") through cross-modal sensory integration, by exploiting the neural plasticity of the human brain.

Soundscape representation P. B. L. Meijer, ``An Experimental System for Auditory Image Representations,'' IEEE Transactions on Biomedical Engineering, 39, , 1992.An Experimental System for Auditory Image Representations

Soundscape NY Times Dec 8, 2005 Seeing is something that most of us expect to do with our eyes. But what if you are born blind or lose your sight later in life? Peter Meijer suggests you consider seeing with your ears instead.

Soundscape [Economist, Nov. 2004] A group led by Petra Stoerig of the University of Dusseldorf, has reported that people can learn to "see" using a system that converts images into sound. Example of one volunteer asked to draw what she heard. Day one: she said it was a "cacophony". Day eight: ‘a natural object but don’t know what’. Day 15: she "saw" that it was a broad-leafed pot plant and drew the pot and leaves. Day 21: she clearly saw a plant and could discern that she was looking at it from the top down. When they replaced the pot plant with a daffodil, she recognized the difference. Qualia?

Soundscape fMRI LOC is activated by visual objects, and also by tactile objects (Malach et al). Consistent with prior results, LOC / LOtv is robustly activated by object palpation, but shows significantly less, minimal activation to the typical sounds made by objects. Critically, this area shows robust activation to the soundscapes of visual- to-auditory transformed objects (‘The vOICe’ condition), as prominent as that for object palpation Amedi et al, Human Brain Mapping Meeting Florence 2006

Soundscape Evidence for possible use after long training Evidence of LOC activation What about sensations (qualia)? –In congenital blind: new qualia? –In recent blind: invoking visual qualia?

Acquiring new qualia Range of studies: Extending existing senses –(rubber hand, end of cane) Switching senses –(sight and hearing) Entirely new qualia

Creating new qualia? Konig et al: Beyond sensory substitution – learning the sixth sense. J. Neural Engineering 2 (2005).

Magnetic North: Homing experiments

The measure is NI-CI NI: No-information, CI: Correct-information. Increased with training, consistent with making use of the belt.

Nystagmus eye movements Significantly more fast eye movements, as if the perception of rotation improved.

Subjective reports Two subjects reported no change, two reported changes. Two reported effects: –Enlarged space (restricted with belt off) –Memory of relative orientations No direct perception of North

Subjective reports ‘I was intuitively aware of the direction of my home or of my office. For example, I would wait in line in the cafeteria and spontaneously think: I’m living over there.’ After coming back, I could retrieve the relative orientation of all places, rooms and buildings, even if I did not pay attention while I was actually there. ‘After removing the belt, my living space shrank quickly: the world appeared smaller and more chaotic’ Marginal improvement in performing spatial tasks, some physiological changes No new quale for north Some changes in existing perceptions Re-doing it with fMRI

On the ‘Active Theory’ of Consciousness Motivation – testing the ‘Active Theory’ of Noë and O’Regan ‘A sensory-motor account of visual consciousness’ If you receive a consistent signal and use it to control action, it becomes conscious. This did not happen.

O’Regan and Noë 2001: A sensorimotor account of vision and visual consciousness Many current approaches to vision rest on the idea that when we see, the brain produces an internal representation of the world. The activation of this internal representation is assumed to give rise to the experience of seeing. An alternative proposal is made here. We propose that seeing is a way of acting. It is a particular way of exploring the environment. The advantage of this approach is that it provides a natural and principled way of accounting for visual consciousness Expectation for the ‘active theory’ was: if you receive a consistent signal and use it to control action, it becomes conscious. This did not happen.

Extending existing senses: Rubber-hand Switching senses: Switching sight and hearing Entirely new qualia: Magnetic north

Extending existing senses Re-mapping location, to an external location (rubber hand) Perceived location is changed, even to an external artifact Perception of physical ownership can change Related to activity in pre-motor cortex Bi-modal cells representing tools as body parts

Switching Senses Switching sight and hearing Brain re-mapping: Development of new organization, some behavioral results In humans: can be used behaviorally. No clear change in sensation.

Entirely new qualia Magnetic north Some evidence for behavioral use Some evidence for incorporation in eye- movement control No clear change in sensation. No support for fundamentally changing or creating new qualia