Presentation is loading. Please wait.

Presentation is loading. Please wait.

Neural Visuomotor Controller for a Simulated Salamander Robot Biljana Petreska Diploma Thesis – March 2004 Responsible Prof. Auke Jan Ijspeert.

Similar presentations

Presentation on theme: "Neural Visuomotor Controller for a Simulated Salamander Robot Biljana Petreska Diploma Thesis – March 2004 Responsible Prof. Auke Jan Ijspeert."— Presentation transcript:

1 Neural Visuomotor Controller for a Simulated Salamander Robot Biljana Petreska Diploma Thesis – March 2004 Responsible Prof. Auke Jan Ijspeert

2 Goals of the Project Investigate through simulations tightly coupled with neurobiological data, the neural mechanisms underlying visually guided behaviour in amphibians Implement a closed-loop with the environment onto the existing neuromechanical simulation developed by Ijspeert, by adding biologically inspired models for parts of the salamander brain Develop a controller that accounts for observations in feeding behaviour, including prey localization and prey recognition Study a model proposed by Ijspeert of structured mapping between the optic tectum (primary visual processing center) and the brain stem (motor centers) as a solution to the visuomotor coordination

3 Interests Relevant for perceptual robotics decoding the brain processes, assigning meaning to complex patterns of sensor stimuli may lead to the solution of many robotics tasks Test bed for probing neurobiological contributions ideal for the validation or refutation of new theories

4 Overview Short introduction on relevant topics and previous works Implemented Models Respective Results Conclusion and Future Work

5 Everything you’ve always wanted to know on Salamanders Amphibians Great variety of species (3924 indexed so far), sizes (from 16mm to 1.5m), aspects and lifestyles (terrestrial and/or aquatic). A relatively simple neural circuitry that presents all main vertebrate features Tractable from an experimental point of view: an important amount of behavioral, biological and neurological data exists

6 Visually Guided Behavior Vision is by far the most important feeding guiding sense. Under good visual conditions the other signals such as olfactory are overridden Feeding strategies (some species can switch from one to another): “hunter” strategy: active search for prey. Prerequisites are a short massive tongue and poor visual capacities. “ambush” strategy: wait until prey comes close. Prerequisites are a highly specialized projectile tongue (up to 80% body length), evolved visual system and frontally oriented eyes

7 Visually Guided Behaviour Sequence of feeding behavior orienting approach olfaction tests gaze stabilization snapping Prey preferences (in order of importance) stimulus size stimulus velocity stimulus-background contrast stimulus shape movement pattern experience-dependant

8 Morphology of the Salamander Brain Functional differentiation of the brain: structurally different regions accomplish different tasks Global top to bottom visual information processing Principal components: photoreceptors, retina, optic tectum, nucleus isthmi, pretectum,thalamus, medulla oblongata and brain stem

9 Retinal Ganglion Cells First layer of visual processing, transfers visual signals to the brain via the optic nerve 3 Types of retinal ganglion cells that project to particular layers in the optic tectum

10 Optic Tectum Main visual processing center. Integrates also multimodal perception, such as ascending somatosensory, auditory, olfaction and vestibular Stratification in 9 layers, first three are retinal afferents Six morphological neuron types identified (one interneuron) Topographic representation of the visual field Viewed as a set of partial overlapping maps due to the different types of tectal projection neurons Number of tectal cells in Hydromantes Italicus : 92 000 and 3300 out of 5000 projection neurons are descending Projection patterns: from and to the retina, pretectum, thalamus, nucleus isthmi and medulla (reaching the spinal cord) Distribution and Receptive Field Sizes of Tectal Neurons in H.Italicus

11 Pretectum Has been ascribed a role in optokinetic nystagmus, figure-background discrimination, pupillary reflex, fixation, phototaxis, and prey-enemy distinction. Properties of pretectal neurons: Homogenous arborisation (no classification was possible) Divergent projections (including to the tectum and spinal cord) Large receptive fields Receive direct and indirect (from tectum) retinal input Direction-sensitive neurons (predominantly in temporonasal direction) Respond to stimuli in the contralateral visual field

12 Lesion Experiments Give insight of the function of the destroyed brain region: Lesion of the optic tectum: both visual prey-catching and predator avoidance fail to occur. Local lesions produce scotoma, total blindness for a part of the visual field corresponding to the size of the lesion. Lesion of the pretectum: locally facilitates feeding and abolishes prey- predator discrimination, attack everything that moves including their own extremities and threatening stimuli Lesion of the thalamus: unable to avoid collision to a vertically stripped barrier, affects the binocular field Lesion of the medulla oblongata: affects distance, elevation or horizontal eccentricity estimates, overshoots prey or snaps only in frontal positions => different components of the stimulus position are handled through different pathways Difficulty: in some cases the animal recovers shortly after the lesion and the relative precision of lesions may induce errors

13 Previous Works Based upon the principle of coarse coding (Eurich et al, 1997): Motivation: the high sensory resolution observed in nature seems incompatible with the large size of receptive fields of tectal neurons Definition: population-coding using mapping combinatorics of intersecting receptive fields A non-firing neuron conveys as much information as a firing neuron. All neurons participate at the information coding Weakness: likely to suffer from metamery (convergence of information channels) Simulander I Feedforward network with only 100 neurons, trained by an evolution strategy for the specific task of head orienting (implies prey localization) Distribution and sizes of receptive fields of tectal neurons are respected and firing rates have been adapted Unstructured mapping: follows the prey with high accuracy Simulander II Similar to Simulander I, but trained for the specific task of frontal tongue projection (implies depth perception)

14 Addressed Questions How can the stimulus location and depth estimates be extracted from the tectum maps? What sensorimotor transformations occur at the level of the optic tectum, the brainstem and the pathways between them? Can a structured mapping provide an accurate visual tracking? Which type of a tectum-brainstem mapping explains the typical curved approach in monocularized salamanders? How is the visual perception influenced by head motion during the approach toward a stimulus? Are additional mechanisms necessary for dealing with the remaining shifts in the visual background? Which mechanism implements the release of the snapping behavior? And how is the tongue controlled?

15 Neural Networks Restrictions (performance motivated): Uniformly distributed neuron units Square receptive fields Specification: Center receptive field (in degrees of visual field) : determines the size of the neural network Surround receptive field (in degrees of visual field) : determines the overlap and redundancy feature Weights matrix, activation function and thresholds Features: Reduction (biologically motivated) Visualization (extremely practical)

16 Eyes of the Simulated Salamander Virtual cameras: extract views using provided OpenGL functions Correct the view using a spherical projection Photoreceptors are equivalent to pixel grey values Scalable visual field

17 Retinal Ganglion Cells of Type 1 Properties: small size excitatory (2-3°) and strong inhibitory (12-16°) receptive fields no response to change in light involved in local contrast calculation => edge detector project to the contralateral spinal cord => obstacle avoidance? give rise to a fine grained representation of the visual field in the retina Modelled with the laplacian of a gaussian filter: Classic edge detector in computer vision and confirmed by the study of a larval tiger salamander retina receptive field

18 Retinal Ganglion Cells of Type 2 Type 2 retinal ganglion cells respond only to moving objects => motion detectors Detection of change: compare the corresponding pixels at different times, using a linear difference function: where τ is a threshold, j and k are moments in time, x and y are the pixel positions in the frame Biological inspirations: Reflects signals with delayed pathways that give rise to a simultaneous representation of the same object at different times in the brain Flat weights: the activity sharply increases when an object enters the receptive field variation Tectal neurons are contrast-sensitive: linear function

19 Retinal Ganglion Cells of Type 3 Properties: large receptive fields (10-20°) tonic response to change in light intensity respond to overall luminosity (dimming detectors) respond also at low contrast and velocity Model: flat weights simple summing network Predator detectors among other

20 Optic Tectum Model Principal biological inspirations: Retinotopic map in the optic tectum: electrical stimulations result in turning movements that roughly correspond to this map Only two synapses between the retina and the brain stem: the tectum directly projects onto the brain stem Input: retinal ganglion cells of type 2. Motion is a necessary prerequisite for a stimulus to be interpreted as prey Structured mapping: different strengths along the rostro-caudal axis, reflects the stimulus eccentricity Motoneuron activation function (integrating weighted tectal activity): where x is the change in light intensity of the pixel at positions i and j Linear weights function: where α and β are parameters + -

21 Optic Tectum Model II Version with ipsilateral input (contribution from both eyes)

22 Normalizing tectal activity. The modified model is robust to changes in the stimulus parameters and visual scene: Biological reference: TO4 neurons, arborize in RGC2 TO2 neurons, large receptive fields both project to the nucleus isthmi Optic Tectum Model III

23 Pretectum Model Large stimuli: based on RGC3 => dimming detectors with three times larger receptive fields Motion: compare direct and indirect (via tectum) RGC3 responses Direction-sensitive neurons: Why temporonasal sensitivity? Based upon separating the ON and OFF channels Hypothesis: only sensitive to dark objects (biologically consistent)

24 Snapping Model Relevant for depth estimation Tongue mechanism (biologically consistent): 4 muscles, protraction and retraction times modulated by the stimulus position We proposed a mechanism for frontal snapping based upon divergent projections of the tectal neurons

25 Results Find optimal α and β parameters of the linear weights function through an exhaustive search of the parameter space Cost function: difference between the stimulus direction and the salamander orienting movement Experimental conditions Ewert experiment: the stimulus is moved on a semi-circular trajectory with a constant speed in front of the animal (task of head orienting) Body muscles were inhibited (only neck muscles) The stimulus parameters (size, speed, distance,...) and network parameters (number of neurons) were fixed according to values found in literature. Both single stimulus and complex background were used

26 Optimal Values Many combinations of values give similar results Good results are also achieved without ipsilateral input

27 α and β Parameters Regular parameter space. With different fixed values the aspect is conserved and the minimal error area (in black) is shifted β parameters are not essential, the minimal error area is centered in point (0,0) α contralateral (x-axis) and α ipsilateral (y-axis) with optimal β parameters β contralateral (x-axis) and β ipsilateral (y-axis) with optimal α parameters

28 Performance Results An accuracy of less than 3° (real value) is achieved for small stimulus velocity values with 20000 RGC2 and 2000 (less than 3300) tectal neurons. Robustness : stable reaction to change in stimulus parameters and visual scene

29 With Complex Background The salamander has difficulties with following the prey stimulus as the amount of “noise” is considerable. It discriminates between objects with same apparent angular size, however orients at "average flies" The model should be coupled with a selective visual attention mechanism (enhanced retinal signals in the area containing the prey stimulus) and/or optokinetic or vestibucollic image stabilization reflexes (antagonistic head movements that compensate for body undulations) Integrating approach is trivial with a unique prey stimulus

30 Pretectum The salamander discriminates between a small prey object and a large predator object When the pretectum is abolished, escape behavior fails to occur Delayed response: the salamander escapes for a longer time than the predator is visible Weakness: based upon angular size, close prey may be interpreted as predator. Therefore the threshold is essential (arbitrary as no data exists on predation)

31 Snapping No additional neurons, based upon divergent patterns of tectal neurons projections Consistent with biological lesion data: codes for “closeness” realistic precision (about 30%) Depends on the movement direction

32 Reproduced Phenomena Lesion and stimulation experiments: Lesion and stimulation of the optic tectum Lesion of the pretectum Generation of saccadic movements Monocularized salamanders Prey preferences

33 Saccadic Movements Pursuit movements such as the head accelerates for a few seconds, until maximum velocity is reached, and then is released We attribute them to the tectal cells resolution

34 Monocularized Salamander With one eye covered, H.Italicus shows a conspicuous approach behavior toward a prey stimulus. It takes a curved path and bends its body toward the side of a seeing eye, compensating by turning the head between 60° and 90°

35 Monocularized Salamander

36 Prey preferences All preferences are inherent to the network!!

37 Comparison to Previous Works Simulander I More neurons, but still biologically plausible (2000 vs. 100) Less accurate, more realistic (2°-6° vs. 1°) Inherent preferences vs. a function reflecting the stimulus size and velocity (corresponds to the observer’s knowledge) No real distribution of tectal neurons, respected in Simulander Faster reaction No positions in which stationery prey elicit orienting behavior Simulander II Lower precision, but more realistic (90%, real success rate 40%) In Simulander far objects elicit more activity, double inconsistency (should code for “closeness” and further objects seem smaller)

38 Response to questions Extraction of stimulus localization and depth estimates can be achieved with a structured mapping between the optic tectum and the brain stem The sensorimotor transformation of the horizontal angular distance of the tectum neurons to muscle activity can provide an accurate prey localization. Direct observation of the influence of head motion during the approach is provided. Additional mechanisms for dealing with the self-motion visual shifts are necessary The investigated tectum model accounts for the typical curved approach in monocularized salamanders A plausible mechanism that acts as a releaser for the snapping behavior is proposed

39 Conclusion We have implemented models of the three types of retinal ganglion cells, the optic tectum, the pretectum and a tongue projection mechanism that account for the typical feeding sequence and escape behavior The optic tectum model reproduces many experimental data Everything is observable Warning: data and methodology dependant Our salamander resembles a newly born salamander thrown in the world

40 Future Work Study a tectum model with nonlinear weights functions Use the real distribution and receptive fields sizes of tectal neurons Time-dynamics vs. discrete time steps Study the effect of overlapping fields (redundancy => error resistant, maybe emerging properties) Implement a visual attention model Implement experience-based models such as habituation Further development of the pretectum model Extend the model to other brain areas such as the nucleus isthmi or thalamus (obstacle avoidance) Develop a more elaborate model for depth estimation (not only frontal) Work on an object-background discrimination with respect to self- motion shifts of the visual input

Download ppt "Neural Visuomotor Controller for a Simulated Salamander Robot Biljana Petreska Diploma Thesis – March 2004 Responsible Prof. Auke Jan Ijspeert."

Similar presentations

Ads by Google