1. 9 January 20149 January 20149 January 2014David Pierre Leibovitz (Carleton University)Vision, Spiders & Time - 1 Vision, Spiders & Time David Pierre Leibovitz January 9, 2014January 9, 2014January 9, 2014

2. 9 January 20149 January 20149 January 2014David Pierre Leibovitz (Carleton University)Time Matters - 2 Summary How is vision perception related to imagination and planning What is the role of attention (saccades) Can smart spiders shed light on human cognition? –They have severe engineering restrictions –They they take a long time to think –How does that affect cognition I will relate spider time to practical matters Hopefully, you will also come to appreciate spiders as well Hopefully, you will also come to appreciate spiders as well

3. Human Vision Like a Camera? Static Camera vs. Video –Do we process changes? Process entire scene or saccade/attend to others Is it process-interpret-act or a top- down/bottom-up iterative thing What about imagination? 9 January 20149 January 20149 January 2014David Pierre Leibovitz (Carleton University)Time Matters - 3

4. 9 January 20149 January 20149 January 2014David Pierre Leibovitz (Carleton University)Time Matters - 4 Vision Movies @ 24hz (42ms/frame) –But flicker at higher rates? Major saccades @ 5hz (20-200ms/move) –Can miss

5. 9 January 20149 January 20149 January 2014David Pierre Leibovitz (Carleton University)Time Matters - 5 Saccades

6. 9 January 20149 January 20149 January 2014David Pierre Leibovitz (Carleton University)Time Matters - 6 Saccades Major (Clinically observable) Minor (Hard to observe) LargeSmallMini/Flick Micro/Trem or Angular Distance >6.2° 6.2° - 1.2° 1.2° – 2 40 Occurrence.05 -.3hz Up to 5hz 40hz - - 200hz Time200ms20-200ms10-20ms<10ms Reason Head, Body Orientation Changes View other features of object, next character grouping Refresh Rod/Cone Signalling Extra Detail?

7. 9 January 20149 January 20149 January 2014David Pierre Leibovitz (Carleton University)Time Matters - 7 Saccadic Eye Motion Major Saccades –Large Saccades –Enables the building up of a high resolution mental map from interesting parts of a scene –The fastest part of the human body (1000°/s). Microsaccades (20 arc seconds @ ~90Hz) serve to refresh image on rod and cone cells which respond to changes in luminance (otherwise a fixed stare would cause an image to slowly disappear) –It is possible that this also improves resolution –Jitter camera –1 pixel camera During saccade, details are masked (mirror experiment). Motion of eye, blur of image nor time gap are perceived. –Cant see flash of light during saccade –Cant see object moved during saccade –However, masking stops if blur stops, i.e., when saccade follows a moving object. During a 1ms a snapshot can be taken (+250ms for later processing). Suppressing (magnocellular) M-pathway responsible for motion responding to transient, high- velocity stimuli of low spatial frequency Enhancing (parvocellular) P-pathway responsible for colour information tradeoff to differentially process changes over time due to motion tradeoff to differentially process changes over time due to motion –Optokinetic reflex – ability to follow (slower) motion with eyes fixed

8. 9 January 20149 January 20149 January 2014David Pierre Leibovitz (Carleton University)Time Matters - 8

9. 9 January 20149 January 20149 January 2014David Pierre Leibovitz (Carleton University)Time Matters - 9 Jitter Camera Same as interlacing Related to micro-saccades Jitter (which doubles the sensing time) doubles perceptual resolution

10. 9 January 20149 January 20149 January 2014David Pierre Leibovitz (Carleton University)Time Matters - 10 Visual Neglect In Visual Neglect, Patients can only see, or imagine half a scene. However, they can mentally rotate 180° and then imagine the other half (from the other perspective) This lends credence to the view that each side GENERATEs a view.

11. 9 January 20149 January 20149 January 2014David Pierre Leibovitz (Carleton University)Time Matters - 11 Scanning Tunneling Microscope Example of a 1 Pixel Camera Not even optical! Not limited by diffraction Highest resolution microscope (0.2 nm) Gold (Au) showing individual atoms

12. 9 January 20149 January 20149 January 2014David Pierre Leibovitz (Carleton University)Time Matters - 12 Jumping Spider - Portia labiata 1 pixel animal Cognitive Abilities –Scene building –Route planning/plotting (15-60 minutes) –Plan execution (several hours) Goal may be invisible –Lure via adaptive mimicry Tickle web to mimic other animals (or try different patterns) –Tactical Attack decisions If prey carrying egg-sac, mount frontal assault (dont worry about spitting spider) Else backwards assault Top assault? Background –Jumping (venomous) spider –Eats other spiders E.g., Scytodes pallida, a spitting (poison coated silk) spider that also eats other spiders –600,000 neurons –Better vision than most mammals (but narrow)

13. 9 January 20149 January 20149 January 2014David Pierre Leibovitz (Carleton University)Time Matters - 13 Brain Size Neurons (not connectivity) Humans:100 billion Mouse:70 million Honeybee:1 million Portia:½ million Housefly:¼ million In Portia, limited size will force a lengthy and complex (algorithmic) visual scan to convert scene into a more compact (plan oriented?) representation. There just isnt enough neurons to rebuild a visual scene

14. 9 January 20149 January 20149 January 2014David Pierre Leibovitz (Carleton University)Time Matters - 14 Spider Eyes Most spiders have 8 eyes and poor vision. Portia sees very well – it jumps far and makes long circuitous routes to distant prey Postero-Lateral Eyes (PLE) Postero-Lateral Eyes (PLE) –Wide Angle –Motion Sensing (Far) –Blurry Vision Postero Median Eyes (PME) Postero Median Eyes (PME) –Often tiny, Vestigial Antero-Lateral Eyes (ALE) Antero-Lateral Eyes (ALE) –Judging Distance – Blurry Vision Antero-Median Eyes (AME) Antero-Median Eyes (AME) –Narrow Angle –Excellent Vision –Colour –Swivel

15. 9 January 20149 January 20149 January 2014David Pierre Leibovitz (Carleton University)Time Matters - 15 Spider Eyes – Fields of View ALE overlap enables accurate judgement of distance (binocular vision) AME do not overlap –Narrow telephoto systems

16. 9 January 20149 January 20149 January 2014David Pierre Leibovitz (Carleton University)Time Matters - 16 Acuity Portia visual acuity is better than cats and pigeons –Measured in angular resolution, not via light sensitivity But slow… Best Insect

17. 9 January 20149 January 20149 January 2014David Pierre Leibovitz (Carleton University)Time Matters - 17 Spider – Field of View Retina: several thousand photoreceptors Layered for different colours (green, blue, ultraviolet) Layer 1 (green) most acute, stacked for various focus distances. Swinging tube also helps Fovea: several hundred photoreceptors (1 pixel) (human eye has 150 million photoreceptors; fovea less than 1% of retina but uses 50% visual cortex)

18. 9 January 20149 January 20149 January 2014David Pierre Leibovitz (Carleton University)Time Matters - 18 Spider – Route Scanning & Planning Can see everything from the top of the dowel –15-60 minutes to scan and plan (anticipatory maze learning) Circuitous route (detour) Simulate? its future movements or place memory? –Adaptive (goal directed) scanning vs. one- pixel camera (non-adaptive, entire scene) Attend more to the correct route Visually following tracks backwards from goal Specific scanning patterns may be used for finding/resolving specific objects –Can find previously eaten prey faster Execution –Up to several hours to execute (in wild) –Lure could not be seen once down –Correct route chosen 75% Even when further away Even if longer Even if both routes had lures, but one had a gap –Wrong route abandoned more often –On par with in complexity with vertebra –Select secondary objectives if needed?

19. 9 January 20149 January 20149 January 2014David Pierre Leibovitz (Carleton University)Time Matters - 19 Principles Time can be used to accumulate smaller bits of information into larger bits (scanning) –Scanning may be algorithmic However, to reduce scanning time (as compared to an actual 1 pixel camera) it must be adaptive, i.e. selective attention based on goals affected by memory Moreover, to reduce representation size, scanning must be done in conjunction with planning –It is possible that the plan is the representation of perception –We see and attend to affordances Perception and decision making (goal system) are part of a dynamic system and cannot be divorced –It is only when the timescales of both are similar that this is made apparent –Seeing the big picture is a more human example. Information needs to be synthesized in order to make good plans Even in humans, visual neglect shows that perception is goal oriented – a visuomotor system

20. 9 January 20149 January 20149 January 2014David Pierre Leibovitz (Carleton University)Time Matters - 20 Spider Gallery - portia africana, 10mm @ Kenya

21. 9 January 20149 January 20149 January 2014David Pierre Leibovitz (Carleton University)Time Matters - 21 Spider Gallery - epeus flavobilineatu, 7mm @ Malaysia

22. 9 January 20149 January 20149 January 2014David Pierre Leibovitz (Carleton University)Time Matters - 22 Spider Gallery - unknown species, 7mm @ Thailand

23. 9 January 20149 January 20149 January 2014David Pierre Leibovitz (Carleton University)Time Matters - 23 Spider Gallery - myrmarachne plataleoides, 8mm @ Thailand Ant or Spider?

24. 9 January 20149 January 20149 January 2014David Pierre Leibovitz (Carleton University)Time Matters - 24 Spider Gallery - chrysilla lauta, 6mm @ Malaysia

25. 9 January 20149 January 20149 January 2014David Pierre Leibovitz (Carleton University)Time Matters - 25 Spider Gallery - viciria praemandibularis, 10mm @ Malaysia

26. 9 January 20149 January 20149 January 2014David Pierre Leibovitz (Carleton University)Time Matters - 26 Spider Gallery - thiania bhamoensis, 7mm @ Malaysia

27. 9 January 20149 January 20149 January 2014David Pierre Leibovitz (Carleton University)Time Matters - 27 Spider Gallery - Lysomannes salticidae @ China

28. 9 January 20149 January 20149 January 2014David Pierre Leibovitz (Carleton University)Time Matters - 28 One Pixel Camera Compressive Imaging/Sensing –Rather than take mega-pixel images and then compress (which is extremely resource intensive), build up your image smartly –Like our saccadic system? –Compressive classification What we know (possible classifications) can influence what we are looking for, i.e., look quicklyfor distinctive salient features, look more at discriminants Scanning is Goal directed perception –Single pixel can be extremely specialized (sensitive, terahertz) Initial designs are not adaptive (no memory). –Quick scans of slowly (temporal & spatial) changing areas, more detailed elsewhere

29. 9 January 20149 January 20149 January 2014David Pierre Leibovitz (Carleton University)Time Matters - 29 Compression vs. Integration Top Row –64x64 bit image with various wavlet compressions (throwing data out) Bottom Row –320x240 DMD with various random samplings Existing system quite dumb. Previous sample (pixel) does not direct next one. Leads to cognitive/dynamic systems Leads to cognitive/dynamic systems Does it take longer to reduce a mega pixel image to an internal representation than to build one up pixel by pixel?

30. 9 January 20149 January 20149 January 2014David Pierre Leibovitz (Carleton University)Time Matters - 30 Terahertz Travels in line of sight Non- ionizing (safe unlike x-rays) Penetrates non- conductors

31. 9 January 20149 January 20149 January 2014David Pierre Leibovitz (Carleton University)Time Matters - 31 One-Pixel CRT Although One-Pixel Cameras (perception) is not common (what about saccades?), what about generation (imagination)

32. 9 January 20149 January 20149 January 2014David Pierre Leibovitz (Carleton University)Time Matters - 32 Jitter Interlacing Jitter & Interlacing can reduce bandwidth, but They increase motion artifacts

33. How do Artists Draw? Is it similar to how they eye-track? 9 January 20149 January 20149 January 2014David Pierre Leibovitz (Carleton University)Time Matters - 33