1. FRS 123: Technology in Art and Cultural Heritage Perception

2. Low-Level or “Early” Vision Considers local properties of an image “There’s an edge!”

3. Mid-Level Vision Grouping and segmentation “There’s an object and a background!”

4. High-Level Vision Recognition “It’s a chair!”

5. Image Formation Human: lens forms image on retina, sensors (rods and cones) respond to lightHuman: lens forms image on retina, sensors (rods and cones) respond to light Computer: lens system forms image, sensors (CCD, CMOS) respond to lightComputer: lens system forms image, sensors (CCD, CMOS) respond to light

6. Intensity Perception of intensity is nonlinearPerception of intensity is nonlinear Amount of light Perceived brightness

7. Modeling Nonlinear Intensity Response Perceived brightness (B) usually modeled as a logarithm or power law of intensity (I)Perceived brightness (B) usually modeled as a logarithm or power law of intensity (I) Exact curve varies with ambient light, adaptation of eyeExact curve varies with ambient light, adaptation of eye I B

8. CRT Response Power law for Intensity (I) vs. applied voltage (V)Power law for Intensity (I) vs. applied voltage (V) Other displays (e.g. LCDs) contain electronics to emulate this lawOther displays (e.g. LCDs) contain electronics to emulate this law

9. Cameras Original cameras based on Vidicon obey power law for Voltage (V) vs. Intensity (I):Original cameras based on Vidicon obey power law for Voltage (V) vs. Intensity (I): Vidicon + CRT = almost linear!Vidicon + CRT = almost linear!

10. CCD Cameras Camera gamma codified in NTSC standardCamera gamma codified in NTSC standard CCDs have linear response to incident lightCCDs have linear response to incident light Electronics to apply required power lawElectronics to apply required power law So, pictures from most cameras (including digital still cameras) will have  = 0.45So, pictures from most cameras (including digital still cameras) will have  = 0.45

11. Contrast Sensitivity Contrast sensitivity for humans about 1%Contrast sensitivity for humans about 1% 8-bit image (barely) adequate if using perceptual (nonlinear) mapping8-bit image (barely) adequate if using perceptual (nonlinear) mapping Frequency dependent: contrast sensitivity lower for high and very low frequenciesFrequency dependent: contrast sensitivity lower for high and very low frequencies

12. Contrast Sensitivity Campbell-Robson contrast sensitivity chartCampbell-Robson contrast sensitivity chart

13. Bits per Pixel – Scanned Pictures 8 bits / pixel / color 6 bits / pixel / color Marc Levoy / Hanna-Barbera

14. Bits per Pixel – Scanned Pictures (cont.) 5 bits / pixel / color 4 bits / pixel / color Marc Levoy / Hanna-Barbera

15. Bits per Pixel – Line Drawings 8 bits / pixel / color 4 bits / pixel / color Marc Levoy / Hanna-Barbera

16. Bits per Pixel – Line Drawings (cont.) 3 bits / pixel / color 2 bits / pixel / color Marc Levoy / Hanna-Barbera

17. Color Two types of receptors: rods and conesTwo types of receptors: rods and cones Rods and cones Cones in fovea

18. Rods and Cones RodsRods – More sensitive in low light: “scotopic” vision – More dense near periphery ConesCones – Only function with higher light levels: “photopic” vision – Densely packed at center of eye: fovea – Different types of cones  color vision

19. Electromagnetic Spectrum Visible light frequencies range between...Visible light frequencies range between... – Red = 4.3 x 10 14 hertz (700nm) – Violet = 7.5 x 10 14 hertz (400nm)

20. Color Perception 3 types of cones: L, M, S 3 types of cones: L, M, S Tristimulus theory of color S L M

21. Tristimulus Color Any distribution of light can be summarized by its effect on 3 types of conesAny distribution of light can be summarized by its effect on 3 types of cones Therefore, human perception of color is a 3-dimensional spaceTherefore, human perception of color is a 3-dimensional space Metamerism: different spectra, same responseMetamerism: different spectra, same response Color blindness: fewer than 3 types of conesColor blindness: fewer than 3 types of cones – Most commonly L cone = M cone

22. Color CRT

23. Preattentive Processing Some properties are processed preattentivelySome properties are processed preattentively (without need for focusing attention). Important for art, design of visualizationsImportant for art, design of visualizations – what can be perceived immediately – what properties are good discriminators – what can mislead viewers Preattentive processing sildes from Healey http://www.csc.ncsu.edu/faculty/healey/PP/PP.html

24. Example: Color Selection Viewer can rapidly and accurately determine whether the target (red circle) is present or absent. Difference detected in color.

25. Example: Shape Selection Viewer can rapidly and accurately determine whether the target (red circle) is present or absent. Difference detected in form (curvature)

26. Pre-attentive Processing < 200–250 ms qualifies as pre-attentive< 200–250 ms qualifies as pre-attentive – eye movements take at least 200ms – yet certain processing can be done very quickly, implying low-level processing in parallel If a decision takes a fixed amount of time regardless of the number of distractors, it is considered to be preattentiveIf a decision takes a fixed amount of time regardless of the number of distractors, it is considered to be preattentive

27. Example: Conjunction of Features Viewer cannot rapidly and accurately determine whether the target (red circle) is present or absent when target has two or more features, each of which are present in the distractors. Viewer must search sequentially.

28. Example: Emergent Features Target has a unique feature with respect to distractors (open sides) and so the group can be detected preattentively.

29. Example: Emergent Features Target does not have a unique feature with respect to distractors and so the group cannot be detected preattentively.

30. Asymmetric and Graded Preattentive Properties Some properties are asymmetricSome properties are asymmetric – a sloped line among vertical lines is preattentive – a vertical line among sloped ones is not Some properties have a gradationSome properties have a gradation – some more easily discriminated among than others

31. SUBJECT PUNCHED QUICKLY OXIDIZED TCEJBUS DEHCNUP YLKCIUQ DEZIDIXO CERTAIN QUICKLY PUNCHED METHODS NIATREC YLKCIUQ DEHCNUP SDOHTEM SCIENCE ENGLISH RECORDS COLUMNS ECNEICS HSILGNE SDROCER SNMULOC GOVERNS PRECISE EXAMPLE MERCURY SNREVOG ESICERP ELPMAXE YRUCREM CERTAIN QUICKLY PUNCHED METHODS NIATREC YLKCIUQ DEHCNUP SDOHTEM GOVERNS PRECISE EXAMPLE MERCURY SNREVOG ESICERP ELPMAXE YRUCREM SCIENCE ENGLISH RECORDS COLUMNS ECNEICS HSILGNE SDROCER SNMULOC SUBJECT PUNCHED QUICKLY OXIDIZED TCEJBUS DEHCNUP YLKCIUQ DEZIDIXO CERTAIN QUICKLY PUNCHED METHODS NIATREC YLKCIUQ DEHCNUP SDOHTEM SCIENCE ENGLISH RECORDS COLUMNS ECNEICS HSILGNE SDROCER SNMULOC

32. SUBJECT PUNCHED QUICKLY OXIDIZED TCEJBUS DEHCNUP YLKCIUQ DEZIDIXO CERTAIN QUICKLY PUNCHED METHODS NIATREC YLKCIUQ DEHCNUP SDOHTEM SCIENCE ENGLISH RECORDS COLUMNS ECNEICS HSILGNE SDROCER SNMULOC GOVERNS PRECISE EXAMPLE MERCURY SNREVOG ESICERP ELPMAXE YRUCREM CERTAIN QUICKLY PUNCHED METHODS NIATREC YLKCIUQ DEHCNUP SDOHTEM GOVERNS PRECISE EXAMPLE MERCURY SNREVOG ESICERP ELPMAXE YRUCREM SCIENCE ENGLISH RECORDS COLUMNS ECNEICS HSILGNE SDROCER SNMULOC SUBJECT PUNCHED QUICKLY OXIDIZED TCEJBUS DEHCNUP YLKCIUQ DEZIDIXO CERTAIN QUICKLY PUNCHED METHODS NIATREC YLKCIUQ DEHCNUP SDOHTEM SCIENCE ENGLISH RECORDS COLUMNS ECNEICS HSILGNE SDROCER SNMULOC Text NOT Preattentive

33. Preattentive Visual Properties [Healey 97] length Triesman & Gormican [1988] length Triesman & Gormican [1988] width Julesz [1985] width Julesz [1985] size Triesman & Gelade [1980] size Triesman & Gelade [1980] curvature Triesman & Gormican [1988] curvature Triesman & Gormican [1988] number Julesz [1985]; Trick & Pylyshyn [1994] number Julesz [1985]; Trick & Pylyshyn [1994] terminators Julesz & Bergen [1983] terminators Julesz & Bergen [1983] intersection Julesz & Bergen [1983] intersection Julesz & Bergen [1983] closure Enns [1986]; Triesman & Souther [1985] closure Enns [1986]; Triesman & Souther [1985] colour (hue) Nagy & Sanchez [1990, 1992]; D'Zmura [1991] Kawai et al. [1995]; Bauer et al. [1996] colour (hue) Nagy & Sanchez [1990, 1992]; D'Zmura [1991] Kawai et al. [1995]; Bauer et al. [1996] intensity Beck et al. [1983]; Triesman & Gormican [1988] intensity Beck et al. [1983]; Triesman & Gormican [1988] flicker Julesz [1971] flicker Julesz [1971] direction of motion Nakayama & Silverman [1986]; Driver & McLeod [1992] direction of motion Nakayama & Silverman [1986]; Driver & McLeod [1992] binocular lustre Wolfe & Franzel [1988] binocular lustre Wolfe & Franzel [1988] stereoscopic depth Nakayama & Silverman [1986] stereoscopic depth Nakayama & Silverman [1986] 3-D depth cues Enns [1990] 3-D depth cues Enns [1990] lighting direction Enns [1990] lighting direction Enns [1990]

34. Accuracy Ranking of Quantitative Perceptual Tasks Estimated; only pairwise comparisons have been validated [Mackinlay 88 from Cleveland & McGill]

35. Visual Illusions People don’t perceive length, area, angle, brightness they way they “should”People don’t perceive length, area, angle, brightness they way they “should” Some illusions have been reclassified as systematic perceptual errorsSome illusions have been reclassified as systematic perceptual errors – e.g., brightness contrasts (grey square on white background vs. on black background) – partly due to increase in our understanding of the relevant parts of the visual system Nevertheless, the visual system does some really unexpected thingsNevertheless, the visual system does some really unexpected things

36. Illusions of Linear Extent Mueller-Lyon (off by 25-30%)Mueller-Lyon (off by 25-30%) Horizontal-VerticalHorizontal-Vertical

37. Illusions of Area Delboeuf IllusionDelboeuf Illusion Height of 4-story building overestimated by approximately 25%Height of 4-story building overestimated by approximately 25%

38. Low-Level Vision Hubel

39. Retinal ganglion cellsRetinal ganglion cells Lateral Geniculate Nucleus – visual adaptation?Lateral Geniculate Nucleus – visual adaptation? Primary Visual CortexPrimary Visual Cortex – Simple cells: orientational sensitivity – Complex cells: directional sensitivity Further processingFurther processing – Temporal cortex: what is the object? – Parietal cortex: where is the object? How do I get it?

40. Low-Level Vision Net effect: low-level human vision can be (partially) modeled as a set of multiresolution, oriented filtersNet effect: low-level human vision can be (partially) modeled as a set of multiresolution, oriented filters

41. Low-Level Computer Vision Filters and filter banksFilters and filter banks – Implemented via convolution – Detection of edges, corners, and other local features – Can include multiple orientations – Can include multiple scales: “filter pyramids” ApplicationsApplications – First stage of segmentation – Texture recognition / classification – Texture synthesis

42. Texture Analysis / Synthesis MultiresolutionOriented Filter Bank Original Image ImagePyramid

43. Texture Analysis / Synthesis OriginalTexture SynthesizedTexture [Heeger and Bergen]

44. Low-Level Computer Vision Optical flowOptical flow – Detecting frame-to-frame motion – Local operator: looking for gradients ApplicationsApplications – First stage of tracking

45. Optical Flow Image #1 Optical Flow Field Image #2

46. Low-Level Depth Cues FocusFocus VergenceVergence StereoStereo Not as important as popularly believedNot as important as popularly believed

47. 3D Perception: Stereo Experiments show that absolute depth estimation not very accurateExperiments show that absolute depth estimation not very accurate – Low “relief” judged to be deeper than it is Relative depth estimation very accurateRelative depth estimation very accurate – Can judge which object is closer for stereo disparities of a few seconds of arc

48. 3D Perception: Illusions Block & Yuker

49. 3D Perception: Illusions Block & Yuker

50. 3D Perception: Illusions Block & Yuker

51. 3D Perception: Illusions Block & Yuker

52. 3D Perception: Illusions Block & Yuker

53. 3D Perception: Illusions Block & Yuker

54. 3D Perception: Illusions Block & Yuker

55. 3D Perception: Illusions Block & Yuker

56. 3D Perception: Illusions Block & Yuker

57. 3D Perception: Illusions Block & Yuker

58. 3D Perception: Conclusions Perspective is assumedPerspective is assumed Relative depth orderingRelative depth ordering Occlusion is importantOcclusion is important Local consistencyLocal consistency

59. Low-Level Computer Vision Shape from XShape from X – Stereo – Motion – Shading – Texture foreshortening

60. 3D Reconstruction Tomasi+Kanade Debevec,Taylor,Malik Phigin et al. Forsyth et al.

61. Mid-Level Vision Physiology unclearPhysiology unclear Observations by Gestalt psychologistsObservations by Gestalt psychologists – Proximity – Similarity – Common fate – Common region – Parallelism – Closure – Symmetry – Continuity – Familiar configuration Wertheimer

62. Gestalt Properties Gestalt: form or configurationGestalt: form or configuration Idea: forms or patterns transcend the stimuli used to create themIdea: forms or patterns transcend the stimuli used to create them – Why do patterns emerge? Under what circumstances? Why perceive pairs vs. triplets?

63. Gestalt Laws of Perceptual Organization [Kaufman 74] Figure and GroundFigure and Ground – Escher illustrations are good examples – Vase/Face contrast Subjective ContourSubjective Contour

64. More Gestalt Laws Law of ProximityLaw of Proximity – Stimulus elements that are close together will be perceived as a group Law of SimilarityLaw of Similarity – like the preattentive processing examples Law of Common FateLaw of Common Fate – like preattentive motion property move a subset of objects among similar ones and they will be perceived as a groupmove a subset of objects among similar ones and they will be perceived as a group

65. Grouping Cues

69. Events of Interest /@rts lecture series on “interrelations of new media, technology and traditional forms and practices of arts and humanities” http://www.princeton.edu/slasharts//@rts lecture series on “interrelations of new media, technology and traditional forms and practices of arts and humanities” http://www.princeton.edu/slasharts/ Scott McCloud “Comics: An Art Form in Transition” Thursday, October 5, 4:30 pm Jimmy Stewart Theater 185 Nassau StreetScott McCloud “Comics: An Art Form in Transition” Thursday, October 5, 4:30 pm Jimmy Stewart Theater 185 Nassau Street

70. Events of Interest Digital Stone ProjectDigital Stone Project – Local facility for automated creation of stone sculpture from 3D computer models – Computer-contolled milling machines, lathes Friday, October 6, 1:30-4:00 Meet in Computer Science building, 2 nd floor “tea room”, 1:00 sharpFriday, October 6, 1:30-4:00 Meet in Computer Science building, 2 nd floor “tea room”, 1:00 sharp