1. Is perception continuous with cognition? The cognitive impenetrability of vision See my BBS article on my web site: ruccs.rutgers.edu/faculty/pylyshyn.html

2. The answer goes along with intellectual (and political) fashions The Zeitgeist in the second half of the 20 th century was one of populist/democratic values which emphasized egalitarianism and the limitless possibility of the human mind. In keeping with this spirit, many scholars mistakenly repudiated innateness and, emphasizing plasticity, embraced learning as the defining character of human nature. It was in this era that the belief arose that everything one saw or thought was relative to some cultural or linguistic context. Hence the Sapir-Worf Hypothesis, Bruner’s New Look in perception and other deconstruction movements.

3. In computational vision (artificial intelligence): Minsky & Papert’s: “Heterarchy, not hierarchy” Riseman & Hanson (1987): “It appears that human vision is fundamentally organized to exploit the use of contextual knowledge and expectations in the organization of spatial primitives… Thus the inclusion of knowledge-driven processes at some level in the image interpretation task, where there is still a great degree of ambiguity in the organization of the visual primitives, appears inevitable (286).”

4. In computational vision (artificial intelligence): David Marr (1982): “Our view is that although some top-down information is sometimes used and necessary, it is of only secondary importance in early visual processing.” “The principle of least commitment… requires not doing something that may later have to be undone, and I believe that it applies to all situations in which performance is fluent. It states that algorithms that are constructed according to a hypothesize-and-test strategy should be avoided because there is probably a better method.”

5. In psychology: Irvin Rock (1983): “Perception differs from thought primarily because it is rooted in and constrained by the necessity of accounting for the proximal stimulus. The major difference between perception and thought is that perception is based on a rather narrow range of internalized knowledge, as far as inference and problem solving are concerned… Perception must rigidly adhere to the appropriate internalized rules, so that it often seems unintelligent and inflexible in its imperviousness to other forms of knowledge (p 340)”.

6. The connection between vision and the world An understanding of how the visual system connects with the world rests on three empirical pillars : 1. Evidence that perceptual representations must be primarily conceptual/descriptive (Seeing, chapter 1) 2. Evidence that vision has at least three major architectural stages (Seeing, chapters 2 & 3) ; a) a preconceptual and preattentive stage at which the world is parsed and different parts/properties are picked out or selected for subsequent conceptual encoding b) a modular informationally encapsulated stage (early vision) c) a cognitive stage where the outputs of perception are interpreted and conceptualized and enter into the process of belief fixation 3. There are reasons (both empirical and logical) why vision can’t do without concepts but also why it can’t do with only concepts: It needs a causally actuated indexing mechanism

7. The connection between vision and the world An understanding of how the visual system connects with the world rests on three empirical pillars : 1. Evidence that perceptual representations must be primarily conceptual/descriptive (Chapter 1)  Evidence that vision has at least three major architectural stages ; a) a preconceptual and preattentive stage in which the world is parsed and different parts/properties are picked out or selected for subsequent conceptual encoding b) a modular informationally encapsulated stage (early vision) c) a cognitive stage where the outputs of perception are interpreted and conceptualized and plausible ones retained as beliefs 3. Reasons (both empirical and logical) why vision can’t do without concepts but also why it can’t do with only concepts: It needs a causally actuated indexing mechanism.

8. Some remarks on the architecture of the visual system What is “architecture”?  Constants vs empirical parameters  Micro and macro architecture  Modular architecture – and Marr’s principle of least commitment What do we know about the architecture of vision?  Early Vision and Natural Constraints

9. The idea of cognitive architecture In order to take the Computational Theory of Mind seriously we must have a notion of strong equivalence of processes; not just a model that mimics behavior, but one that generates the behavior in the same information- processing manner – i.e. by the same algorithm. The idea of same algorithm only has meaning in the context of assumptions about the kind of machine it runs on – in other words we need to know something about the architecture of the machine on which the algorithm is executed because algorithms are architecturally-relative. Information-processing explanations may appeal to the architecture of a system or to its algorithm and data structures (to its representations) or both. This distinction will play a major role in our analysis of mental imagery theories later.

10. A playful example to illustrate the difference between appealing to architecture and appealing to representational content In considering the following example, think about how you could tell whether the regular pattern of observed behavior is due to the architecture or to the representational content. How could you generalize this example to cognitive functions?

11. An illustrative example: Mystery Code Box What does this behavior pattern tell us about the nature of the box?

12. The Moral – Explaining regularities in behavior may need to appeal to either: 1. The inherent nature of the system (to its structure), or 2. The nature of what the system represents (what it “knows”). Explanation (1) appeals to what we call the architecture of the system, while explanation (2) is an intentional explanation that appeals to the content of representations. This distinction will play a pivotal role in our later discussion

13. The first architectural property I will be concerned with here is one that places constraints on information transmission When information flow is restricted by the nature of the system’s architecture (as opposed to more ephemeral reasons – like habits) the system is said to be encapsulated or modular. Fodor, J. A. (1983). The Modularity of Mind: An Essay on Faculty Psychology. Cambridge, Mass.: MIT Press, a Bradford Book. In the course of this class we will ask whether certain proposed modules can be empirically supported. Consider these options, each of which has been defended in the literature:  Vision – early vision  Mental imagery  Visual-motor coordination  Amodal spatial competence (a geometrical module?)

14. The cultural zeitgeist gave strong support to the view that vision is highly plastic and at the mercy of cognition Through the last half of 1900s both the general public and the social science community assumed that vision and cognition were continuous – that you could not distinguish between the two.  Examples from social science, biology, and psychology. The influence of a political/cultural zeitgest.  Bruner’s New Look in Perception: Perception modeled on Science – hypothesis formation, verification & modification  Effect on Philosophy of Science: Feyerabend, Hanson, Kuhn,...  The Intuitive appeal of the New Look (social/cultural effects, linguistic effects (Worf/Sapir), effect of expectation (e.g., magic), perceiving patterns in noise, perceptual completion, perceptual learning (expertise).

15. What are some reasons for thinking that vision is cognitively penetrable (and therefore not a modular system) Expectancy and the perception of patterns  Perception in noise of words vs nonwords, sentences, statistical properties of sequences  Assimilation of perception to the norms (Postman)  Explanation in terms of readiness: Seeing as… (Potter)  Perceptual expertise (bird watchers, wine tasters…) Neuroscience evidence for top-down effects (Churchland) The experience in computer vision (Heterarchy –Shirai) Experiments on the effect of instructions, hints, …on closure figures or on stereograms Why should we doubt the continuity thesis?  Illusions….  The need to distinguish perception from belief fixation  Methodological concerns (signal detection theory)

16. Completion of incomplete patterns

17. Meaning and the reconstruction of partially hidden sounds Signal detection theory helps to isolate stages in information processing Signal detection theory helps to isolate slages in information processing VERNALITINTERVALTRLAVNEI

18. Meaning and phonetic recognition 1. The pitcher’s thoughts about the dangerous batter made him nervous 2. The pitcher’s thoughts about the dangerous battle made him nervous 3. The pitcher’s thoughts about the dangerous batt▒ made him nervous 4. The soldier’s thoughts about the dangerous batter made him nervous 5. The soldier’s thoughts about the dangerous battle made him nervous 6. The soldier’s thoughts about the dangerous batt▒ made him nervous  Signal detection analysis of responses shows that the effect is connected to the response selection stage* *Samuel, A. G. (1981). Phonemic restoration: Insights from a new methodology. Journal of Experimental Psychology: General, 110(4), 474-494.

19. Do expectations, hints, etc make affect the “closure” of fragmented figures?

20. According to Reynolds (1985) the only thing that makes a difference to ease of closure is knowing that there is a sensible reading of the fragmented figure. Knowing what the figure is about does not help.

21. Many reasons for thinking that vision is cognitively penetrable Expectancy and the perception of patterns  Perception in noise of words vs nonwords, sentences, statistical properties of sequences  Assimilation of perception to the norms  Explanation in terms of readiness: Seeing as…  Perceptual learning (bird watchers, wine tasters…) Neuroscience evidence for top-down effects The experience in computer vision Experiments on the effect of instructions, hints, …on closure figures or on stereograms Why should we doubt the continuity thesis?  Illusions….  Methodological concerns (signal detection theory)

22. The Blackboard Architecture used in many AI applications is highly non- modular because all parts can communicate with one another.

23. Pandemonium An early architecture, similar to the blackboard architecture, was proposed by Oliver Selfridge in 1959. This idea continues to be at the heart of many psychological models, including ones implemented in contemporary connectionist or neural net models.

24. From a function perspective it makes sense that the front end of vision should be built to be fast and generally (though not necessarily always) veridical. There is a lot of prima facie evidence that vision works independently of what we believe and what we expect. In order to explain why that seems to be so we need to distinguish between the part of vision that is unique to vision and the part that is shared by all intellectual processes. The unique part is called Early Vision Here are some explanations for the apparent cognitive influences on vision… Is there any reason to think that vision is a modular process?

25. Independence of principles of visual organization and cognition You can’t make illusions go away by changing what you believe! Perception often belies rational or simplicity principles Evidence from brain-damage: failure to see without loss of pattern-recognition (Humphreys & Riddoch, 1987. To see but not to see; a case study of visual agnosia) Statistical expectancy effects are post-perceptual (SDT) Visual expertise arises from learning where to attend (pre- visual) and which patterns are diagnostic (post-visual).

26. Independence of principles of visual organization and cognition You can’t make illusions go away by changing what you believe! Perception often belies rational or simplicity principles Evidence from brain-damage: failure to see without loss of pattern-recognition (Humphreys & Riddoch, 1987. To see but not to see; a case study of visual agnosia) Statistical expectancy effects are post-perceptual (SDT) Visual expertise arises from learning where to attend (pre- visual) and which patterns are diagnostic (post-visual).

27. Illusions are cognitively impenetrable

28. What about the other arguments that cognitive influence is needed in vision Noninvertibility of 3D-to-2D mapping There is an important difference between constraints on visual interpretation being built in to through evolutionary pressures and knowledge about the likelihood of particular scenes contents being used in the visual interpretation of that scene  David Marr and Natural Constraints  Natural constrains involve only optical-geometrical properties – not physical constrains or statistical properties

29. Every perspective projection of edges is infinitely ambiguous, yet is almost always perceived univocally Any set of 2D edges could have arisen for an unlimited number of 3D configurations. What makes the perception unique is the assumption that certain configurations are non-accidental – i,e. they would not change with a small change in perspective.

30. Whether these figures are seen as dimples or mounds depends on whether they are viewed in this orientation or upside-down (relative to the head)

31. Many natural constraints take the form of label-propagation

32. A simple example of a natural constraint (details in the book)

33. The label consistency requirement can explain why this figure is impossible … … the interpretation of this figure depends on where you look first

34. The label consistency requirement can also explain why this figure is ambiguous…because it has two globally consistent sets of labels

35. Perceiving a real figure as an impossible one! Because the visual system applies Natural Constraints blindly, it can be tricked in reverse – to see an impossible figure knowing that it is real – as this illusory figure constructed by Richard Gregory shows.

36. Apparent Motion – another Natural Constraint (which of these two matches will vision choose?)

38. Other kinds of apparent influence of cognition on vision The non-invertibility of 3D-2D  Natural constraints vs inference from knowledge “Problem-solving” in vision (Rock examples) Micro-modules in vision (color vs motion vs contour) (Cavanagh examples) Whatever restrictions on inverse-mapping that are build- in are not based on frequency of occurrence. They appear to be confined to spatial and optical rather than physical properties Is there a visuo-motor “module”? Milner & Goodale; patient DF. Ventral-dorsal pathways. Postural, grasping, eye-movement responses are not subject to illusions and errors that influence conscious vision.

39. Apparent “problem solving” in vision

40. Does the circle appear to move or are there two circles being covered?

42. Apparent “problem solving” in vision

43. Another example from Rock

44. Does this circle move?

45. Other kinds of apparent influence of cognition on vision The non-invertibility of 3D-2D  Natural constraints vs inference from knowledge Whatever the build-in constraints on inverse-mapping they are not based on frequency of occurrence and appear to be confined to spatial and optical rather than physical properties “Problem-solving” in vision (Rock examples) Is there a visuo-motor “module”? Milner & Goodale; DF. Ventral-dorsal pathways. Postural, grasping, eye- movement responses are not subject to illusions and errors that influence conscious vision.

46. Maybe cognition has a post- perceptual selection function? Swinney study of resolution of lexical ambiguity ContextAmbiguousUnambiguous None The man was not surprised when he found several bugs ▲ in the corner of his room. The man was not surprised when he found several insects ▲ in the corner of his room. Biased The man was not surprised when he found several spiders, roaches, and other bugs ▲ in the corner of his room. The man was not surprised when he found several spiders, roaches, and other insects ▲ in the corner of his room. Visual reading task at ▲ : Ant (related), Spy (inappropriate), Sew (unrelated)

47. The Pulfrich Pendulum illusion shows that a basic physical principle, such as the impenetrability of solid objects, is not part of the build-in constraints on visual interpretation. Here the visual system readily yields an impossible interpretation (from A Leslie).

48. Modularity is a common principle in nature (Simon’s Partially Decomposable Systems) The parable of the blind watchmaker Vision has submodules for color, motion, contour, stereopsis… and there is even evidence for larger modules such as face recognition

49. The visual system consists of minimodules that are restricted in their intercommunication

50. Not all contours are alike: Equiluminous boundaries do not lead to an interpretation as shadows

51. Equiluminous boundaries also don’t yield a depth percept

52. Where does this leave us? There is a significant part of vision, which we call early vision (after Marr) that is impervious to direct cognitive influences –It can be affected by vision at two loci: prior to vision, where attention may operate as selection, and after vision, where cognition may select which possible interpretation to keep This part of vision appears to deliver the layout of the visible surfaces, their depth orientations etc and constructed from luminance, color, motion, stereo and other sources of information While this part of vision is modular and cognitively impenetrable, it contains more than people had assumed. In fact there is evidence that it contains lookup keys perhaps in the form canonical shape, that allow rapid recognition (cf Ellsworth & Henderson)

53. How does this connect with classical issues in the philosophy of mind? Early vision may correspond to what some psychologists and philosophers have called the sensorium. But it seems to include more than has been assumed to be part of sensory information since it involves inference-like processes. However, this stage does not permit contact with information in memory so it does not allow recognition of known objects. The question of whether representations at this level are conceptual has not been broached yet. I believe the answer may be tied to other distinctions that we have not yet made – in particular to the distinction between personal and subpersonal (subdoxastic) representations, and architectural vs intentional or representation-governed processes. Also the issue of accessibility to consciousness has not been raised even though it is central to some views of sensation (or sentience) We will come back to this question later.