2. Is Cognitive Science Special? In what way is it special? What’s in the mind that we may know it? (cf Shakespeare’s ‘What's in the brain, that ink may character?’) http://ruccs.rutgers.edu/faculty/pylyshyn.html Zenon Pylyshyn, Rutgers Center for Cognitive Science Sept 2, 2014

3. Cognitive science is a delicate mixture of the obvious and the incredible Granny was almost right: Behavior really is governed by what we believe and what we want (together with the mechanisms for representing and for drawing inferences from these)

4. The central role of representation presents some serious problems for a natural science  What our beliefs are about is what matters  But how can the fact that a belief is about some particular thing have an observable consequence? How can the presence of “holy grail” (or “Santa Claus”) in a belief determine behavior when these objects do not exist? Intentionality is the name philosophers give to the aboutness of mental states  In a natural science if “X causes Y” then X must exist and be connected to Y by natural law(s)! It’s even worse than that; even when does X exist, it is not X’s physical properties that are relevant to determining behavior! It’s how X is represented (i.e., its what X is represented as -- its intentional property).  e.g., the North Star & navigation

5. This dilemma is sometimes referred to as Brentano’s problem, or  The Problem of Intentionality  What determines what we do is what our mental states are about, but aboutness is not a category of natural science. It does not occur in any natural law.  Our behavior follows our beliefs and desires!  That is why Brentano (and Wolfgang K ӧ hler after him) concluded that psychology was beyond the reach of natural science.

6. Is it hopeless to think we can have a natural science of cognition? Along comes The computational theory of mind “the only straw afloat”

7. What we have learned: Intelligent systems behave the way they do because of what they represent (what they represent things as being) But in order to function under principles of natural science, the representations must have physical (including chemical & mechanical) properties that encode the content of the representation (i.e., its behavior must be caused by physical codes). How to encode knowledge in physical properties is by first encoding it in symbolic form (Symbolic Logic tells us how) and then instantiating those symbolic codes physically (computer science tells us how).

8. Cognitive Science and the Tri-Level Hypothesis Intelligent systems are inherently organized at three (or more) distinct levels 1. The physical or biological level 2. The symbolic or syntactic level 3. The knowledge or semantic level This is not just a metter of convenience in describing a complex system: It’s how the system really is organized. So explaining different regularities may require appeal to different properties and principals.

9. Does intentionality (and the trilevel hypothesis) only apply to high- level processes such as reasoning? An example from color vision: “Red light and yellow light mix to produce orange light” This remains true for any way of creating red and yellow light: e.g. yellow may be light of 580 nanometer wavelength, or it may be a mixture of light of 530 nm and 650 nm wavelengths.  So long as one light looks yellow and the other light looks red this “law” of color mixing will hold.

10. Does the trilevel hypothesis only apply to high-level processes such as recognition of familiar things? An example from vision – linkages among interpretations provides the basis for the natural constraint methodology

11. If cognitive processes are at a different level of organization from the physical level, how can we ever discover what they are?  Answer: We are limited only by the imagination of the experimenter, e.g.,  Relative complexity evidence (RT, error rates…)  Intermediate state evidence Eye tracking Verbal Protocols  Event Related Potentials (EEG)  fMRI, clinical observations of brain damage, Psychophysical methods (SDT), Etc… Skip to 23 or 17 ?

12. Example: Sternberg’s rapid memory search  Give subjects a set of 2 to 6 letters to memorize (this is the memory set). Next tasks will refer to these letters.  Present subjects with a single letter (this is the probe).  Subjects must respond or to indicate whether the probe letter was one of the memory set letters (usually respond by pressing a button, the time of which is recorded).  Use Reaction Time (RT) as a measure of how much processing went on during the search.  Plot RT against Independent (manipulated) Variables (e.g. number of letters in the set, whether the probe was in the memory set or not – ‘present’ vs ‘absent’)  Results ……

13. Possible Result 1: Sternberg memory search task What do they tell us about how people do it? Is this Input-Output equivalent or is it strongly equivalent to human performance? Consider the following possible outcomes (#1): What does it suggest about the nature of the cognitive process. Option 1

14. Possible Result 2: Sternberg memory search task What do they tell us about how people do it? Is this Input-Output equivalent or is it strongly equivalent to human performance? Consider the following possible outcome (#2): What does it suggest about the nature of the cognitive process. Reaction time (seconds) Option 2

15. Results of the Sternberg memory search task What do the results tell us about how people do it? Is this Input-Output equivalent or is it strongly equivalent to human performance? Self-terminating search Exhaustive search

16. Related Core Topic ● The most important concept in cognitive science, and the one that is most likely to unite behavioral and neurological research, is the concept of Cognitive Architecture (see Newell’s Unified Theories of Cognition and Gallilstel’s account of spatial representation and navigation, Gallistel, C. R. & King, A. P. (2009). Memory and the computational brain: Why cognitive science will transform neuroscience. New York: Wiley/Blackwell..) ● The approach I am taking is designed to cast light on certain intuitively tempting proposals concerning different forms of mental representation (especially the intuitive idea of mental pictures).

17. Aside on scientific explanation The difference between explanations that appeal to (relatively fixed) architecture and those that appeal to (relatively malleable) representations. Suppose we observe and summarize some reliable behavioral regularity. What does it tell us about the nature of the system, or how it works?  Note that a systematic record might allow us to extrapolate and make predictions. But the reason we study science (esp cognition) is not mainly to make predictions, but to understand how something works.

18. The parable of the found mystery box A Cognitive Scientist, out walking in a field one day, comes upon a black box which happens to have a meter and recorder that records the meter’s changes over time (like an EEG record). The Cognitive Scientist examines lots of records generated by the box and finds the pattern to be quite systematic. It follows the following regular pattern: Clarifying the Obligatory requirement An example to illustrate the concept of capacity

19. An illustrative example: Mystery Code Box What does this behavior pattern tell us about the nature of the box?

20. An illustrative example: continued What does this behavior pattern tell us about the nature of the box? Careful study reveals that pattern #2 only occurs in this special context when it is preceded by pattern A

21. Regularities in behavior may tell us nothing about how a system works because the pattern may be due to one of two different sources: 1. The inherent nature of the system (to its structure or its capacity)  its architecture  2. The nature of what the system represents (what it “knows”) about the outside world.  It this example, the reason why this pattern was observed, as opposed to some other pattern that the system might equally have exhibited, is external to the system itself.  Note: Science is about understanding what could be – it is about counterfactuals – not what is typical.

22. Where it matters: I will briefly discuss the architecture vs knowledge distinction to understanding what goes on when we reason using what seems like different forms of thought i.e.,  Thinking using mental imagery

23. Should these examples of experimental findings be attributed to properties of images or to tacit knowledge? Mental Color mixing ? The effect of image size ? Scanning mental images ?

24. Mental color-mixing example This example is typical of almost all research on how mental images are used in reasoning. They rely on people’s use of what they tacitly know would happen in the world and then making their imaginings mimic that.

25. The important distinction between the architecture and the content of representations  It is important to ask why certain patterns (e.g. color mixing) occur – whether the pattern is caused by fixed properties of the architecture or because of properties of what is represented (i.e., what the observer tacitly knows about the behavior of that which is represented)  If they occur only because the theorist says they do, then that is a free empirical parameter (a wild card).  The important consequence is that if we allow one theory to stipulate that a certain pattern occurs, without there being a principle reason for it, then any other theory can add the same assumption as well. Such unconstrained theories explain nothing.  This failure of image theories is quite general – all picture theories suffer from the same lack of principled constraints.

26. A famous example: Mental Scanning Hundreds of experiments have now been done demonstrating that it takes longer to scan attention between places that are further apart in the imagined scene. In fact the time-distance relation is linear. These experiments have been reviewed in:  Denis, M., & Kosslyn, S. M. (1999). Scanning visual mental images: A window on the mind. Cahiers de Psychologie Cognitive / Current Psychology of Cognition, 18(4), 409-465.  Rarely cited are experiments by Liam Bannon and me (described in Pylyshyn, 1981) which I will summarize for you. A window on the mind

27. What is assumed in the mental picture explanation of mental scanning? ● In actual vision, it takes longer to scan a greater distance in the world because real distance, real motion, and real time is involved, therefore this equation holds due to natural law: Time = distance speed  But what ensures that a corresponding relation holds in an image? The tacit assumption is that the image is laid out in real space!  But what if that option is blocked for empirical reasons? You might try to weasel out by appealing to a “Functional Space,” which theorists liken to a matrix data structure in which some pairs of cells are closer and others further away, and to move from one to another it is natural that you pass through intermediate cells.  But such constraints are not inherent in the architecture if they can be changed by changes in beliefs – which they invariably can be: You can visit every 3 rd or 4 th cell – or none at all – in ‘scanning’ across the matrix!  The central problem with imagistic explanations… 

28. Studies of mental scanning Does it show that images have metrical space? (Pylyshyn & Bannon. Described in Pylyshyn, 1981) Conclusion: The image scanning effect is Cognitively Penetrable  i.e., it depends on Tacit Knowledge.

29. The intentional fallacy failure to recognize an ambiguity: A representation of X with property P can mean: [1] A representation of (X with property P)  In this case it is X that has property P, A representation of a red circle may be neither red nor round! [2] (A representation of X) with property P  In that case the representation has property P A part of a picture of a red circle is usually red and circular

31. So why does it feel like we are not computing, but just observing some spontaneous events? ● Because the content of our conscious experience is a very poor guide to what causes our cognitive processing. Science is concerned with causes, not just correlations. ● Because we can’t assume that the way things seem has much to do with how it works (e.g., language understanding).  As in most sciences, the essential causes are far from obvious (e.g., Why did people think that the earth went around the sun rather than that the earth rotates?). What does it look like?  In the case of cognition, what is going on is a delicate mixture of the obvious (what Granny or Shakespeare knew about why people do what they do) and the incredible.

32. So what is in the brain? Shakespeare asked: What's in the brain, that ink may character? The best hypothesis so far (i.e., the only one that has not been shown to be clearly wrong) is that the brain is a species of computer in which representations of the world are encoded in the form of symbol structures, and actions are determined by calculations (i.e., inferences) on these symbolic encodings. Additional detailed discussion can be found in: Fodor, J. A. 1975. The Language of Thought. New York: Crowell. (Second edition LOT2, 2013) Fodor, J. A. (2009). Introduction: Computation, Cognition and Pylyshyn: So what's so good about Pylyshyn? In D. Dedrick & L. M. Trick (Eds.), Computation, Cognition, and Pylyshyn (pp. ix-xvii). Cambridge, MA: MIT Press. Fodor, J. A., & Pylyshyn, Z. W. (2014). Minds Without Meanings. Cambridge, MA: MIT Press. Pylyshyn, Z. W. (1984). Computation and cognition: Toward a foundation for cognitive science. Cambridge, MA: MIT Press. Pylyshyn, Z. W. (2007). Things and Places: How the mind connects with the world (Jean Nicod Lecture Series). Cambridge, MA: MIT Press. Pylyshyn, Z. W. (2009). Perception, Representation and the World: The FINST that binds. In D. Dedrick & L. M. Trick (Eds.), Computation, Cognition and Pylyshyn. Cambridge, MA: MIT Press.