1. Topic: Cognitive Architecture The fixed parts of cognitive processes
No matter what your favorite computational theory may be, it always assumes certain fixed properties of the system within which it functions

2. Among the architectural properties assumed by any computational theory:
The form of representations. Computational theories typically assume that representations consist of symbol structures. But how do they represent magnitudes? Numerals or analogs?
Available operations. The building blocks of processes that combine to form algorithms.
Information flow and constraints on subprocesses (e.g., encapsulated modules?)
Fixed capacities determined by the innate structures of the mind/brain, as opposed to those that depend on their niche environment.

3. The form of representation: The case of visual perception
Distinguishing form and content
What is the form of mental representations?
In memory? In perception? In thought?
Lectures by R.C. Gallistel & B. McLaughlan
In imagination (in mental imagery).
Does vision depend on cognition, or is it encapsulated so it cannot use knowledge?
Pylyshyn, Z. W. (1999). Is vision continuous with cognition? The case for cognitive impenetrability of visual perception. Behavioral and Brain Sciences, 22(3), )

4. The Form of visual representation
Our subjective impressions (our intuitions) of what our representations are like are seriously unreliable and misleading. We do not experience the form of a representation, only of its content – what it is about or what it represents
A representation of a large square object is not large and square
But the demands of scientific explanation are quite different; and they almost always lead us to unfamiliar and counterintuitive conclusions

5. This is what our conscious experience suggests goes on in vision…

6. This is what the demands of explanation suggests must be going on in vision…

7. Some reasons given for believing that there is a “pictorial” representation in vision
Visual input is highly incomplete (moving peephole view), but visual system operates over filled-in (completed) displays
Representations are in world, not retinal, coordinates
Our representation not only completes the missing retinal information, but it also represents panoramic (broad) and dynamic (moving) information
It’s obvious that we experience a rich and finely detailed 3D perceptual world – so it must be recreated in the brain as a 3D display

8. Standard view of saccadic integration by superposition

9. Partial patterns are seen as completed …
Where’s Waldo?

10. Vision patterns can present many different percepts
Seeing as: It’s what you see the figure as that determines behavior – not its physical properties.
What you see one part as determines what you see another part as.
There must be a representation that contains the interpreted pictorial pattern

11. Is it possible to specify a set of ways of physically presenting a visual stimulus for it to be perceived in a certain way?

12. Can you think of other ways of presenting a stimulus so it is perceived as e.g., a Necker Cube?

13. Errors in recall suggest how visual information is encoded
Children have very good visual memory, yet often make egregious errors of recall
Errors in relative orientation often take a canonical form
Errors in reproducing a 3D image preserve 3D information

14. Errors in recall suggest how visual information is encoded
Children more often confuse left-right than rotated forms
Errors in imitating actions is another source of evidence

15. Ability to manipulate and recall patterns depends on their conceptual, not geometric, complexity
Difficulty in superimposing shapes depends on how they are conceptualized
Look at first two shapes and superimpose them in your mind; then draw or select one that is their superposition ?

16. Many studies have shown that memory for shapes is dependent on the conceptual vocabulary available for encoding them e.g., recall of chess positions by beginners and masters

17. Other examples showing that it is how you represent something that is relevant to cognitive science
Example from color vision
“Red light and yellow light mix to produce orange light”
This remains true for any way of getting red light and yellow light:
e.g. yellow may be light of 580 nanometer wavelength, or it may be a mixture of light of 530 nm and 650 nm wavelengths…
So long as one light looks yellow and the other looks red the “law” will hold.

18. What is the evidence against an inner pictorial display?
We will see that constraints on explanation demand a symbolic form of representation.
Causal explanation requires strong equivalence
Some fields of study, such as History, can only provide retrospective analyses and “reasons” not causal models: Is Psychology like History?

19. If cognitive processes are at a different level of organization from the physical level, how can we ever find out what they are – i.e., how can we discover what algorithm is being used?
We are limited only by the imagination of the experimenter, e.g.,
Relative complexity evidence (RT, error rates…)
Intermediate state evidence
Verbal Protocols, Eye tracking
Stage analysis (additive factors method)
Event Related Potentials (EEG)
fMRI
clinical observations of brain damage
Psychophysical methods (SDT)
Etc…

20. Strong Equivalence and the role of cognitive architecture
For two models to be in strong equivalence they must have the same architecture (at least in theoretically relevant ways).

21. The concept of cognitive architecture
If differences among behaviors (including differences among individuals) is to be attributed to different beliefs or to different algorithms, then there must be some common set of basic operations and mechanisms. This is called the Cognitive Architecture
The concept of a particular algorithm, or of being “the same algorithm” is only meaningful if two systems have the same architecture. Algorithm is architecture-relative.
The architecture is the part of the system that does not change when beliefs change. So it defines the system’s Cognitive Capacity.

22. Recall our earlier example of a model of the Sternberg task
Store memory set as a list L. Call the list size = n
Read target item, call it 
Check if  is one of the letters in the list L
If found in list, assign Resp = “yes” otherwise Resp = “no”
If Resp =“yes”, set  = K * n  Random(20  x  50)
If Resp =“no”, set  = K * n  Random(20  x  50)
Print , Print 
Go to 2
Is this the way people do it? How do you know?

23. The program outputs Yes/No response and Time
Notice that for this to be a model, various aspects of the architecture have to be specified
Store memory set as a list L. Call the list size = n
Read target item, call it 
Check if  is one of the letters in the list L
If found in list, then assign =“yes” else  =“no”
If  =“yes”, then set  = K set * n  Rand(20  x  50)
If  =“no”, then set  = K * n  Rand(20  x  50)
Print , Print 
Go to 2
 HOW?
The program outputs Yes/No response and Time

24. Tacit assumptions made in constructing a computational model
But there are many other properties of algorithms that constitute assumptions about the cognitive architecture. One class of properties seems so natural that it goes unquestioned – it’s the control structure
Operations are carried out in sequence. No operation can begin until the previous one is completed. This seems so natural that it goes unnoticed as an assumption.
Another fundamental property that is assumed is that control is passed from one operation to another (e.g., “go to”), as opposed to being grabbed in a “recognize-act” cycle in architectures called Production System or Blackboard System Rules are: Condition  Action (or IF  THEN)

25. Recall the other considerations that are special to cognitively determined behavior
The Cognitive Penetrability of most cognitive processes. A regularity that is based on representations (knowledge) can be systematically altered by imparting new information that changes beliefs.
The critical role of "Cognitive Capacity". Because of an organism's ecological or social niche, only a small fraction of its behavioral repertoire is ever actually observed. Nonetheless an adequate cognitive theory must account for the behavioral repertoire that is compatible with the organism's structure, which we call its cognitive capacity.
That’s why “variance accounted for” is a poor measure of a theory’s explanatory value.

26. An even more fundamental reason why it is essential to get the cognitive architecture right is the critical role of "Cognitive Capacity"
Because an organism may remain in its ecological or social niche, only a small fraction of its behavioral repertoire is ever actually observed. But an adequate explanatory theory must reveal an organism's structure, or its cognitive capacity. To do that it must account for the organism’s entire potential behavioral repertoire.
That’s why “variance accounted for” is a poor measure of a theory’s explanatory value.

27. The productivity and systematicity of systems of mental representation
The productivity and systematicity of systems of mental representation. Systems of mental representation are structured so that if they are capable of representing certain situations then they are also capable of representing an unbounded number of other related situations. This leads to the requirements that representations be compositional, and that they have constituent structure. <More on this later!>

28. Observed regularity versus capacity: The difference between explanations that appeal to mental architecture and those that appeal to tacit knowledge
The parable of a found mysterious box:
Suppose we observe some robust behavioral regularity of an unknown system. What does it tell us about the nature of the system or about its intrinsic properties?

29. The parable of a found mystery box:
Observed regularity versus capacity: The difference between explanations that appeal to mental architecture and those that appeal to tacit knowledge
The parable of a found mystery box:
Walking through a field one day, a scientist comes upon a mysterious box. The box has a meter that records some aspect of its behavior. Suppose that after many days of recording, the scientist finds some robust behavioral regularity of the box’s recording. What does it tell him about the nature of the box or about its intrinsic causal properties?

30. An illustrative example: Mystery Code Box
What does this behavior pattern tell us about the nature of the box?

31. An illustrative example: Mystery Code Box
Careful study reveals that pattern #2 only occurs in this special context when it is preceded by pattern A
What does this behavior pattern tell us about the nature of the box?

32. It tells us (very nearly) nothing about the nature of the system under study
Why? Because the observed behavior, although it is an objective true record, is but a small part of what the box is capable of.
The sample we observed is attributable to the environment – including what the box is used for – rather than to the fixed structure (the architecture) or the nature of the box.

33. How can an objective record of behavior not tell you about the nature of a system? What more is there?
In this example, what the scientist found happens to be a box that transmits English messages in international Morse code.
Two short blips is a code for i, one blip is an e and a long-short-long-short sequence is a c.
Thus the regularity that the scientist found can be explained by the spelling rule in English: “i before e except after c”!
Nothing inside the box can explain that because the box has a capacity not revealed by its usual ecological behavior.

34. The Moral: Regularities in behavior may be due to either:
The inherent nature of the system or its structure or architecture.
The content of what the system represents (what it “knows”).

35. Why it matters:
A great many regular patterns of behavior reveal nothing more about human nature than that people do what follows rationally from what they believe.
An example from language understanding
The example of human conditioning

36. An example from language understanding
Examples from language.
John gave the book to Fred because he finished it
John gave the book to Fred because he wanted it
The city council refused to give the workers a permit for a demonstration because they feared violence
The city council refused to give the workers a permit for a demonstration because they were communists

37. Another example where it matters: The study of mental imagery
Application of the architecture vs knowledge distinction to understanding what goes on when we reason using mental images

38. Examples of behavior regularities attributable to tacit knowledge
Color mixing, conservation of volume
The effect of image size ?
Scanning mental images ?

39. Color mixing example

40. Conservation of volume example

41. Our studies of mental scanning
(Pylyshyn & Bannon. See Pylyshyn, 1981)
There is even reason to doubt that one can imagine scanning continuously (Pylyshyn & Cohen, 1998)

42. Can you rotate a mental image?
Which pair of 3D objects is the same except for orientation?

43. Do mental images have size. Imagine a very small mouse
Do mental images have size? Imagine a very small mouse. Can you see its whiskers? Now imagine a huge mouse. Can you see its whiskers?
Which is faster?

45. Why do so many people deny these obvious facts about mental imagery?
The power of subjective experience (phenomenology). The mind-body problem is everywhere: but subjective experience does not cause behavior! (e.g., conscious will)
The failure to make some essential distinctions
Content vs form (the property of images vs the property of what images are about) {compare the code box example}
An image of X with property P can mean
(An image of X) with property P or
An image of (X with property P)
Capacity vs typical behavior: Architecture vs knowledge

46. Are all the things we thought were due to internal pictures actually due to tacit knowledge?
Other reasons for imagery phenomena:
Task demands: Imagine that X = What would it be like if you saw X?

47. Are there pictures in the brain?
There is no evidence for cortical displays of the right kind to explain visual or imaginal phenomena

48. So what is in the brain?
The best hypothesis so far (i.e., the only one that has not been shown to be clearly on the wrong track) is that the brain is a species of computer in which representations of the world are encoded in the form of symbol structures, and actions are determined by calculations (i.e., inferences) based on these symbolic encodings.

49. So why does it not feel like we are doing computations?
Because the content of our conscious experience is a very poor guide to what is actually going on that causes our experiences and our behavior. Science is concerned with causes, not just correlations.
Because we can’t assume that the way things seem has much to do with how it works (e.g., language understanding)
As in most sciences, the essential causes are far from obvious (e.g., why does the earth go around the sun? What is this table made of ? etc.).
In the case of cognition, what is going on is a delicate mixture of the obvious (what Granny or Shakespeare knew about why people do what they do) and the incredible

50. Often we get the wrong story because we have the wrong methods or instruments. Conscious experience may be one of those

51. If all else fails there is always parsimony and generality…(they worked well in physics and linguistics!)