Cognitive Computing 2012 The computer and the mind SEARLE THE CHINESE ROOM ARGUMENT Professor Mark Bishop

(c) Bishop: The philosophy of A.I. & A-Life Background Although there is still tremendous controversy over its success – with perhaps the majority of Cognitive Scientists and Philosophers holding that the argument fails - there is a great deal of consensus over the importance of John Searle’s Chinese Room argument: In a recent article (Minds & Machines, volume 7, 1997), Larry Hauser has called it, “perhaps the most influential and widely-cited argument against the claims of Artificial Intelligence”. Stevan Harnad, editor of Behavioural and Brain Sciences (possibly damming with faint praise) asserted that it has, “already reached the status of a minor classic”. And Anatol Rapaport claims that it, “rivals the Turing test as a touchstone of philosophical inquiries into the foundations of AI”. In your opening, establish the relevancy of the topic to the audience. Give a brief preview of the presentation and establish value for the listeners. Take into account your audience’s interest and expertise in the topic when choosing your vocabulary, examples, and illustrations. Focus on the importance of the topic to your audience, and you will have more attentive listeners. 28/03/2017 (c) Bishop: The philosophy of A.I. & A-Life

(c) Bishop: The philosophy of A.I. & A-Life Central claims The central claims of Searle’s argument - first presented in his 1980 paper ‘Minds, Brains & Programs’ - are that computations alone cannot, in principle, give rise to cognitive states, and that they therefore cannot explain human cognition. Searle uses the distinction between syntax and semantics to argue that while computers can follow purely formal rules, they cannot be said to know the meaning of the symbols they are manipulating, and therefore cannot be credited with an understanding of the programs those symbols compose. They can simulate intelligent performances, but not duplicate them. 28/03/2017 (c) Bishop: The philosophy of A.I. & A-Life

Searle: two fallacies of computing The ‘information processing’ fallacy: In the CRA Searle famously asserts that a computer is “all syntax, no semantics”. I.e. The computer does not do ‘information processing’ as it does not know exactly what it is processing; a ‘bit’ of information might represent the day of the week, the weight of a body or the speed of a bullet. NB. In later writings Searle moves to deny that a computer has even [observer independent] syntax. The ‘simulation’ fallacy: Given that a computer simulation of a fire doesn't burn the neighbourhood down, or a simulation of gold make you rich, why should a computer simulation of understanding, actually understand? 28/03/2017 (c) Bishop: The philosophy of A.I. & A-Life

(c) Bishop: The philosophy of A.I. & A-Life Weak and Strong AI John Searle’s classic [and ubiquitous] taxonomy of AI systems: Weak AI “the principal value of the computer in the study of the mind is that it gives us a very powerful tool”. Strong AI “the appropriately programmed computer really is a mind”; it is conscious and has intentional states (i.e. the internal states have an ‘about-ness’ and ‘direction’ to them with respect to the entities to which they refer); i.e. the distinguishing property of mental phenomena is that of being necessarily directed upon an object, whether real or imaginary. 28/03/2017 (c) Bishop: The philosophy of A.I. & A-Life

(c) Bishop: The philosophy of A.I. & A-Life Implications Searle’s argument - if sound - would clearly undermine the very foundation of much cognitive science grounded, as it is, on a ‘Strong AI’ supposition. The importance of Searle’s article is thus: Philosophical Can a computational theory of mind explain human understanding? Is the Turing Test an adequate test for machine intelligence? Practical – (in Computing/Cybernetics/Cognitive Science) Are computational approaches to AI a good idea? … and this has ensured that it has been widely attacked - and defended - with almost religious fervour, by exponents of both disciplines. 28/03/2017 (c) Bishop: The philosophy of A.I. & A-Life

Schank and Abelson’s research using ‘scripts’ In 1977 Schank and Abelson published information on a program they had created, which could accept a story and then answer questions about it, using a large set of rules, heuristics and scripts, (Schank, R.C. & Abelson, R.P. (1977). Scripts, Plans, Goals & Understanding, Hillsdale, NJ:Erlbaum). A script is a detailed description of a stereotypical event unfolding through time. For example, a system dealing with restaurant stories would have a set of scripts about typical events that happen in a restaurant: entering; choosing a table; ordering food; paying etc. And the script for a lecture on “The computer and the mind”? If you have several points, steps, or key ideas use multiple slides. Determine if your audience is to understand a new idea, learn a process, or receive greater depth to a familiar concept. Back up each point with adequate explanation. As appropriate, supplement your presentation with technical support data in hard copy or on disc, e-mail, or the Internet. Develop each point adequately to communicate with your audience. 28/03/2017 (c) Bishop: The philosophy of A.I. & A-Life

SAM – the Script Applier Mechanism A more recent application of scripts is found in a program called SAM. To answer questions about stories, SAM needs to be able to expand them. e.g. “John went to a restaurant. He asked a waitress for coq au vin. He left a very large tip.” SAM INFERS: “John went to a French restaurant. He sat down at a table. He read a menu. He ordered coq au vin from the waitress. He ate the coq au vin. He enjoyed the meal. He left a large tip. He didn’t pay cash. He exited from the restaurant.” 28/03/2017 (c) Bishop: The philosophy of A.I. & A-Life

FRUMP: the Fast Reading Understanding and Memory Program FRUMP takes as input live news feed, (from United Press International wire service) and automatically produces news summaries. Proponents of ‘Strong AI’ have claimed that: Programs like SAM and FRUMP actually understand the stories they operate upon. The (scripting) mechanisms that SAMP and FRUMP embody actually explain human comprehension. 28/03/2017 (c) Bishop: The philosophy of A.I. & A-Life

(c) Bishop: The philosophy of A.I. & A-Life The Chinese room A modern critic of Strong AI is John Searle. Searle’s most famous work on machine understanding, is the Chinese Room gedankenexperiment (Searle, J. “Minds, Brains & Programs”, The Behavioural and Brain Sciences, 3 (1980): 417-24). In the paper Searle describes a situation where he is locked in a room where he is presented with a batch of paper on which are Chinese symbols that he does not understand. Searle is then given a second batch of Chinese symbols together with a set of rules in English that describe an algorithm for correlating the second with the first. Finally he is given a third batch of Chinese symbols together with another set of rules in English to enable him to correlate the third batch with the first two, and these rules instruct him how to return certain sets of Chinese symbols in response to the symbols given in the third batch. 28/03/2017 (c) Bishop: The philosophy of A.I. & A-Life

Scripts, stories and questions Unknown to Searle, the people outside the room call the first batch of Chinese symbols, the script; the second set the story; and the third questions about the story. The symbols he returns they call answers to the questions about the story; and the set of rules Searle is obeying, the program. To complicate matters further the people outside also give him stories in English and ask him questions about them in English to which he can reply in English. 28/03/2017 (c) Bishop: The philosophy of A.I. & A-Life

Understanding Chinese? After a while Searle gets so good at manipulating the symbols and the ‘outsiders’ get so good at supplying the rules which he follows, that the answers he gives to the questions in Chinese symbols become indistinguishable from those a true Chinese man might give. From the external point of view, the answers to the two sets of questions, one in English the other in Chinese, are equally good, but in the Chinese case, he behaves ‘like a computer’, as an instantiation of a computer program. Searle. Searle trenchantly points out that he has passed the Turing test and hence that the claim of Strong AI is that he can ‘think in’ and ‘understand’ Chinese, whereas he still does not understand a word of the language! 28/03/2017 (c) Bishop: The philosophy of A.I. & A-Life

(c) Bishop: The philosophy of A.I. & A-Life The Chinese room .. 28/03/2017 (c) Bishop: The philosophy of A.I. & A-Life

From ‘low-level grammars’ … Historically real A.I. practitioners have been incredulous at the extreme simplicity of the symbol correlation rules described by Searle; rules that simply … … “correlate one set of formal symbols with another set of formal symbols … merely by their shape”; … such that [typically very trivial combinations of] un-interpreted symbols – Squiggles - map simply onto others - squoggles. It has always seemed likely to A.I. engineers that any machine understanding program with a claim to real-world generality would require a very large and complex rule-base (program) typically applying very high-level rules (functions).

… to high level programs However it is also clear from MBP Searle intended the CRA to be general, applicable to any possible A.I. program (grammar based; rule based; neural network; Bayesian etc) [JS] “I can have any formal program you like, but I still understand nothing”. So if the CRA succeeds, it must succeed against even the most complex high-level systems. So, in a spirit of cooperation (between A.I. practitioners and Searle) let us consider a more complex formal program/rule-book-system which has (as one high-level- rule) a call to, say, Google-translate. Hence the internal representations used by monoglot Searle (as the ‘man in the room’) scribbled on bits of paper could now perhaps maintain partial interpretations of the [unknown] Chinese input symbols as “symbol-strings-in- English”.

High-level English: the English [language] reply revisited Because Searle speaks English, by default he brings to the room an understanding of: (A) the high-level rules [as defined in English] in the rule-book; (B) any English language internal representations that the rules book requires - Searle’s scribbles on paper; Hence it is likely that the repeated application of such [externally- grounded] high-level-rules to Chinese text could foster the emergence of semantics and understanding in even a monoglot English speaker like Searle … … via a process analogous to ones gradual understanding of a Chinese text via the repeated use of a Chinese-English dictionary .

“Escaping from the Chinese room” “in (Boden, M “Escaping from the Chinese room” “in (Boden, M.) (ed) ‘The philosophy of A.I.’, (1988)” But does a computer CPU ‘understand’ its program and variables (internal representations) as coded as raw binary signals? In her 1988 paper Maggie Boden suggests that, unlike say [the human-driven manipulations of] formal logic, it does ... ... because unlike the rules of logic, the execution of a computer program ‘actually causes events to happen (e.g. data read/writes to memory and peripherals)’. and such ‘causal semantics’ enable Boden (contra Searle) to suggest that a CPU processing Chinese symbols actually does have a ‘toe-hold on semantics’ … … “It is a mistake to regard [executing] computer programs as pure syntax and no semantics”. The analogy Boden draws is to to Searle’s understanding of the English rule-book ; if correct, this suggests the [extended; high-level] ‘English reply’ holds.

On the execution of a computer program However it is not Searle’s command of the [extended] English rule-book alone that enables the [extended] English reply to have force; it is the combination of (a) Searle’s understanding of the [extended] English rules and (b) the dynamic [English language] data on which they operate. But in contrast to Searle [reading ‘rules-in-English’ on ‘data-in -English’]: (a) the computer CPU does not understand its rule-book [program] anymore than water in a stream understands its flow down-hill; both processes are strictly entailed by their current state and that of their environment (data); (b) and for the CPU, the strings of binary data that it processes exactly constitute un- interpreted ‘formal symbols’ of the type Searle described in his original exposition of the CRA. Hence even the extended English reply fails; “all syntax and no semantics” any physical computer (as it executes its machine-code program on un-interpreted binary data) is merely analogous to [monoglot] Searle’s attempts to understand an unknown Chinese text using only a Chinese-Chinese dictionary..

The ‘Systems Reply’ (Berkeley) Over the 30+ years since its inception, perhaps the most widely deployed response to the CRA has been ‘The Systems Reply’. This states that “It is a mistake to limit understanding to the animate simulator, but rather it belongs to the system {JS; Rule-book; Google-translate; scraps-of-paper etc.} as a whole”. JS: “Internalise all the components [of the high-level system] so that they are all in Searle’s mind” … … “Now there isn't anything at all to the system that [Searle] does not encompass” and yet Searle continues to trenchantly insist that he doesn’t understand a word of Chinese, even though his answers to questions posed to him in Chinese continue to be indistinguishable to those a native Chinese speaker might give. In response to this ‘internalisation’ move some (e.g. John Haugeland) insist that such internalisation produces two distinct cognitive systems - (a) Searle and (b) Searle-as- CRA - with only the latter genuinely ‘understanding Chinese’.

John Haugeland’s response to Searle’s ‘systems reply’ “ What we are to imagine in the internalisation fantasy is something like a patient with a multiple personality disorder: one ‘personality’ Searle is fluent in English (both written and spoken), doesn't know a word of Chinese, and is otherwise perfectly normal (except that he has the calculative powers of a mega idiot savant); the other ostensible personality - let's call him Hao - is fluent in Chinese (though only written, not spoken), has no English, and, moreover, apart from seeming to be able to read and write, is deaf, dumb, blind, and paralysed. Why, exactly, should we conclude that Hao doesn't understand the Chinese that he appears to be reading and writing ‘automatically’, as it were ? ”

A tale of two jokes ... In response to Haugeland, contrast Searle responding to a joke in English and Hao responding to a joke in Chinese. Although, ex-hypothesi, the external responses will be the same in both cases ... “Ha ha” ... there is a clear ontological distinction between the state of being John Searle and that of being Hao with only the former actually ‘understanding’ the joke. In other words, merely generating appropriate [external] behaviour is no guarantee of the presence of relevant underlying cognitive state(s).

Ontological and epistemological considerations It is this basic, fundamental distinction … between epistemic concerns: The knowledge we deploy to check for the presence of a cognitive state in a known cognitive system (e.g. in a human); AND ontological: On being in [instantiating] that state (e.g. in a machine); ... that continues to lie at the root of much confusion in the on- going Chinese room debate.

(c) Bishop: The philosophy of A.I. & A-Life The robot reply “But what if we built a robot, controlled by a computer, that could directly inter-react with the world? By directly interacting with the world such a robot would have genuine understanding and mental states”. Cf. Harnad’s ‘Total Turing Test’ : the TTT involves demonstrating successful performance in both ‘linguistic’ and ‘robotic’ behaviours and incorporates the same range of empirical data that is available in the human case [i.e. over a lifetime]. JS. It is interesting to note that now the cognitive scientists are retreating from their original claim that only appropriate computations are necessary for understanding. Nonetheless, if we rebuilt the Chinese room inside the ‘head’ of the [TTT] robot, with symbolic [sense] inputs and symbolic [motor] outputs, the original scenario is unchanged. 28/03/2017 (c) Bishop: The philosophy of A.I. & A-Life

The connectionist reply “Build a computer that simulates the firing of real brain neurons when it communicates in Chinese”. JS. Again, it is interesting to note that now the cognitive scientists are retreating from their original symbolic theory of mind to a sub-symbolic, connectionist theory of mind, where knowledge is represented as a distributed pattern of weights. We could design a network of water pipes to mimic the neural network and emulate the neural network by applying rules to control the opening and closing of the pipe valves. However the operation of such a system is still formally defined by a set of rules and hence remains vulnerable to the CRA; Searle imagines himself manically running around the network of pipes, opening and closing appropriate valves [and hence producing ‘appropriate Chinese responses], but still not understanding a word of Chinese. 28/03/2017 (c) Bishop: The philosophy of A.I. & A-Life

(c) Bishop: The philosophy of A.I. & A-Life The combination reply “Link all of the above together forming a bi-pedal robot, controlled by neural networks, that interacts directly with the world. Surely this forms a convincing and decisive argument…” JS. If we knew nothing more then perhaps we would attribute intentionality, understanding and intelligence to such a machine, just as we do for other humans, apes etc. However given that we know it is controlled by a computer program, we must still continue to think of the robot as nothing more than a sophisticated dummy. 28/03/2017 (c) Bishop: The philosophy of A.I. & A-Life

(c) Bishop: The philosophy of A.I. & A-Life The other minds reply “How do you know that people understand Chinese or anything else; only by their behaviour…” JS. It is a base axiom of cognitive science that humans do possess cognitive states, genuine understanding and intentionality … … in much the same way that a physicist knows the reality, (and the fundamental know-ability of), physical matter. 28/03/2017 (c) Bishop: The philosophy of A.I. & A-Life

The many mansions reply “John Searle only considers today's analogue & digital computers; would his argument apply to the computers of tomorrow?” E.g. Biological, Quantum computers ... JS. Yes, assuming such systems remain bound by the Church/Turing thesis... 28/03/2017 (c) Bishop: The philosophy of A.I. & A-Life

Conclusions The CRA suggests that merely generating appropriate [external] behaviour is no guarantee of the presence of relevant underlying cognitive state(s). And it is this basic, fundamental distinction between epistemic concerns AND ontological … … that continues to lie at the root of much confusion in the on-going Chinese room debate. 28/03/2017 Dr. Mark Bishop