By Rohana Mahmud (NLP week 1-2) Communication & NL By Rohana Mahmud (NLP week 1-2)

Study of Language Language: Written: Long-term record of knowledge from one generation to another Spoken: primary mean of coordinating day-to-day behavior with others Natural (eg. Malay, English) vs. Artificial (Java, Prolog, Coding) Communication Use sign / natural language/ body language Sender and Receiver Studied in several disciplines: Linguist: structure of language PsychoLinguist: the process of human language production and comprehension Philospher: how words can mean anything & how they identify object in the world, what it means to have belief, goals and intention, cognitive capabilities relate to language CL: to develop a computational theory of Language (using the notions of algorithm & data structure from CS)

History Speech & Language Processing: overlapping fields: Computational Linguistics (Linguistics) Natural Language Processing (CS) Speech Recognition (EE) Computational Psycholinguistics (Psychology)

History 1940s & 1950s – Foundational Insights: the automaton and probabilistic/ information-theoretic models (Turing1936, Shannon 1948) Formal language theory (chomsky 1956) Probabilistic algorithm 1957-1970: 2 Camps- Symbolic and Stochastic 1970-1983: 4 paradigms – Stochastic, Logic-based, NLU (SHRDLU Winograd, LUNAR Woods), Discourse Modelling (grosz) & Believe-Desire-Intention Current: The Field comes together – probabilistic & data model, increase in speed and memory of computers & applied to Augmentative and Alternative Communication(AAC) and the rise of Web, need for language-based IR & IE.

Goal Scientific goal: Cognitive science – interdisciplinary research Technological/practical goal - NLP revolutionize the computers are used Computer that understand NL could access all information (human knowledge) NL interfaces to computers: allow complex system to be accessible to everyone. More flexible and intelligent

Speech and Language Processing Understand human: speech recognition & NLU (lip-reading) Communicating with human: NLG & Speech Synthesis Information retrieval, information extraction, inference (draw conclusions based on facts) Spelling corrections, grammar checking, machine translation Data processing vs. language processing (knowledge of language – what it means to be a word)

Application of NLU It represents the meaning of sentences in some representation language that can be used later for further processing Text-based applications Written text processing (books,newpaper, reports, manual, email, sms) = reading-based tasks Searching/finding from database of text Extracting information from text Translating documents Summarizing texts for certain purpose Story understanding

Application of NLU Dialogue-based applications – involve human-machine communication (spoken / keyboard/mouse/ recognizer) Q&A systems, eg. Query database Automated customer service (phone) Tutoring systems (interaction with students) Spoken language control of machine General cooperative problem-solving system Speech recognition <> Language understanding system (only identify the word spoken from a given speech signal, not – how words are used to communicate) Discuss ELIZA system

ELIZA system Mid-1960s, MIT, a Therapist (system) & patient (user), Weizenbaum, 1966 Algorithm: Has a Dbase of particular words (keywords) For each keyword -> store an integer, a pattern to match against the input and a specification of the output Given Sentence(S), find a keyword in S whose pattern matches S If > 1 keyword, pick the one with highest integer value Use the output specification that is associated with this keyword to generate next sentence If there are No keywords, generate an innocuous continuation statement, eg: Tell me more, Go on. (figure 1.2, 1.3 Allen)

Representations and Understanding Computing a representation of the meaning of sentences and texts (Notion of representation) Why can’t use the sentence itself as a representation of its meaning? Most words have multiple meanings (Senses). eg. Cook, bank, still (verb or noun), I made her duck. I saw a man in the park with a telescope Thus, ambiguity inhibit system from making the appropriate inferences needed to model understanding (need to resolve or disambiguate: eg. Use Lexical disambiguation: POS, word-sense disambiguation) A program must explicitly consider each senses of a word to understand a sentence

Represent meaning: must have a more precise language Mathematics & Logic and the use of formally specified representation languages (formal language) – notion of an atomic symbol Useful representation languages have 2 properties: Precise and unambiguous Capture the intuitive structure of the natural language sentences that it represents

Models and Algorithm Toolkit: state machines, formal rule systems, logic, probability theory, machine learning States, transitions among states and inpur representation Basic procedural models: deterministic, Non-Deterministic finite-state automata, finite-state transducer -> weighted automata, markov model, HMM Formal rules: Regular grammar, regular relations, context free grammar (phonology, morphology, syntax) Involve a search through a state of spaces representing by hypotheses about an input : depth first, best-first and A* search Logic – first order logic and predicate calculus, semantic network and conceptual dependency – logical representation (semantic, pragmatic and discourse)

Bibliography ACL (Association for CL) / EACL COLING (int conference of CL) Applied NLP Workshop on Human Language Technology Journal: CL & NLE IEEE ICASSP: Acoustic, Speech and Signal Processing IEEE Transactions on Pattern Analysis and Machine Intelligence IJCAI: Int Joint Conference on AI Journal: AI, Computational Intelligence, Cognitive Science

TUTORIAL – WEEK 2 Submit your first week Tutorial Presents your findings Definitions (NLP/NLG/NLU) (CL) (LE) History? Key works / research done in the area References

Tutorial 2 - task Practical Hands-on with ELIZA system or similar system Software? Algorithm? Domain?

PROJECT / COURSEWORK System Development Human-Computer Interface Database Retrieval Expert System Interface Word Analyzer (morpheme, suffix) CALL Dictionary & Senses Meaning Postulate Back-end Engine Discourse Segmentation (Essay – Paragraph – sentence – word) POS Recognizer (Malay / English) Information Retrieval (web/ book) Frequency-checker Parser

Assessment 25 % out of 50% (5% - tutorial, 20% mid-exam/test) Topic = Open System = 15 % (Demonstration) Report = 10% (SD, TE) Using HLL – Java, Prolog, C & MM softwares Showing the NL processor tasks and the interface (major/minor) Submit: Week 3/4 – topic & proposal Week 9 – demonstration & report