Presentation on theme: "Outline: Where have we been and were are we going? Were making consistent progress, or Were running around in circles, or –1950s: Empiricism (Information."— Presentation transcript:
Outline: Where have we been and were are we going? Were making consistent progress, or Were running around in circles, or –1950s: Empiricism (Information Theory, Behaviorism) –1970s: Rationalism (AI, Cognitive Psychology) –1990s: Empiricism (Data Mining, Statistical NLP, Speech) –2010s: Rationalism (TBD) Were going off a cliff… –Dont worry; be happy No matter what happens, its goin be great! Rising tide of data lifts all boats
Rising Tide of Data Lifts All Boats If you have a lot of data, then you dont need a lot of methodology 1985: There is no data like more data –Fighting words uttered by radical fringe elements (Mercer at Arden House) 1995: The Web changes everything All you need is data (magic sauce) –No linguistics –No artificial intelligence (representation) –No machine learning –No statistics –No error analysis –No data mining –No text mining
It never pays to think until youve run out of data – Eric Brill Banko & Brill: Mitigating the Paucity-of-Data Problem (HLT 2001) Fire everybody and spend the money on data More data is better data! No consistently best learner Quoted out of context Moores Law Constant: Data Collection Rates Improvement Rates
The rising tide of data will lift all boats! TREC Question Answering & Google: What is the highest point on Earth?
The rising tide of data will lift all boats! Acquiring Lexical Resources from Data: Dictionaries, Ontologies, WordNets, Language Models, etc. http://labs1.google.com/sets http://labs1.google.com/sets EnglandJapanCatcat FranceChinaDogmore GermanyIndiaHorsels ItalyIndonesiaFishrm IrelandMalaysiaBirdmv SpainKoreaRabbitcd ScotlandTaiwanCattlecp BelgiumThailandRatmkdir CanadaSingaporeLivestockman AustriaAustraliaMousetail AustraliaBangladeshHumanpwd
Applications What good is word sense disambiguation (WSD)? –Information Retrieval (IR) Salton: Tried hard to find ways to use NLP to help IR –but failed to find much (if anything) Croft: WSD doesnt help because IR is already using those methods Sanderson (next two slides) –Machine Translation (MT) Original motivation for much of the work on WSD But IR arguments may apply just as well to MT What good is POS tagging? Parsing? NLP? Speech? Commercial Applications of Natural Language Processing, CACM 1995 –$100M opportunity (worthy of government/industrys attention) 1.Search (Lexis-Nexis) 2.Word Processing (Microsoft) Warning: premature commercialization is risky Dont worry; Be happy
Sanderson (SIGIR-94) http://dis.shef.ac.uk/mark/cv/publications/papers/my_papers/SIGIR94.pdf http://dis.shef.ac.uk/mark/cv/publications/papers/my_papers/SIGIR94.pdf Not much? Could WSD help IR? Answer: no –Introducing ambiguity by pseudo-words doesnt hurt (much) Short queries matter most, but hardest for WSD F Query Length (Words) 5 Ian Andersons
Sanderson (SIGIR-94) http://dis.shef.ac.uk/mark/cv/publications/papers/my_papers/SIGIR94.pdf http://dis.shef.ac.uk/mark/cv/publications/papers/my_papers/SIGIR94.pdf Resolving ambiguity badly is worse than not resolving at all –75% accurate WSD degrades performance –90% accurate WSD: breakeven point Soft WSD? Query Length (Words) F
Some Promising Suggestions (Generate lots of conference papers, but may not support the field) Two Languages are Better than One –For many classic hard NLP problems Word Sense Disambiguation (WSD) PP-attachment Conjunction Predicate-argument relationships Japanese and Chinese Word breaking –Parallel corpora plenty of annotated (labeled) testing and training data –Dont need unsupervised magic (data >> magic) Demonstrate that NLP is good for something –Statistical methods (IR & WSD) focus on bags of nouns, Ignoring verbs, adjectives, predicates, intensifiers, etc. –Hypothesis: Ignored because perceptrons cant model XOR –Task: classify comments into good, bad and neutral Lots of terms associated with just one category Some associated with two –Depending on argument Good & Bad, but not neutral: Mickey Mouse, Rinky Dink –Bad: Mickey Mouse(us) –Good: Mickey Mouse(them) –Current IR/WSD methods dont capture predicate- argument relationships
Web Apps: Document Language Model Query Language Model Documents –Function Words –Adjectives –Verbs –Predicates Queries –Typos –Brand Names –Celebrities –Named Entities –Slower Vocab Growth Technical Op: Reduce IR to Translation Promising Apps: Web Spam, Frame Problem
Speech Data Mining & Call Centers: An Intelligence Bonanza Some companies are collecting information with technology designed to monitor incoming calls for service quality. Last summer, Continental Airlines Inc. installed software from Witness Systems Inc. to monitor the 5,200 agents in its four reservation centers. But the Houston airline quickly realized that the system, which records customer phone calls and information on the responding agent's computer screen, also was an intelligence bonanza, says André Harris, reservations training and quality-assurance director.
Speech Data Mining Label calls as success or failure based on some subsequent outcome (sale/no sale) Extract features from speech Find patterns of features that can be used to predict outcomes Hypotheses: –Customer: Im not interested no sale –Agent: I just want to tell you… no sale Inter-ocular effect (hits you between the eyes); Dont need a statistician to know which way the wind is blowing
Outline Were making consistent progress, or Were running around in circles, or –Dont worry; be happy Were going off a cliff… According to unnamed sources: Speech Winter Language Winter Dot Boom Dot Bust
Sample of 20 Survey Questions (Strong Emphasis on Applications) When will –More than 50% of new PCs have dictation on them, either at purchase or shortly after. –Most telephone Interactive Voice Response (IVR) systems accept speech input. –Automatic airline reservation by voice over the telephone is the norm. –TV closed-captioning (subtitling) is automatic and pervasive. –Telephones are answered by an intelligent answering machine that converses with the calling party to determine the nature and priority of the call. –Public proceedings (e.g., courts, public inquiries, parliament, etc.) are transcribed automatically. Two surveys of ASRU attendees: 1997 & 2003
Hockey Stick Business Case Last Year This Year Next Year
2003 Responses 1997 Responses + 6 Years (6 years of hard work No progress)
Wrong Apps? New Priorities –Increase demand for space >> Data entry New Killer Apps –Search >> Dictation Speech Google! –Data mining Old Priorities –Dictation app dates back to days of dictation machines –Speech recognition has not displaced typing Speech recognition has improved But typing skills have improved even more –My son will learn typing in 1 st grade –Sec rarely take dictation –Dictation machines are history My son may never see one Museums have slide rulers and steam trains –But dictation machines?
Great Challenge: Annotating Data Produce annotated data with minimal supervision Active learning –Identify reliable labels –Identify best candidates for annotation Co-training Bootstrap (project) resources from one application to another Borrowed Slide: Jelinek (LREC) Self-organizing Magic Error Analysis Great Strategy Success
Grand Challenges ftp://ftp.cordis.lu/pub/ist/docs/istag040319-draftnotesofthemeeting.pdf ftp://ftp.cordis.lu/pub/ist/docs/istag040319-draftnotesofthemeeting.pdf
Roadmaps: Structure of a Strategy (not the union of what we are all doing) Goals –Example: Replace keyboard with microphone –Exciting (memorable) sound bite –Broad grand challenge that we can work toward but never solve Metrics –Examples: WER: word error rate Time to perform task –Easy to measure Milestones –Should be no question if it has been accomplished –Example: reduce WER on task x by y% by time t Accomplishments v. Activities –Accomplishments are good –Activity is not a substitute for accomplishments –Milestones look forward whereas accomplishments look backward Serendipity is good! Small is beautiful –Quantity is not a good thing –Awareness –1-slide version if successful, you get maybe 3 more slides Size of container –Goal: 1-3 –Metrics: 3 –Milestones: a dozen Mostly for next year: Q1-4 Plus some for years 2, 5, 10 & 20 –Accomplishments: a dozen Broad applicability & illustrative –Dont cover everything –Highlight stuff that Applies to multiple groups Forward-Looking / Exciting
Resources Apps & Techniques Grand Challenges Goal: Reduce barriers to entry Goals: 1.The multilingual companion 2.Life log Goal: Produce NLP apps that improve the way people communicate with one another Evaluation
Summary: What Worked and What Didnt? Data It is the data, stupid! –Stay on msg: It is the data, stupid! WVLC (Very Large) >> EMNLP (Empirical Methods) If you have a lot of data, –Then you dont need a lot of methodology Rising Tide of Data Lifts All Boats Methodology –Empiricism means different things to different people 1.Machine Learning (Self-organizing Methods) 2.Exploratory Data Analysis (EDA) 3.Corpus-Based Lexicography –Lots of papers on 1 EMNLP-2004 theme (error analysis) 2 Senseval grew out of 3 Substance: Recommended if… Magic: Recommended if… Promise: Recommended if… Short term Long term Lonely Whats the right answer? Therell be a quiz at the end of the decade…