1. 1 CS 178H Introduction to Computer Science Research What is CS Research?

2. Discovery of new knowledge of computing through mathematical analysis and experimental evaluation of algorithms and computer software. 2

3. Epistemology (definitions from Wikipedia) Epistemology (from Greek επιστήμη - episteme, "knowledge" + λόγος, "logos") or theory of knowledge is the branch of philosophy concerned with the nature and scope (limitations) of knowledge. It addresses the questions:Greeklogosphilosophy knowledge –"What is knowledge?" –"How is knowledge acquired?" –"What do people know?" –"How do we know what we know?" 3

4. Rationalism Rationalism is "any view appealing to reason as a source of knowledge or justification" (Lacey 286). In more technical terms it is a method or a theory "in which the criterion of the truth is not sensory but intellectual and deductive" (Bourke 263).reasontheorydeductive Originated with Socrates (469 BC–399 BC) and Plato (428/427 BC – 348/347 BC). 4

5. Empiricism Empiricism is a theory of knowledge which asserts that knowledge arises from experience. Empiricism emphasizes the role of experience and evidence, especially sensory perception, in the formation of ideas.knowledgeexperience evidencesensory perception Originated with Aristotle (384 BC – 322 BC) 5

6. Rationalism in CS (Theoretical CS) Programs are formal mathematical objects. Therefore, important properties of algorithms/software can be proven mathematically. –Termination –Correctness (satisfies a formal specification) –Computational Complexity (time and space requirements) 6

7. Theoretical CS Research Algorithm Design and Analysis –Design a new (more efficient) algorithm for some well- defined problem (e.g. sorting, longest-common- subsequence) –Mathematically prove the correctness and improved complexity of the new algorithm. Theoretical Analysis –Form a mathematical conjecture about a computational problem (e.g. graph isomorphism is NP-complete) –Mathematically prove the conjecture as a theorem. 7

8. Limits of Rationalism in CS Sometimes software is too complex to analyze theoretically. Sometimes correctness cannot be characterized formally and depends on natural or human behavior. –Protein folding –Handwriting/speech recognition Sometimes software behavior on real data depends on unknown natural properties of this data. –Locality affecting paging performance 8

9. Empiricism in CS (Experimental CS) Behavior of software can be studied experimentally. Anecdotal evidence (running a few sample cases) is insufficient. Collect data (e.g. accuracy, run-time) on running programs many times on large, real- world benchmark collections. Verify hypotheses about behavior using controlled experiments. Statistically analyze results for significance. 9

10. Scientific Method (steps from Wikipedia) 1) Define the question 2) Gather information and resources (observe) 3) Form hypothesis 4) Perform experiment and collect data 5) Analyze data 6) Interpret data and draw conclusions that serve as a starting point for new hypothesis 7) Publish results 8) Retest (frequently done by other scientists) 10

11. 1) Define the question Example from My Research: Search Query Disambiguation from Short Sessions –Can a web search engine disambiguate queries? 11 scrubs Search ?

12. 2) Gather information and resources Obtained web search session data from Microsoft Find instances of ambiguous queries Find contextual clues that might help disambiguate queries 12

13. Context can Aid Disambiguation 98.7 fm kroq scrubs www.star987.com www.kroq.com ??? huntsville hospital ebay.com scrubs www.huntsvillehospital.com www.ebay.com ??? scrubs-tv.com scrubs.com

14. 3) Form Hypothesis Previous queries and clicks in a session can help disambiguate queries by relating them to previous sessions involving the same query (where we know what result was clicked). 14

15. 4) Perform Experiment and Collect Data Build system that uses prior context and previous session data to predict clicked results for new user. Reorder results from existing search engine based on predicted probability of clicking on a result. –Should reduce number of results user needs to examine before finding a relevant one. Test on unseen data and compare predictions to actual results clicked. 15

16. Using Relational Information with a Markov Logic Network (MLN) huntsville hospital ebay scrubs huntsvillehospital.org ebay.com ??? huntsville school... hospitallink.com scrubs scrubs-tv.com … ebay.com scrubs scrubs.com

17. Controlled Experiment Performance of experimental system must be compared to some baseline or control. Controls are necessary to demonstrate the system is improving over some naïve method (strawman) or current best system for a problem. –For example, in the old joke, someone claims that they are snapping their fingers "to keep the tigers away"; and justifies this behavior by saying "see - its working!" While this "experiment" does not falsify the hypothesis "snapping fingers keeps the tigers away", it does not really support the hypothesis - not snapping your fingers does not keep the tigers away as well (Wikipedia: Experiment) 17

18. Control for Query Disambiguation Simple control is to order results from search engine randomly. Another baseline is to just use ordering from existing (non-personalized) search engine. 18

19. Performance Metrics Need quantitative measure of system’s performance (runtime or accuracy). Compare quantitative performance of experimental system to baseline control system. To measure accuracy of ordering of web search results we measure AUC-ROC –Percentage of irrelevant results not seen by user before finding a relevant result (if scan results from top) 19

20. 5) Analyze Data Do results support the hypothesis? Are differences statistically significant? –Use statistical test to determine if observed differences are unlikely to be due only to random variation, i.e. probability of null hypothesis <.05. 20

21. Results (AUC-ROC) * * * * Indicates statistically significant improvement over previous result

22. 6) Interpret data and draw conclusions that serve as a starting point for new hypothesis Is random ordering the best baseline to compare to? What if just order results based on popularity (i.e. how many people clicked on a particular result after submitting a given ambiguous query). 22

23. New Baseline Results 23

24. Refine System Develop MLN that incorporates popularity information. Rerun experiment to obtain results for revised version and verify the hypothesis that it performs better than the popularity baseline. 24

25. Results for Revised System 25

26. 7) Publish Results Paper submitted to the international data mining conference. –KDD-09: Paris, June 28 – July 1, 2009 26