1. 1 Search strategy & tactics Governed by effectiveness & feedback Tefko Saracevic tefkos@rutgers.edutefkos@rutgers.edu; http://comminfo.rutgers.edu/~tefko/http://comminfo.rutgers.edu/~tefko/

2. Central ideas Searches are all about being effective – finding what is needed, desired To measure effectiveness common measures of precision and recall are used Various tactics are used to affect effectiveness Application of tactics for a desired results is a second nature of professional searchers 2 Tefko Saracevic Knowing searching = also knowing tactics to reach toward desired effectiveness As a searcher you are striving toward doing effective searches

3. ToC 1.Major concepts 2.Measuring effectiveness 3. Search tactics 4.Feedback types 3 Tefko Saracevic

4. Effectiveness, relevance 1. Major concepts Tefko Saracevic 4 can it be done?

5. 5 Some definitions Search statement (query): – set of search terms with logical connectors and attributes - file and system dependent Search strategy (big picture): – overall approach to searching of a question selection of systems, files, search statements & tactics, sequence, output formats; cost, time aspects Search tactics (action choices): – choices & variations in search statements terms, connectors, attributes Tefko Saracevic

6. 6 Some definitions (cont.) Move : – modifications of search strategies or tactics that are aimed at improving the results e.g. from searching for digital AND libraries to digital(w)libraries (Dialog) or “digital libraries” (Scopus) Cycle (particularly applicable to systems such as Dialog): – set of commands from start (begin) to viewing (type) results, or from a viewing to a viewing command Tefko Saracevic

7. Search strategy – big picture all kinds of actions from start to end The entire approach to a search – selection of – files and sources to use – approaches in proceeding to search & combining search terms operators to use fields to search – formats for viewing results – alternative actions if search yields too much too little – problem-solving heuristics Tefko Saracevic 7

8. Search tactics – specific actions connected with a given search as it progresses A query - command line entered into a system in order to retrieve relevant information – terms, operators & attributes as allowed by a given system – vocabulary & syntax used in conjunction with connectors &/or limiters to search a system Again: depends on a system how it is done – for example, a search statement might be: in Dialog: b 47; ss (garbanzo? or chickpeas) and (hum?us or humus) in Scopus you can enter: (garbanzo? or chickpeas) and (hum?us or humus) how would you do that in ? Tefko Saracevic 8

9. 9 Search performance is expressed in terms of: Effectiveness : – performance as to objectives to what degree did a search accomplish what desired? how well done in terms of relevance? Efficiency : – performance as to costs at what cost and/or effort, time? But here we concentrate on effectiveness Tefko Saracevic Both are KEY concepts & criteria for selection of strategy, tactics & evaluation

10. 10 Effectiveness criteria Search tactics chosen & changed following some criteria of accomplishment, such as: – none - no thought given – hard to imagine but happens sometimes – relevance (most often) – is it relevant to a need, task, problem? – magnitude (also very often)- is it to much retrieved to start with? – output attributes- is it trustworthy? Authorities? Understandable? … – topic – is it on the topic of the question? Tactics are chosen or altered interactively to match given criteria - feedback plays an important role Tefko Saracevic Knowing what choice of tactics may produce what results is key to professional searcher

11. 11 Relevance: key concept in IR & key criterion for assessing effectiveness “Relevant: having significant and demonstrable bearing on the matter at hand.” “Relevance: the ability (as of an information retrieval system) to retrieve material that satisfies the needs of the user.” Merriam Webster (2005) Attribute/criterion reflecting effectiveness of exchange of inf. between people (users) & IR systems in communication contacts, based on valuation by people Tefko Saracevic

12. Variation in names Relevance or relevant can be expressed in different terms – utility, pertinence, appropriate, significant, useful, germane, applicable, valuable … But the concept still remains relevance That which we call relevance by any other word would still be relevance Relevance is relevance is relevance is relevance thank you Shakespeare and Gertrude Stein Tefko Saracevic 12

13. Some attributes of relevance – in IR - user dependent – users are the final judges at the end – multidimensional or faceted – users asses relevance on a number of factors, not only topic, but also authority, novelty, source … – dynamic – users may change relevance assessments & criteria as search progresses or they learn or the problem is modified – not only binary – users assess object not only as relevant/not relevant, but also more on a continuum (scale) - partially relevant included – intuitively well understood – nobody has to explain to users what it is Tefko Saracevic 13

14. Consistency Individual differences – consistency of assessment varies over time & between people to a (sometimes significant) degree – but (in)consistency is similar as in other processes e.g. indexing, classifying Subject expertise affects consistency of relevance judgments. Higher expertise results in higher consistency and stringency. Lower expertise results in lower consistency and more inclusion. – relevance is not fixed; individual differences may be large It is most helpful to discuss with a user what kind of documents they may asses as relevant, what are their criteria and then try to incorporate that in searching & own assessments Tefko Saracevic 14

15. 15 Two major types of relevance – leading to a dichotomy of systems- & users-view of relevance: – Systems or algorithmic relevance the way a system or algorithm assesses relevance relation between a query & objects in the file of a system as retrieved or failed to be retrieved by a given algorithm – User relevance the way a user or user surrogate (searcher, specialist …) assesses relevance relation between information need or problem at hand of a user and objects retrieved or out there in general – But Topical relevance can be considered by both system & user the degree to which topics or subjects of a query & topics or subjects of objects (documents) in a file or retrieved match relation between topic in the query & topic covered by the retrieved objects, or objects in the file(s) of the system, or even in existence Tefko Saracevic

16. 16 User relevance can be further distinguished as to: – Cognitive relevance or pertinence: possible changes in user’s cognitive state due to objects retrieved relation between state of knowledge & cognitive information need of a user and the objects provided or in the file(s) – Motivational or affective relevance matching & satisfying user’s intentions, purposes, rationales, emotions relation between intents, goals & motivations of a user & objects retrieved by a system or in the file, or even in existence. Satisfaction – Situational relevance or utility: value of given objects or information for user’s situation or changes in situation relation between the task or problem-at-hand & the objects retrieved (or in the files). Relates to usefulness in decision-making, reduction of uncertainty... Tefko Saracevic

17. Precision and recall 2. Measuring effectiveness Tefko Saracevic 17

18. How? The basic way by which effectiveness is established in IR searching is to compare Tefko Saracevic 18 User assessment of relevance System assessment of relevance Where user assessment is the gold standard

19. 19 Effectiveness measures Two measures are used universally: Precision: – of the stuff retrieved & given to user how much (what %) was relevant? – more formally: probability that given that an object is retrieved it is relevant, or the ratio of relevant items retrieved to all items retrieved Recall: – of the stuff that is relevant in the file how much (what %) was actually retrieved? – more formally: probability that given that an object is relevant it is retrieved, or the ratio of relevant items retrieved to all relevant items in a file Tefko Saracevic

20. 20 Items judged RELEVANT Items judged NOT RELEVANT Items RETRIEVED a No. of items retrieved & judged relevant b No. of items retrieved & judged not relevant (junk) Items NOT RETRIEVED c No. of items not retrieved & relevant (missed relevant) d No. of items not retrieved & not relevant (missed junk) Precision = a a + b Recall = a a + c High precision = maximize a, minimize b High recall = maximize a, minimize c Calculation

21. Examples of calculation If a system retrieved 16 documents and only 4 were assessed as relevant by a user then precision is 25% If a system had 40 documents in the file that were relevant but managed to retrieve only 12 of them then recall is 30% Precision is easy to establish, recall is not union of retrievals is used as a “trick” to establish relative recall you do a number of searches or use a number of tactics or algorithms then you take together all that was retrieved (union) and have that assessed; then you can calculate a relative recall of each search, tactic or algorithm in respect to the union & see which one provides better or worse relative recall Tefko Saracevic 21

22. 22 Interpretation: PRECISION Precision= percent of relevant stuff you have in your set of answers retrieved – or conversely percent of junk (false drops) in the answer set – high precision = most stuff relevant – low precision = a lot of junk Some users demand high precision – do not want to wade through much stuff – but it comes at a price: relevant stuff may be missed Tradeoff almost always: high precision = low recall we will get to tradeoff a bit later, but it is VERY important to consider Tefko Saracevic

23. 23 A file may have a lot of relevant stuff Recall = percent of that relevant stuff in the file that you retrieved in your answer set – conversely percent of stuff you missed – high recall = you missed little – low recall = you missed a lot Some users demand high recall (e.g. PhD students doing a dissertation; patent lawyers; researchers writing a proposal or article) – want to make sure that important stuff is not missed – but will have to pay a price of wading through a lot of junk Tradeoff almost always: high recall = low precision Interpretation: RECALL Tefko Saracevic

24. Tradeoff between precision and recall Using it for different tactics 3. Search tactics Tefko Saracevic 24

25. Aim of search tactics Since there is no such thing as a perfect search: – the aim is to search in a way that will insure a given, chosen or desired, level of effectiveness That means that we have to agree on – what do we mean by effectiveness in searching? general agreement is that retrieval of relevant answers is the major criterion for effectiveness of searching – what measures do we use to express achievement of effectiveness in terms of relevance general agreement is that we use measures of precision and recall (or derivatives) Tefko Saracevic 25 Thus many search tactics are geared toward achieving certain level of precision or recall

26. 26 Precision-recall trade-off USUALLY: precision & recall are inversely related – higher recall usually means lower precision & vice versa 100 % 0 Ideal Usual Improvements Precision Recall Tefko Saracevic

27. 27 Precision-recall trade-off It is like in life, usually: you get some, lose some; you can't have your cake and eat it tooyou can't have your cake and eat it too Usually, but not always (keep in mind these are probabilities) – when you have high precision most stuff you got is relevant or on the target but you also missed other stuff (could be a lot) that is relevant – it was left behind – when you have high recall you did not miss much but you also got junk (could be a lot) - you have to wade through it There is price to pay either way – but then a lot of users are perfectly satisfied with and are aiming at high precision give me a few good things, that is all what I need Tefko Saracevic

28. Effectiveness (Cleverdon’s) “laws ” High precision in retrieval is usually associated with low recall. High precision = low recall High recall in retrieval is usually associated with low precision. High recall = low precision You – with your user – have to decide if you are aiming toward high recall or high precision & you have to be aware of & explain tradeoffs Tefko Saracevic 28 Originally formulated by Cyril Cleverdon, Cyril Cleverdon, a UK librarian, in 1960s, in first IR tests done in Cranfield, UK, thus “Cranfield tests” You use different tactics for high recall from those for high precision

29. Search tactics Tefko Saracevic 29 VariableWhat variation possible? 1. LOGICchoice of connectors among terms AND, OR, NOT, W … 2. SCOPEno. of terms linked - ANDs A AND B vs A AND B AND C 3.EXHAUSTIVITYfor each concept no. of related terms - OR connections A OR B vs. A OR B OR C 4. TERM SPECIFICITYfor each concept level in hierarchy broader vs narrower terms 5. SEARCHABLE FIELDSchoice for text terms & non-text attributes e.g. titles only, limit as to years, various sources 6. FILE OR SYSTEM SPECIFIC CAPABILITIES using capabilities, options available e.g. given fields, ranking, sorting, linking several ‘things’ or variables in a query can be selected & changed to affect effectiveness

30. Actions and consequences ActionConsequence SCOPE - adding more ANDs Output size: down Recall: down Precision: up EXHAUSTIVITY - adding more ORs Output size: up Recall: up Precision: down USE OF NOTs - adding more NOTs Output size: down Recall: down Precision: up USING BROAD TERMS - low specificity Output size: up Recall: up Precision: down USING PHRASES - high specificity Output size: down Recall: down Precision: up Tefko Saracevic 30 BUT: each variation has consequence in output if you do X then Y will happen

31. Tactics: what to do? These “laws” lead to precision & recall devices – tactics to increase/decrease precision or recall Tefko Saracevic 31 To increase precisionTo increase recall SCOPE -add more ANDs -- you restrict EXHAUSTIVITY -add more ORs - you enlarge NOTs -add more NOTs -you eliminate BROAD TERMS - you broaden terms PHRASES - you become more specific USE MORE TRUNCATION -you use grammatical variants of terms

32. Precision, recall devices Tefko Saracevic 32 NARROWING Higher precision BOADENNING Higher recall More ANDs Fewer ORs More NOTs Less free text More controlled Less synonyms Narrower terms More specific Less truncation More qualifiers More limits Building blocks Fewer ANDs More ORs Fewer NOTs More free text Fewer controlled More synonyms Broader terms Less specific More truncation Fewer qualifiers Fewer limits Citation growing With experience use of these devices will become second nature

33. 33 Other tactics (but that gets us in the next unit on advanced searching) Citation growing: – find a relevant document – look for documents cited in – look for documents citing it – repeat on newly found relevant documents Building blocks – find documents with term A – review – add term B & so on Using different feedbacks – a most important tool Tefko Saracevic

34. Types. Berry picking 4. Feedback Tefko Saracevic 34

35. Feedback in searching Extremely important! And constantly used, consciously or unconsciously Formally: – the process in which part of the output of a system is returned to its input in order to regulate its further output Simply put: – you search, find something that may be relevant, then look at it, then on the basis of AHA! change your next query or tactic to get better or more stuff Tefko Saracevic 35

36. 36 Feedback in searching … Any feedback implies loops – a completion of a process provides information for modification, if any, for the next process – information from output is used to change previous or create new input In searching: – some information taken from output of a search is used to do something with next query (search statement) examine what you got to decide what to do next in searching – a basic tactic in searching Several feedback types used in searching – each used for different decisions Tefko Saracevic

37. 37 Feedback types Content relevance feedback – judge relevance of items retrieved – make decision what to do next switch files, change exhaustivity … Term relevance feedback – find relevant documents – examine what other terms used in those documents – search using additional terms also called query modification & in some systems done automatically Magnitude feedback – on the basis of size of output make tactical decisions often the size so big that documents are not examined but next search done to limit size Tefko Saracevic

38. 38 Feedback types (cont.) Tactical review feedback – after a number of queries (search statements) in the same search review tactics as to getting desired outputs review terms, logic, limits … – change tactics accordingly Strategic review feedback – after a while (or after consultation with user) review the “big” picture on what searched and how sources, terms, relevant documents, need satisfaction, changes in question, query … – do next searches accordingly – used in reiterative searching Tefko Saracevic

39. But then There are other ways to look at searching where many of these things are combined Here is one way that looks at the whole search as a process of complex wandering Shifting exploration & feedback are implied Tefko Saracevic 39

40. 40 Bates’ Berry-picking model of searchingBerry-picking model of searching “…moving through many actions towards a general goal of satisfactory completion of research related to information need.” – query is shifting (continually) as search progresses queries are changing different tactics are used – searcher (user) may move through a variety of sources new files, resources may be used – think of your last serious search isn’t this what you were doing? Tefko Saracevic Elaborated by Marcia Bates UCLA

41. 41 Berry-picking … – closer to the real behavior of information searchers – new information may provide new ideas, new directions feedback is used in various ways – question is not satisfied by a single set of answers, but by a series of selections & bits of information found along the way results may vary & may have to be provided in appropriate ways & means – you go as if berry-picking in a field Tefko Saracevic

42. A berry-picking evolving search (from the article) Tefko Saracevic 42 “1) The nature of the query is an evolving one, rather than single and unchanging, and 2) the nature of the search process is such that it follows a berrypicking pattern, instead of leading to a single best retrieved set.”

43. Conclusions search tactics have to be mastered Users are not concerned about searching but about finding Effective searching is a prerequisite for finding the right stuff Search tactics are critical for effective searching They need to be understood and followed if you do X you can expect Y in order to get to Y you have to know what X to do Tefko Saracevic 43

44. Tefko Saracevic 44 Thank you M. C. Escher