1. Information Retrieval
Unit 1
Seema Chandak

2. Unit 1 : Objective & Content
To deal with IR representation, storage, organization & access to information items.

3. Unit 1 : Content(contu..) Content :: Basic Concepts of IR,
Data Retrieval & Information Retrieval,
IR system block diagram.
Automatic Text Analysis,
Luhn's ideas,
Conflation Algorithm,
Indexing and Index Term Weighing,
Probabilistic Indexing,

4. Unit 1 : Content(contu…)
Automatic Classification.
Measures of Association,
Different Matching Coefficient,
Classification Methods,
Cluster Hypothesis.
Clustering Algorithms,
Single Pass Algorithm,
Single Link Algorithm,
Rochhio's Algorith
Dendogram

5. What is IR Information retrieval:
Subfield of computer science that deals with automated retrieval of infromaition (especially text) based on their content and context.
The term Information Retrieval was first coined by Calvin Moores (1950).
“ It is concerned with the representation, storage, and organization and accessing of information items .“

6. Need for IR
Information is considered as the most important source, for most of the activities.
Example : Timely Weather reports.
Timely sharing of information.
The timely retrieval of information plays a major role, keeping with the motto “right information at the right time”.

7. Types of IR Structured (All Database management systems)
Unstructured (Search engines)
Semi structured(Datawarehouses)

8. IR Based on Structured Data
Recollect Terms related to DBMS ..
Data Organization in the form of schema, keys, index, metadata….
Query structure
Results set
….. ….

9. Why IR ?Why not Database?
What are some limitations of Database Systems?

10. IR Vs. DR
Information Retrieval System: a system that allows a user to retrieve documents that match her “information need” from a large corpus.
Example: Get documents about Java, except for ones that are about the Java coffee.
Data Retrieval System: a system that allows a user to retrieve all documents that match her query from a large corpus.
Example: Get all documents containing the term “Java” but no containing the term “coffee”.

11. IR Vs. DR
1. Matching.
In data retrieval we are normally looking for an exact match, that is, we are checking to see whether an item is or is not present in the file.
Eg.Select * from Student where per >= 75.0
In information retrieval more generally we want to find those items which partially match the request and then select from those a few of the best matching ones.
Eg. Student having 75 or >75 percentage from student of pict college.

12. IR Vs. DR
2. Inference
In data retrieval is of the simple deductive kind, that is, aRb and bRc then aRc.
In information retrieval it is of inductive inference;
Relations are only specified with a degree of certainty or uncertainty and hence our confidence in the inference is variable.
3. Model
Data retrieval is deterministic but information retrieval is probabilistic.
Frequently Bayes' Theorem is invoked to carry out inferences in IR, but in DR probabilities do not enter
into the processing.

13. IR Vs. DR 4 .Classification:
In DR most likely monothetic classification is used.
That is, one with classes defined by objects
possessing attributes both necessary and sufficient to belong to a class.
In IR such a classification is not very useful.
A polythetic classification is mostly used.
Each individual in a class will possess only a proportion of all the attributes possessed by all the members of that class.
Hence no attribute is necessary nor sufficient for membership to a class.

14. IR Vs. DR 5. Query Language: 6. Query Specification :
The query language for DR is one with restricted syntax and vocabulary.
In IR we prefer to use natural language although there are some notable exceptions.
6. Query Specification :
In DR the query is generally a complete specification of what is wanted,
In IR it is invariably incomplete.

15. IR Vs. DR 7. Items wanted : 8. Error response :
In IR we are searching for relevant documents as opposed to exactly matching items in DR.
8. Error response :
DR is more sensitive to error in the sense that, an error in matching will not retrieve the wanted item which implies a total failure of the system.
In IR small errors in matching generally do not affect performance of the system significantly

16. Information Retrieval (IR) Matching Exact match
IR Vs. DR
Data Retrieval (DR)
Information Retrieval (IR)
Matching
Exact match
Partial match, best match
Inference
Deduction
Induction
Model
Deterministic
Probabilistic
Classification
Monothetic
Polythetic
Data
Database tables, structured
Free text, unstructured
Query language
Artificial, SQL, relational algebras.
Natural, Keywords, free text
Query specification
Complete
Incomplete
Items wanted
Relevant

17. Information Retrieval
IR vs.DR
Data Retrieval
Information Retrieval
Sensitive
Insensitive
Error Response
Exact matches
Approximate matches
Results
Unordered
Ordered by relevance
Knowledgeable users or automatic processes
Non-expert humans
Accessibility

18. Issues with Information Retrieval?
Information Retrieval deals with uncertainty and vagueness in information systems.
• Uncertainty: available representation does typically not reflect true semantics/meaning of objects (text, images, video, etc.)
• Vagueness: information need of user lacks clarity, is only vaguel expressed in query, feedback or user actions.
• Differs conceptually from database queries!

19. Re Call the Definition What Is IR ?
“ Finding some desired information in large data sets or store of information “
Means :
Searching for documents
Searching for information in documents
Searching for metadata which describes documents
Searching within database
Web search engines like Google and Lycos are the most visible
IR applications.
IR systems are used to reduce information overload.

20. Definition Automatic Information Retrieval
Automatic – as against ‘manual’.
Information – as against ‘data’.
Defn : An information retrieval system does not inform (i.e.change the knowledge of) the user on the subject of his inquiry.
It merely informs on the existence (or non-existence) and whereabouts of documents relating to his request.

21. Media – Where Does Information Reside?
• Text documents: web pages, books, articles , papers, s etc.
• Manuscripts
• Graphics & Images
• Speech & Video
• Maps & Satellite Imagery
• Local Information, Yellow Pages
• Mismatch: given representation in specific medium vs.
semantic description of information (semantic gap)

22. Scale - How Much Information is out there?
• World Wide Web
Tens or hundreds billions of documents?
Approx. 10KB/doc of 100s of TB
• Then there is everything else
, personal files, proprietary databases,
broadcast media, print
• Estimated 5 Exabytes p.a. (growing at 30%)
• MB p.a. and person
• Web is just a tiny starting point….

23. IR problem
It is mainly dealing with a very large , mostly unstructured data set
IR problem consists of :
building efficient indexes.
processing user queries with high performance.
improve ‘quality’ of answer set.

24. Basic Concepts Information retrieval is directly affected by the :
User Tasks
Document Logical view

25. User Tasks Interaction of the user with retrieval system. Retrieval
Documents
Browsing

26. User Tasks Classical information retrieval system allows IR
Hypertext system are usually tuned for quick Browsing.
Modern digital lib. and Web interfacing might attempt to combine these tasks.

27. Logical view of the document
Documents are represented either by Keywords or Indexes is known as logical view of the documents.
Keywords are either extracted directly from the text of document or specified by human.
Modern computers represents doc by its set of :
Full words.
Small words.
Stopwords : elimination of articles and connectives.
steaming : (reduces distinct words to their common grammatical roots.)

28. Introduction… Information Retrieval System: Feedback Queries Processor
Sample retrieval
Queries
Input
Processor
Output
Documents
A typical IR system

29. Introduction… Information Retrieval System:
Input: Store only a representation of the document (or query) which means that the text of a document is lost once it has been processed for the purpose of generating its representation.
A document representative could be a list of extracted words considered to be significant.
The user has to use the language in which he/she can express the needed information in the language.
Processor: Involve in performing actual retrieval function, executing the search strategy in response to a query.
Feedback: Improving the subsequent run after a sample retrieval.
Output:A set of document numbers. And the evaluation can be done.

30. Introduction Information Retrieval Process How is the query How is
need
Collections
How is
the query
constructed?
How is
the text processed?
Pre-process
text input
Query
Index
Parse
Rank

31. Definitions
Searching: Seeking for specific information within a body of information. The result of a search is a set of hits.
Browsing: Unstructured exploration of a body of information.
Linking: Moving from one item to another following links, such as citations, references, etc.

32. The Basics of Information Retrieval
Query: A string of text, describing the information that the user is seeking. Each word of the query is called a search term.
A query can be a single search term, a string of terms, a phrase in natural language, or a stylized expression using special symbols.
Full text searching: Methods that compare the query with every word in the text, without distinguishing the function of the various words.
Fielded searching: Methods that search on specific bibliographic or structural fields, such as author or heading.

33. SORTING AND RANKING HITS
When a user submits a query to a search system, the system returns a set of hits. With a large collection of documents, the set of hits maybe very large.
The value to the use depends on the order in which the hits are presented.
Three main methods:
• Sorting the hits, e.g., by date
• Ranking the hits by similarity between query and
document
• Ranking the hits by the importance of the documents

34. Examples of Search Systems
Find file on a computer system (Spotlight for Macintosh).
Library catalog for searching bibliographic records about books and other objects (Library of Congress catalog).
Abstracting and indexing system for finding research information about specific topics (Medline for medical information).
Web search service for finding web pages (Google).