Presentation is loading. Please wait.

Presentation is loading. Please wait.

資訊檢索與擷取 Information Retrieval and Extraction

Similar presentations

Presentation on theme: "資訊檢索與擷取 Information Retrieval and Extraction"— Presentation transcript:

1 資訊檢索與擷取 Information Retrieval and Extraction
陳信希 Hsin-Hsi Chen 台大資訊系

2 Information Retrieval
generic information retrieval system select and return to the user desired documents from a large set of documents in accordance with criteria specified by the user functions document search the selection of documents from an existing collection of documents document routing the dissemination of incoming documents to appropriate users on the basis of user interest profiles

3 Detection Need Definition a set of criteria specified by the user which describes the kind of information desired. queries in document search task profiles in routing task forms keywords keywords with Boolean operators free text example documents ...

4 Example <head> Tipster Topic Description <num> Number: 033
<dom> Domain: Science and Technology <title> Topic: Companies Capable of Producing Document Management <des> Description: Document must identify a company who has the capability to produce document management system by obtaining a turnkey- system or by obtaining and integrating the basic components. <narr> Narrative: To be relevant, the document must identify a turnkey document management system or components which could be integrated to form a document management system and the name of either the company developing the system or the company using the system. These components are: a computer, image scanner or optical character recognition system, and an information retrieval or text management system.

5 Example (Continued) <con> Concepts:
1. document management, document processing, office automation electronic imaging 2. image scanner, optical character recognition (OCR) 3. text management, text retrieval, text database 4. optical disk <fac> Factors: <def> Definitions Document Management-The creation, storage and retrieval of documents containing, text, images, and graphics. Image Scanner-A device that converts a printed image into a video image, without recognizing the actual content of the text or pictures. Optical Disk-A disk that is written and read by light, and are sometimes associated with the storage of digital images because of their high storage capacity.

6 search vs. routing The search process matches a single Detection Need against the stored corpus to return a subset of documents. Routing matches a single document against a group of Profiles to determine which users are interested in the document. Profiles stand long-term expressions of user needs. Search queries are ad hoc in nature. A generic detection architecture can be used for both the search and routing.

7 Search retrieval of desired documents from an existing corpus
Retrospective search is frequently interactive. Methods indexing the corpus by keyword, stem and/or phrase apply statistical and/or learning techniques to better understand the content of the corpus analyze free text Detection Needs to compare with the indexed corpus or a single document ...

8 Document Detection: Search

9 Document Detection: Search(Continued)
Document Corpus the content of the corpus may have significant the performance in some applications Preprocessing of Document Corpus stemming a list of stop words phrases, multi-term items ...

10 Document Detection: Search(Continued)
Building Index from Stems key place for optimizing run-time performance cost to build the index for a large corpus Document Index a list of terms, stems, phrases, etc. frequency of terms in the document and corpus frequency of the co-occurrence of terms within the corpus index may be as large as the original document corpus

11 Document Detection: Search(Continued)
Detection Need the user’s criteria for a relevant document Convert Detection Need to System Specific Query first transformed into a detection query, and then a retrieval query. detection query: specific to the retrieval engine, but independent of the corpus retrieval query: specific to the retrieval engine, and to the corpus

12 Document Detection: Search(Continued)
Compare Query with Index Resultant Rank Ordered List of Documents Return the top ‘N’ documents Rank the list of relevant documents from the most relevant to the query to the least relevant

13 Routing

14 Routing (Continued) Profile of Multiple Detection Needs
A Profile is a group of individual Detection Needs that describes a user’s areas of interest. All Profiles will be compared to each incoming document (via the Profile index). If a document matches a Profile the user is notified about the existence of a relevant document.

15 Routing (Continued) Convert Detection Need to System Specific Query
Building Index from Queries similar to build the corpus index for searching the quantify of source data (Profiles) is usually much less than a document corpus Profiles may have more specific, structured data in the form of SGML tagged fields

16 Routing (Continued) Routing Profile Index Document to be routed
The index will be system specific and will make use of all the preprocessing techniques employed by a particular detection system. Document to be routed A stream of incoming documents is handled one at a time to determine where each should be directed. Routing implementation may handle multiple document streams and multiple Profiles.

17 Routing (Continued) Preprocessing of Document
A document is preprocessed in the same manner that a query would be set-up in a search The document and query roles are reversed compared with the search process Compare Document with Index Identify which Profiles are relevant to the document Given a document, which of the indexed profiles match it?

18 Routing (Continued) Resultant List of Profiles
The list of Profiles identify which user should receive the document

19 Summary Generate a representation of the meaning or content of each object based on its description. Generate a representation of the meaning of the information need. Compare these two representations to select those objects that are most likely to match the information need.

20 an Information Retrieval System
Basic Architecture of an Information Retrieval System Documents Queries Document Representation Query Representation Comparison

21 Research Issues Given a set of description for objects in the collection and a description of an information need, we must consider Issue 1 What makes a good document representation? What are retrievable units and how are they organized? How can a representation be generated from a description of the document?

22 Research Issues (Continued)
Issue 2 How can we represent the information need and how can we acquire this representation either from a description of the information need or through interaction with the user? Issue 3 How can we compare representations to judge likelihood that a document matches an information need?

23 Research Issues (Continued)
Issue 4 How can we evaluate the effectiveness of the retrieval process?

24 Information Extraction
Generic Information Extraction System An information extraction system is a cascade of transducers or modules that at each step add structure and often lose information, hopefully irrelevant, by applying rules that are acquired manually and/or automatically.

25 Information Extraction (Continued)
What are the transducers or modules? What are their input and output? What structure is added? What information is lost? What is the form of the rules? How are the rules applied? How are the rules acquired?

26 Example: Parser transducer: parser
input: the sequence of words or lexical items output: a parse tree information added: predicate-argument and modification relations information lost: no rule form: unification grammars application method: chart parser acquisition method: manually

27 Modules Text Zoner turn a text into a set of text segments
Preprocessor turn a text or text segment into a sequence of sentences, each of which is a sequence of lexical items, where a lexical item is a word together with its lexical attributes Filter turn a set of sentences into a smaller set of sentences by filtering out the irrelevant ones Preparser take a sequence of lexical items and try to identify various reliably determinable, small-scale structures

28 Modules (Continued) Parser input a sequence of lexical items and perhaps small-scale structures (phrases) and output a set of parse tree fragments, possibly complete Fragment Combiner turn a set of parse tree or logical form fragments into a parse tree or logical form for the whole sentence Semantic Interpreter generate a semantic structure or logical form from a parse tree or from parse tree fragments

29 Modules (Continued) Lexical Disambiguation turn a semantic structure with general or ambiguous predicates into a semantic structure with specific, unambiguous predicates Coreference Resolution, or Discourse Processing turn a tree-like structure into a network-like structure by identifying different descriptions of the same entity in different parts of the text Template Generator derive the templates from the semantic structures

30 Topics 1. Introduction to Information Retrieval and Extraction
2. Conventional Text-Retrieval Systems (Salton, Chapter 8) - Database Management and Information Retrieval - Text Retrieval Using Inverted Indexing Methods - Extensions of the Inverted Index Operations - Typical File Organization - Text-Scanning Systems 3. Automatic Indexing (Salton, Chapter 9) - Indexing Environment - Indexing Aims - Single-Term Indexing Theories - Term Relationships in Indexing - Term-Phrase Formulation - Thesaurus-Group Generation

31 Topics (Continued) 4. Advanced Information-Retrieval Models (Salton, Chapter 10) - The Vector Space Model - Automatic Document Classification - Probabilistic Retrieval Model - Extended Boolean Retrieval Model 5. File Structures (Frakes & Baeza-Yates, Chapters 3-5) - Inverted Files - Signature Files - PAT trees 6. Term and Query Operations (Frakes & Baeza-Yates, Chapters 7-9,10) - Lexical Analysis and Stoplists - Stemming Algorithms - Thesaurus Construction - Relevance Feedback 7. Evaluation Metrices (Jones & Willett, Chapter 4) - The Pragmatics of Information Retrieval Experimentation, Revisited - The TREC Conferences

32 Topics (Continued) 8. IR on the World Wide Web (Cheong, Chapter 4)
- Spiders for Indexing the Web - Web Indexing Spiders - WebCrawler: Finding What People Want - Lycos: Hunting WWW Information - Harvest: Gathering and Brokering Information - WebAnts: Hunting in Packs - Issues of Web Indexing - Spiders of the Future 9. Cross-Language Information Retrieval (Hsin-Hsi Chen) 10. Information Extraction (Jerry R. Hobbs) - What information extraction is - What is involved in building information extraction systems, and some how to? - What kinds of resources and tools are needed, and how to access them

33 Information Sources Books
Salton, G. (1989) Automatic Text Processing. The Transformation, Analysis and Retrieval of Information by Computer. Reading, MA: Addison-Wesley. Frakes, W.B. and Baeza-Yates, R. (Eds.) (1992) Information Retrieval: Data Structures and Algorithms. Englewood Cliffs, NJ: Prentice Hall. Cheong, F. (1996) Internet Agents: Spiders, Wanderers, Brokers, and Bots. Indianapolis, IN: New Riders, 1996. Karen Sparck Jones and Peter Willett (1997) Readings in Information Retrieval, CA: Morgan Kaufmann Publishers.

34 Information Sources Conference Proceedings Journals
ACM SIGIR Annual International Conference on Research and Development in Information Retrieval (1978-) Journals ACM Transactions on Information Systems Information Processing and Management (formerly Information Storage and Retrieval) Journal of the American Society for Information Science (formerly American Documentation) Journal of Documentation

Download ppt "資訊檢索與擷取 Information Retrieval and Extraction"

Similar presentations

Ads by Google