Classification & Your Intranet: From Chaos to Control Susan Stearns Inmagic, Inc. E-Libraries E204 May, 2003

Look familiar? ?#$%&* Why cant I find anything on the Intranet? How do we manage all the information we want to publish on the Intranet? User Content Manager

The issue: We know that we have documents in-house that contain hugely valuable information The problem: How do users find the right information at the right time? The answer: Automated spider and meta- classification software that allow an enterprise to automatically build and maintain a completely searchable database of critical content.

Automated Spidering Software A spider that crawls specified in- house servers and Web sites Extracts content from most popular file types and formats – HTML, Text, MSOffice, PDF – even Content can be loaded into a database

Key features to look for in a spider Document types: Microsoft Office, PDF, other formats (IFilter compatible) Zip files and folders can be crawled Remote administration Can be scheduled to run multiple times a day Web crawling can be set up to n levels deep Easy to create an XML transform to your database design Integrates with automated classification software

How a spider works File system crawl Database Web crawl The Spider Native document cache Extracted text cache XML load files Content Manager can add value (e.g. add additional meta-tags, etc.) Users can search and access Gathered content via a Web-browser

We Love Search: We Hate Search Search is ubiquitous but insufficient Only one slice into content Missing relationships across information Few are skilled at searching

The search engine paradox: Regardless of the product or a user's ability to use it, effective searches require the user to know the terms they need to use before they type them into the search engine. The Delphi Group

The Solution: Meta-Classification Enrich the content with meta-data Leverage XML and integrate content from multiple sources Extract other useful concepts Give users browse-able directories in addition to a search box

What is Meta-Classification? Automated meta-data extraction Meta-data includes subject information as well as names of people, company names, acronyms, key noun phrases Auto-classification of documents using a predefined taxonomy This meta-data can be mapped to a database along with the full-text of the document or a URL link

Why Meta-Classification? Creates structured information from unstructured data Allows local terminology to be reflected in searching Provides a browse-able directory Greatly enhances search through controlled vocabulary

How does it work? Spider/crawl the documents to create a corpus Automated software Identifies key words and phrases Maps them to known topics in taxonomy Scores the topics and derives a central theme Repeat for the sub-themes

Step 1: Identify words and phrases in the text Microsoft NASDAQ:MSFT, which won a round in its antitrust fight against the government today, launched its Microsoft.Net initiative that could someday replace computer hardware with software. Via XML (extensible markup language), Microsoft.net will enable use of much larger computers accessible on the Internet for storage of programs, word processing files and other data.

Step 2:Map them to topics within the taxonomy Government: government Computer Science: Internet, XML Hardware: computers Storage: data, storage Application Files: programs, language Word Processing Files: word processing files Software Companies: Microsoft Microsoft: Microsoft.Net

Step 3: Determine themes End results of classification of this story are: Central Theme: Microsoft Sub-theme 1: Word Processing Files Sub-theme 2: Software Companies Microsoft is a good match for central theme Microsoft.Net would have been the best classification Original taxonomy didnt know this topic Will be added to taxonomy

Customize the Taxonomy 1 million node taxonomy often too large Develop a custom taxonomy A subset of the large taxonomy Selected nodes to match business needs A set of rules to aggregate from the low level topics in the large taxonomy to the custom taxonomy

The Result Very large corpus of content can be classified in automated fashion Meta-data is used to create browse-able directories Meta-data is used for searching End user is given clues for finding the right information

Other features to consider Document summaries/abstracts Including external content Spidered from Web sites Integrated from licensed content sources User submissions User ratings/reviews

To Control Web Content Meta Data: Captured centrally User Interaction Word Doc Document Properties, Classification Search and Browse Content Collection (spider) Context (entity extraction/auto-classification) Corporate Intranet From Chaos

Thank You.