NITISH MANOCHA

Platforms §AIX workstation §OS/390 §Sun Solaris §Windows NT

Tools to Use §Topic categorization tool l Categorizing s l Categorizing Web Pages

Text Analysis Tool §Topic Categorization Tool

Text Analysis Tool §Topic Categorization Tool l Category 1 (AI Schedule)

Text Analysis Tool l Category2 (Database Schedule)

Text Analysis Tool §Target Category ( Data Mining Schedule)

Text Analysis Tool §Result - Category 2 (Databases)

Tools to Use §Clustering Tool (Finding Similar Information) l Dividing Documents into Groups l Identifying hidden similarities in documents l Identifying duplicate documents from a collection l Finding Documents that are out of place

Text Analysis Tool §Hierarchical Clustering - imzhclst

Text Analysis Tool §Binary Clustering - imzcrlst

Text Analysis Tool §Results

Text Analysis Tool §Results

Tools to Use §Feature Extraction Tool l Name Extraction l Abbreviation Extraction l Relation Extraction

Text Analysis Tool §Using Feature Extraction tool to extract names l imzxrun -b 2 -f C -x n -o faculty.out faculty.htm

Text Analysis Tool

Tools to Use §Language Identification Tool l Organize collection of documents by language l Restrict Search Results to documents in a particular language

Text Analysis Tool §Using Language Identification tool l imzlgini -b 2 -v < mydoc.htm

Text Analysis Tool §Language Identification Tool Results l Supports 13 Languages, New Languages Can be trained

Text Analysis Tool §Using Summarizer tool l imzsum -l 4 project.html

Text Analysis Tool §Summarizer tool - Results

Tools to Use §Web Crawler l Follows the Link topology for a fast search l Produces a Web Site Map l Use to Recognize the Authoritative pages l Provides a filtered collection of pages

Web Crawler §imyclean - to define a web space l Created include.re, exclude.re, types.re §imycrawl - to crawl a defined web space l imycrawl url webspace §imystat - to track what happens during a crawl

Tools to Use §Text Search Engine l Complicated Text Search l Powerful Linguistic Capabilities l Fuzzy searches l Query based on structure of document

Text Search Engine §Operates on a Previously based index

Text Search Engine §Types of Index l Linguistic Index (bought as buy) l Feature Index (Linguistics + Names) l Precise Index (bought as bought) l Normalized Precise Index (Case Insensitive) l Ngram Index

Combining Tools for Solutions §Searching with Categories l combining Text Search Engine and Topic Categorization Tool §Surviving a flood of l by using Topic Categorization Tools §Selectively indexing Web Pages l by combining Web Crawler, Topic Categorization Tool & Text Search Engine

Views of the Tool §Command Line (Good for Unix) §Not very useful on Windows NT §Not a good stand-alone Tool §Should be viewed as a Library