Cyborg Categorization Salvation for Search? Tom Reamy Information Architect Charles Schwab © 2001 Charles Schwab & Co., Inc., member NYSE/SIPC. All rights reserved. ( )

Categorization Explosion l Autonomy l Semio l Verity l Inxight l Topical Net l Mohomine l Simile l H5Technologies l GammaSite l MetaTagger l Applied Semantics l Sageware l SmartLogik l Quiver l PurpleYogi l Other - Tacit

Categorization: Why Now? l Forrester: Must Search Stink? l Browse and Search l Need a Taxonomy l Problem: Expensive to develop Taxonomies l Buy Search to get Categorization

News Feeds - Corporate Intranets l News Feeds and Content providers –uniform content, size and structure –professional writers –Simple or standard vocabulary l Corporate intranet –Wildly varied content –Mix of good, bad, and ugly writers –Tower of Babel: Acronyms, special meanings © 2001 Charles Schwab & Co., Inc., member NYSE/SIPC. All rights reserved. ( )

Auto-Categorization: the How l Rules l Catalog by Example l Statistical Clustering l Support Vector Machines l Machine Learning l World Knowledge © 2001 Charles Schwab & Co., Inc., member NYSE/SIPC. All rights reserved. ( )

Automatic vs. Humanatic l Humans are better, but not as consistent –General bin, understandable mistakes –Bring outside contexts to the document l Purpose, similar documents, common sense l Computers are faster and cheaper. –Faster yes, Cheaper ? –Cost of poorer quality categorization l Intranet: 20,000 users taking 60 seconds longer = $20,000 a week © 2001 Charles Schwab & Co., Inc., member NYSE/SIPC. All rights reserved. ( )

The Answer is Cyborg l Integration not Assimilation l Human and Computer Integration –Iterative, distributed work flow, ease of use l Cyborg and Content Management –Categorization and keywords by Subject Matter Experts l Cyborg and Search –Computers and people learn from each other

Create the Taxonomy l Top Level Taxonomy Categories –Human intensive, Cluster - random creativity l Grow the Taxonomy - 2nd - 3rd Levels –Humans - create rules, select training sets –Computers - Taxonomy Builders, Refine rules or training sets l Essential Feature –White Box Categorization –Customize algorithm, not just results © 2001 Charles Schwab & Co., Inc., member NYSE/SIPC. All rights reserved. ( )

Refine the Taxonomy l Initial Phase: Information Architect Effort l Suggest –Provisional Categorization, Meta Data –Automatic Summarization l Support –Distributed Work flow –Visualization of taxonomic relationships © 2001 Charles Schwab & Co., Inc., member NYSE/SIPC. All rights reserved. ( )

Maintain the Taxonomy l Intranets - ongoing human efforts –Can’t pass on the cost to your customers - they work for the same company as you l Continue and Improve Refinement –Collaborative Categorization l Features: –Smart Learning categorization –Integration - Content management, Search © 2001 Charles Schwab & Co., Inc., member NYSE/SIPC. All rights reserved. ( )

Apply the Taxonomy l Integration of Search and Categorization –Browse and Search –Real time clustering, customiztion of results –support collaborative filtering l Integration with Content Management –Integrated Distributed Work Flow –Support Taxonomic Publishing Model l Integration with Expertise & Processes © 2001 Charles Schwab & Co., Inc., member NYSE/SIPC. All rights reserved. ( )

Lessons Learned l Out of the Box, Out of Your Mind l Play well with others l Brain surgery is funl l World revolves around you l Quality counts and size matters l Let a Hundred flowers Bloom l The End © 2001 Charles Schwab & Co., Inc., member NYSE/SIPC. All rights reserved. ( )

The END l Really.