Integration Issues IMT 589 February 4, 2006

2/4/2006IMT 589-Applied and Structural Metadata2

2/4/2006IMT 589-Applied and Structural Metadata3 Solving the Integration Problem Need to find a way to bring multiple metadata schemas together Examples from readings this week show several approaches Tannenbaum has examples of standalone and distributed repositories, metadata interexchange, and enterprise portals Hunter discusses mapping elements through a shared term thesaurus Bedford talks about metadata integration at the World Bank After looking at some lessons from Bedford, we’ll discuss in more depth the MSWeb example in Rosenfeld and Morville’s article

2/4/2006IMT 589-Applied and Structural Metadata4 Enterprise Architecture Basics Design your Enterprise Architecture to support your goals Enterprise implies integration and context High level reference model must take into account the following Functional Architecture Technical Architecture Content Architecture Presentation Architecture From Bedford, Denise. Presentation to American Society of Indexers Annual Conference – Arlington Virginia – May 15, 2004

2/4/2006IMT 589-Applied and Structural Metadata5 Facilitate integration and repurposing of content - Provide broad search and retrieval capabilities - Increase reuse and decrease redundancy across content providers Increase the value and quality of content - Build intelligent relationships among disparate content sources using concepts and metadata - Define, enforce, monitor processes/procedures on content collections to ensure quality Consistent information security and disclosure enforcement - Bank records must be consistent in order to facilitate disclosure policy compliance and information sharing for partners Simplify and complete the content life-cycle - Reduce the number of user- facing content entry points by using already existent business processes - Manage content end-to-end from initial inception to final disposition What are the Goals of the World Bank Enterprise Architecture? From Bedford, Denise. Presentation to American Society of Indexers Annual Conference – Arlington Virginia – May 15, 2004

2/4/2006IMT 589-Applied and Structural Metadata6 The ECA Taxonomy View Thesaurus Topics Language From Bedford, Denise. Presentation to American Society of Indexers Annual Conference – Arlington Virginia – May 15, 2004

2/4/2006IMT 589-Applied and Structural Metadata7 Bank Metadata – Purpose & Taxonomies Identification/ Distinction Search & Browse Use Management Compliant Document Management Flat Taxonomy Hierarchical Taxonomy Network Taxonomy Faceted Taxonomy From Bedford, Denise. Presentation to American Society of Indexers Annual Conference – Arlington Virginia – May 15, 2004

2/4/2006IMT 589-Applied and Structural Metadata8 Identification/ Distinction Use Management Compliant Document Management Human Capture Inherit from Structured Content Programmatic Capture Inherit from System Context Extrapolate from Business Rules Search & Browse Metadata Capture Methods From Bedford, Denise. Presentation to American Society of Indexers Annual Conference – Arlington Virginia – May 15, 2004

2/4/2006IMT 589-Applied and Structural Metadata9 Search As a Service A project to formalize and productize MSWeb offerings in the enterprise search arena Included search, metadata support, search metrics, and optional UI Also included search and vocabulary management tools, formal documentation and processes for customer support and change control The first step toward an object-based portal

2/4/2006IMT 589-Applied and Structural Metadata10 Metadata in Search

2/4/2006IMT 589-Applied and Structural Metadata11 Preliminary Architecture There are three layers of possible interaction between vocabularies, metadata tag sets and the search/tagging interface. Examples of these interactions are shown in Figure 1, and described briefly below. Using a common data model for vocabulary construction allows the most powerful interactions between the layers; however, any vocabulary registering its metadata tag set can be exposed to a user at the time of search or browse. Vocabularies using this repository can: 1) Reuse terms in other vocabularies. 2) Reuse terms with structure in other vocabularies. 3) Map entire vocabularies to metadata tags. 4) Map parts of vocabularies to metadata tags. 5) Expose their terms and synsets to non-core users. Vocabularies not using this repository can: 6) Register metadata tags in the metadata registry. 7) Use terms and synsets from core vocabularies. All vocabularies associated with metadata tags can: 8) Be linked to tags from other tag sets through registry mapping. All registered metadata tag sets can: 9) Expose metadata tags and associated vocabularies for publishing tools. 10) Segment results in searches through matches to predetermined tags and vocabularies at time of query. 11) Expose metadata tags before search as an advanced user interface, or during search as an intermediate query refinement assist. 12) Expose vocabularies and tags in whole or in part as browsing structures for content (as in Yahoo!).

2/4/2006IMT 589-Applied and Structural Metadata12 Products Included Search Query and results Best Bets Catalogs Metadata Management Tools Metadata Registry Unified Catalog Service Vocabularies Core vocabularies Other vocabularies Categories I Need Tos Showcase Support Documentation Demos, screenshots Code

2/4/2006IMT 589-Applied and Structural Metadata13 Quick Return In order to show customers and users what could be done, the MSWeb team focused on creating a service that used editorial selection, tagging and the core vocabularies to create a small set of highly relevant content for MSWeb and sub-portals This was leveraged in: Navigation through categories Search results (Best Bets) Customizable versions of both for sub-portals

2/4/2006IMT 589-Applied and Structural Metadata14 What Did This Involve? Less than 100 categories Fewer than 1000 tagged surrogate records for high demand search content Those 1000 records are used in categories, Best Bets, and other areas on site Takes only a few hours a week to maintain and update database Results enabled users to directly navigate and search for high-use content

2/4/2006IMT 589-Applied and Structural Metadata15 Common Elements

2/4/2006IMT 589-Applied and Structural Metadata16 How It Works From Peter Morville-

2/4/2006IMT 589-Applied and Structural Metadata17 Moving to the Enterprise Once MSWeb had shown what was possible, the next step was to deploy to the sub-portals. To make this happen, they: Turned user’s (whether an end-user or a sub-portal search page) query into an XML request which returns an XML response of search results. Leveraged taxonomy work to provide a deep resource for search and browse Enabled rapid customization by embedding parameters into XML schema Provided consultation and assistance with category building and tagging skills Result- everyone was on the same platform, and using core vocabularies

2/4/2006IMT 589-Applied and Structural Metadata18 How It Works Vocabulary and Schema Database Site Server indexes Search DLL Modified string Search results Input query XML

2/4/2006IMT 589-Applied and Structural Metadata19 Query String Parsing Query: XML in SQL2000

2/4/2006IMT 589-Applied and Structural Metadata20 MSWeb Search

2/4/2006IMT 589-Applied and Structural Metadata21 MSW All-Intranet Schema Query type & schema (from query parser) Results properties definition Collection definition Sort parameters

2/4/2006IMT 589-Applied and Structural Metadata22 Category Schema Results properties definition Collection definition (Category set) Hierarchy display parameters Sort parameters

2/4/2006IMT 589-Applied and Structural Metadata23 What This Does Puts all processing on server side- client (the sub-portal server) just needs a few lines of code to pass and receive XML streams Client (sub-portal server) site is insulated from code changes Simple parameter changes allow customization of collections, query type, indexed properties, etc.

2/4/2006IMT 589-Applied and Structural Metadata24 The Old Way

2/4/2006IMT 589-Applied and Structural Metadata25 The New Way

2/4/2006IMT 589-Applied and Structural Metadata26 NTServer

2/4/2006IMT 589-Applied and Structural Metadata27 ITGWeb

2/4/2006IMT 589-Applied and Structural Metadata28 WordTest

2/4/2006IMT 589-Applied and Structural Metadata29 PGPortal

2/4/2006IMT 589-Applied and Structural Metadata30 Measures for Success For MSWeb, the goal was to increase user’s ability to navigate easily and find information more quickly For sub-portals, the goal was to have the ability to leverage MSWeb’s resources locally in a sustainable way Here’s some results

2/4/2006IMT 589-Applied and Structural Metadata31 Results for End-Users Key measureQ4 99Q1 00Q2 00 Total number of registered sites Average # Best Bets returned with 20 top search strings Modal # BB with top Median # BB with top Percentage of all top search strings that return Best Bets69%85%98% Percentage of 50 top search strings that return BBs82%84%98% Percentage of 20 top search strings that return BBs90%80%100% Number of all top search strings returning 10 or more Best Bets18125 Number of top50 search strings returning 10 or more BB6105 Number of top 20 search strings returning 10 or more BB364

2/4/2006IMT 589-Applied and Structural Metadata32 User Satisfaction Usability testing provided the following before and after numbers: A 62% reduction in the number of clicks An average of 16 seconds saved per task An 11% increase in task success rate High employee satisfaction with the site  42% VSAT in year 2000 field survey  Only 4% DSAT on same survey

2/4/2006IMT 589-Applied and Structural Metadata33 Portals Using MSWeb Services In the first three months of the offering, 9 sub-portals implemented search on their sites 2 of those created site-specific categories for their navigation All leveraged the MSWeb Best Bets results in their custom search No increase in staff at MSWeb Equivalent to a cost savings of 45 person years in avoided work

2/4/2006IMT 589-Applied and Structural Metadata34 What Worked? Providing a clear example of taxonomy value in MSWeb search and navigation Building a taxonomy management tool that used an extensible data model Empowering portal owners to do their own management of site navigation, and separating from the core shared taxonomy Divorcing presentation from delivery through use of XML and XSL Leveraging the taxonomy through tools to support all of the above