Presentation on theme: "The Challenges of Building Enterprise Content Taxonomies and the Role of Classification Technologies in Maintaining Their Effectiveness Reginald J. Twigg,"— Presentation transcript:
1 The Challenges of Building Enterprise Content Taxonomies and the Role of Classification Technologies in Maintaining Their EffectivenessReginald J. Twigg, Ph.D.Capture, Classification and Taxonomy, IBM ECMThis session provides an overview of the role of taxonomy and classification management in ECM, outlines IBM’s strategy for enabling ECM (starting with FileNet P8) classification and taxonomy management and introduces the offering. This session is a Classification for ECM 101, starting high level with key challenges for managing unstructured enterprise content, then moving into more detail on the CM maturity model, and finally, the offering.Nowhere, can any other vendor provide such a broad set of capabilities and amass so many experts in ECM.This is a key IBM unique. And a customer value.
2 Agenda The Challenge of Unstructured Content Key Concepts and Terms Taxonomy, Classification and ECM AdoptionClassification Technologies for ECM
4 80% of Enterprise Data is Unstructured Billing statementsClaims imagesCustomer correspondenceMortgage docsContractsSigned BOLsHealthcare EOBsMarketing collateralWebsite contentVoice authorizationsSignature cardsCredit enrollmentsMaterial Safety Data SheetsISO 9000 docsPlant schematicsProduct imagesSpec sheets….and much more!There are two key categories of enterprise information – data and content. Data are managed by relational databases. The advent of the relational database ushered in a wave of enterprise applications, starting with ERP, then CRM, Supply Chain Management, etc. These business solutions leveraged the database to help organizations manage their structured transactional business processes.However, most enterprise information – especially that used in many core business process for services, government and other sectors – is unstructured. According to Gartner, at least 80% of enterprise information is unstructured, and many organizations report growth at over 100% per year. Because unstructured content is the substance of many critical line of business processes, it has become necessary to deploy new technologies for managing its processes.Databases
5 What is Enterprise Content? According to Forrester, unstructured content falls into 3 main categories – transactional, business and persuasive. Imaging customers are most familiar with transactional content, such as forms, faxes, reports and scanned images. Business content includes process-centric and collaborative document management, while persuasive includes digital assets and web-based content.Unstructured enterprise content is tightly coupled with business processes. Content is created in processes, changed through business processes and also drives those processes. Finding ways to manage these content forms drives the effectiveness of many core business processes in customer service, case management, document management, etc.
6 Where do I start?Organizing the explosion of unstructured content becomes critical:We’ve got 600 GB of content from basic content services all over the enterprise. How can we get this content efficiently mapped into our ECM taxonomy?We’ve been managing our content without classifying it for a few years now. How can our users navigate amongst this existing content in a way that’s intuitive for our business?The lawyers have to review 400,000 electronic documents for their case. How can we make sure they don’t waste their time?You’re likely familiar with the statistic by now – 80% of the information in the enterprise is unstructured (as opposed to structured, database content). By bringing that content under management with ECM and cataloguing it, enterprises are working hard to both save time and cost by further leveraging the content, but also manage risk and enforce the compliance of that content.To meet those ends, we’ve seen our customers begin to struggle with organizing this content. Organizing the content, lending structure to this unstructured tangle of documents and leaves enterprises thinking, “Where do I start?”. Getting this content organized is rising as a hurdle in our customers adoption of ECM. Some examples:Customers find that they simply cannot classify content as they bulk ingest content into ECM. There is simply no one to manually address the problem of getting hundreds and hundreds of gigabytes of content mapped into their standard ECM taxonomyIt’s not just the new content – it’s also the content that you might already have under management – there wasn’t a classification solution that you had the time or budget for in the past, and now you’re left with content that’s unclassified. Yes, you have it under management, but its still relatively unstructured – you did the bare minimum. Now you’re having trouble finding this content and have better access to this content in an intuitive manner for your business.Finally, you can’t always proactively organizing your content – new applications (like a legal discovery review process) might demand new ways of approaching the information. Maybe you need to prioritize your content and create workflows that make sure your lawyers are devoting the bulk of their attention to the right area.
7 Business Value of Classification for ECM Key Business DriversECM Taxonomy and ClassificationCompliance, Records, Legal DiscoveryIn Process ClassificationMessage Tagging, Classification and Monitoring2134Increase accessibility of content under managementAutomated, High Scale ClassificationClassify at ingestion and/or re-classify over timeTaxonomy Evolution ToolsEnhanced AccessibilityTaxonomy ProposerIncrease legal discovery review effectiveness while reducing riskLegal Discovery Prioritization and Workflow AssignmentRecords Classification and Exception HandlingStorage and Retention Policy AssignmentIncrease worker productivity and automate content related decisionsAd Hoc Category SuggestionContent-Based Workflow SelectionContent Based Decision MakingReduce inquiry costs, automate message routing and increase customer satisfaction, Chat RoutingAgent Response SuggestionSupervision and MonitoringAutomatic Customer ResponseFrom a broad perspective, there are a variety of applications of classification, both within ECM and outside the guises of ECM. Classification, at its core, provides the enterprise with automated decision making capability. This decision making capability can in turn lend itself to a wide set of automatic actions: assigning categories or taxonomies, filtering, routing, answer suggestions. These automated actions can be leveraged in a variety of use cases. Lets review some of them:1) For the customer who needs to accurately catalog content, we’ve got a solution for automating this process – automating the process of classifying content as it is brought under management or as it needs to be reclassified.Beyond this application, there are other applications of classification2) In the compliance space, you can use classification to automate the records declaration process and save every employee time as they struggle to decide what “category” their content fits into.In the eDiscovery space you can you use classification to better organize your case content.Alternatively, you can organize your content in a manner that’s unique to a particular legal case for better navigation. You could divide up the content and categorize it in a way that makes sense for assignment of workloads to different lawyers with different areas of expertise or even generate a list of high priority documents for review by your most valuable legal minds – leaving the drudgery to the lower priced knowledge workers.3) There is intersection with BPM in the sense that classification can be used to identify the proper workflow for a content or make a workflow more efficient by automatically populating attributes or making strong suggestions to the executor of the workflow4) Finally, classification has applications in handling – whether it be to automatically respond to a customer inquiry, route an incoming or monitor it for inappropriate content.<CLICK>But in this presentation, we’re going to focus on the value of the first use case -- #1 – and classification’s application to the problem of taxonomies in ECM . . .
8 Ability to Structure Content with Databases There is an inverse relationship between the growth of unstructured enterprise content and the ability to get value out of it using traditional relational databases and data-centric enterprise applications. The challenge of managing unstructured content and using it in key business processes depends upon providing ways to rationalize, bring order to, and enable users to gain access to the right content at the right time.Ability to Structure Content with DatabasesPercent of corporate information value managed in traditional databasesUnstructured DataData Creation And DemandStructuredDataOLTP and BI (narrow scope)Application TypesCompliance, Competitive Intelligence (wide scope)Source: Gartner
9 Multiple Repositories Make Access Difficult 1 repository“The Future of Content in the Enterprise,” Connie Moore and Robert MarkhamDon't know5%17%36%More than 15repositories2-5 repositories25%14%According to Forrester, global 2000 organizations have multiple content repositories. Each repository has one or more content management application built on top of it. Most notable is that a quarter of large enterprises have more than 15 content repositories with dependent applications. This can be a real barrier to setting policies and standards for the enterprise, since the organization must interpret the different organizational concepts and schema across different content management repositories.Please note that these statistics include mainstream content management repositories (e.g., FileNet, IBM, OTEX, DCTM) and not other content management systems.10-15 repositories6-10 repositories4%Base: 81 North American decision-makers(multiple responses accepted)
10 And Then There’s SharePoint, File Shares and . . . Which leads us to SharePoint. The ease of deployment has made MSFT SharePoint the choice for many departmental collaboration and simple content management solutions. These solutions, including Quickr, Team Rooms and other departmental content management applications, are loosely-organized, relying on users to classify and manage the lifecycle of their content.The challenge is to gain access to, and manage this content without sacrificing productivity. Being able to classify this content is foundational to being able to manage it.They look out over that horizon, look out over their own enterprise to find all that other content they’re determined to bring under management. But its not just one file system, or one sharepoint instance. These repositories, these silos, they’re everywhere. There could be hundreds of sharepoint repositories. There could be hundreds of file shares. Silos associated with basic content services can litter the enterprise.Maybe this content is classified. Probably not.And to make matters worse, even if there is a taxonomy for each of these content silos, and the content is all classified, it isn’t likely to match the standard information model you’ve built into your ECM architecture. Each of them can come with their own designed taxonomy Or an organic taxonomy – a “Folksonomy”.
11 Key Concepts and TermsNow we turn to classification and taxonomy management. Before diving deeper into the role of taxonomy and classification management in ECM, it is important to level-set on the key concepts and definitions.Although many of you have experience with classification (here ask how many audience members are currently working on class/tax projects), the terminology can vary. This section attempts to provide some clarity on these concepts.
12 Key ConceptsMetadata: a means of describing, locating, cataloging, and activating content as objects in a software ecosystem (literally, data about data).Enterprise Catalog: a centralized and normalized metadata model for unstructured content for the purposes of providing consistent services across all ECM applications.Taxonomy: a hierarchical structure of information components, any part of which can be used to classify a content item in relation to other items in the structure.Classification: a coding of content items as members of a group for the purposes of cataloging them or associating them with a taxonomy.Taxonomy - Paraphrased from Gartner: “A taxonomy is a classification, typically hierarchical, of information components (for example, terms, concepts, graphics and sounds) and the relationships among them. In a hierarchical structure, it reflects increasing levels of specificity the further down the hierarchy a particular element lies. Taxonomies may be used to represent membership in various domains, and can support information organization, discovery, presentation and access. The taxonomic organization and its labels serve as metadata for the content they organize.” Hype Cycle for the High Performance Workplace 2006, Rita E. Knox et al. 14 July 2006 (ID: G ), p. 34.Classification - Classification is rooted in natural and library sciences for the purposes of identification and location. Wikipedia provides the most useful definition and analogy for ECM classification: “A library classification is a system of coding and organizing library materials (books, serials, audiovisual materials, computer files, maps, manuscripts, realia) according to their subject and allocating a call number to that information resource. Similar to classification systems used in biology, bibliographic classification systems group entities that are similar together typically arranged in a hierarchical tree structure (assuming none-faceted system) Classification of a piece of work consists of two steps. Firstly the 'aboutness' of the material is ascertained. Next, a call number based on the classification system will be assigned to the work using the notation of the system Classification systems in libraries generally play two roles. Firstly they facilitate subject access (See Cutter) by allowing the user to find out what works or documents the library has on a certain subject. Secondly, they provide a known location for the information source to be located (e.g where it is shelved).”
13 Taxonomy Is . . . Not turning animals into trophies A system for organizing the corpus of business contentBefore the builds, point out that we often get asked about what taxonomy is. I have been asked if it is about stuffing animals. Hit the first build – what taxonomy is not.Second build explains what it is in simple terms. The library image also provides the organizing principles for ECM and the use of the enterprise catalog for organizing and providing access to the corpus of content.
14 Taxonomy and Classification in ECM Classification Examples:Document ClassingFolderingTaxonomy Examples:Enterprise Content CatalogIndustry Standard Document Taxonomies (ISO, XMI)Methods:Rules-Based: Applies pre-determined rules for ‘if, then’ classification of text and propertiesAnalytics-Based: Applies algorithms to interpret classes in order to apply classification rules to themThese are the most fundamental types of classification and taxonomy. Most ECM customers will be familiar with these concepts, so it should provide a baseline for the remaining discussion.2 key methods of classification:Rules-based is most common and very useful for classification. Rules-based classification allows setting pre-determined rules for classing content, then determining the appropriate actions to take on it. The ‘if-then’ logic of rules-based approaches quite literally performs classification (‘if this’ classifies a content item) in the process of taking action on it. Records Crawler is a solid rules-based engine for classification and provides a useful tool for many classification tasks.Analytics-based classification provides content intelligence as a way of understanding content that can not be easily classified through pre-defined rules. Using advanced analytical algorithms, analytics-based classification can learn a taxonomy and interpret unknown content to automate its classification. These approaches have the benefits of reducing the number of exceptions to manage, as well as help classify content in file shares and collaboration solutions where little or inconsistent metadata are common.These approaches are complementary – rules-based classification can address most content, while analytics-based classification helps manage exceptions. Today, the IBM Classification Module (available through the OmniFind offering) provides out-of-the-box analytics classification.
15 ECM Taxonomy Illustrated Before we go any further, lets pause to define some terms. Taxonomy can mean different things to different people. In the spirit of “a picture is worth a thousand words”, we’ve thrown up some examples of taxonomies.Lets build up to a definition of a taxonomyMetadata is data about content. Think of metadata as an attribute of a piece of content or a tag associated to a piece of content.In turn, a taxonomy is a collection of the different metadata values, for a particular attribute (or category). All the different values an attribute could have are collected into a metadata structure called a taxonomy.Some easy examples of taxonomies within the context of P8 are folders or document classes. But really, any other attribute that has been designed into your ECM deployment makes up a taxonomy.The act of deciding where in a particular taxonomy a document should be placed is the act of classification. And that decision – that action of classification – is at the basis of the solution we’ll discuss today.==================Other notes to the speaker:Some outright, formal definitions:Metadata: A means of describing, locating, cataloging, and activating content as objects in a software ecosystem (literally, data about data)Taxonomy: A hierarchical structure of information components, any part of which can be used to classify a content item in relation to other items in the structureClassification: A coding of content items as members of a group for the purposes of cataloging them or associating them with a taxonomy===========================Taxonomy - Paraphrased from Gartner: “A taxonomy is a classification, typically hierarchical, of information components (for example, terms, concepts, graphics and sounds) and the relationships among them. In a hierarchical structure, it reflects increasing levels of specificity the further down the hierarchy a particular element lies. Taxonomies may be used to represent membership in various domains, and can support information organization, discovery, presentation and access. The taxonomic organization and its labels serve as metadata for the content they organize.” Hype Cycle for the High Performance Workplace 2006, Rita E. Knox et al. 14 July 2006 (ID: G ), p. 34.Classification - Classification is rooted in natural and library sciences for the purposes of identification and location. Wikipedia provides the most useful definition and analogy for ECM classification: “A library classification is a system of coding and organizing library materials (books, serials, audiovisual materials, computer files, maps, manuscripts, realia) according to their subject and allocating a call number to that information resource. Similar to classification systems used in biology, bibliographic classification systems group entities that are similar together typically arranged in a hierarchical tree structure (assuming none-faceted system) Classification of a piece of work consists of two steps. Firstly the 'aboutness' of the material is ascertained. Next, a call number based on the classification system will be assigned to the work using the notation of the system Classification systems in libraries generally play two roles. Firstly they facilitate subject access (See Cutter) by allowing the user to find out what works or documents the library has on a certain subject. Secondly, they provide a known location for the information source to be located (e.g where it is shelved).”
16 Taxonomy, Classification and ECM Adoption Now let’s connect the dots – how classification and taxonomy play in ECM.
17 Drive New Business Value from Content Improve Content AccessOrganize Unstructured ContentContent Classification SolutionsDerive Business Insight
18 Business Drivers for ECM Taxonomy Management Proliferating departmental solutionsContent ManagementCollaboration (SP, Quickr, Team Rooms, Wikis)User-based classification and high workforce turnoverProductivity declines as knowledge disappearsLegal discovery is a secondary concernMergers and Acquisitions – need to reconcile disparate content management practices, repositories and processesChallenges – the growing number of departmental solutions is an obvious challenge. How do organizations balance the need for knowledge-worker productivity with establishing and maintaining corporate standards for business process and governance.High workforce turnover risks a loss of knowledge and understanding of the uses of content. Although lawsuits are an obvious example (an employee leaves and then sues the company), the loss of productivity that occurs from the loss of knowledge about the location and uses of content is a higher, more measurable cost. Studies have suggested that the average company invests $50,000 to train new knowledge workers. Add to this cost the cost of recreating documents that may be ‘out there somewhere’, and turnover can add significant expense to relying on users to manage classification.M&A is another obvious challenge. Not only must different content be accessed and integrated, but this content is always borne of different processes, users and work cultures. Being able to understand acquired taxonomies is essential to integrating them into the new ECM system.
19 Classification is Hard Work Key Business ChallengesECM Taxonomy and ClassificationMost organizations face content taxonomy pain – especially as they standardize around ECMMapping content to taxonomy during ingestionReclassifying content under managementEvolving taxonomies as new types of content emergeIntegrating folksonomies (SharePoint) into a master taxonomyIncrease accessibility of content under managementAutomated, High Scale ClassificationClassify at ingestion and/or re-classify over timeTaxonomy Evolution ToolsEnhanced AccessibilityTaxonomy Proposer11
20 Organization is the Root Cause Most organizations face content taxonomy barriers – especially as they standardize around ECMAssigning categories en masseReclassifying existing content as taxonomies evolveMerging taxonomiesIntegrating the wisdom of folksonomiesIn Enterprise Content Management, taxonomies ensure that content is accurately catalogued and easily accessible. Having consistent and reliable access to unstructured content is arguably the foundation to realizing the business benefits of ECM, and all subsequent content-centric enterprise applications will realize their ROI by leveraging this essential capability.Enterprise information architects see the standardization of content under a single set of rules and policies, as a key driver for ECM adoption. ECM platform standardization on IBM FileNet P8 enables the management of unstructured content across the enterprise by metadata management into a single, unified catalog. Standardization also raises a unique challenge – namely, how to manage content mapped under widely different metadata structures (i.e. different taxonomies) spread across multiple departments, repositories, and applications.This manifests itself in some pain points that you yourself might be pondering or tackling right now.1) The easiest, most basic use case to think of is bringing new content under management into ECM. Its stored on a fileshare and you need to accurately catalog it – thousands of documents coming in ‘en masse’, requiring classification.2) The next pain point is related – what if you’ve already brought content under management, and either because you didn’t have the facility before or the need didn’t exist at ingestion time, the content doesn’t have the proper classification. You need to reclassify your content.3) The business scenarios go on You might have merged with a company with its own ECM. How are you going to rectify the conflicting taxonomies?4) You’ve got taxonomies distributed through the enterprise – they don’t exactly match your standard taxonomy – but they’ve got interesting aspects to them – how are you going to normalize your standard taxonomy to take this wisdom into account?
21 Challenges and Impacts of Merging Taxonomies Misclassification – change is constant, and master taxonomies must manage multiple custom taxonomies for each content source“Folksonomies” from departmental collaboration solutions are created by users and unmanaged by ECM standardsImpact:Unreliable Metadata – Inconsistencies lose or mislabel contentProcess Misfires – Poor metadata triggers incorrect events and workflowsInformation challenges.Unreliable metadata is particularly a challenge with BCS (Basic Content Services) and local collaboration solutions (SharePoint, Quickr). These solutions rely on users to manage metadata, if it is managed at all. Without consistent metadata, traditional classification methods break down.“Folksonomies” are user-created, purpose-built taxonomies. Systems like SharePoint and Team Rooms allow users to create and use their own classifications. While folksonomies might have their work purpose, they create a challenge for applying consistent standards to their content.Classification flux is the fact that taxonomies are dynamic and constantly subject to change. For this reason classification is not a single-shot but must be performed on a regular basis to ensure that taxonomies are not out of synch.Scale is the Challenge – Automation is Essential
22 Classification Barriers to ECM Maturity The keynote session provided an introduction to the ECM maturity model. Classification challenges can pose hurdles to moving up the maturity curve, specifically:Build 1 (ingestion): moving from a siloed to a more integrated content management environment poses the challenge of being able to interpret and classify content across multiple silos. Getting metadata into FileNet P8 from multiple content silos is a major challenge, since understanding how to classify it often requires manual effort. At this stage, automation is needed to manage the volume and types of content.Build 2 (Standardization): Establishing an ECM platform as the enterprise standard for content-centric business applications has demonstrable ROI in reducing operating costs and allowing the consistent capturing and application of policies across the organization. The hurdle to standardization is applying standards across multiple taxonomies. The inability to reconcile different taxonomies can lead to failure to standardize.Build 3 (Enforcement): Once standards are set, classification is necessary to provide ongoing enforcement of content management policies across the organizations. Content must be understood in order to apply both policies and process to it. Classification and taxonomy serve as the basis for this understanding. Compliance typically surfaces the pain of classification and enforcement, although many content management applications raise the requirement.Note: Often the challenge of taxonomy management is less about defining an enterprise taxonomy than about applying it to the different taxonomies that exist in work processes and different forms of content. In this case, taxonomy does not need to be imposed as much as it needs to interpret, understand, and classify content in different applications and sources.Classification Barriers to ECM MaturityScale / ScopeTruly Enterprise Class SystemsLevel 5Federated Activation & Policy BasedLevel 4Federation & ActivationLOB and DepartmentalLevel 3Search & DiscoveryClassification hurdle #3 - EnforcementZero-click Policy based ComplianceActive Storage And RetrievalEnd-user and Process DrivenNormalized and federated securityRe-use enabledAbility to find and use content across departmentsLOB SystemsClassification hurdle #2 - StandardizationEnd-user and Process DrivenNormalized and federated securityRe-use enabledAbility to find and use content across departmentsFederated RetentionLevel 2Silos & StorageAd-hoc usageNormalized securitySome re-useDirectory Server based securityEnd-user DrivenLoosely defined responsibilityAbility to find content across departmentsClassification hurdle #1 - IngestionLevel 1ChaosAd-hoc usageFragmented securityLittle re-useApplication based, siloed workflowEnd-user DrivenLoosely defined responsibilityNo re-use across departmentsAd-hoc usageNo re-useNo workflowEnd-user DrivenUndefined ResponsibilityFile system securityTechnology & CapabilitiesUnified RepositoryContent/Process FusionActive enforcement of ComplianceIntegrated Content, Process and Compliance capabilitiesAvailable as a serviceFederatedShared DriveMultiple disparate content repositoriesPaperStand-aloneMaturityEvolution Over Time
23 Lessons Learned From ERP Adoption Getting Classification Right: ‘Garbage in = garbage out’ is often used in metadata management projects to describe the problem of building a metadata model on inconsistent sources.Driving Process on Taxonomies: ERP systems depending on 3 master taxonomies – material, vendor and customer. These taxonomies drive events, workflow definition and the development of transaction-centric business process applicationsMastering Metadata: The ability to deploy new enterprise applications depends upon the re-usability, scalability and integrity of the metadata modelSystem of Record is Required for Standardization:Establishes an enterprise standard that can be auditedForms the foundation for building demonstrable best practicesEnforces consistency of data capture and outputECM has many lessons to learn from the mass adoption of ERP. SAP arguably became the market leader in this space by closely managing classification, taxonomy, metadata and process. ECM has similar challenges today.Getting Classification Right: Managing data input requires discipline. ERP systems tightly controlled the classification of data and data entry to maintain the integrity of their metadata models. Much of the pain of ERP implementation was training users to enter data in a highly-structured, disciplined way. Classification is at the core of this process. ECM will have special challenges, however, since much unstructured content is produced by knowledge workers in their own processes. The challenge for ECM will be to classify in ways that maintain and enforce process and policies without interfering with productivity.Driving Process on Taxonomies: SAP built the R/3 system on 3 master taxonomies – material, vendor and customer. The relationship among these 3 taxonomies formed the basis of their event model, workflow and application stack. This provided a consistent way of applying taxonomy to improve the efficiency of their business processes. The challenge for ECM adoption is to maintain productivity while providing the benefits of master taxonomies.Mastering Metadata: Not a new concept to FileNet P8 users, mastering metadata for unstructured content ensures consistent access to, and interaction with, that content across different business functions and sources.Establishing a System of Record: First used in data management in the 1980s, a ‘system of record’ provides a basis of accountability for all of the information contained in the system. This means it can pass audit (internal, external, regulatory, industry) and serves as the basis for demonstrating policy compliance. ERP systems established themselves as the ‘system of record’ for transactional business processes – most notably General Ledger, costing, inventory, procurement, etc. The challenge for ECM is to establish a system of record for unstructured content and content-centric business processes, records management, document management and other ECM processes.
24 Customer Lessons for Mastering ECM Taxonomies ‘Master’ taxonomy of record required forComplianceBusiness process applicationsMerged master taxonomies become large and unwieldyMultiple taxonomies require integration and translationCentralized, decentralized, or hybrid?Intelligent Classification increasingly is used to manage:Taxonomy merging from multiple use casesTaxonomy/folksonomy translation from distributed content sources
25 A Look at ECM Classification Technologies This final section introduces the strategy and offering for classification and taxonomy management for FileNet P8.
26 State of Classification Management Technologies ECM Classification/Taxonomy is an emerging disciplineIndustry standard taxonomies:Focus on business function or transaction typesHave not reached the enterprise levelClassification best practices:Content ingestionApplication development reclassificationClassification software focuses on content ingestion:Electronic content ( , Office documents, free-form text)Paper content (document images) requires OCRSearch is not enough – must drive value in the business process
27 Criteria For ECM Classification Management Solutions Integrate with and support the ECM metadata modelInterpret a highly-federated content ecosystemGo beyond search to catalog and manage contentBuild on advanced analytic technologies – rules alone are not enoughInterpret content to extract meaningful (meta)dataEmploy multiple methods (engines) for classificationIntegrate teaching/learning
28 Common Platform for Electronic Content Classification QueueClassification and MonitoringCompliance, Records, Legal DiscoveryClassificationPlatformECM Taxonomy and ClassificationIn Process Classification
29 IBM Classification/Taxonomy Strategy for ECM Enterprise Services for Active ContentClassification/Reclassification at CaptureFile SharesSharePoint, QuickrFederated RepositoriesTaxonomy Management forExposing P8 taxonomies (Enterprise Manager) for classifying enterprise contentExtending taxonomies as enabling services for Content-Centric BPM ApplicationsEstablish System of Record for Master Content ManagementIBM ECM’s approach to classification and taxonomy management is to provide these as an enterprise service. This service helps to ensure the integrity of metadata from different content sources and access to key content from applications built on the P8 platform. This service, which we call Master Content Management, is designed to provide classification and taxonomy as a foundational, universal capability of which all ECM/BPM applications can take advantage.This solution, using existing IBM products in the Discovery portfolio (specifically, the classification analytics and integration products acquired through iPhrase), is available today as part of the OmniFind offering.We are completing an integration of this analytics product with FileNet P8 to provide a solution for classifying content in file shares, collaboration solutions (such as SharePoint) and other content sources.An analytics-based engine, this classification/taxonomy management solution for P8 goes beyond keyword search to provide syntactic, semantic and contextual interpretation of content (metadata and full text) for classification. With the ability to learn from teaching documents and user actions, this analytics-based classification engine increases the quality of classification with use. Multiple algorithms are provided out of the box, as well as rapid deployment of integrations with different content sources.This product will be announced as the ‘ECM Taxonomy Server’ in May, 2007.
30 IBM Classification Module for Electronic Content Organize your ECM contentAutomated classification and filteringCombines text analytics understanding with rulesAcquires domain specificity from your own contentUnique learning technology for adaptive classificationSuggests new categories or even seeds an entirely new taxonomyRectifies conflicting taxonomiesMarket proven, scalable platformWith that in mind, lets introduce to you the solution from IBM – the IBM Classification Module for IBM FileNet P8.At its core, it can automate the classification of content as its ingested or content that is already under management. It can also filter out content that doesn’t fit the profile of managed content.It does so though a combination of text analytics-based understanding with rules. We’ll spend more time on this, but in summary, text analytics understanding of your content means that the solution reads your content – all of the text in the document or – and assigns metadata to it based on the full body of the content. Its not just looking for one magic word or following a set of policies (though it can if you need it to). Its reading the whole document and taking the context of the whole document into account.When it makes classification decisions, its able to do so in a manner unique to your business because it trains itself from your content (especially the content already classified and under management in P8!).And as you use the software more and more, the solution learns from feedback you provide to it. What you see on the left hand side of the slide here is a snapshot of the Classification Review Tool. It lets you audit the automated actions as well as handle the exception cases. With the auditing, you’re able to monitor the automation – and as you gain confidence in the quality of the decisions being made by the solution, you can ramp down the level of auditing you execute.Through the course of reviewing and auditing the automated actions, these “manual” actions are providing feedback for understanding your content, the solution takes that feedback into account and in turn adapts its understanding in realtime. So the very next classification request will learn from the actions that you’ve taken. And as it learns in this manner, it takes more recent teachings into account heavier than older ones, allowing the system to adapt and evolve its understanding of how to classify content, as your business adapts and evolves.In terms of helping your taxonomy evolve, the solution can also review content items that don’t fit into the current taxonomy and in turn suggest new groups and new categories to augment your existing taxonomy. Furthermore, you can take the full set of content and ask it to recommend not just a new group of values, but an entirely new taxonomy.The core of the solution is built on the IBM Classification Module, a market proven product from IBM that has been in the market for over years.
31 Understanding Content with Text Analytics The strategic value of this market is paramount to IBMAIP is essentialClassificationEngineALegal is currently requiring full approvalTraining (Teach)BEngineering requires clear requirementsMatchingC Strategy isImportant to the marketing teamCategories list andRelevancies (Scores)FeedbackCorpus (Categorized)The strategic value of this market is paramount to IBMC The core market for this new product has been defined as such by IBMAuditC: 97%,B: 54%,A: 12%Here, lets dive down a little deeper into the IBM Classification Technology – the core service for interpreting different classifications and normalizing their expression in the central enterprise catalog. This service is a combination of analytics, linguistic and statistical methods and tools.Lets focus on the right hand side first:A document is sent to the classification, the text is “read”. In turn a classification (or set of suggested classifications) are returned for use by the ECM system via text analysis. The content is run through natural language processing to identify the key concepts involved in the document and in turn this concept profile is compared against the training set for statistical similarity. The content is classified as part of the category (or categories) it is most similar to.Each classification, at application runtime, is also paired with a confidence level. This confidence level is used in a variety of ways. Primarily, it is used to set a level of automation. The higher confidence level you require of automated action, the lower the amount of your automation will be. Other applications can use this confidence level to throttle the levels of automation.Moving to the left hand side, the system itself is trained using real content, each associated to a category in the taxonomy. Your actual business content is used to create this statistical profile of your taxonomy and it learns from the best possible examples – your real content. It encompasses the “messiness” of real content and the subtleties that more rule-based approaches might miss. With actual content, we get not only the main topic of the document, but also the full context to help us differentiate similar categories.The capability of understanding the meaning of unstructured text and adapting to changing environments in real-time is what makes IBM Classification Technology unique. The technology understands not only the words used, but also the context of the language, as well as associated metadata. Unlike other technologies, IBM Classification Technology self-learns, becoming more accurate over time without requiring human adjustment.=============================Other Notes:IBM Classification Technology currently supports language processing and identification in 11 languages, including most west European languages, Japanese, Chinese, and Korean. It can also execute classification in a generic manner for many other languages.The broader solution leverages FileNet P8 to train the system and in turn automate the assignment of classifications.
32 Classification Workflow: Accelerating Content Organization Classification ReviewToolSend to taxonomy proposerExisting Unclassified Managed ContentAutomatically categorize majority of contentFileSystemClassifierThe workflow for the ingestion use case starts with the ingestion process where content (or all of your content) is identified for classification from its wide variety of sources.In any case, the Classifier reviews the incoming documents and attempts to automatically classify them in the ECM system, decide where to put them, under which folder, using which document class and/or what attribute.If it feels confident about a document, it can auto-classify it into the right document class and folder. And this level of confidence is confifgurable.If, on the other hand, it feels confident that the document doesn’t belong in the ECM system at all, it doesn’t seem to belong to any existing category in the repository, it can filter out the document, simply remove it and not insert it into P8. (this step, is of course, optional)And in all other cases, it forwards the document to the Classification Review Tool to have it manually reviewed by the user.A fourth scenario exists as well: content that it has confidently classified, but is siphoned off to the review tool for auditing purposes. You can set the level of auditing to meet your needs. We’d typically recommend to start with a high level of auditing to build confidence that you trust the automated actions and then turn down the auditing to lower levels as time moves on.The bulk of the content will be automatically handled in this workflow. But for the content that does flow into the review tool, there are a couple of potential outcomes:The content was properly classified and the reviewer simply confirms the automated decision. Positive feedback is generated for realtime learning.The content was improperly classified (or wasn’t effectively classified at all) and the user assigns the content to an existing category in P8 or creates a new category in P8 and assigns the content to that category. Again, feedback is generated and the system learns in realtime.Finally, the content can be sent for further review in the taxonomy proposer, and used for creating new categories for your taxonomyFilter out documentsBasic Content ServicesReference: Integration ComponentsClassifier (Runtime Application)Classification Review (UI)Taxonomy Proposer (UI)Content Extractor (training based on P8)
33 Components of the Solution for Text Classification ClassifierAutomatically classifies and filters out documentsMoves some documents for manual reviewClassification Review ToolAllows user to manually review documentsContent ExtractorExtracts content from the ECM system for trainingTaxonomy ProposerUser workflow to identify and name new categories or apply existing taxonomy from P8Features available today include:Automatic classification of source documents. This classification tool learns from the corpus of content it touches (both native P8 and external sources) to enhance the automation of classification.Review Tool for managing exceptions. The Review Tool also learns from expert interaction to provide additional automation over time.Content Extractor for evaluating content for learning purposes.Taxonomy Proposer uses workflow to enable users and the system to identify new categories/classes and name them for future reference.
34 Classification for Paper Documents Classification of paper documents occurs in capture processUse cases for paper document classificationRecognition using OCR/ICRClassification to associate to folders or doc classSeparation to reduce costs and improve process
35 Three Primary Types of Images – The Document Recognition Problem Less AdvancedMore AdvancedSemi-StructuredStructuredUn-StructuredStructuredStructured dataKnown page layoutConsistent formatsLimited data fieldsGood quality documentsSemi-StructuredSemi-Structured dataUnknown page layoutVariable formatsTabular dataComplex/multi-page documentsVariable quality documentsUnstructuredUnstructured dataUnknown/complex page layoutStructured formsapplication formsbenefit formstax formsairline ticketsRemittancescheckspaying in slipsSemi-structured formsinvoicesUnstructured formscorrespondenceProcessing Unstructured and Semi-Structured Forms When it comes to recognizing unstructured or semi-structured forms and other documents types, these same steps still apply, although in a different order and manner. On unstructured forms such as invoices, EOBs, and transportation documents, locating variably-formatted data is the number one priority. The presence of random differences over a series of functionally similar forms means that the forms in question cannot be processed using the traditional template-based approach, in which one software template matches each and every data field on each and every form in a presorted batch. Multiple approaches are required, approaches that create alternatives with results that are compliant with classifying similar data elements. Given the incredible computing horsepower and enormous amount of memory that reside in the average desktop PC, a variety of techniques, including ICR brute force, can be run in parallel to achieve remarkably accurate results that were impossible to achieve only a few years ago. Sometimes, morphological analysis techniques involving blob analysis, edge detection, multi-line character segmentation, and long-line detection can be used to find form objects, columns, and data fields. At other times, the geometrical and spatial relationships between the text data elements, such as rows or subheadings (rather than graphical objects), can locate the places where data most likely will be found. In fact, the data location process need not involve character recognition at all; the text can be treated as integral patterns of blobs.Arthur Gingrande Jr., IMERGE ConsultingSteps in the Template-Based Processing of Structured Forms Just as documents must be prepared in order to be fed into a scanner by removing staples, smoothing wrinkles, positioning them for optimal registration, etc., so the image of a form document must be prepared by following these steps before it can be intelligently recognized:Document scanning—Pages of forms are scanned and converted into bit-mapped (usually TIFF) images of forms which are either compressed and stored for later batch processing, or are passed immediately in an uncompressed format to an ICR engine for recognition.Image analysis—The document image is cleaned up. Character image quality is improved, using image enhancement techniques. Background “noise” is removed from the form.Form alignment—The image is registered and deskewed by the ICR software, which automatically aligns the form by locating special symbols on the document called registration marks as guides.Form identification—The document is identified by certain predefined characteristics that the ICR software is trained to look for, so that the zones containing the fields designated for recognition can be located by a customized, predefined ICR template. Form ID attributes can include form numbers, corporate logos, or the name of the form itself imprinted somewhere on the form.Form background removal—This stage is not necessary if the document is a form that was originally printed in a colored (“drop out”) ink that is invisible to the scanner being used. If colored ink is not used, the form image may contain lines, boxes, fine print, and other form attributes—passive data—that tend to confuse the ICR engine. These form attributes must be extracted from the image of the form, so that only the character images—the active data—are left behind. Broken and fragmented characters are automatically repaired and restored to their original shapes.Character field location—The predefined ICR template automatically locates the fields that contain character data. The template identifies which individual fields on the form image require character recognition, and what the nature of those fields are—hand print, machine print, numeric, alphabetic, alphanumeric, etc. The template also identifies which areas are barcodes or check box recognition zones.Character segmentation—Sophisticated software routines analyze, separate, and break down the character fields into isolated characters. If the form is “ICR –friendly,” characters are segmented with the aid of graphic devices such as boxes, tick-marks, and connected boxes called “combs” that serve to force the form user to legibly separate the characters from one another.Character classification— Individual characters are classified by ICR algorithms according to their ASCII category and assigned a confidence value, which is an index of how “certain” the ICR engine “feels” about the selection it has made. Alternate character choices are ranked according to those values, so that they can be incorporated into editing procedures that improve ICR accuracy. For example, the alternate choice “1” might be used instead of the first-ranked choice “I” when contextual analysis reports that the field is all-numeric.Post-processing—The initial or “raw” recognition results are validated using edit procedures such as grammatical rules, spell-checkers, dictionaries, check-sum routines, and look-up tables. Ambiguous and erroneous data fields—the “rejects”— are identified and sent to data entry operators at workstations for manual correction.Manual correction of rejected character fields—The manner in which the data entry operator is presented the rejected data for correction can dramatically impact both the speed and the accuracy of the reject repair process. In particular, the data entry GUI is important because the ergonomics of data entry are what enable a given data entry operator to reach his or her maximum correction speed.What is interesting is that only one of the steps—character classification—is specifically concerned with identifying character data. The rest of the steps have to do with either preparing the imaged characters for classification or interpreting the results of character classification. With so much opportunity for error increasing at each successive step of the way, it is remarkable that ICR accuracy rates can attain (and sometimes exceed) human performance levels. Processing Unstructured and Semi-Structured Forms When it comes to recognizing unstructured or semi-structured forms and other documents types, these same steps still apply, although in a different order and manner. On unstructured forms such as invoices, EOBs, and transportation documents, locating variably-formatted data is the number one priority. The presence of random differences over a series of functionally similar forms means that the forms in question cannot be processed using the traditional template-based approach, in which one software template matches each and every data field on each and every form in a presorted batch. Multiple approaches are required, approaches that create alternatives with results that are compliant with classifying similar data elements. Given the incredible computing horsepower and enormous amount of memory that reside in the average desktop PC, a variety of techniques, including ICR brute force, can be run in parallel to achieve remarkably accurate results that were impossible to achieve only a few years ago. Sometimes, morphological analysis techniques involving blob analysis, edge detection, multi-line character segmentation, and long-line detection can be used to find form objects, columns, and data fields. At other times, the geometrical and spatial relationships between the text data elements, such as rows or subheadings (rather than graphical objects), can locate the places where data most likely will be found. In fact, the data location process need not involve character recognition at all; the text can be treated as integral patterns of blobs.
36 The Document Separation Problem in Image Capture Separation of documents is a significant expense for a high-volume capture systemTypical ‘structured’ recognition technologies are not applicableManual insertion of separator sheets is the primary workaround today50% of document preparation labor is spent sorting documents and inserting separator pages – source: TAWPIWhere does one document stop and the next begin?Here?Here?Here?Here?
37 Classification Methods for Paper Content (Images) Image Classificationbased on the overall layout and structure of a documentIncludes lines, boxes, logos and placement of textText Classificationbased on detailed analysis of the text content of a pageRules-Based Classificationperformed by searching for specific data or keywordsindependent of layoutTemplated Classificationdetermined by the presence of one or more marks, barcodes or items of text in pre-defined locations
38 Waterfall Approach to Classification and Separation Two-pass system:1st pass: Classificationoptimizes performance by using fastest classification techniques firstAdvanced Text Classification final “catch-all12345678Page #BarcodeRecognition:FirstForm X???????1 msImageClassification:N/A??FirstForm Y?FirstForm Z??20 msRulesBased :N/A?LastForm XLastForm Y?LastForm Z200 msText Classification:N/AMiddleForm XMiddleForm Z1000 ms
39 Why Invest in Automated Classification? Accelerate the time to value in your investment in ECMEnsure more accurate content catalogsHaving consistent and reliable access to unstructured content is arguably the foundation to realizing the business benefits of ECM, and all subsequent content-centric enterprise applications will realize their ROI by leveraging this essential capabilityAt its core, classification will accelerate your time to value when it comes to your investment in ECM. It will be easier to bring more content under management, quickly.When you do bring it under management it will be accurately catalogued and as a consequence you’ll be lowering your risk and improving your down stream efficiencyMore content under management and having it catalogued accurately logically leads to content that is more likely to be found, more accessible, more likely to be re-used, more likely to be further leveraged. A key driver to investing in ECM across the enterprise is making your content more accessible – and accurately cataloguing information is key to that value propositionFinally, the last benefit is the one that speaks to every organization – more time. Your knowledge workers not only will be more productive because they’re wasting less time looking for information, but also your true authoring experts – the truly valuable people in your organization – will be freed up from manually tagging and classifying the contentMake your content easier to find and leverageFree up your subject matter experts
40 IBM Classification Module for IBM FileNet P8 SummaryAccelerate ECM StandardizationPoor content classification undermines ECM value – maximize your ECM potential and time-to-value with automated classificationAutomating Classification Always PaysTypical employees spend 10 hours/week searching for information – slash that time and increase productivityClassification Technologies Automate Classification to Drive Development of Best PracticesIBM Classification Module for IBM FileNet P8Automatically organizing your content by understanding it
41 Questions&AnswersContact Reggie Twigg for more information or to arrange a demonstration