Presentation on theme: "Ontology Assessment – Proposed Framework and Methodology"— Presentation transcript:
1 Ontology Assessment – Proposed Framework and Methodology
2 Which one is an ontology? Why/not? Biological Classification SchemeAMS Classification SchemeNASA ThesaurusLibrary of Congress Subject HeadingsDublin Core Metadata SchemeOrganizational ChartISO Country ListMetadata repository schemeMaster Data RepositoryContent architecture models (OO models)SCORMXML Schema for DirectoryRecords Classification SchemeSocial Network RepresentationFolksomonyDomain Knowledge MapVisual representation of concept clustersFinancial ratiosEconomic indicatorsMathematical formulaXML structured electronic journal issueWordNetWhy do we care whether one is or isn’t an ontology? What’s the point?
3 Personal Recommendation We need to distinguish between an ontology and applications that use ontologiesWe need to suspend our heavy reliance on different domain terminologies that describe applications that use ontologies and adopt a neutral mental modelMinimize our references to dictionaries, synonym rings, taxonomies, thesauri, class schemes, knowledge maps, content schema, etc.Rather, compare the specific applications and standards to a neutral framework – this will facilitate more intelligent conversations, and will also help us to better communicate with others outside the fieldFramework must reflect the multi-dimensionality of ontologies, though – a single, linear representation of applications does not serve as a framework
4 Goals of the FrameworkBefore we can do this, though, we need to explicitly agree on the ‘end game’ of a the frameworkDo we agree that the goal is to develop a neutral, well defined, quantifiable, multidimensional framework against which any ‘thing’ that any one is calling an ontology could be evaluated?Anyone who has anything they’re calling an ontology should be able to use the framework to judge:whether it is or is not an ontologyWhich essential components it is missingWhere it ranks on a scale of informal to formal ontologiesWhat they can do to improve or enhance itEnable assessment of any proposed ontology for the purpose ofinforming users about an ontologyProviding developers with methodology for comparison and improvementEnable definition of:Minimum standards for what is/is not an ontologyThresholds for formal and informal ontologies
5 First StepsIf we agree on the goal, we need to start by defining the basic dimensions of an ontology.Dimensionality proposed in the framework includes:StructureExpressivenessrepresentational granularityintended useautomated reasoningdescriptive/prescriptivedesign methodologyDo the proposed framework dimensions accomplish this goal?Are these theoretical or practical dimensions?Do they work at a representational or on an analytical level?How easy would it be for people who are developing ontologies to understand them?Do they allow everyone who is working in an ontology space to play, or do they automatically exclude some?Do they support the ‘end game’ of communication and use?
6 Framework Recommendations I would suggest that the framework still requires both simplification and elaborationSimplification in terms of how it groups factors, and elaboration in terms of coverage of the factors that matter to those who are developing or using ontologies‘Pre-tested’ the framework with some colleagues in different areas of responsibility – none of them could understand the framework because it was too theoreticalNeed to bring it to a practical levelNeed to describe the dimensions in terms of formal lists of factors and concrete definitions for those factorsAs we develop the framework, we must also define the analytical methodsThe test of the framework is our ability to leverage it as an analysis method that allows us to neutrally characterize any ‘thing’ as an ontology and to be able to explain the characterization so that anyone can understandSuggest that we should consider using a simple factor analysis for representation and analysis
7 Simplify the Framework I would suggest that the following framework accomplishes this ‘end game’ more effectivelyConcepts – the nature of the content or values that are delivered or accessed through the ontology such as type, granularity, etc.Relationships – nature, type, extent, specification of relationships, logic associated with relationshipsContext – the context for which the ontology was developed and in which it may be used, including knowledge domain, application domain,Governance – control and management of the concepts, relationships and context exercised by the developer or current userDimensions are orthogonal but yet sufficiently well defined that they allow us to include factors which are important to different kinds of ontology applications
8 Factor AnalysisStatistical method used to describe variability of factors in which the factors are modeled as linear combinations.A single factor in the model would represent a set of ‘like’ variables which otherwise would be too complex to modelFactor analysis might help us to synthesize a set of variables into a single factor – to represent in this case a dimension of an ontologyChallenges:Agree on dimensions (synthesis of factors)Develop a method for quantifying factors appropriate to the dimensionDefine the method of factor analysisAdvantages:it might help us to focus our discussions on actual factors and away from argumentationAllows everyone and anyone to play in the ontology spaceAllows everyone and anyone to characterize their ontology as a starting point for conversations and interoperabilityWe can keep the analysis simple since we are only using this to ‘characterize’ and ‘communicate’ – not to predict or to explain factors and
9 Example of Factor Analysis Methodology is currently used to calculate and visually display factors whichContribute to the development or knowledge economies. Helps economists to compare and define knowledge economies.
10 Sample List of Innovation Factors Let’s take as an example the ICT factor as it relates to knowledge economiesWhat factors might define the ICT Dimension?Access to computersTelecommunications developmentLevel of education achievedInvestment in technology development (Tech R&D)
11 Proposed Ontology Assessment Methodology Factor analysis for ontologies would involve …defining the essential dimensions of an ontologydefining those factors which characterize each dimensionquantifying the factorsanalyzing the factors for any given application (factor analysis) or comparisonvisually representing the analysis for a single ‘ontology’ and/or for comparisons of ‘ontologies’Let me explain how factor analysis might be usedIf we can define the dimensions of an ontology, each dimension could then be represented as a composite measureThe composite measure is made up of scores for a set of factors that define that dimensionHaving a composite score for each dimension would allow us to use a very simple analytical method that would characterize or compare specific ontology applications
12 Representation of Ontological Assessments Another Dimensionality FrameworkDimension 1Index of FactorsDimension 3Dimension 2Index of FactorsIndex of FactorsIndex of FactorsDimension 4Methodology could be used to generate a factor index for ontologies, to rank and compare ontologies.
13 Factor Analysis Factor analysis could be conducted: At the component level on that subset of factorsAt the ontology level, across all factorsDevelopers or users could determine what the optimal dimensionality was for their particular useSummit members and the Ontology community could identify minimun factor scores that define what is/is not an ontology, and what constitutes a full, formal ontologyUltimately, this may provide us with an ecumenical vs. evangelical approach to ontological standards development and assessment
14 Representation of Ontological Assessments Dimensionality Suggested in the Framework PaperStructureExpressivenessIntendedUseRepresetationalGranularityUse of AutomatedReasoningDescriptive vs.PrescriptiveCritical Question: Are these dimensions orthogonal, mutually exclusive and clean enough for analysis?
15 Representation of Ontological Assessments Another Dimensionality FrameworkRelationshipsConceptsContextGovernanceMethodology could be used to generate an ontological factor index for ontologies, and to rank and compare ontologies.
16 Representation of Ontological Assessments Sample assessment of a folksonomyRelationshipsContextConceptsGovernanceMethodology could be used to generate an ontological factor index for‘ontological things’, and to rank and compare ontologies.
17 Representation of Ontological Assessments Sample assessment of a medical disease classification schemeRelationshipsContextConceptsGovernance
18 Representation of Ontological Assessments Sample assessment of an institutional records classification schemeRelationshipsContextConceptsGovernance
19 Defining and Quantifying Factors For each component an orthogonal, independent set of factors must be definedFactors must be independent of any particular pre-existing ontology (neutral)Each factor must have a quantifiable method of representation that lends itself to ‘scoring’, analysis and comparisonFactors must have agreed upon definitions, be easily interpreted by people and machines, and be inclusive in their coverage of values/conditionsTo illustrate the idea, selected examples are presented in following slides
20 Selected Examples of Concept Factors Concept typesData/numbersCalculation/ratiosWordsGrammatical fragmentLogical statementRule expressionEngineering equationsDegree of ambiguityContext sensitivity/insensitivity of definitionRepresentational formUsable encoding methodAvailability of representational specifications (Strings vs. syntax)Degree of conceptualization/ specificationTheoretical to commitalWhat else…?
21 Selected Examples of Relationship Factors Simple expressive form of relationshipsGrammaticalMathematicalLogicalRelationship behaviorMembership dependenceRepresentation or instanceEquivalenceCausal dependenceDerivational dependenceDegree of Relationship Validation/RigorFully SubjectiveGrammatical validationMathematical validationLogical rigor/validationWhat else?
22 Selected Examples of Context Factors Knowledge ContextFormal vs. informal knowledge domainApplication ContextSystem vs. human application/ consumptionManaged/standardized application vs. home grownFunctional contextSearchMathematical or statistical analysisLogical inferenceClassificationDynamic clusteringMetadata representationConcept indexingWhat else…?
23 Selected Examples of Governance Factors Standards AvailabilityPublished formal vs. guidelines vs. ad hoc conceptsPublished formal vs. guidelines vs. ad hoc relationshipsPrescriptive vs. Descriptive GovernanceEnforcement of standardsDesign GuidelinesTop-down (model) vs. Bottom-up (empirical)ExtensibilityDegree to which others can add to or extend either the concepts or the relationshipsCurrencyDegree to which the concepts and/or relationships represent our current view or knowledge of the contextWhat else…?