Presentation is loading. Please wait.

Presentation is loading. Please wait.

Ontology Assessment – Proposed Framework and Methodology.

Similar presentations


Presentation on theme: "Ontology Assessment – Proposed Framework and Methodology."— Presentation transcript:

1 Ontology Assessment – Proposed Framework and Methodology

2 Which one is an ontology? Why/not? Biological Classification Scheme AMS Classification Scheme NASA Thesaurus Library of Congress Subject Headings Dublin Core Metadata Scheme Organizational Chart ISO Country List Metadata repository scheme Master Data Repository Content architecture models (OO models) SCORM XML Schema for Directory Records Classification Scheme Records Classification Scheme Social Network Representation Social Network Representation Folksomony Folksomony Domain Knowledge Map Domain Knowledge Map Visual representation of concept clusters Visual representation of concept clusters Financial ratios Economic indicators Mathematical formula XML structured electronic journal issue WordNet Why do we care whether one is or isnt an ontology? Whats the point?

3 Personal Recommendation We need to distinguish between an ontology and applications that use ontologies We need to distinguish between an ontology and applications that use ontologies We need to suspend our heavy reliance on different domain terminologies that describe applications that use ontologies and adopt a neutral mental model We need to suspend our heavy reliance on different domain terminologies that describe applications that use ontologies and adopt a neutral mental model Minimize our references to dictionaries, synonym rings, taxonomies, thesauri, class schemes, knowledge maps, content schema, etc. Minimize our references to dictionaries, synonym rings, taxonomies, thesauri, class schemes, knowledge maps, content schema, etc. Rather, compare the specific applications and standards to a neutral framework – this will facilitate more intelligent conversations, and will also help us to better communicate with others outside the field Rather, compare the specific applications and standards to a neutral framework – this will facilitate more intelligent conversations, and will also help us to better communicate with others outside the field Framework must reflect the multi-dimensionality of ontologies, though – a single, linear representation of applications does not serve as a framework Framework must reflect the multi-dimensionality of ontologies, though – a single, linear representation of applications does not serve as a framework

4 Goals of the Framework Before we can do this, though, we need to explicitly agree on the end game of a the framework Do we agree that the goal is to develop a neutral, well defined, quantifiable, multidimensional framework against which any thing that any one is calling an ontology could be evaluated? Anyone who has anything theyre calling an ontology should be able to use the framework to judge: whether it is or is not an ontology Which essential components it is missing Where it ranks on a scale of informal to formal ontologies What they can do to improve or enhance it Enable assessment of any proposed ontology for the purpose of informing users about an ontology Providing developers with methodology for comparison and improvement Enable definition of: Minimum standards for what is/is not an ontology Thresholds for formal and informal ontologies

5 First Steps If we agree on the goal, we need to start by defining the basic dimensions of an ontology. Dimensionality proposed in the framework includes: Structure Expressiveness representational granularity intended use automated reasoning descriptive/prescriptive design methodology Do the proposed framework dimensions accomplish this goal? Are these theoretical or practical dimensions? Do they work at a representational or on an analytical level? How easy would it be for people who are developing ontologies to understand them? Do they allow everyone who is working in an ontology space to play, or do they automatically exclude some? Do they support the end game of communication and use?

6 Framework Recommendations I would suggest that the framework still requires both simplification and elaboration I would suggest that the framework still requires both simplification and elaboration Simplification in terms of how it groups factors, and elaboration in terms of coverage of the factors that matter to those who are developing or using ontologies Simplification in terms of how it groups factors, and elaboration in terms of coverage of the factors that matter to those who are developing or using ontologies Pre-tested the framework with some colleagues in different areas of responsibility – none of them could understand the framework because it was too theoretical Pre-tested the framework with some colleagues in different areas of responsibility – none of them could understand the framework because it was too theoretical Need to bring it to a practical level Need to bring it to a practical level Need to describe the dimensions in terms of formal lists of factors and concrete definitions for those factors Need to describe the dimensions in terms of formal lists of factors and concrete definitions for those factors As we develop the framework, we must also define the analytical methods As we develop the framework, we must also define the analytical methods The test of the framework is our ability to leverage it as an analysis method that allows us to neutrally characterize any thing as an ontology and to be able to explain the characterization so that anyone can understand The test of the framework is our ability to leverage it as an analysis method that allows us to neutrally characterize any thing as an ontology and to be able to explain the characterization so that anyone can understand Suggest that we should consider using a simple factor analysis for representation and analysis Suggest that we should consider using a simple factor analysis for representation and analysis

7 Simplify the Framework I would suggest that the following framework accomplishes this end game more effectively Concepts – the nature of the content or values that are delivered or accessed through the ontology such as type, granularity, etc. Relationships – nature, type, extent, specification of relationships, logic associated with relationships Context – the context for which the ontology was developed and in which it may be used, including knowledge domain, application domain, Governance – control and management of the concepts, relationships and context exercised by the developer or current user Dimensions are orthogonal but yet sufficiently well defined that they allow us to include factors which are important to different kinds of ontology applications Dimensions are orthogonal but yet sufficiently well defined that they allow us to include factors which are important to different kinds of ontology applications

8 Factor Analysis Statistical method used to describe variability of factors in which the factors are modeled as linear combinations. Statistical method used to describe variability of factors in which the factors are modeled as linear combinations. A single factor in the model would represent a set of like variables which otherwise would be too complex to model A single factor in the model would represent a set of like variables which otherwise would be too complex to model Factor analysis might help us to synthesize a set of variables into a single factor – to represent in this case a dimension of an ontology Factor analysis might help us to synthesize a set of variables into a single factor – to represent in this case a dimension of an ontology Challenges: Challenges: Agree on dimensions (synthesis of factors) Agree on dimensions (synthesis of factors) Develop a method for quantifying factors appropriate to the dimension Develop a method for quantifying factors appropriate to the dimension Define the method of factor analysis Define the method of factor analysis Advantages: Advantages: it might help us to focus our discussions on actual factors and away from argumentation it might help us to focus our discussions on actual factors and away from argumentation Allows everyone and anyone to play in the ontology space Allows everyone and anyone to play in the ontology space Allows everyone and anyone to characterize their ontology as a starting point for conversations and interoperability Allows everyone and anyone to characterize their ontology as a starting point for conversations and interoperability We can keep the analysis simple since we are only using this to characterize and communicate – not to predict or to explain factors and We can keep the analysis simple since we are only using this to characterize and communicate – not to predict or to explain factors and

9 Example of Factor Analysis Methodology is currently used to calculate and visually display factors which Contribute to the development or knowledge economies. Helps economists to compare and define knowledge economies.

10 Sample List of Innovation Factors Lets take as an example the ICT factor as it relates to knowledge economies Lets take as an example the ICT factor as it relates to knowledge economies What factors might define the ICT Dimension? What factors might define the ICT Dimension? Access to computers Access to computers Telecommunications development Telecommunications development Level of education achieved Level of education achieved Investment in technology development (Tech R&D) Investment in technology development (Tech R&D)

11 Proposed Ontology Assessment Methodology Factor analysis for ontologies would involve … Factor analysis for ontologies would involve … 1. defining the essential dimensions of an ontology 2. defining those factors which characterize each dimension 3. quantifying the factors 4. analyzing the factors for any given application (factor analysis) or comparison 5. visually representing the analysis for a single ontology and/or for comparisons of ontologies Let me explain how factor analysis might be used Let me explain how factor analysis might be used If we can define the dimensions of an ontology, each dimension could then be represented as a composite measure If we can define the dimensions of an ontology, each dimension could then be represented as a composite measure The composite measure is made up of scores for a set of factors that define that dimension The composite measure is made up of scores for a set of factors that define that dimension Having a composite score for each dimension would allow us to use a very simple analytical method that would characterize or compare specific ontology applications Having a composite score for each dimension would allow us to use a very simple analytical method that would characterize or compare specific ontology applications

12 Representation of Ontological Assessments Methodology could be used to generate a factor index for ontologies, to rank and compare ontologies. Dimension 3Dimension 2 Dimension 1 Dimension 4 Another Dimensionality Framework Index of Factors

13 Factor Analysis Factor analysis could be conducted: Factor analysis could be conducted: At the component level on that subset of factors At the component level on that subset of factors At the ontology level, across all factors At the ontology level, across all factors Developers or users could determine what the optimal dimensionality was for their particular use Developers or users could determine what the optimal dimensionality was for their particular use Summit members and the Ontology community could identify minimun factor scores that define what is/is not an ontology, and what constitutes a full, formal ontology Summit members and the Ontology community could identify minimun factor scores that define what is/is not an ontology, and what constitutes a full, formal ontology Ultimately, this may provide us with an ecumenical vs. evangelical approach to ontological standards development and assessment Ultimately, this may provide us with an ecumenical vs. evangelical approach to ontological standards development and assessment

14 Representation of Ontological Assessments Structure Represetational Granularity Expressiveness Descriptive vs. Prescriptive Dimensionality Suggested in the Framework Paper Intended Use Use of Automated Reasoning Critical Question: Are these dimensions orthogonal, mutually exclusive and clean enough for analysis?

15 Representation of Ontological Assessments Methodology could be used to generate an ontological factor index for ontologies, and to rank and compare ontologies. ConceptsContext Relationships Governance Another Dimensionality Framework

16 Representation of Ontological Assessments Methodology could be used to generate an ontological factor index for ontological things, and to rank and compare ontologies. Concepts Context Relationships Governance Sample assessment of a folksonomy

17 Representation of Ontological Assessments Concepts Context Relationships Governance Sample assessment of a medical disease classification scheme

18 Representation of Ontological Assessments Concepts Context Relationships Governance Sample assessment of an institutional records classification scheme

19 Defining and Quantifying Factors For each component an orthogonal, independent set of factors must be defined For each component an orthogonal, independent set of factors must be defined Factors must be independent of any particular pre-existing ontology (neutral) Factors must be independent of any particular pre-existing ontology (neutral) Each factor must have a quantifiable method of representation that lends itself to scoring, analysis and comparison Each factor must have a quantifiable method of representation that lends itself to scoring, analysis and comparison Factors must have agreed upon definitions, be easily interpreted by people and machines, and be inclusive in their coverage of values/conditions Factors must have agreed upon definitions, be easily interpreted by people and machines, and be inclusive in their coverage of values/conditions To illustrate the idea, selected examples are presented in following slides To illustrate the idea, selected examples are presented in following slides

20 Selected Examples of Concept Factors Concept types Concept types Data/numbers Data/numbers Calculation/ratios Calculation/ratios Words Words Grammatical fragment Grammatical fragment Logical statement Logical statement Rule expression Rule expression Engineering equations Engineering equations Degree of ambiguity Degree of ambiguity Context sensitivity/insensitivity of definition Context sensitivity/insensitivity of definition Representational form Representational form Usable encoding method Availability of representational specifications (Strings vs. syntax) Degree of conceptualization/ specification Degree of conceptualization/ specification Theoretical to commital What else…? What else…?

21 Selected Examples of Relationship Factors Simple expressive form of relationships Simple expressive form of relationships Grammatical Grammatical Mathematical Mathematical Logical Logical Relationship behavior Relationship behavior Membership dependence Membership dependence Representation or instance Representation or instance Equivalence Equivalence Causal dependence Causal dependence Derivational dependence Derivational dependence Degree of Relationship Validation/Rigor Degree of Relationship Validation/Rigor Fully Subjective Grammatical validation Mathematical validation Logical rigor/validation What else? What else?

22 Selected Examples of Context Factors Knowledge Context Knowledge Context Formal vs. informal knowledge domain Formal vs. informal knowledge domain Application Context Application Context System vs. human application/ consumption System vs. human application/ consumption Managed/standardized application vs. home grown Managed/standardized application vs. home grown Functional context Functional context Search Mathematical or statistical analysis Logical inference Classification Dynamic clustering Metadata representation Concept indexing What else…? What else…?

23 Selected Examples of Governance Factors Standards Availability Standards Availability Published formal vs. guidelines vs. ad hoc concepts Published formal vs. guidelines vs. ad hoc concepts Published formal vs. guidelines vs. ad hoc relationships Published formal vs. guidelines vs. ad hoc relationships Prescriptive vs. Descriptive Governance Prescriptive vs. Descriptive Governance Enforcement of standards Enforcement of standards Design Guidelines Design Guidelines Top-down (model) vs. Bottom-up (empirical) Top-down (model) vs. Bottom-up (empirical) Extensibility Extensibility Degree to which others can add to or extend either the concepts or the relationships Currency Currency Degree to which the concepts and/or relationships represent our current view or knowledge of the context What else…? What else…?


Download ppt "Ontology Assessment – Proposed Framework and Methodology."

Similar presentations


Ads by Google