Presentation is loading. Please wait.

Presentation is loading. Please wait.

Controlled Ontology Evolution Through Semiotic-based Ontology Evaluation International Workshop on Ontology Dynamics IWOD 2008 Renata Dividino

Similar presentations


Presentation on theme: "Controlled Ontology Evolution Through Semiotic-based Ontology Evaluation International Workshop on Ontology Dynamics IWOD 2008 Renata Dividino"— Presentation transcript:

1 Controlled Ontology Evolution Through Semiotic-based Ontology Evaluation International Workshop on Ontology Dynamics IWOD 2008 Renata Dividino Daniel Sonntag

2 Overview Motivation Semiotic-based Ontology Evaluation Controlled Ontology Evolution through Evaluation Semiotic-based Ontology Evaluation Tool Evaluation Results Conclusion

3 Motivation = Reason for changes! not good for my application? Apply changes! not good for my intend of use? Apply changes! Inconsistent? Apply changes! not aligned to dependent ontologies? Apply changes!

4 Motivation = Evaluation criterion! Which version is the best one for my app? Is the ontology still good (or better) for my intend of use? Is the ontology consistent? Can I apply these changes without affecting the dependent ontologies?

5 Motivation Explicit Requirements Reasons for changes = Evaluation criterion Ontology changes captured by ontology evaluation process Support controlled ontology evolution by using evaluation methods and theories

6 Overview Motivation Semiotic-based Ontology Evaluation Semiotics Semiotics & Ontology Semiotics & Ontology Evaluation Controlled Ontology Evolution through Evaluation Semiotic-based Ontology Evaluation Tool Evaluation Results Conclusion

7 Yojo Concept Symbol Object The Meaning Triangle: (Ogden and Richards 1923) Semiotics is composed of three fundamental components: (Moris, 1938) Syntax Semantics Pragmatics

8 (Niles & Pease, 2001) Language Syntax Semantics Pragmatics OntologySemiotic Object Semantics Pragmatics Syntax

9 Symbol Object Concept Symbol Ontology Graph graph-like structures containing terms and their inter-relationships Object Intended Conceptualization represent an intended conceptualization. Concept Communication Context Rep The ontologys representation is interpretable by some agent (Gangemi at al, 2006)

10 SyntaxSemanticsPragmatics Semiotic-based Ontology Evaluation Defining Criterion Structural Evaluation Functional Evaluation Pragmatics Evaluation

11 Semiotic-based Ontology Evaluation Defining Criterion Structural Evaluation Functional Evaluation Pragmatics Evaluation The quality of ontologies is critical for the success of their improvements, cost reduction, and maintenance! Semiotic-based Evaluation: semiotic levels inter-dependencies interactive evaluation steps – incremental improvements

12 Structural Measures Depth Breath Modularity … Syntax (Topological Dimension) Consistency Complexity Concept Satisfiability … Formal Semantics (Logical Dimension) Semiotic Measures Pragmatics Measures Experts judgments User satisfaction Agreement satisfaction Data Set Task Assessment Modularity Assessment … Precision-Recall Based Measures (Functional Dimension) Functional Measures Assesing the ontology pragmatics Assesing the ontology cognitive semantics Assesing the ontology syntax and formal semantics Annotations/Documentation: Structural-Related: Depth, Breath, Modularity. Functional-Related: Experts Judgments, Data Set. User-Oriented: Deployment, Commercial, History/Review, Version. Ontology profile (Usability Dimension)

13 Overview Motivation Semiotic-based Ontology Evaluation Controlled Ontology Evolution through Evaluation Semiotic-based Ontology Evaluation Tool Evaluation Results Conclusion

14 Controlled Ontology Evolution through Evaluation applying quality assessment at each evolution step (incrementally changes) continual improvements (continual changes) Controlled Ontology Evolution through Evaluation Capturing Representation Semantics of change PropagationImplementation Validation

15 Controlled Ontology Evolution through Evaluation Semiotic Ontology Evaluation Process Capturing Representation Semantics of change PropagationImplementation Validation Controlled Ontology Evolution through Evaluation (Stojanovic, 2004)

16 Controlled Ontology Evolution through Evaluation Semiotic Ontology Evaluation Process Capturing Representation Semantics of change PropagationImplementation Validation Controlled Ontology Evolution through Evaluation Explicit Requirements Reasons for changes = Evaluation Criteria

17 Controlled Ontology Evolution through Evaluation Capturing Representation Semantics of change PropagationImplementation Validation Controlled Ontology Evolution through Evaluation Structural Evaluation

18 Controlled Ontology Evolution through Evaluation Capturing Representation Semantics of change PropagationImplementation Validation Controlled Ontology Evolution through Evaluation Structural Evaluation Making the ontology changes visible in a form of an adequate representation Ontology changes need to be managed such that the ontology remains consistent

19 Controlled Ontology Evolution through Evaluation Capturing Representation Semantics of change PropagationImplementation Validation Controlled Ontology Evolution through Evaluation Structural Evaluation Functional Evaluation

20 Controlled Ontology Evolution through Evaluation Capturing Representation Semantics of change PropagationImplementation Validation Controlled Ontology Evolution through Evaluation Structural Evaluation Functional Evaluation Ontology changes need to be managed such that the ontology remains consistent Verify consistency of dependent ontologies

21 Controlled Ontology Evolution through Evaluation Capturing Representation Semantics of change PropagationImplementation Validation Controlled Ontology Evolution through Evaluation Structural Evaluation Functional Evaluation Pragmatics Evaluation

22 Controlled Ontology Evolution through Evaluation Capturing Representation Semantics of change PropagationImplementation Validation Controlled Ontology Evolution through Evaluation Structural Evaluation Functional Evaluation Pragmatics Evaluation User is able to approve the changes applied or to reverse them

23 Controlled Ontology Evolution through Evaluation Capturing Representation Semantics of change PropagationImplementation Validation Controlled Ontology Evolution through Evaluation Structural Evaluation Functional Evaluation Pragmatics Evaluation

24 Overview Motivation Semiotic-based Ontology Evaluation Controlled Ontology Evolution through Evaluation Semiotic-based Ontology Evaluation Tool Implementation Evaluation Results Conclusion

25 Semiotic-based Ontology Evaluation Tool Structural Evaluation Functional Evaluation Pragmatics Evaluation

26 Semiotic-based Ontology Evaluation Tool Structural Evaluation Functional Evaluation Pragmatics Evaluation User recognition (documentation, versioning) Consistent intended use, context or usage (domain coverage, performance, etc) Adequate representation form (syntactic correct) Formal consistent

27 SyntaxSemanticsPragmatics Semiotic-based Ontology Evaluation Tool

28 Representation Semantics of change PropagationImplementation Validation

29 Structural Measures Semiotic Measures Pragmatics Measures Functional Measures Formal Semantics: Consistency Checking Cognitive Semantics: Task-based Approach Pragmatics: Annotation Analysis Implementation: Semiotic-based Ontology Evaluation Tool

30 Semiotic Measures Consistency Checking: onto changes remains consistent Task-based Approach: onto. changes max. performance Annotation Analysis: changes reported for versioning Implementation: Semiotic-based Ontology Evaluation Tool Validation Semantics of change Propagation Representation Semantics of change

31 Use of the reasoners RACER System and Pellet. …a logical theory is consistent if it does not contain a contradiction, or, more precisely, for no proposition φ are both φ and ¬φ provable. PersonSeal Disjoint (Person, Seal) Shark (primitive class) Animal and eats some (Person and Seal) Inconsistent Consistency Checking

32 Task-based Approach How effective a given ontology is in the light of a well- defined task (Porzel, 2004; Maedche & Staab,2002) Task Application Performance Results Ontology Compare with Gold Standard Answers Improvements ?

33 Task-based Approach Evaluate different ontology versions! Ontology V0.1 Ontology V0.2 changes Is my evolved ontology still good (or better) for my intend of use? max. performance for a specific task!

34 Task-based Approach How efficient is the system to answers questions using just ontologies Improvements Question-Answering SmartWeb Performance Results Compare to Gold Standard Answers SWIntO V0.2 V0.1

35 Task-based Approach SmartWeb Performance Results Original (GS) Gold Standard Answers Generation 1.Plug the Gold Standard Ontology into the system 2.Query the system 3.Check Time Performace 4.Gold Standard Answers Generation

36 Task-based Approach 1.Plug the Evolved Ontology 4. Compare with GS Answers 2.Query the system 3.Check Time Performance 5.Make Report Lexicon Taxonomy Semantic Relations SmartWeb Performance Results Compare with Gold Standard Answers Evolved Improvements 6.Apply changes!

37 Task-based Approach Comparing Ontologies (Maedche & Staab,2002; Dellschaft & Staab, 2006) root accomodation area city hotelyouth hostel hotelarea wellness hotel city Lexicon(hotel, hotel) = 1 Taxonomy(hotel) = {hotel, accomodation} {hotel,wellness hotel} = Semantic(located-at)= {hotel, accomodation}{hotel} * {city,area}{area} = ½ * ½ = ¼ located-at

38 Usability-Related Evaluation Annotation Analysis: Quantitative analysis of the amount of metadata linked to the tag rdf:comments … Teacher Class

39 Overview Motivation Semiotic-based Ontology Evaluation Controlled Ontology Evolution through Evaluation Semiotic-based Ontology Evaluation Tool Evaluation Results Conclusion

40 Evaluation Results SWIntO Ontology (SmartWeb Project*) Foundational (DOLCE) and general (SUMO) knowledge Domain- and task-specific knowledge Football (soccer) entities and events Navigation Linguistic information Discourse Multimedia * SmartDOLCE:Entity SmartSUMO:Attribute SmartSUMO:SocialRole SportEvent:FootballPlayer SportEvent:FootballOrganizationPerson … …… …… … …

41 Consistency Checking

42 Functional Evaluation I

43 SWIntO V Q1:Which matches took place in the semifinals in 1954? Q2:Who was the world champion in 1990? Time-performance:31,10 ms GS-performance:26,23 ms Vocabulary Overlap = 100% Hierarchy Overlap = 87% Relation Overlap = 45% Evaluated Relation: GS Relation: SWIntO V Q1:Which matches took place in the semifinals in 1954? Q2:Who was the world champion in 1990? Time-performance:31,10 ms GS-performance:26,23 ms Vocabulary Overlap = 100% Hierarchy Overlap = 87% Relation Overlap = 45% List of Overlap Descriptions: Evaluated Relation: GS Relation: Functional Evaluation II

44 Functional Evaluation III

45 Annotation Analysis

46 Conclusion Evaluation framework to support and control ontology evolution Apply changes to an ontology keeping its quality with respect to the purpose of the ontology (or the purpose of the ontology changes) Controlled evolution by assessing the quality of the ontology with respect to all semiotic dimensions -> Ontology changes captured by ontology evaluation process Implementation = choose three measures which are essential in any ontology evolution/evaluation process Structural Dimension: Consistency Checking Functional Dimension: Task-based Evaluation Usability Dimension: Annotation Analysis Future Work level of granularity & integration

47 Thank you for your attention!

48 RACER System, Renamed abox and concept expression reasoner. R. Porzel and R. Malaka, A task-based approach for ontology evaluation, M. Ciaramita J. Lehmann A. Gangemi, C. Catenacci, Modelling ontology evaluation and validation. V. Sugumaran A. Burton-Jones, V. C. Storey and P. Ahluwalia, A semiotic metrics suite for assessing the quality of ontologies, Janez Brank, Marko Grobelnik and Dunja Mladenic, A Survey of Ontology Evaluation Techniques. W. Wahlster, Smartweb: Mobile applications of the semantic web, In Proceedings of Informatik. 2004s. P. Buitelaar and A. Frank,Ontology-driven Predicate-Argument Structure Analysis for Event Annotation John F. Sowa, Ontology, Metadata, and Semiotics, P. Cimiano,Text Analysis and Ontologies, C.W. Morris, Foundations of a theory of signs. In: International Encyclopedia of Unified Science (O. Neurath, R. Carnap & C. Morris, eds), Chicago University Press, Chicago, pp I. Niles and A. Pease.Towards a Standard Upper Ontology, Y. Sure and R. Studer. On-to-knowledgemethodology - final version. Technical Report Deliverable 18, Institute AIFB, University of Karlsruhe, A. Gangemi, C. Catenacci, M. Ciaramita, and J. Lehmann. Qood grid: A metaontology-based framework for ontology evaluation and selection. In Proceedings of EON2006, N. Guarino.. Towards a Formal Evaluation of Ontology Quality, pages 1541–1672. IEEE Intelligent Systems, D.Orbele et al. Dolce ergo sumo: On foundational and domain models in swinto (smartweb integrated ontology), N. Noy. Evaluation by ontology consumers. pages 1541–1672. IEEE Intelligent Systems, K. Dellschaft and S. Staab. On How to Perform a Gold Standard Based Evaluation of Ontology Learning, 2006 A. Maedche and S. Staab.Measuring Similarity between Ontologies, L. Stojanovic. Methods and tools for ontology evolution. PhD thesis, University of Karlsruhe (TH), References


Download ppt "Controlled Ontology Evolution Through Semiotic-based Ontology Evaluation International Workshop on Ontology Dynamics IWOD 2008 Renata Dividino"

Similar presentations


Ads by Google