A review of ontology editor evaluation Presenter: Yujie Cao Paper Sharing A review of ontology editor evaluation Presenter: Yujie Cao

Catalog 01 Research Method 02 Main Findings 03 Recommendations 04 Questions

01 Research Method – literature review

02 Main Findings 01 Doesn’t exist any standard approach Integrative analysis of the main methods and parameters 03 An evaluation framework and benchmark Evaluation content and process analysis from every study 02 Main evaluation parameters and methods An overview of ontology editor evaluation 01 Doesn’t exist any standard approach

02 Main Findings- parameters and methods General Ontology Edit Cooperative reliability knowledge representation interoperability simplicity creating taxonomies discussion threads learnability creating relations tracking of changes inference content reviewing clipping of views libraries zooming Testing Inspection Inquiry co-discovery learning heuristic evaluation questionnaire cognitive walkthroughs field observation interview

02 Main Findings- framework & benchmark Standard should contain: 1)a set of common steps 2)questions/checklist/heuristics 3)a standard formula

03 Recommendations 1) Common evaluation process of ontology editors 2) Related information need to be collected 3) Possible comparative tables for ontology editors 4) Questions need to be considered in evaluation

Step1: Identify evaluation objectives Step2: Introduce the tool to users Step3: Use ontology editors Step4: Obtain feedback Step5: compute evaluation

Protégé OntoSaurus …… User domain 5(medical) 1(unrelated) — experience 1 Specific PARM ontologies taxonomies relations inference interoperability tracking Logger record mistakes -1 -2 Overall PARM reliability 4 3 simplicity 5 Total score 8.5 7 Compute formula: 01 Set specific levels of parameters and give values to each level 02 Set weight for each parameter in the form based on Evaluation objectives

03 Recommendations- questions 1) What are the key features that should be analyzed? 2) What is the appropriate number and levels of participants? 3) How could the parameters be conducted by tasks? 4) How to abstract a standard from evaluations with different objectives? 5) What is the relationship between the evaluation result of ontology editor and the needs of users?

04 Questions about this paper 1) Is there any quantitative parameters in the ontology editors evaluation？ General Ontology Edit Cooperative reliability knowledge representation interoperability simplicity creating taxonomies discussion threads learnability creating relations tracking of changes inference content reviewing clipping of views libraries zooming

2) Is there any specialized evaluation technologies or methods for ontology editors?

Yujie Cao Joint Phd student Thank you! Yujie Cao Joint Phd student