1. A review of ontology editor evaluation Presenter: Yujie Cao
Paper Sharing
A review of ontology editor evaluation
Presenter: Yujie Cao

2. Catalog 01 Research Method 02 Main Findings 03 Recommendations
04 Questions

3. 01 Research Method – literature review

4. 02 Main Findings 01 Doesn’t exist any standard approach
Integrative analysis of the main methods and parameters
03 An evaluation framework and benchmark
Evaluation content and process analysis from every study
02 Main evaluation parameters and methods
An overview of ontology editor evaluation
01 Doesn’t exist any standard approach

5. 02 Main Findings- parameters and methods
General
Ontology Edit
Cooperative
reliability
knowledge representation
interoperability
simplicity
creating taxonomies
discussion threads
learnability
creating relations
tracking of changes
inference
content reviewing
clipping of views
libraries
zooming
Testing
Inspection
Inquiry
co-discovery learning
heuristic evaluation
questionnaire
cognitive walkthroughs
field observation
interview

6. 02 Main Findings- framework & benchmark
Standard should contain:
1)a set of common steps 2)questions/checklist/heuristics 3)a standard formula

7. 03 Recommendations 1) Common evaluation process of ontology editors
2) Related information need to be collected
3) Possible comparative tables for ontology editors
4) Questions need to be considered in evaluation

8. Step1: Identify evaluation objectives
Step2: Introduce the tool to users
Step3: Use ontology editors
Step4: Obtain feedback
Step5: compute evaluation

10. Protégé
OntoSaurus ……
User
domain
5(medical)
1(unrelated) —
experience 1
Specific PARM
ontologies
taxonomies
relations
inference
interoperability
tracking
Logger record
mistakes -1 -2
Overall PARM
reliability 4 3
simplicity 5
Total score
8.5 7
Compute formula:
01 Set specific levels of parameters and give values to each level
02 Set weight for each parameter in the form
based on
Evaluation objectives

11. 03 Recommendations- questions
1) What are the key features that should be analyzed?
2) What is the appropriate number and levels of participants?
3) How could the parameters be conducted by tasks?
4) How to abstract a standard from evaluations with different objectives?
5) What is the relationship between the evaluation result of ontology editor and the needs of users?

12. 04 Questions about this paper
1) Is there any quantitative parameters in the ontology editors evaluation？
General
Ontology Edit
Cooperative
reliability
knowledge representation
interoperability
simplicity
creating taxonomies
discussion threads
learnability
creating relations
tracking of changes
inference
content reviewing
clipping of views
libraries
zooming

13. 2) Is there any specialized evaluation technologies or methods for ontology editors?

14. Yujie Cao Joint Phd student
Thank you!
Yujie Cao
Joint Phd student