1. Ontology Evaluation, Metrics, and Metadata in NCBO BioPortal Natasha Noy Stanford University

2. What Are We Trying to Do? You’ve built an ontology, how do you let the world know? You need an ontology, where do you go to get it? How do you know whether an ontology is any good?

3. BioPortal: A Community-Based Ontology Repository http://bioportal.bioontology.org

4. BioPortal Today 195 ontologies in biomedical domain (as of 4/28/2010) Ontologies in six different formats ~ 1.7 million classes ~ 2 million mappings

5. Evaluating Ontologies in BioPortal Ontology metrics – Computed automatically User reviews – Submitted by users: star ratings and free-text reviews along different dimensions Other metadata – “Stamp of approval” from a specific community

6. Ontology Metrics You can find the details here: http://www.bioontology.org/wiki/index.php/Ontology_Metrics Statistics: – Number of classes – Number of instances – Number of properties – Maximum depth of the class hierarchy – Maximum number of siblings – Average number of siblings Metrics assessing conformance to best practices: – Classes with a single subclass – Classes with a large number of subclasses (more than 20) – Classes with no definition Auditing information: – Contributors and authors of individual classes

7. The Metrics Page in BioPortal You can view metrics for any ontology

8. Peer Review of Ontologies The people who know the answer to these question are – (maybe) ontology authors – other users of the ontology Our solution: Allow users to provide ratings for ontologies Which ontology is appropriate for my task?

9. Reviewers Provide General review and rating Usage information – Which applications have successfully used the ontology? – What problems were encountered? Coverage – Does it cover the domain properly? – Are there major gaps? – Are some parts developed better than others?

10. Projects in BioPortal Users can describe their ontology-based projects in BioPortal Projects are linked to ontologies they are using Ontology reviews can be done in the context of projects

11. Dimensions of a Review Degree of formality Documentation and support Usability Domain coverage Correctness Quality of content

12. Organizational Stamps of Approval BioPortal ontologies belong to different groups: – OBO Foundry members and candidates – WHO Family of International Classifications – UMLS – caBIG – We can add other groups…. Each ontology can be in 0, 1, or more groups We should (but currently don’t) link to the criteria for inclusion in each group

13. Maintaining Metadata through Ontology Versioning We attach reviews to a specific version of an ontology We expose the reviews for all subsequent versions – We plan to add an indicator “This review was submitted for an earlier version of this ontology” Potentially, any part of the description can change

14. Representing Metrics and Reviews Ontology Metadata Vocabulary (OMV): – OMV is a metadata schema that captures salient information about an ontology – OMV is implemented as an OWL ontology BioPortal Metadata ontology extends OMV to add NCBO-specific metadata and evaluation metrics

15. BioPortal Metadata Ontology

16. Discussion We can extend the BioPortal Metadata ontology to represent – Ontology profiles – Additional metrics – Sets of metrics and criteria for a specific organization All information in BioPortal is accessible through REST services – Other applications can use the metadata – For example, there can be an external application that compares different profiles All metadata is specific to an ontology version, but it is accessible through a version-independent ontology id