1. September 30, 2002EON 2002Slide 1 Integrating Ontology Storage and Ontology-based Applications A lesson for better evaluation methodology Peter Mika:

2. September 30, 2002EON 2002Slide 2 Contents The context: A case study in ontology- based knowledge management The problem area What did we find? Some observations The solution and the lessons learned Significance for evaluation methodology

3. September 30, 2002EON 2002Slide 3 Context of the development On-To-Knowledge –Contributions include OIL, Methodology for ontology-based KM, Tools for the extraction, editing, sharing, storage and control of ontologies, plus ontology-based reasoning, search, navigation and ontology visualization, Case studies –No single product, only a loose-coupling between tools The EnerSearch case –Virtual organization which disseminates scientific content through its website –Goal: improved access to information through ontology-based search and navigation –Technology and user-focused evaluation of the On-To-Knowledge approach: Does integration through loose coupling really work?

4. September 30, 2002EON 2002Slide 4 Ontology storage –The Sesame RDF storage and query facility Ontology-based applications –Ontology-based portal generation with Spectacle –Ontology-based search engine using QuizRDF The integration Sesame Portal gen. app Spectacle Server Publishing API Quiz RDF Results Query Results

5. September 30, 2002EON 2002Slide 5 Ontology storage architecture Basic operations –Upload, extract, remove, clear –Query, browse One of the functional modules is the RDF inference engine (RDF- MT) Not only a storage facility any more!

6. September 30, 2002EON 2002Slide 6 Observations 1/2 Small, frequent queries result in poor performance Limited expressivity, inefficient evaluation for some queries Transformations are required based on client-side semantics

7. September 30, 2002EON 2002Slide 7 Observations 2/2 Server-side transformations are inefficient: –The larger the repository, the slower it is –Admin., communication costs are significant for small datasets –Transformations weigh heavily on storage functionality

8. September 30, 2002EON 2002Slide 8 The solution Move query/transformation work from server to client where it results in greater efficiency Mechanism for shipping query/transformation knowledge: Portable Inference Modules (PIM) –Wrapper for inference knowledge described in procedural form –Software module with a standardized interface, working on a predefined data model –Portable, self-descriptive, trusted

9. September 30, 2002EON 2002Slide 9 Applications Use in On-To-Knowledge: –Simple queries/transformations on the client-side –Significantly improved performance, allowing applications to scale to the size of a fairly large ontology (~150,000 statements) Future work – Extending Sesame with a PIM container Enrich the expressivity of your ontology language in a plug- and-play fashion! –Compiling PIMs from declarative forms of rules

10. September 30, 2002EON 2002Slide 10 The evaluation methodology lesson What did we see on the previous slides? –An ontology-based system built from loosely coupled components –Performance, scalability issues start to appear when integrating components –Missing pieces of the architecture The message for web services developers: watch out! Lesson for the On-To-Knowledge methodology for technology-focused evaluation: The evaluation of ontology-based tools should be complemented with the evaluation of frameworks or architectures of ontology-based systems.

11. September 30, 2002EON 2002Slide 11 Maybe next year… Evaluation of Ontology-based Frameworks? (EOF)