[jws13] Evaluation of instance matching tools: The experience of OAEI A. Ferrara, A. Nikolov, J. Noessner , F. Scharffe

Outlines Introduction State of the art The real-data benchmark The automatically generated benchmark Current issues and open problem Concluding remarks

Introduction Abstract: Problem: The availability of large collections of data Techniques and tools capable of linking data together Retrieve potentially useful relations among data Associate together data representing the same or similar real objects Problem: A methodology and a set of benchmarks to understand the quality of results produced by matching process A framework where tools can be compared with others on the same data

Introduction Solution: Goal: Organize the Instance Matching track of the Ontology Alignments Evaluation Initiative (IM@OAEI) Goal: Discover where improvements and new solutions are possible and needed in the matching techniques and tools.

Introduction: The instance matching problem Informal definition: A special case of the relation discovery which takes two collections of data as input and produce a set of mappings between entities of them as output High-level formal definition: D = relational database, I = primary key value D = RDF graphs, I = URIs, use classification schema

Introduction: The instance matching problem Three main categories in the instance matching: Value matching: Basic building blocks of data linking tools, identify equivalence between property values of instances (i.e. String similarity metrics, Jaro-Winkler). Individual matching: Decide whether two individuals represent the same real-world objects, utilize the aggregation of similarities between property values. Dataset matching: Construct an optimal alignments between whole sets of individuals, rely on results of the individual matching and further refine them, utilize methods like similarity propagation, optimization algorithms, logical reasoning, etc.

Introduction: Requirements for the evaluation of instance matching & data linking approaches Main goal of common evaluation: Validate different proposed methods Identify most promising techniques and directions for improvements Guide further research in the area & developments of robust tools for real-world tasks

Introduction: Requirements for the evaluation of instance matching & data linking approaches Requirements of the evaluation procedure 1st requirement: Representative capabilities of the evaluation approach -- provide useful information about expected performance of techniques & tools, compare different methods and choose the best suited ones. (Benchmark & Criterion) 2nd requirement: Pragmatic -- Requirements are mutually contradictory to some extent, hardly possible to satisfy all to the full extent, aim at a reasonable compromise.

Introduction: Requirements for the evaluation of instance matching & data linking approaches Benchmarking Comprehensive: Include as many challenges in real-world tasks as possible (e.g. diversity of data formats, attributes & schemas) Illustrative: Reflect the distribution of different data features similar to the most likely parameters of real-world tasks (i.e. feature rarely occurs should not dominate data) Criterion Precision: proportion of correct mappings among the method results Recall: proportion of correct mappings identified by the tool among all actual mappings Other

Introduction: Instance matching at OAEI Process Preparation phase: Provide datasets to be matched & reference alignments. Execution phase: Use systems to automatically match the instance data from test cases. Evaluation phase: Standard evaluation measures are precision and recall computed against reference alignments, use weighted harmonic means (weights, size of true positives).

State of the art: Evaluation initiatives in the database community Evaluation test sets Real-world data sources: Include two or more publicly available datasets originate from different sources but describe the same domain, gold standard created or validated manually after an initial automatic generation Artificially generated datasets: Created by taking one reference dataset in advance and introduce artificial distortions in a controlled way(e.g. by removing/adding attributes and changing values randomly)

State of the art: Evaluation initiatives in the database community Earlier Stages of research Databases in the domain of scientific publications Citation matching Publicly availability (Cora, ACM-DBLP, etc.) Advantage of reusing Possibility to compare with the techniques developed in the database community Disadvantage Not fully representative to the challenges of the linked data Lack of version consistency

State of the art: Evaluation initiatives in the database community Solution: Create benchmarks representing realistic matching challenges & maintain “canonical” versions of benchmark datasets – primary motivations of OAEI Valid quantitative evaluation measures: Maximum F-Measure: Harmonic mean between pairwise precision and recall Pairwise accuracy for the optimal number of pairs Percentage of the correct equivalence classes Proportions of true matching pairs at different error rate levels Precision-recall curves over the whole range of possible threshold values Conclusion: Precision & Recall measure most informative, F-measure a single quantitative indicator & precision-recall curves a more fine-grained illustration

State of the art: Evaluation of ontology matching tools Development Originally, a single artificial benchmark Serve well in checking the capabilities of schema matching tools to handle presence or absence of features in ontologies Less suited for comparing the overall performance of tools Extension Include some realistic benchmarks involving real-world ontologies covering the same topics (particularly Conference & Anatomy) Conclusion To achieve effective evaluation of the tools, benchmark tests have to utilize both artificial & real-world datasets

State of the art: Evaluation of ontology matching tools Important differences between tasks not suited to be reused Larger dataset Large number of literal data values Identity & similarity Different role of names & property values Different kinds of data heterogeneities Structural differences between ontology & instances as graphs Relations between datasets & the real-world Mutual relations between ontology & instance matching

The real-data benchmark Background The need to develop a different set of benchmark specially for the instance matching task Establish the instance matching evaluation as a separate subtrack with the OAEI, has been performed 3 times (in 2009, 2010, & 2011)

The automatically generated benchmark (IIMB) Idea Automatically acquiring a potentially large set of data from an existing data source & to represent data in form of an OWL Abox, serialized in RDF Programmatically introduce several kinds of data transformations to produce a final set of Aboxes in a controlled way Match each of the transformed Aboxes against the initial one to find the correct mappings between them Advantage: have a control over the type and strength of each transformation

Creation of the benchmark Approach: SWING (Semantic Web Instance Generation)

Data acquisition techniques Operations Add super classes & super properties Convert attributes to class assertions Determine disjointness restrictions Enrich with inverse properties Specify domain & range restrictions

Data transformation techniques Test cases Deletion/addition of individuals Data value transformation Data structure transformation Data semantics transformation

Data transformation techniques

Data evaluation techniques Automatically create a ground-truth as a reference alignment for each test case Reference alignment: mappings between the reference Abox individuals and the corresponding transformed individuals in the test case

Current issues & open problems Future directions for IM@OAEI Larger datasets Identity & similarity Different role of names & property values Different kinds of data heterogeneities Structural differences between ontology & instance as graphs Relations between datasets & the real-world Mutual relations between ontology & instance matching

Current issues & open problems

Concluding remarks Approach: base on the idea of combining real-data and automatically & programmatically generated data for the evaluation Provide a realistic context for instance matching tools Provide a framework where we can reproduce different causes of data heterogeneity to analytically & programmatically verify the points of strength and weakness of each evaluated tool Future work Study of new measures for the evaluation Improve benchmarks to evaluate the behavior of the instance matching tools with respect to some crucial problems in the field

Thank You!