1. [jws13] Evaluation of instance matching tools: The experience of OAEI
A. Ferrara, A. Nikolov, J. Noessner , F. Scharffe

2. Outlines Introduction State of the art The real-data benchmark
The automatically generated benchmark
Current issues and open problem
Concluding remarks

3. Introduction Abstract: Problem:
The availability of large collections of data
Techniques and tools capable of linking data together
Retrieve potentially useful relations among data
Associate together data representing the same or similar real objects
Problem:
A methodology and a set of benchmarks to understand the quality of results produced by matching process
A framework where tools can be compared with others on the same data

4. Introduction Solution: Goal:
Organize the Instance Matching track of the Ontology Alignments Evaluation Initiative
Goal:
Discover where improvements and new solutions are possible and needed in the matching techniques and tools.

5. Introduction: The instance matching problem
Informal definition:
A special case of the relation discovery which takes two collections of data as input and produce a set of mappings between entities of them as output
High-level formal definition:
D = relational database, I = primary key value
D = RDF graphs, I = URIs, use classification schema

6. Introduction: The instance matching problem
Three main categories in the instance matching:
Value matching: Basic building blocks of data linking tools, identify equivalence between property values of instances (i.e. String similarity metrics, Jaro-Winkler).
Individual matching: Decide whether two individuals represent the same real-world objects, utilize the aggregation of similarities between property values.
Dataset matching: Construct an optimal alignments between whole sets of individuals, rely on results of the individual matching and further refine them, utilize methods like similarity propagation, optimization algorithms, logical reasoning, etc.

7. Introduction: Requirements for the evaluation of instance matching & data linking approaches
Main goal of common evaluation:
Validate different proposed methods
Identify most promising techniques and directions for improvements
Guide further research in the area & developments of robust tools for real-world tasks

8. Introduction: Requirements for the evaluation of instance matching & data linking approaches
Requirements of the evaluation procedure
1st requirement: Representative capabilities of the evaluation approach -- provide useful information about expected performance of techniques & tools, compare different methods and choose the best suited ones. (Benchmark & Criterion)
2nd requirement: Pragmatic --
Requirements are mutually contradictory to some extent, hardly possible to satisfy all to the full extent, aim at a reasonable compromise.

9. Introduction: Requirements for the evaluation of instance matching & data linking approaches
Benchmarking
Comprehensive: Include as many challenges in real-world tasks as possible (e.g. diversity of data formats, attributes & schemas)
Illustrative: Reflect the distribution of different data features similar to the most likely parameters of real-world tasks (i.e. feature rarely occurs should not dominate data)
Criterion
Precision: proportion of correct mappings among the method results
Recall: proportion of correct mappings identified by the tool among all actual mappings
Other

10. Introduction: Instance matching at OAEI
Process
Preparation phase: Provide datasets to be matched & reference alignments.
Execution phase: Use systems to automatically match the instance data from test cases.
Evaluation phase: Standard evaluation measures are precision and recall computed against reference alignments, use weighted harmonic means (weights, size of true positives).

11. State of the art: Evaluation initiatives in the database community
Evaluation test sets
Real-world data sources: Include two or more publicly available datasets originate from different sources but describe the same domain, gold standard created or validated manually after an initial automatic generation
Artificially generated datasets: Created by taking one reference dataset in advance and introduce artificial distortions in a controlled way(e.g. by removing/adding attributes and changing values randomly)

12. State of the art: Evaluation initiatives in the database community
Earlier Stages of research
Databases in the domain of scientific publications
Citation matching
Publicly availability (Cora, ACM-DBLP, etc.)
Advantage of reusing
Possibility to compare with the techniques developed in the database community
Disadvantage
Not fully representative to the challenges of the linked data
Lack of version consistency

13. State of the art: Evaluation initiatives in the database community
Solution: Create benchmarks representing realistic matching challenges & maintain “canonical” versions of benchmark datasets – primary motivations of OAEI
Valid quantitative evaluation measures:
Maximum F-Measure: Harmonic mean between pairwise precision and recall
Pairwise accuracy for the optimal number of pairs
Percentage of the correct equivalence classes
Proportions of true matching pairs at different error rate levels
Precision-recall curves over the whole range of possible threshold values
Conclusion: Precision & Recall measure most informative, F-measure a single quantitative indicator & precision-recall curves a more fine-grained illustration

14. State of the art: Evaluation of ontology matching tools
Development
Originally, a single artificial benchmark
Serve well in checking the capabilities of schema matching tools to handle presence or absence of features in ontologies
Less suited for comparing the overall performance of tools
Extension
Include some realistic benchmarks involving real-world ontologies covering the same topics (particularly Conference & Anatomy)
Conclusion
To achieve effective evaluation of the tools, benchmark tests have to utilize both artificial & real-world datasets

15. State of the art: Evaluation of ontology matching tools
Important differences between tasks not suited to be reused
Larger dataset
Large number of literal data values
Identity & similarity
Different role of names & property values
Different kinds of data heterogeneities
Structural differences between ontology & instances as graphs
Relations between datasets & the real-world
Mutual relations between ontology & instance matching

16. The real-data benchmark
Background
The need to develop a different set of benchmark specially for the instance matching task
Establish the instance matching evaluation as a separate subtrack with the OAEI, has been performed 3 times (in 2009, 2010, & 2011)

17. The automatically generated benchmark (IIMB)
Idea
Automatically acquiring a potentially large set of data from an existing data source & to represent data in form of an OWL Abox, serialized in RDF
Programmatically introduce several kinds of data transformations to produce a final set of Aboxes in a controlled way
Match each of the transformed Aboxes against the initial one to find the correct mappings between them
Advantage: have a control over the type and strength of each transformation

18. Creation of the benchmark
Approach: SWING (Semantic Web Instance Generation)

19. Data acquisition techniques
Operations
Add super classes & super properties
Convert attributes to class assertions
Determine disjointness restrictions
Enrich with inverse properties
Specify domain & range restrictions

20. Data transformation techniques
Test cases
Deletion/addition of individuals
Data value transformation
Data structure transformation
Data semantics transformation

21. Data transformation techniques

22. Data evaluation techniques
Automatically create a ground-truth as a reference alignment for each test case
Reference alignment: mappings between the reference Abox individuals and the corresponding transformed individuals in the test case

23. Current issues & open problems
Future directions for
Larger datasets
Identity & similarity
Different role of names & property values
Different kinds of data heterogeneities
Structural differences between ontology & instance as graphs
Relations between datasets & the real-world
Mutual relations between ontology & instance matching

24. Current issues & open problems

25. Concluding remarks
Approach: base on the idea of combining real-data and automatically & programmatically generated data for the evaluation
Provide a realistic context for instance matching tools
Provide a framework where we can reproduce different causes of data heterogeneity to analytically & programmatically verify the points of strength and weakness of each evaluated tool
Future work
Study of new measures for the evaluation
Improve benchmarks to evaluate the behavior of the instance matching tools with respect to some crucial problems in the field

26. Thank You!