1. e-mail: david.shotton @ zoo.ox.ac.uk © David Shotton, 2007 David Shotton Image BioInformatics Research Group Oxford e-Research Centre and Department of Zoology University of Oxford, UK UK Electronic Information Group Image Management in Bio- and Environmental Sciences: New Directions John Rylands Library, University of Manchester Thursday 31st May 2007 Image semantics

2. The nature of images Images capture aspects of the real world, and form a vital part of the scholarly record for which words are no substitute Images acquisition is often costly and time consuming The storage requirement for digital images is large, particularly if they are multidimensional (e.g. videos or 3D spatial images)

3. The problem with images Despite popular misconception, the central problem with images is not their size or dimensional complexity It is that, unlike text documents, images are not self-describing  While images may be readily interpretable by humans, they are not readily amenable to automatic interpretation or indexing by present technologies Since their internal semantics are not easily extractable, descriptive metadata annotations are usually required to bridge this ‘semantic gap’ Without such metadata, on-line digital image repositories face the risk of becoming little more than meaningless and costly data graveyards

4. How to describe and classify things... “On those remote pages it is written that animals are divided into: a. those that belong to the Emperor b. embalmed ones c. those that are trained d. suckling pigs e. mermaids f. fabulous ones g. stray dogs h. those that are included in this classification i. those that tremble as if they were mad j. innumerable ones k. those drawn with a very fine camel's hair brush l. others m. those that have just broken a flower vase n. those that resemble flies from a distance" From The Celestial Emporium of Benevolent Knowledge, Jorge Luis Borges

5. Structuring metadata Free text tagging, as in the previous example A controlled vocabulary (a word list with no internal structure) A hierarchical taxonomy of parent-offspring ‘is_a’ relationships  e.g. a crow is a bird, a bird is a vertebrate A thesaurus, in which additional relationships between terms may be defined An ontology, in which such relationships are, ideally, defined in such a manner as to permit computers to make semantic inferences and undertake logical reasoning over the data A helpful definition of an ontology has been given by Tom Gruber as  The formal explicit specification of a shared conceptualisation The role of an ontology is thus to facilitate the formal sharing and re-use of knowledge through the construction of an explicit domain model

6. Animal is_a Vertebrate is_a Mammal is_a Rodent is_a Mouse An ontology is richer than a taxonomic hierarchy Here all the relationships are of a single type, that of being a sub-class, where each sub-class has only one ‘parent’. Phylogenetic trees are typical constructs using this relationship Hierarchies have the advantage that each sub-class (e.g. rodent) inherits all class properties previously defined for its parent class (e.g. mammal), such as the possession of four legs and fur – this is called subsumption However, in an ontology one can express more complex relationships about a mouse, other than just its taxonomy

7. Group of Mus musculus Rodent organisms is_a is_a Colony has_species_name member_of Mouse proper_part_of has_ID Leg has_mode_ 667 (has_cardinality: 4) of_locomotion (has_position: front / rear) (has_handedness: left / right) (has_length: number unit) used_for Locomotion type proper_ Running is_a part_of Fur hypothesised_ (default_colour: white) function (mean_length: number unit) (mean_density: number per unit area) Escape A partial ontology of ‘mouse’ Ontologies that permit only very few relationship types are limited in their expressiveness... but easier to share This is a directed acylic graph with many relationship types

8. The Semantic Web Tim Berners-Lee’s vision of “the Web of integrated data” The Semantic Web extends the web by providing a data representation that has both syntactic consistency and a semantic framework, enabling both interoperability and computational inferencing It involves three technologies, each resting hierarchically on the previous one:  The eXtenstible Markup Language (XML) that permits one to define the meaning of terms using XML tags, with XML Schema providing syntactical structure  The Resource Description Framework (RDF) that permits one to make simple logical statements (subject-verb-object, or entity-attribute- value) written in XML, for describing objects and the relationships between them, with RDF Schema providing semantic structure  The Web Ontology Language OWL, itself expressed as a set of RDF / RDFS statements, to specify the supporting ontologies that provide semantic definitions of the RDF terms

9. How to make ontological statements using RDF An RDF triple might state that a mouse is_a mammal, informing the computer that an entity ‘ mouse ’ is included in the more general category of ‘ mammal ’ By using several RDF entity-attribute-value triples referring to the same entity, multiple attributes can be defined: Subject (Entity) = Mouse (class)or This mouse (instance) Property (Attribute) = is_a / has_location / has_identifier Object (Value) = Mammal / Oxford / 667 In RDF, the statement “ This mouse is located in Oxford ” is simply: Oxford

10. How to build an ontology Relationships in an ontology take the form of a directed acyclic graph (DAG), in which an entry can have more than one ‘parent’ An OWL ontology can conveniently be written in RDF, the subject-verb- predicate of an RDF triple equating to an single node-link-node in the DAG Tools such as Protégé-OWL make the task of ontology building much easier: Part of the ImageStore Ontology of the BioImage Database, visualized in Protégé-OWL

11. Just how big is the ‘semantic gap’? To what extent is it now possible for computers to identify objects within images by direct inspection of the pixel information? The results I am about to show you are from two state-of-the-art automated methods for  object detection  semantic segmentation Independently they produce good results, and in combination they are remarkable Credits: Jamie Shotton (2007) Contour and Texture for Visual Recognition of Object Categories. Ph. D. Thesis, University of Cambridge

12. Object detection using contour fragments These results are obtained using the first method, based upon contour fragments, used here to detect the presence of horses in images The algorithm has been ‘educated’ using a set of training images, and has then been let loose on these and other test images, which it has analysed automatically On the left of each pair, the green boxes surround the detected horses, while on the right the contour fragments used in the detection are shown

13. This method works well on a variety of objects It gives few false positives and few false negatives, with almost perfect results for motorbikes and cows! However, it does require training, and has not yet been tested on biological research images

14. Automatic image segmentation using texture The second method combines texture, colour, shape and context It learns from a set of 591 training images pre-labelled for 21 object classes

15. Results of the ‘texture’ method Results of the ‘texture’ method for the semantic segmentation of test images

16. ....but the method is not perfect As Jamie says in his conclusion, concerning the capabilities of machine vision: “While we are still a considerable way from accurately recognizing the tens of thousands of classes that humans effortlessly distinguish, despite incredible variations in appearance, we believe that this thesis has taken a positive step towards a solution” So the semantic gap between the capabilities of machine vision and the necessity for human metadata annotation is perhaps not as wide as I made out initially!

17. The end