New York State Center of Excellence in Bioinformatics & Life Sciences R T U New York State Center of Excellence in Bioinformatics & Life Sciences R T U Seminar on the Role of Ontologies in Clinical Medicine Assessment instruments and biomedical reality: examples in the pain domain June 13, 2012 – Ramada Inn, Buffalo NY Werner CEUSTERS, MD Center of Excellence in Bioinformatics and Life Sciences, Ontology Research Group and Department of Psychiatry University at Buffalo, NY, USA

New York State Center of Excellence in Bioinformatics & Life Sciences R T U New York State Center of Excellence in Bioinformatics & Life Sciences R T U Presentation overview Context of the work: OPMQoL Background on assessment instrument How Ontological Realism and Referent Tracking might improve the design and use of assessment instruments 2

New York State Center of Excellence in Bioinformatics & Life Sciences R T U New York State Center of Excellence in Bioinformatics & Life Sciences R T U OPMQoL 3

New York State Center of Excellence in Bioinformatics & Life Sciences R T U New York State Center of Excellence in Bioinformatics & Life Sciences R T U Context of this work OPMQoL: Ontology for Pain-related disablement, Mental health and Quality of Life, Funded by grant 1R01DE A1 from the National Institute of Dental and Craniofacial Research (NIDCR). The content of this presentation is solely the responsibility of the author and does not necessarily represent the official views of the NIDCR or the National Institutes of Health.

New York State Center of Excellence in Bioinformatics & Life Sciences R T U New York State Center of Excellence in Bioinformatics & Life Sciences R T U Werner Ceusters – Richard Ohrbach UB (PIs) Mike T. John – Eric L. Schiffman University of Minnesota Vishar Aggarwal Manchester, UK Joanna Zakrzewska London, UK Thomas List Malmö, Sweden Rafael Benoliel Hadassah, Israel Collaborators

New York State Center of Excellence in Bioinformatics & Life Sciences R T U New York State Center of Excellence in Bioinformatics & Life Sciences R T U Project goals to obtain better insight into: –the complexity of pain disorders, pain types as well as pain-related disablement and –its association with mental health and quality of life, to develop an ontology for this subdomain incorporating a broad array of measures consistent with a biopsychosocial perspective regarding pain, to integrate five existing datasets that broadly encompass the major types of pain in the oral and associated regions. 6

New York State Center of Excellence in Bioinformatics & Life Sciences R T U New York State Center of Excellence in Bioinformatics & Life Sciences R T U Considered datasets ‘US Dataset’ (724 patients) resulted from the NIH funded RDC/TMD Validation Project, ‘Hadassah Dataset’ (306 patients) from the Orofacial Pain Clinic at the Faculty of Dentistry, Hadassah, ‘German Dataset’ (416 patients) of patients seeking treatment for orofacial pain at the Department of Prosthodontics and Materials Sciences, University of Leipzig, ‘Swedish Dataset’of 46 consecutive Atypical Odontalgia (AO) patients recruited from 4 orofacial pain clinics in Sweden as well as data about age- and gender-matched control patients, ‘UK Dataset’ (168 patients) of facial pain of non dental origin present for a minimum of three months. 7

New York State Center of Excellence in Bioinformatics & Life Sciences R T U New York State Center of Excellence in Bioinformatics & Life Sciences R T U Most important challenge The data sets cover more or less the same domain, but … the data within each data set are collected independently from each other, with distinct, partially overlapping data collection and data organization tools. 8

New York State Center of Excellence in Bioinformatics & Life Sciences R T U New York State Center of Excellence in Bioinformatics & Life Sciences R T U Tools used SourcesData generationData organization Data collection sheets Instruction manuals Interpretation criteria Diagnostic criteria Assessment instruments Terminologies Data validation procedures Data dictionaries Ontologies If not used for data collection and organization, these sources can be used post hoc to document, and perhaps increase, the level of data clarity and faithfulness in and comparability of existing data collections.

New York State Center of Excellence in Bioinformatics & Life Sciences R T U New York State Center of Excellence in Bioinformatics & Life Sciences R T U Ontologies and data collections: simplified view Linking the variables of distinct data collections to a realism-based ontology.

New York State Center of Excellence in Bioinformatics & Life Sciences R T U New York State Center of Excellence in Bioinformatics & Life Sciences R T U term concept 1 broader narrower 1..* used in uses 0..* 1..* means expressed- by data collection assessment instrument 0..1 expresses 0..* ontology data dictionary uses used-in 1 1..* uses 1..* 1 reference ontology 1..* used-for 1..* used for 1 uses bridging axiom used-for used for 0..* uses 1..* application ontology Terminology component Data component Ontology component data item representational artifact 1..* data collection ontology assessment instrument ontology 1 uses 0..* terminology 1..* uses 1 used-for entity 1 denotes 0..* denoted by * uses 1..* used for 0..* 1 corresponds-to 1 explained-in 1..* explains uses 1..* used for 0..* denotator Linking data organization tools

New York State Center of Excellence in Bioinformatics & Life Sciences R T U New York State Center of Excellence in Bioinformatics & Life Sciences R T U Linking data collections using distinct assessment instruments

New York State Center of Excellence in Bioinformatics & Life Sciences R T U New York State Center of Excellence in Bioinformatics & Life Sciences R T U Assessment instruments 13

New York State Center of Excellence in Bioinformatics & Life Sciences R T U New York State Center of Excellence in Bioinformatics & Life Sciences R T U McDowell, Ian & Newell, Claire Measuring health a guide to rating scales and questionnaires 14 Free 2006 copy:

New York State Center of Excellence in Bioinformatics & Life Sciences R T U New York State Center of Excellence in Bioinformatics & Life Sciences R T U Assessment instrument design strategy Identify the (human) feature of interest, Select questions answers to which provide a means to quantify the presence in a subject, Design a scoring algorithm, Identify baselines, Assess the quality of the instrument towards its design objectives. 15

New York State Center of Excellence in Bioinformatics & Life Sciences R T U New York State Center of Excellence in Bioinformatics & Life Sciences R T U Example: Perceived Stress Scale (PSS) 16 Cohen, S., Kamarck, T., and Mermelstein, R. (1983). A global measure of perceived stress. Journal of Health and Social Behavior, 24,

New York State Center of Excellence in Bioinformatics & Life Sciences R T U New York State Center of Excellence in Bioinformatics & Life Sciences R T U Perceived Stress Scale: score calculation 1.In the last month, how often have you been upset because of something that happened unexpectedly? In the last month, how often have you felt that you were unable to control the important things in your life? In the last month, how often have you felt nervous and “stressed”? In the last month, how often have you felt confident about your ability to handle your personal problems? In the last month, how often have you felt that things were going your way? In the last month, how often have you found that you could not cope with all the things that you had to do? In the last month, how often have you been able to control irritations in your life? In the last month, how often have you felt that you were on top of things? In the last month, how often have you been angered because of things that were outside of your control? In the last month, how often have you felt difficulties were piling up so high that you could not overcome them? ∑ Cohen, S., Kamarck, T., and Mermelstein, R. (1983). A global measure of perceived stress. Journal of Health and Social Behavior, 24,

New York State Center of Excellence in Bioinformatics & Life Sciences R T U New York State Center of Excellence in Bioinformatics & Life Sciences R T U Perceived Stress Scale: norm table 18 Cohen, S. and Williamson, G. Perceived Stress in a Probability Sample of the United States. Spacapan, S. and Oskamp, S. (Eds.) The Social Psychology of Health. Newbury Park, CA: Sage, 1988.

New York State Center of Excellence in Bioinformatics & Life Sciences R T U New York State Center of Excellence in Bioinformatics & Life Sciences R T U Quality of instruments Validity: –how well does the instrument measure what it is intended to measure?  property of the instrument –what can one conclude given a certain score  property of the interpretation Reliability (consistency): –how well reproduces the instrument the same results? 19

New York State Center of Excellence in Bioinformatics & Life Sciences R T U New York State Center of Excellence in Bioinformatics & Life Sciences R T U Validity versus reliability This instrument measures very reliably but has poor validity; However, it might be measuring something else ! 20 Assessing validity – especially in absence of a gold standard – is a difficult and error-prone procedure driven by phenomenological and statistical considerations.

New York State Center of Excellence in Bioinformatics & Life Sciences R T U New York State Center of Excellence in Bioinformatics & Life Sciences R T U Construct validity ‘For variables such as pain, quality of life, or happiness, gold standards do not exist and thus validity testing is more challenging. For such abstract constructs, validation of a measurement involves a series of steps known as “construct validation.” This begins with a conceptual definition of the topic (or construct) to be measured, indicating the internal structure of its components and the way it relates to other constructs. These may be expressed as hypotheses indicating, for example, what correlations should be obtained between a quality of life scale and a measure of depression, or which respondents should score higher or lower on quality of life. None of these challenges alone proves validity and each suffers logical and practical limitations, although when systematically applied, they build a composite picture of the adequacy of the measurement.’ 21 Ian McDowell. Measuring Health: A Guide to Rating Scales and Questionnaires, Third Edition. Oxford University Press 2006; p34

New York State Center of Excellence in Bioinformatics & Life Sciences R T U New York State Center of Excellence in Bioinformatics & Life Sciences R T U Evidence for validity of PSS Higher PSS scores were associated with (for example): –failure to quit smoking, –failure among diabetics to control blood sugar levels, –greater vulnerability to stressful life-event-elicited depressive symptoms, –more colds. 22

New York State Center of Excellence in Bioinformatics & Life Sciences R T U New York State Center of Excellence in Bioinformatics & Life Sciences R T U Epistemic value commitments for constructs ‘values involved in making and advancing epistemologically-relevant claims, such as scientific ones’: Coherence Consistency Comprehensiveness Fecundity Simplicity Instrumental efficacy Originality Relevance Precision JZ. Sadler. Epistemic Value Commitments in the Debate over Categorical vs. Dimensional Personality Diagnosis. Philosophy, Psychiatry, & Psychology 3.3 (1996)

New York State Center of Excellence in Bioinformatics & Life Sciences R T U New York State Center of Excellence in Bioinformatics & Life Sciences R T U Many caveats f.i.: way questions are phrased creates bias or confusion; e.g.: “In the last month, how often have you been upset because of something that happened unexpectedly? ” –‘0’ can mean: nothing unexpectedly happen there were unexpected events but none caused being upset 24

New York State Center of Excellence in Bioinformatics & Life Sciences R T U New York State Center of Excellence in Bioinformatics & Life Sciences R T U Construct validity – additional notes ‘For variables such as pain, quality of life, or happiness, gold standards do not exist and thus validity testing is more challenging. For such abstract constructs, validation of a measurement involves a series of steps known as “construct validation.” This begins with a conceptual definition of the topic (or construct) to be measured, indicating the internal structure of its components and the way it relates to other constructs. These may be expressed as hypotheses indicating, for example, what correlations should be obtained between a quality of life scale and a measure of depression, or which respondents should score higher or lower on quality of life. None of these challenges alone proves validity and each suffers logical and practical limitations, although when systematically applied, they build a composite picture of the adequacy of the measurement.’ 25 Ian McDowell. Measuring Health: A Guide to Rating Scales and Questionnaires, Third Edition. Oxford University Press 2006; p34

New York State Center of Excellence in Bioinformatics & Life Sciences R T U New York State Center of Excellence in Bioinformatics & Life Sciences R T U Hypothesis 26 networks of constructs represent correlations in reality

New York State Center of Excellence in Bioinformatics & Life Sciences R T U New York State Center of Excellence in Bioinformatics & Life Sciences R T U But: are we discovering or inventing disorders? 27

New York State Center of Excellence in Bioinformatics & Life Sciences R T U New York State Center of Excellence in Bioinformatics & Life Sciences R T U Real correspondence to entities in reality? 28 denotes?

New York State Center of Excellence in Bioinformatics & Life Sciences R T U New York State Center of Excellence in Bioinformatics & Life Sciences R T U Hypothesis Ontological Realism and Referent Tracking hold – once again – a key to a solution 29

New York State Center of Excellence in Bioinformatics & Life Sciences R T U New York State Center of Excellence in Bioinformatics & Life Sciences R T U Methodology 30

New York State Center of Excellence in Bioinformatics & Life Sciences R T U New York State Center of Excellence in Bioinformatics & Life Sciences R T U Remember: Assessment instrument design Identify the (human) feature of interest, Select questions answers to which provide a means to quantify the presence in a subject, Design a scoring algorithm, Identify baselines, Assess the quality of the instrument towards its design objectives. 31

New York State Center of Excellence in Bioinformatics & Life Sciences R T U New York State Center of Excellence in Bioinformatics & Life Sciences R T U Starting point: IASP definition for ‘pain’ ‘an unpleasant sensory and emotional experience associated with actual or potential tissue damage, or described in terms of such damage’; what asserts: –a common phenomenology (‘unpleasant sensory and emotional experience’) to all instances of pain, –the recognition of three distinct subtypes of pain involving, respectively: 1.actual tissue damage, 2.what is called ‘potential tissue damage’, and 3.a description involving reference to tissue damage. 32

New York State Center of Excellence in Bioinformatics & Life Sciences R T U New York State Center of Excellence in Bioinformatics & Life Sciences R T U Thus rather: five pain-related phenomena! 33 Smith B, Ceusters W, Goldberg LJ, Ohrbach R. Towards an Ontology of Pain. In: Mitsu Okada (ed.), Proceedings of the Conference on Logic and Ontology, Tokyo: Keio University Press, February 2011:23-32.

New York State Center of Excellence in Bioinformatics & Life Sciences R T U New York State Center of Excellence in Bioinformatics & Life Sciences R T U Linking the instruments and other tools analyze data dictionaries, assessment instruments, study criteria and corresponding terminologies, build realism-based application ontologies to link these sources to realism-based reference ontologies.

New York State Center of Excellence in Bioinformatics & Life Sciences R T U New York State Center of Excellence in Bioinformatics & Life Sciences R T U Example: assessing TMJ Anatomy

New York State Center of Excellence in Bioinformatics & Life Sciences R T U New York State Center of Excellence in Bioinformatics & Life Sciences R T U Panoramic X-ray of mouth

New York State Center of Excellence in Bioinformatics & Life Sciences R T U New York State Center of Excellence in Bioinformatics & Life Sciences R T U Radiology RDC/TMD Examination: data collection sheet

New York State Center of Excellence in Bioinformatics & Life Sciences R T U New York State Center of Excellence in Bioinformatics & Life Sciences R T U RDC/TMD: a collaborator’s data dictionary Fieldnames in that collaborator’s data collection Allowed values for the fields

New York State Center of Excellence in Bioinformatics & Life Sciences R T U New York State Center of Excellence in Bioinformatics & Life Sciences R T U Anybody sees something disturbing ?

New York State Center of Excellence in Bioinformatics & Life Sciences R T U New York State Center of Excellence in Bioinformatics & Life Sciences R T U This data dictionary alone is not reliable! That these variables are about the condylar head of the TMJ is ‘lost in translation’!

New York State Center of Excellence in Bioinformatics & Life Sciences R T U New York State Center of Excellence in Bioinformatics & Life Sciences R T U ‘meaning’ of values in data collections ‘The patient with patient identifier ‘PtID4’ is stated to have had a panoramic X-ray of the mouth which is interpreted to show subcortical sclerosis of that patient’s condylar head of the right temporomandibular joint’ 1 meaning

New York State Center of Excellence in Bioinformatics & Life Sciences R T U New York State Center of Excellence in Bioinformatics & Life Sciences R T U Objectives of the ‘sources’ analysis Find for each value V in the data collections all possible configurations of entities (according to our best scientific understanding) for which the following can be true: – V –‘it is stated that V’ Describe these possible configurations by means of sentences from a formal language that mimic the structure of reality.

New York State Center of Excellence in Bioinformatics & Life Sciences R T U New York State Center of Excellence in Bioinformatics & Life Sciences R T U Objectives of the ‘sources’ analysis (2) For example, –for the value stating that ‘The patient with patient identifier ‘PtID4’ has had a panoramic X-ray of the mouth which is interpreted to show subcortical sclerosis of that patient’s condylar head of the right temporomandibular joint’ to be true, –this statement must have been made, –for the statement to be true, there must have been that patient, an X-ray, etc, … –BUT! It is not necessarily true that that patient has indeed the sclerosis as diagnosed.

New York State Center of Excellence in Bioinformatics & Life Sciences R T U New York State Center of Excellence in Bioinformatics & Life Sciences R T U Methodology 1.Formulate for each variable in the data collection a sentence explaining as accurately as possible what the variable stands for, 2.list the entities in reality that the terms in the sentence denote, 3.list recursively for all entities listed further entities that ontologically must exist for the entity under scrutiny to exist, 4.classify all entities in terms of realism-based ontologies (RBO), 5.specify all obtaining relationships between these entities, 6.outline all possible configurations of such entities for the sentence to be true.

New York State Center of Excellence in Bioinformatics & Life Sciences R T U New York State Center of Excellence in Bioinformatics & Life Sciences R T U etiological processdisorderdiseasepathological process abnormal bodily featuressigns & symptomsinterpretive processdiagnosis producesbearsrealized_in producesparticipates_inrecognized_as produces RBO (1): Ontology of General Medical Science Scheuermann R, Ceusters W, Smith B. Toward an Ontological Treatment of Disease and Diagnosis AMIA Summit on Translational Bioinformatics, San Francisco, California, March 15-17, 2009;:

New York State Center of Excellence in Bioinformatics & Life Sciences R T U New York State Center of Excellence in Bioinformatics & Life Sciences R T U No conflation of diagnosis, disease, and disorder The disorder is thereThe diagnosis is here The disease is there

New York State Center of Excellence in Bioinformatics & Life Sciences R T U New York State Center of Excellence in Bioinformatics & Life Sciences R T U RBO (2): (cleaned up) Ontology of Biomedical Investigations

New York State Center of Excellence in Bioinformatics & Life Sciences R T U New York State Center of Excellence in Bioinformatics & Life Sciences R T U Step 1: formulate a statement ‘The patient with patient identifier ‘PtID4’ is stated to have had a panoramic X-ray of the mouth which is interpreted to show subcortical sclerosis of that patient’s condylar head of the right temporomandibular joint’ 1 meaning

New York State Center of Excellence in Bioinformatics & Life Sciences R T U New York State Center of Excellence in Bioinformatics & Life Sciences R T U Step 2 (1): list the entities denoted 1(The patient) with 2(patient identifier ‘PtID4’) 3(is stated) 4(have had) a 5(panoramic X-ray) of 6(the mouth) which 7(is interpreted) to 8(show) 9(subcortical sclerosis of 10(that patient’s condylar head of the 11(right temporomandibular joint)))’ CLASSINSTANCE IDENTIFIER personIUI-1 patient identifierIUI-2 assertionIUI-3 technically investigatingIUI-4 panoramic X-rayIUI-5 mouthIUI-6 interpretingIUI-7 seeingIUI-8 diagnosisIUI-9 condylar head of right TMJIUI-10 right TMJIUI-11 notes: colors have no meaning here, just provide easy reference, this first list can be different, any such differences being resolved in step 3

New York State Center of Excellence in Bioinformatics & Life Sciences R T U New York State Center of Excellence in Bioinformatics & Life Sciences R T U Step 2 (2): provide directly referential descriptions CLASS INSTANCE IDENTIFIERDIRECTLY REFERENTIAL DESCRIPTIONS personIUI-1 the person to whom IUI-2 is assigned patient identifierIUI-2 the patient identifier of IUI-1 assertionIUI-3 'the patient with patient identifier PtID4 has had a panoramic X-ray of the mouth which is interpreted to show subcortical sclerosis of that patient’s right temporomandibular joint' technically investigatingIUI-4the technically investigating of IUI-6 panoramic X-rayIUI-5the panoramic X-ray that resulted from IUI-4 mouthIUI-6the mouth of IUI-1 interpretingIUI-7the interpreting of the signs exhibited by IUI-5 seeingIUI-8the seeing of IUI-5 which led to IUI-7 diagnosisIUI-9the diagnosis expressed by means of IUI-3 condylar head of right TMJIUI-10the condylar head of the right TMJ of IUI-1 right TMJIUI-11the right TMJ of IUI-1

New York State Center of Excellence in Bioinformatics & Life Sciences R T U New York State Center of Excellence in Bioinformatics & Life Sciences R T U Step 3: identify further entities that ontologically must exist for each entity under scrutiny to exist. assigner roleIUI-12the assigner role played by the entity while it performed IUI-21 assigningIUI-21the assigning of IUI-2 to IUI-1 by the entity with role IUI-12 assertingIUI-20the asserting of IUI-3 by the entity with asserter role IUI-13 asserter roleIUI-13the asserter role played by the entity while it performed IUI-20 investigator roleIUI-14the investigator role played by the entity while it performed IUI-4 panoramic X-ray machine IUI-15the panoramic X-ray machine used for performing IUI-4 image bearerIUI-16the image bearer in which IUI-5 is concretized and that participated in IUI-8 interpreter roleIUI-17the interpreter role played by the entity while it performed IUI-7 perceptor roleIUI-18the perceptor role played by the entity while it performed IUI-8 diagnostic criteriaIUI-19the diagnostic criteria used by the entity that performed IUI-7 to come to IUI-9 study subject roleIUI-22the study subject role which inheres in IUI-1

New York State Center of Excellence in Bioinformatics & Life Sciences R T U New York State Center of Excellence in Bioinformatics & Life Sciences R T U Step 3: some remarks interpreter role, perceptor role, … –reference to roles rather than the entity in which the roles inhere because it may be the same entity and one should not assign several IUIs to the same entity each description follows similar principles as Aristotelian definitions but is about particulars rather than universals

New York State Center of Excellence in Bioinformatics & Life Sciences R T U New York State Center of Excellence in Bioinformatics & Life Sciences R T U Step 4: classify all entities in terms of realism-based ontologies requires more ontological and philosophical skills than domain expertise or expertise with Protégé, not just term matching CLASSHIGHER CLASS personBFO: Object patient identifierIAO: Information Content Entity assertionIAO: Information Content Entity technically investigating OBI: Assay panoramic X-rayIAO: Image mouthFMA: Mouth interpretingMFO: Assessing seeingBFO: Process diagnosisIAO: Information Content Entity condylar head of right TMJ FMA: Right condylar process of mandible right TMJFMA: Right temporomandibular joint assigner roleBFO: Role assigningBFO: Process study subject roleOBI: Study subject role

New York State Center of Excellence in Bioinformatics & Life Sciences R T U New York State Center of Excellence in Bioinformatics & Life Sciences R T U Step 5: specify relationships between these entities For instance: –at least during the taking of the X-ray the study subject role inheres in the patient being investigated: IUI-23 inheres-in IUI-1 during t1 –the patient participates at that time in the investigation IUI-4 has-participant IUI-1 during t1 These relations need to follow the principles of the Relation Ontology. Smith B, Ceusters W, Klagges B, Koehler J, Kumar A, Lomax J, Mungall C, Neuhaus F, Rector A, Rosse C. Relations in biomedical ontologies, Genome Biology 2005, 6:R46. Relations in biomedical ontologies

New York State Center of Excellence in Bioinformatics & Life Sciences R T U New York State Center of Excellence in Bioinformatics & Life Sciences R T U Step 6: outline all possible configurations of such entities for the sentence to be true (a one semester course on its own) Such outlines are collections of relational expressions of the sort just described, Variant configurations for the example: –perceptor and interpreter are the same or distinct human beings, –the X-ray machine is unreliable and produced artifacts which the interpreter thought to be signs motivating his diagnosis, while the patient has indeed the disorder specified by the diagnosis (the clinician was lucky) –…

New York State Center of Excellence in Bioinformatics & Life Sciences R T U New York State Center of Excellence in Bioinformatics & Life Sciences R T U Conclusion Realism-based ontology has a lot to offer to make data collections comparable and unambiguously understandable. It is hard ! How far one needs to go depends on the purposes. –ideally: an analysis should be such that it can accommodate ALL purposes, i.e. the analysis should be independent of any purpose; distinction between reference ontologies and application ontologies.