Identifying sets and classes: taxonomies as finding aids Alex Haig NHS Education for Scotland 29 th September 2005

A Case Study: medical education

What is a Taxonomy? From the Greek taxis and nomos, (division and law) From the Greek taxis and nomos, (division and law) Division into ordered groups or categories Division into ordered groups or categories Taxonomic schemas can be developed to order almost anything Taxonomic schemas can be developed to order almost anything

Carl Linnaeus ( ) Swedish botanist Swedish botanist 12 volume Systema Naturae 12 volume Systema Naturae Domain; Kingdom; Phylum (animals) or Division (plants); Class; Order; Family; Genus; Species Domain; Kingdom; Phylum (animals) or Division (plants); Class; Order; Family; Genus; Species

What is a Taxonomy? Retain characteristics of classification, but not always based on standards used in library settings Retain characteristics of classification, but not always based on standards used in library settings Taxonomies and vocabularies are structured collections of terms that can serve as values for the meta-data elements. IMS Taxonomies and vocabularies are structured collections of terms that can serve as values for the meta-data elements. IMS

Ironically Ambiguous … Controlled Vocabulary: (usually) enumerative list of all given terms/values in a subject area Controlled Vocabulary: (usually) enumerative list of all given terms/values in a subject area Taxonomy: top-down hierarchical arrangement that does not necessarily define components Taxonomy: top-down hierarchical arrangement that does not necessarily define components Thesaurus: defines components as well as associative relationships; bound by international standards Thesaurus: defines components as well as associative relationships; bound by international standards Ontology: conceptual relationships (self-evident) Ontology: conceptual relationships (self-evident)

Controlled Vocabulary (usually) enumerative list of all given terms/values in a subject area (usually) enumerative list of all given terms/values in a subject area Arbovirus Infections Arbovirus Infections Bronchiolitis Bronchiolitis Encephalitis Encephalitis Eye Infections Eye Infections Fatigue Syndrome Fatigue Syndrome Hepatitis Hepatitis Meningitis Meningitis Pneumonia Pneumonia RNA Virus Infections RNA Virus Infections Sexually Transmitted Diseases Sexually Transmitted Diseases Skin Diseases Skin Diseases Tumor Virus Infections Tumor Virus Infections

Taxonomy top-down hierarchical arrangement that does not necessarily define components top-down hierarchical arrangement that does not necessarily define components Viral Diseases Viral Diseases Hepatitis Hepatitis Hepatitis A Hepatitis A Hepatitis B Hepatitis B Hepatitis C Hepatitis C Chronic Hepatitis C Chronic Hepatitis C Hepatitis D Hepatitis D Hepatitis E Hepatitis E

Thesaurus Chronic Fatigue Syndrome: A syndrome characterized by persistent or recurrent fatigue, diffuse musculoskeletal pain, sleep disturbances, and subjective cognitive impairment of 6 months duration or longer. Symptoms are not caused by ongoing exertion; are not relieved by rest; and result in a substantial reduction of previous levels of occupational, educational, social, or personal activities Chronic Fatigue Syndrome: A syndrome characterized by persistent or recurrent fatigue, diffuse musculoskeletal pain, sleep disturbances, and subjective cognitive impairment of 6 months duration or longer. Symptoms are not caused by ongoing exertion; are not relieved by rest; and result in a substantial reduction of previous levels of occupational, educational, social, or personal activities See Related: FIBROMYALGIA Used For: chronic fatigue syndrome encephalomyelitis, myalgic infectious mononucleosis-like syndrome, chronic postviral fatigue syndrome chronic fatigue disorder chronic fatigue and immune dysfunction syndrome chronic fatigue-fibromyalgia syndrome fatigue syndrome, postviral myalgic encephalomyelitis royal free disease

Ontology Conceptual relationships (self-evident) Conceptual relationships (self-evident) Much more powerful way of describing an entire domain in a variety of methods Much more powerful way of describing an entire domain in a variety of methods Metaphysical origins with the nature and relations of being Metaphysical origins with the nature and relations of being Viral diseases by: aetiology (cause), prognosis, diagnosis, protein regulation, affect Viral diseases by: aetiology (cause), prognosis, diagnosis, protein regulation, affect

Why Use or Create a Taxonomy?

I can call spirits from the vast deep. Why, so can I, or so can any man; but will they come when you do call for them? Shakespeare. Henry VI Part 1 3 i

Where are taxonomies used?

A Case Study: medical education (discipline wide effort)

Best Evidence Medical Education the dissemination of information which allows medical teachers, institutions and all concerned with medical education to make decisions on the basis of the best evidence available the production of appropriate systematic reviews of medical education which reflect the best evidence available and meet the needs of the user, the creation of a culture of best evidence medical education amongst individual teachers, institutions and national bodies.

Searching for evidence in medical education … … the need for a taxonomy … Get a Measure of the Problem.

Association for Medical Education in Europe Evidence retrieval in medical education: obstructions and opportunities. Berlin, 2001.

Methods - topic BEME pilot and consequent review groups (Barcelona/Tel Aviv) Feedback in Assessment

Methods - software software used - Ovid [CGI version 7.8] –permits design of rigorous strategies –consistency

Methods - databases selected Medline Embase ERIC most relevant to medical education

Methods - journal selected limited time and resources required a title that was most comprehensively indexed Academic Medicine present (2001)

Methods - strategies Three levels of strategy: I. standard (most users; limited search syntax) II. enhanced (some use of search syntax) III. expert (full use of search syntax) syntax includes: free-text, controlled vocabulary, term explosions, phrase lists, subheadings, sub strings, filters, proximity operators, etc...

Methods - handsearching...refers to the planned searching of a journal page by page (i.e. by hand), including editorials, letters, etc., to identify all relevant items. time consuming and meticulous produces the gold standard by which search efficiency can be measured

Sensitivity Sensitivity (recall) - percentage of gold standard Sensitivity = total retrieved by search total of the hand-search Gold Standard = 46

Embase Sensitivity (n) Basic 4.3% 2 Enhanced 10.7% 5 Expert 15.2% 7 ERIC Sensitivity (n) Basic 0% 0 Enhanced 4.3% 2 Expert 6.5% 3 MedlineSensitivity (n) Basic 0% 0 Enhanced 10.7% 5 Expert 19.6% 9 GS=46

Specificity Specificity (precision) - positive predictive value Specificity = relevant records identified total retrieved by search

EmbaseSpecificity (n) Basic 40% 5 Enhanced 33% 15 Expert 30.4% 23 ERIC Specificity (n) Basic 0% 0 Enhanced 40% 5 Expert 37.5% 8 MedlineSpecificity (n) Basic 0% 0 Enhanced 31.3% 16 Expert 32.1% 28

A Note of Caution Academic MedicineAcademic Medicine is a journal that specialises in medical education : –more likely to be indexed for context –journal presents information for better retrieval Other journals will fare worse Other specificity scores for BEME pilots (not limited to one journal) ranged from 6 to 34%, with feedback in assessment at 17.8%

Reasons for shockingly poor performance 1. Incomplete coverage of journals 2. No indexed database for medical education 3. Existing controlled vocabularies are inadequate for medical education

Medical Education Taxonomy/Thesaurus Research Organisation Initial meeting in May 2002 Group originally coalesced around special interest group discussing the subject area

Other Driving Factors GMC-driven reforms in 1990s highlight need for life-long learning and professionalisation of university teaching Consequently the literature expands (teachers, managers, researchers and students), with the expansion of medical education itself

NHS Education for Scotland University of Edinburgh University of Newcastle Royal College of Physicians (London) University of Birmingham Hull/York Medical School University College London

Phases of Construction 1.Analysis / planning 2.Design /development/ evaluation 3.Implementation 4.Maintenance

A Diverse Group enriches the entire effort

Funding Applied to LTSN01 for small grant funding £4000 Travel, communication and dissemination

METRO 1 - Scoping

Prospective Applications Primary - entities and processes directly used in a medical educational setting: VLEs and frameworks such as Scottish Doctor & GMCs Tomorrows Doctor Secondary - entities and processes involved with reporting and analysis: description, abstraction and synthesis of data; audit; evaluation Tertiary – applied to philosophical and ontological studies and activities surrounding medical education.

Stakeholder Contexts: education/e-learning Increased dependence on electronically supported activities and contexts RLOs If there are to be efficiency gains there needs to be robust semantic and symbolic representation of entities, activities, knowledge and competencies

Stakeholder Contexts: research BEME example: groups often spread across the world and divided by language and culture Taxonomy would aid: formulation of research question; evidence retrieval, data abstraction, data synthesis, publication and evaluation of results Both Primary and Secondary contexts

Review of Existing Schemas Structure: natural language (not), enumerative lists (e.g. glossary), semantically mono-dimension (taxonomy or ontology), semantic poly-dimension (thesaurus) Purpose: descriptive or indexing Identity: non-controlled or controlled Assche et al. IMS Global Learning Consortia

Review of Existing Schemas BET: modelled on ERIC, health context at high level MeSH: global use; North American bias, freely maintained on behalf of users EMTREE: less educational depth than MeSH SNOMED: clinical specificity EET: generalised educational thesaurus

Review of Existing Schemas IIME Glossary: Institute for International Medical Education in 2000 –Scoped not relational terms (enumerative structure) –Many terms appear elsewhere Ninewells Thesaurus –For a print catalogue in early 1980s –Never piloted/evaluated –Many terms appear elsewhere

Why not use existing schemas? Nothing robust enough to support UK- specific contexts No comprehensive educational terms for undergraduate, postgraduate and CPD phases No comprehensive medical context Yet BET and MESH were most appropriate and stable

Pragmatic Approach Create a set of bridging terms and definitions between MeSH and BET, but only where terms are absent or require new definitions or extensions System must be dynamic –Contexts and cultures shift and evolve –Ongoing service not one-off product –Long-term viability means appropriate rules and procedures

METRO Phase One: Processes Submission of seed terms Discussion and agreement of terms Initial scope notes from debate Voting and resolution of scoped terms and revision Publishing of terms

Collaborative Work Environment CWE based on VLE at UoE Medical School –Commentary –Voting on terms; adding terms –Added and imbedded links and material –Forum –Simultaneous cross searching of MeSH, BET and METRO

Phase One (Terms) 4 months, 180 terms, considered by 16 METRO members CWE enabled extremely valuable discussion Many terms too general/not specific, or synonymous with BET or MESH (dropped)

Phase One: Final Workshop Typology of terms –Generic education (BET) –Generic medicine (MeSH) –Role –Process –Teaching –Learning –Assessment –Periodicity –Design –Artefact

Knowledge Issues Procedures influence the product … Was 16 reviewers enough? Reasoning not always explicit – data would have been illuminating Pre/post coordination of terms Limit to medical education?

Technical Issues How to render, apply and represent terms? Structure hierarchically or flat What data protocols and formats to employ Interoperability is key: Zthes (protocol) and XML (format), but will this accommodate all situations? Necessity to engage with wider interoperability debate for use in subscribing systems and applications

Organisational Issues By in from publishers/editors key UK terms – transferability to other English-speaking countries? Need to encompass yet be sensitive Terms can be divergent and even contradictory within constituent communities Need single coherent entity

Procedural Issues All members were unpaid contributors One model would be to pay members or their organisations for work Relationship with MeSH Relationship with BET

Investment of Time

METRO Phase 2 A second batch of funding from LTSN01 As with Phase 1, there was a core of 6 – 8 members, and slightly more in a supporting capacity Focus was on building the assessment branch of the thesaurus

Phase 2 Decision to submit terms for inclusion in MeSH Dictated the format: –Heading –Tree Number (repeatable) –Annotation –Scope Note –See Also (repeatable) –Entry Term (repeatable) –Allowable Qualifiers –Previous Indexing (repeatable) –History Note

Phase 2 Assessment seemed the logical place to start 104 terms: derived from Good Assessment Guide, Phase One, METRO members and interested parties, RCP seed terms, and existing schemas But where to start … Design / Methods / Process

Working Environment There is software! Blogger Not the most user-friendly Free and universally/publicly accessible (the nearly final version)

Creating Assessment Terms for MESH: Considerations Do we need a new MESH term? Assessment terms can be found in MESH branches other than Education e.g Behavioral Disciplines Existing terms and scope notes can be amended How likely is the concept you are describing likely to be encountered again

Creating Educational Terms for MESH: Considerations (2) General rules: Identify a place in the MESH/METRO hierarchy where your proposed term will sit Terms should represent simple or unitary concepts as far as possible e.g. Portfolio Compound terms can also be created e.g. Test Reliability The number of simple concepts in a compound term should be kept to a minimum

Testing and Development 1 Online Feedback Approximately ten volunteers logged on and recorded comments regarding the applicability of terms, scope notes and overall structure Several set practical exercises to see how the terms faired against their working areas Comments from non-logged-in persons appeared as well Differing backgrounds; debate; consolidation

Testing and Development 2 Interactive Workshop AMEE September 2004 Small group work to match assessment articles against the vocabulary All comments from groups and the plenary discussion were reviewed and key revisions made Comprehensiveness and hierarchy

Testing and Development 3 Inter-rater reliability Following on from the workshop, an exercise was developed to test IRR Five objects (4 articles and 1 assessment instrument) Five volunteers (each evaluate all 5) Complete agreement on 4/5

Lessons from Phase 2 Evolution and establishment of terminology demands flexibility and patience Consistency of description paramount IRR improved by precise unambiguous instruction, as with training Success depends on continuing interest and application by user community

Phases of Construction 1.Analysis / planning 2.Design /development/ evaluation 3.Implementation 4.Maintenance

METRO - products

Publications METRO--the creation of a taxonomy for medical education. Health Info Libr J Dec;21(4): Haig A, Ellaway R, Dozier M, Liu D, McKendree J. METRO taxonomy -- progress report on assessment. Med Teach Mar;27(2): Haig A, Dozier M, Liu D, McKendree J, Roper T, Selai C.

Applications Royal College of Physicians – catalogue at Education Resources Centre BEME Collaboration – data abstraction for research International Virtual Medical School (IVIMEDs) & other academic e-learning architectures NHS Education for Scotland – research database

Other Applications CoBaTrICE (Competency-Based Training in Intensive Care in Europe) AAMC curriculum database (CurrMIT) Ongoing international interest

What Next? Submission to Medline/National Library of Medicine Ensuring the effort continues …

Folksonomy user-generated classification, emerging through bottom-up consensus socially constructed classification schemes or communal categorisation Democratisation …

Folksonomy Del.icio.us., Flickr, and Furl are most famous examples with users categorising with tags (keywords) Reflect the language of users Serendipity Cheap – no training Quick feedback

Fauxonomy? Grab-bag of keywords Imprecision No clearly defined relations between terms Ambiguity of tags Lack of synonym control = same concept, different tags Trees are neat; piles of leaves are not.