Presentation is loading. Please wait.

Presentation is loading. Please wait.

Assessing SNOMED CT for Large Scale eHealth Deployments in the EU Workpackage 2- Building new Evidence Daniel Karlsson, Linköping University Stefan Schulz,

Published byBryce Hodge Modified over 6 years ago

Similar presentations

Presentation on theme: "Assessing SNOMED CT for Large Scale eHealth Deployments in the EU Workpackage 2- Building new Evidence Daniel Karlsson, Linköping University Stefan Schulz,"— Presentation transcript:

1 Assessing SNOMED CT for Large Scale eHealth Deployments in the EU Workpackage 2- Building new Evidence Daniel Karlsson, Linköping University Stefan Schulz, Medical University of Graz

2 Studies performed Annotating clinical models (terminology binding) by terminology experts Manual annotation of clinical narratives by terminology experts Machine annotation of clinical narratives using Natural language processing

3 Annotating clinical models
Assumptions Use of terminologies in clinical information models representative of a major terminology use case Agreement across country, cultural, language, etc. borders representative of EU-wide, cross-border use

4 Annotating clinical models

5 Annotating clinical models
End points Content coverage Inter-annotator agreement Material / Methods 12 information models extracts, 101 elements Full SNOMED CT Set of ICD-10, ATC, LOINC, and MeSH 6 participants from 6 countries (5 EU + US)

6 Comparison SNOMED CT vs. Alternative
Small agreement on coverage assessment (ISO TR 12300) Difference agreement only after basic quality control Krippendorff’s α (95% CI) SNOMED CT 0.61 ( ) Alternative 0.47 ( )

7 Annotating clinical narratives
End points Content coverage  reference terminology Inter-annotator agreement  reference terminology Term coverage  user interface terminology Material / Methods Parallel corpus of 60 clinical text samples in 6 languages representing clinical specialties, document types, text sections 2 human annotators per language standard NLP pipeline per language Three terminology settings compared SNOMED CT (English, Swedish, French, ) A compilation of international terminologies extended subset of UMLS for text annotation A local scenario with German language terminologies

8

9 Annotating clinical narratives
End points Content coverage  reference terminology Inter-annotator agreement  reference terminology Term coverage  user interface terminology Material / Methods Parallel corpus of 60 clinical text samples in 6 languages representing clinical specialties, document types, text sections 2 human annotators per language Three terminology settings compared SNOMED CT (English, Swedish, French, ) A compilation of international terminologies extended subset of UMLS for text annotation A local scenario with German language terminologies

10 Content in SNOMED CT vs. Alternative
SNOMED CT terminology setting Alternative UMLS based setting complete translations partial translations

11 Manual Annotation : concept coverage
English: no significant difference in measured concept coverage Swedish equals English for concept coverage French / Dutch: incomplete SNOMED CT translation shows significant impact in concept coverage; more resources in alternative

12 Concept annotation: inter-annotator agreement
Krippendorff's Alpha SCT ONLY UMLS EXT English 0.40 Swedish 0.35 0.41 Dutch 0.37 0.45 French 0.31 0.30 Fair agreement values Swedish better agreement for alternative scenario (however with much less coverage) French / Dutch: incomplete SNOMED CT translation: no good performance compared to alternative

13 Manual Annotation : term coverage
Only English SNOMED provides an acceptable coverage of user interface terms Fully specified terms only (like in Swedish SNOMED version): less than 50% coverage Existing terminologies: many interface terms for French German as an example for good coverage with alternative international terminologies Finnish as opposing example (typical for small European languages)

14 Machine annotation (NLP)
Heterogeneous results Most reportable: NLP reaches 68% of the human performance in UMLS_ONLY 79% in the SNOMED CT scenario 90% in the abstain scenario (averaged over all six languages)

15 Summary Suitability of SNOMED CT as reference terminology:
for English: comparable to alternative for Swedish: better than alternative Inter-annotator agreement needs improvement (multiple strategies), not specific to SNOMED CT annotations Partial localisations of SNOMED CT not convincing results SNOMED CT as source for interface terms only for English, not ideal Recommendation: interface terminology aspects (synonyms, short forms) to be addressed by separate terminologies, linked to reference terminology

16 User Interface Terminology Terminology: Recommentation SNOMED CT
User interface terminology vs. core reference terminology User Interface Terminology RT4 Core Reference Terminology: Recommentation SNOMED CT RT1 RT2 RT3

17 Thank you! Contact: Daniel Karlsson daniel.karlsson@liu.se
Stefan Schulz

18 Interoperability Ecosystems
Process Models Information Models Terminologies Guideline Models

19 Interoperability Ecosystems
…describe in a neutral, language-independent sense The meaning of domain terms The properties of the objects that these terms denote Representational units are commonly called “concepts” Reference terminologies enhanced by formal-mathematical descriptions often called "Ontologies" Information Models Reference Terminologies Guideline Models

20 Interoperability Ecosystems
…reference terminology that occupies a pivotal role within a terminology ecosystem conceptual coverage linkage with other terminologies In most terminology ecosystems it has to be supplemented by other reference terminologies. Core Reference Terminology Information Models Other Reference Terminologies Guideline Models

21 Interoperability Ecosystems Terminologies (Classifications)
Systems of non-overlapping classes in single hierarchies, for data aggregation and ordering. aka classifications, e.g. the WHO classifications Typically used for health statistics and reimbursement AT3 Core Reference Terminology Information Models AT2 AT1 AT4 Aggregation Terminologies (Classifications) Guideline Models

22 Interoperability Ecosystems
Systems of non-overlapping classes in single hierarchies, for data aggregation and ordering. aka classifications, e.g. the WHO classifications Typically used for health statistics and reimbursement AT3 Core Reference Terminology Information Models AT2 AT1 AT4 Guideline Models

23 Interoperability Ecosystems
Collections of terms used in written and oral communication within a group of users Terms often ambiguous. Entries in user interface terminologies to be further specified by language, dialect, time, sub(domain), user group. User Interface Terminology (language specific) Information Models Guideline Models

24 "Ca" "Kalzium" "Calcium" "Ca" "Krebs" "Karzinom"
User Interface Terminology (German) Reference Terminology "Ca" "Kalzium" "Calcium" [chemistry] | Calcium (substance) | "Ca" "Krebs" "Karzinom" | Carcinoma (morphologic abnormality) | [oncology]

25 Interoperability Ecosystems User Interface Terminology
AT3 RT4 Core Reference Terminology Information Models AT2 RT1 AT1 RT2 AT4 RT3 Guideline Models

26 Interoperability Ecosystems User Interface Terminology
Process Models AT3 RT4 Core Reference Terminology Information Models AT2 RT1 AT1 RT2 AT4 RT3 Guideline Models

27 Interoperability Ecosystems User Interface Terminology
Process Models AT3 RT4 SNOMED CT ? Information Models AT2 RT1 AT1 RT2 AT4 RT3 Guideline Models

Download ppt "Assessing SNOMED CT for Large Scale eHealth Deployments in the EU Workpackage 2- Building new Evidence Daniel Karlsson, Linköping University Stefan Schulz,"

Similar presentations

Ontology Assessment – Proposed Framework and Methodology.

Ontology Assessment – Proposed Framework and Methodology.
WHO Agenda: Classifications – Terminologies - Standards

WHO Agenda: Classifications – Terminologies - Standards
Enhancing Data Quality of Distributive Trade Statistics Workshop for African countries on the Implementation of International Recommendations for Distributive.

Enhancing Data Quality of Distributive Trade Statistics Workshop for African countries on the Implementation of International Recommendations for Distributive.
TC3 Meeting in Montreal 2003 3(Montreal/Secretariat)6 page 1 of 10 Structure and purpose of IEC 82045 ISO - IEC 82045 - Specifications for Document Management.

TC3 Meeting in Montreal (Montreal/Secretariat)6 page 1 of 10 Structure and purpose of IEC ISO - IEC Specifications for Document Management.
Resources Primary resources – Lexicons, structured vocabularies – Grammars (in widest sense) – Corpora – Treebanks Secondary resources – Designed for a.

Resources Primary resources – Lexicons, structured vocabularies – Grammars (in widest sense) – Corpora – Treebanks Secondary resources – Designed for a.
ACCESS TO QUALITY RESOURCES ON RUSSIA Tanja Pursiainen, University of Helsinki, Aleksanteri institute. EVA 2004 Moscow, 29 November 2004.

ACCESS TO QUALITY RESOURCES ON RUSSIA Tanja Pursiainen, University of Helsinki, Aleksanteri institute. EVA 2004 Moscow, 29 November 2004.
SEEA Experimental Ecosystem Accounts: A Proposed Outline and Road Map Sixth Meeting of the UN Committee of Experts on Environmental-Economic Accounting.

SEEA Experimental Ecosystem Accounts: A Proposed Outline and Road Map Sixth Meeting of the UN Committee of Experts on Environmental-Economic Accounting.
Development Principles PHIN advances the use of standard vocabularies by working with Standards Development Organizations to ensure that public health.

Development Principles PHIN advances the use of standard vocabularies by working with Standards Development Organizations to ensure that public health.
Terminology in Health Care and Public Health Settings

Terminology in Health Care and Public Health Settings
1 Betsy L. Humphreys, MLS Betsy L. Humphreys, MLS National Library of Medicine National Library of Medicine National Institutes of Health National Institutes.

1 Betsy L. Humphreys, MLS Betsy L. Humphreys, MLS National Library of Medicine National Library of Medicine National Institutes of Health National Institutes.
TERMINOLOGICAL SERVER Proposal for short term complementary homeworks SEMINOP 28/10/2010.

TERMINOLOGICAL SERVER Proposal for short term complementary homeworks SEMINOP 28/10/2010.
Karen Gibson.  Significant investment in eHealth is underway  Clinical records: ◦ Not only a record for the author ◦ Essential to inform the next person.

Karen Gibson.  Significant investment in eHealth is underway  Clinical records: ◦ Not only a record for the author ◦ Essential to inform the next person.
SNOMED CT Denise Downs Knowledge Management & Education Lead Data Standards, Technology Office Department of Health Informatics Directorate.

SNOMED CT Denise Downs Knowledge Management & Education Lead Data Standards, Technology Office Department of Health Informatics Directorate.
1 EviCare (NLP) Aid clinical work in hospitals using NLP: “Summarize” health records Select and rank recommendations from clinical practice guidelines.

1 EviCare (NLP) Aid clinical work in hospitals using NLP: “Summarize” health records Select and rank recommendations from clinical practice guidelines.
School of Computing FACULTY OF ENGINEERING Developing a methodology for building small scale domain ontologies: HISO case study Ilaria Corda PhD student.

School of Computing FACULTY OF ENGINEERING Developing a methodology for building small scale domain ontologies: HISO case study Ilaria Corda PhD student.
SNOMED CT – Distributed Content Management Stefan Schulz Content Committee April 2, 2009.

SNOMED CT – Distributed Content Management Stefan Schulz Content Committee April 2, 2009.
Survey of Medical Informatics CS 493 – Fall 2004 September 27, 2004.

Survey of Medical Informatics CS 493 – Fall 2004 September 27, 2004.
An Ontology-Based Approach for Sharing Digital Resources in Teacher Education 7 th International Workshop on Ontologies and Semantic Web for E-Learning.

An Ontology-Based Approach for Sharing Digital Resources in Teacher Education 7 th International Workshop on Ontologies and Semantic Web for E-Learning.
A School of Information Science, Federal University of Minas Gerais, Brazil b Medical University of Graz, Austria, c University Medical Center Freiburg,

A School of Information Science, Federal University of Minas Gerais, Brazil b Medical University of Graz, Austria, c University Medical Center Freiburg,
Terminology Services in the OpenHIE. Agenda Terminology Services Overview Terminology Services in Rwanda Distributed Terminology System (DTS) Next Steps.

Terminology Services in the OpenHIE. Agenda Terminology Services Overview Terminology Services in Rwanda Distributed Terminology System (DTS) Next Steps.

Similar presentations

Ads by Google