1. The KnowledgeMap Project: Development of a Concept-Based Medical School Curriculum Database Joshua C. Denny, MD Plomarz R. Irani Firas H. Wehbe, MD Jeffrey D. Smithers, MD Anderson Spickard, III, MD, MS

2. Setting Vanderbilt School of Medicine –104 Medical students in each class –4 local hospitals No electronic repository or course schedule

3. Goals for KM Provide a set of tools to help improve the curriculum and students’ access to it Accommodate a variety of presentation styles Automate document conversion Provide a secure repository of documents that protects intellectual property

4. KM Structure Web application –Apache web server, MySQL database –Written in Perl, VisualC++, and Visual Basic Multiple servers All documents mapped to UMLS concepts via KM Concept Identifier

5. Document Corpus Manually converted 2001-2002 preclinical lecture handouts –“legacy” documents New handouts/presentations uploaded by faculty

6. Pilot Anatomy (Fall) and Cell Biology (Spring) for 2002-2003 –4 th year elective later came online

7. KM Concept Identifier Uses NLP techniques –Abbreviation and acronym extraction –Semantic regularization Score-based –Derivational forms (stenosis  stenotic, lungs  pulmonary) –Document-based disambiguation Word and concept clustering Performs favorably with MetaMap on educational documents (82% Recall, 89% Precision)

8. Document Processing Document uploaded by lecturer, placed in queue Document Conversion Server KM Concept Identifier Document Conversion Server pulls next document off queue, converts to HTML and Text Text version placed in queue Apache Web server HTML & PDF versions Identified concepts indexed for searching

9. Search Processing User enters a search query, example: “Wilson’s disease” Search Concept Identifier MySQL database C0019202 -- “Hepatolenticular Degeneration” C0019202 found in index of curriculum documents

10. Content Coverage Query Created to answer questions such as “Where is Women’s Health taught?” –“metaconcepts” Uses relationships defined in the UMLS to expand queries with related child and child- like concepts

11. Content Coverage Query User enters a “metaconcept” query: “Women’s Health” Search Concept Identifier C0080339 -- “Women’s Health” These concepts found in index of curriculum documents KM finds related UMLS concepts MySQL database

12. Other features Relevant PubMed searches –Based on the document title and the most frequent MeSH concepts in the document Definition searching –Based on UMLS SRDEF file and MedlinePLUS Course management Lecture calendar –Organized by semester and student year

18. Analysis Primary data source was Log Files: –All events in Apache –Key events in KM, including: Logins/Logoffs Searches Documents viewed (by browse or by search) PubMed searches Content Coverage queries (available only to Course Directors and Administrators) –Removed all events generated by a developer or researcher Downtime measured by a separate server that logged any time the system (or a component) was unavailable

19. Calculations Browsed document: any documents accessed via a course home page or via the “browse” function on the toolbar Searched document: any document accessed via a search

20. Results 3271 searches –84% completed with a concept search –15% definition searches (since 5/03) Total of 526 users logged in 15,885 times and viewed 1,143 documents a total of 32,113 times –All members of the first, second, and third year classes have logged on

21. Results 1264 active documents (1489 total) –722 uploaded by 28 faculty members –135 documents uploaded by authors –407 legacy documents Total downtime was about 20 hours, including scheduled downtime –No true downtime since 1/03

22. Start of new academic year

23. * * ***

25. Conclusions KM is being adopted by more classes –Predominant class and student use still first year courses Students are using KM more –Student use precedes classes coming online –All students with courses online have used KM –Initial reactions seem positive –Heaviest use by 1 st and 2 nd year students

26. Future Directions Automation of content coverage queries Expand to more courses and another site More types of media Expansions of search algorithms to include spell checker Support for a PocketPC/Palm-compatible site Student tracking

27. Acknowledgements Randy Miller, M.D. Michel Décary of Cogilex R & D, Inc Dean’s Office Art Dalley, Ph.D. Cathleen Pettepher, Ph.D. For more information http://knowledgemap.mc.vanderbilt.edu/research josh.denny@vanderbilt.edu