City eHealth Research Centre Mednet A methodology for evaluating digital library impact on user knowledge, attitude and behaviour Gemma Madle, Patty Kostkova, Jane Mani-Saada and Julius Weinberg City eHealth Research Centre

Mednet Talk outline 1.Introduction – The role of medical digital libraries 2.Proposed methods a.Transaction log analysis b.Pre & post use questionnaires c.Combining the methods 3.Example results 4.Future research 5.Contacts

City eHealth Research Centre Mednet Introduction Medical digital libraries –Life critical applications –Empower health professionals & patients National electronic Library for Health –Specialist library of Infection –Freely accessible to all via the Internet

City eHealth Research Centre Mednet But is knowledge the enemy of disease?

City eHealth Research Centre Mednet Proposed methods Transaction Log Analysis –Record of the requests for pages of a website Pre/Post Questionnaires –Evaluate changes in user knowledge and attitude –Questions to test knowledge –Questions to test attitude

City eHealth Research Centre Mednet a Transaction log analysis What data do we get? –Path of user through a site –IP address of user What conclusions can we draw? –How users are navigating/searching the site –Most popular pages –Linked pages, relationships between pages What are the limitations? –Identification of crawlers/spiders –Problems relating IP addresses to users, use of proxy servers –Time spent on last page not recorded –Use of back/forward buttons –Limited information about the user –Periods of inactivity within sessions

City eHealth Research Centre Mednet a Transaction log analysis: Examples Microanalysis (Nicholas & Huntingdon) –Analyse use by small number of users –Build a picture of individual users use over time –Problems identifying the same users on separate visits Conclusions from evaluation of NeLI website –Increase in use of site over 18 month period evaluated (More than double) –Most users UK based –Identification of most popular pages –Suggestion of relationship between topics visited and infectious disease outbreaks in the news –Users logging in from hospital or NHS based PCs more likely to return to the site, spend more time on the site and visit more pages

City eHealth Research Centre Mednet b Pre/post use questionnaires What data do we get? –Changes in knowledge and attitudes –More specifically changes in scores for specific questions and/or specific users –Self-reporting of perceived knowledge gain and intended behavioural changes –User demographics What conclusions can we draw? –Are users learning from the site? Which groups? What? –Is it making a difference to their attitudes? –Do they intend to change their behaviour? What are the limitations? –Only evaluating intended behaviour not actual –No indication of how people are using the site

City eHealth Research Centre Mednet b Pre/post use questionnaires: Example The Antimicrobial Resistance Digital Library –Significant improvements in user knowledge –Significant changes in attitudes –Users generally felt they learnt from using the library

City eHealth Research Centre Mednet c Combining the methods What data do we get? –User activity on the website –Changes in knowledge and attitude –Self-reported perception of learning and intended behaviour change –User demographics What conclusions can we draw? –Relationships between sections of the library and knowledge/attitude changes –Relationship between different pages of the library and user demographics –Navigation by different user groups What are the limitations? –Limited qualitative information about users experience of the site –Still only evaluating intended behaviour change

City eHealth Research Centre Mednet Future research What? –Comparing user demographics and changes in knowledge/attitude with transaction logs of user activity Where? –The Antimicrobial Resistance Digital Library at When? –Soon!!!

City eHealth Research Centre Mednet Contacts Gemma Madle City eHealth Research Centre National electronic Library of Infection (NeLI) Antimicrobial Resistance Digital Library Available at Results of evaluation published at ECDL 2003, slides available at