1. City eHealth Research Centre Mednet 20031 A methodology for evaluating digital library impact on user knowledge, attitude and behaviour Gemma Madle, Patty Kostkova, Jane Mani-Saada and Julius Weinberg City eHealth Research Centre

2. Mednet 20032 Talk outline 1.Introduction – The role of medical digital libraries 2.Proposed methods a.Transaction log analysis b.Pre & post use questionnaires c.Combining the methods 3.Example results 4.Future research 5.Contacts

3. City eHealth Research Centre Mednet 20033 1. Introduction Medical digital libraries –Life critical applications –Empower health professionals & patients National electronic Library for Health –Specialist library of Infection –Freely accessible to all via the Internet

4. City eHealth Research Centre Mednet 20034 But is knowledge the enemy of disease?

5. City eHealth Research Centre Mednet 20035 2. Proposed methods Transaction Log Analysis –Record of the requests for pages of a website Pre/Post Questionnaires –Evaluate changes in user knowledge and attitude –Questions to test knowledge –Questions to test attitude

6. City eHealth Research Centre Mednet 20036 2a Transaction log analysis What data do we get? –Path of user through a site –IP address of user What conclusions can we draw? –How users are navigating/searching the site –Most popular pages –Linked pages, relationships between pages What are the limitations? –Identification of crawlers/spiders –Problems relating IP addresses to users, use of proxy servers –Time spent on last page not recorded –Use of back/forward buttons –Limited information about the user –Periods of inactivity within sessions

7. City eHealth Research Centre Mednet 20037 2a Transaction log analysis: Examples Microanalysis (Nicholas & Huntingdon) –Analyse use by small number of users –Build a picture of individual users use over time –Problems identifying the same users on separate visits Conclusions from evaluation of NeLI website –Increase in use of site over 18 month period evaluated (More than double) –Most users UK based –Identification of most popular pages –Suggestion of relationship between topics visited and infectious disease outbreaks in the news –Users logging in from hospital or NHS based PCs more likely to return to the site, spend more time on the site and visit more pages

8. City eHealth Research Centre Mednet 20038 2b Pre/post use questionnaires What data do we get? –Changes in knowledge and attitudes –More specifically changes in scores for specific questions and/or specific users –Self-reporting of perceived knowledge gain and intended behavioural changes –User demographics What conclusions can we draw? –Are users learning from the site? Which groups? What? –Is it making a difference to their attitudes? –Do they intend to change their behaviour? What are the limitations? –Only evaluating intended behaviour not actual –No indication of how people are using the site

9. City eHealth Research Centre Mednet 20039 2b Pre/post use questionnaires: Example The Antimicrobial Resistance Digital Library –Significant improvements in user knowledge –Significant changes in attitudes –Users generally felt they learnt from using the library

10. City eHealth Research Centre Mednet 200310 2c Combining the methods What data do we get? –User activity on the website –Changes in knowledge and attitude –Self-reported perception of learning and intended behaviour change –User demographics What conclusions can we draw? –Relationships between sections of the library and knowledge/attitude changes –Relationship between different pages of the library and user demographics –Navigation by different user groups What are the limitations? –Limited qualitative information about users experience of the site –Still only evaluating intended behaviour change

11. City eHealth Research Centre Mednet 200311 3 Future research What? –Comparing user demographics and changes in knowledge/attitude with transaction logs of user activity Where? –The Antimicrobial Resistance Digital Library at http://www.neli.org.ukhttp://www.neli.org.uk When? –Soon!!!

12. City eHealth Research Centre Mednet 200312 4 Contacts Gemma Madle g.c.madle@city.ac.uk City eHealth Research Centre http://www.city.ac.uk/cerc National electronic Library of Infection (NeLI) http://www.neli.org.uk Antimicrobial Resistance Digital Library Available at http://www.neli.org.ukhttp://www.neli.org.uk Results of evaluation published at ECDL 2003, slides available at http://www.staff.city.ac.uk/~jane/ppt.html