1. National and University Library of Slovenia University of Ljubljana, Faculty of Civil and Geodetic Engineering User-centred evaluation of digital repositories: reUSE experiences LIDA Conference, Dubrovnik 30. May – 3. June 2005 by Mateja Šmid, Darko Majcenovič, Špela Zupanc, Alenka Kavčič Čolić

2. Introduction ObjectivesEvaluation User surveyConclusion  Project reUSE: reuse digital master files of printed materials!  Financial source: EU eContent (nr. 11173)  Partners: 9 partners University and national libraries from 4 different countries: Austria, Estonia, Germany, and Slovenia.  Duration of the project: June 2004 - June 2006 INTRODUCTION Introduction

3. ObjectivesEvaluation User surveyConclusion  UBER – Humboldt University Berlin repository  ONB – Austrian national library  ALO – Austrian literature online  NLE – national library of Estonia Each implementers will establish its own digital trusted repository with its own restrictions, services and policies. REPOSITORY IMPLEMENTERS Introduction

4. ObjectivesEvaluation User surveyConclusion  Repository = organization that intends to maintain information for access and use.  Trusted repository = has the mission to provide reliable long-term access of managed digital resources to its designated community, now and in the future. TRUSTED DIGITAL REPOSITORIES Objectives

5. Introduction ObjectivesEvaluation User surveyConclusion To:  build useful systems  systems optimatization  improve services to users  measure progress  enhance the role of digital repositories OVERALL TO MAKE USER- CENTRED REPOSITORIES PURPOSE OF reUSE EVALUATION Evaluation

6. Introduction ObjectivesEvaluation User surveyConclusion EVALUATION FRAMEWORK Evaluation approach  Methodology: multi-atribute system and SWOT analysis focused on:  Organizational aspects  Technical aspects  User perspective We will use interviews and questionnaires to gain a quantitative and qualitative comprehensive study. Evaluation

7. Introduction ObjectivesEvaluation User surveyConclusion ORGANIZATIONAL ASPECTS  External and internal context in which the repositories were developed and their main functions in it. It emphasizes the concept of trustfulness through:  The organizational strategy which defines the goals and mission of the repository,  The stakeholders interests in its maintaining and financing,  Favourable organizational culture to its development,  The long-term preservation strategy and policy. Evaluation

8. Introduction ObjectivesEvaluation User surveyConclusion TECHNICAL ASPECTS OF THE REPOSITORIES Technical specifications which can show us to what extent the repositories satisfy users requirements and needs. Several subgroups will be evaluated:  General technical information  OAIS compliance  Technical and procedural suitability, security and procedural accountabillity  User friendliness and services Evaluation

9. Introduction ObjectivesEvaluation User surveyConclusion Digital libraries are useful when are in harmony (Bishop, Van House, Butterfield: Digital library use, 2003) The users The documents The system The DLs are seen as:  sociotechnical system  networks of technology  information artifacts  people and practices interacting with the larger world of work and society Evaluation

10. Introduction ObjectivesEvaluation User surveyConclusion Happy user is the main goal. WHO WILL ACCESSWHAT PURPOSE The fact that the user predetermine the collection should be highly considered. It is imporant WHO WILL ACCESS and for WHAT PURPOSE. User survey

11. Introduction ObjectivesEvaluation User surveyConclusion  Purpose:  to evaluate repositories from user perspective  Form:  questionnaire is in national language  one sheet in paper form  Based on:  ISO 9241 Usability standard  ISO 13407 Human-centred design processes for interactive systems  some specific questiones for reUSE repositories …… ADDITIONALY other methods such as simple observation and logs and statistics will be carried out (depands on certain institution) USER PERSPECTIVE User survey

12. Introduction ObjectivesEvaluation User surveyConclusion User survey End-user questionnaire consists of 2 parts: 1. general (demographic) information 2. questiones

13. Introduction ObjectivesEvaluation User surveyConclusion DIFFERENT SET OF QUESTIONES:  content  user’s satisfaction with material in the digital repository  recognition  understandability of the interface  performance and efficiency  efficiency of the repository  personal and subjective perception  personal view and satisfaction with the repository  error tolerance  possible effects on further search User survey

14. Introduction ObjectivesEvaluation User surveyConclusion  2 periods of end user’s evaluation.  June 2005 after repositories are set up  September 2005 after strong dissemination will be done  To get stable and representative sampling  around 100 questionnaires per repository will be needed.  Every 10th visitor will be asked: Whether he/she has ever used the repository?  If yes, he/she is pleased to fill in the questionnaire for end-user.  If no, the person is pleased to fill in short questionnaire for non- user.  Perception/opinion of non-users is also important. Conclusion User survey REALIZATION OF THE SURVEY

15. Introduction ObjectivesEvaluation User surveyConclusion  Different set of questiones will give us several answers.  Important feedback from end-users and non-users as well.  implementers might redesign certain part(s) of repository if it is necessary  clarification of what kind of content is expected by users  Based on those answers we will be able to compose a broader picture about user’s perspective on certain repository.  Usage habits, user’s wish, a need for training,… answers on such questiones will be given. … Cultural differencess might also surface (different backgrounds of national/university libraries, different participating countries…) User survey EXPECTED RESULTS

16. Introduction ObjectivesEvaluation User surveyConclusionCONCLUSION  At the end of the evaluation a comprehensive SWOT analysis for each implemented repository will be made.  The evaluation should have the retroactive effect on the awareness and further dissemination  valuable input for the implementers to improve the repositories if necessary  to improve services to users Conclusion

17. Introduction ObjectivesEvaluation User surveyConclusion THANK YOU FOR ATTENTION! Mateja Šmid (msmid@ikpir.fgg.uni-lj.si) Darko Majcenović (darko.majcenovic@nuk.uni-lj.si)