1. Alexandria Digital Library User and Use Evaluation Experiments with Log Data Analysis Linda Hill Mary Larsgaard Catherine Masi Mary-Anna Rae Philip Sallis Auckland Institute of Technology ASIS Mid-Year Meeting Pasadena, May 24-26, 1999

2. Alexandria Digital Library 2 Hill/Sallis - ASIS Mid-Year, 1999 ADL Digital Library Environment & Evaluation Goals o Evaluation Environment  georeferenced digital library  stable release of the ADL/JiGi client  registration database  session log database o Evaluation Goals  Combine registration data and session log data to characterize groups of users  Design analysis methods for this data  Evaluate the usefulness of the output of these methods  Develop evaluation methods for future application to the operational version of ADL

5. Alexandria Digital Library 5 Hill/Sallis - ASIS Mid-Year, 1999 ADL Search Buckets o High-level containers for metadata from multiple underlying sources  Current Search Buckets (latitude and longitude –type (logical type) –format (“available as” format) –date range –topical text (freetext from selected attributes) –assigned terms (controlled vocabulary subject attributes) –originator –identifier

6. Alexandria Digital Library 6 Hill/Sallis - ASIS Mid-Year, 1999 ADL Collections o MIL Catalog (322,000 records)  Maps, aerial photographs, remote-sensing images, digital map products, datasets in modified FGDC metadata format o ADL Gazetteer (1.5 million entries)  Georeferenced placename set: placename, footprint, and type o GeoRef (Amer. Geological Inst.) (13,500 entries)  Georeferenced bibliographic records o Earthquakes (Southern California) (330 entries) o Volcanoes (Smithsonian) (1500 entries)

7. Alexandria Digital Library 7 Hill/Sallis - ASIS Mid-Year, 1999 Example ADL Search o In the MIL Catalog, find holdings that satisfy the following constraints:  Format is “gif” or “jpeg”  Type is “aerial photograph”  Originator is “Fairchild”  Topical Text includes the phrase “digital elevation model”  Date is between 19000101 and 19891231  Location is contained within (130W, 32N) and (129.5W and 34N)

8. Alexandria Digital Library 8 Hill/Sallis - ASIS Mid-Year, 1999 User Information: Registration Data  Sex/Gender  Primary Areas of Interest  Highest Educational Degree  Proficiency Level in Four Areas – Geospatial Data – Online Searching – WWW – Computers

9. Alexandria Digital Library 9 Hill/Sallis - ASIS Mid-Year, 1999 ADL Session Log Data o Session ID: User ID-Session start time o Time stamps for each action o Actions  Start query  Query statements sent to server –Query areas –Collections –Search Buckets and values –Limit of number of items to return  End query  End session

10. Alexandria Digital Library 1010 Hill/Sallis - ASIS Mid-Year, 1999 Use Information: Session Log Data  ----- Included in this study ----  Collection Use by Group  Search Bucket Use by Group  Search Bucket Use by Collection  Frequency of Use (based on the number of sessions)  Session Duration Intervals by Group

11. Alexandria Digital Library 1 Hill/Sallis - ASIS Mid-Year, 1999 Use Information: Session Log Data  ----- Included in this study -----  Collection Use by Group  Search Bucket Use by Group  Search Bucket Use by Collection  Frequency of Use (based on the number of sessions)  Session Duration Intervals by Group  ----- Not included in this study -----  Search Bucket Values  Geographic areas used for query areas  Requests for full metadata reports  Requests for data downloads  Use of Workspace functions and other functions of the client

12. Summary Statistics for Groups

13. Alexandria Digital Library 1313 Hill/Sallis - ASIS Mid-Year, 1999 Analyses o Descriptive Statistics  Excel Spreadsheet o User Feedback to Descriptive Statistics o Neural Net  Viscovery

14. Alexandria Digital Library 1414 Hill/Sallis - ASIS Mid-Year, 1999 Flow of work

24. Alexandria Digital Library 2424 Hill/Sallis - ASIS Mid-Year, 1999 Interesting Findings of Descriptive Statistics o Interesting differences in the levels of confidence reported by the members of the two groups o Patterns of reported proficiency for WWW and Computers are similar - perhaps we don’t need to continue asking for both of them o Format Search Bucket strongly associated with Researchers and with searching the Catalog o Some Search Buckets are rarely used - can be factored into Client design o We can say that the non-staff LIS group, who are primarily Masters level professionals and who are experts in online searching, are using ADL’s spatial searching capability to search GeoRef

25. Alexandria Digital Library 2525 Hill/Sallis - ASIS Mid-Year, 1999 We are interested in your reaction to data presented here and our summary findings. 1. What is most interesting to you about this study and why? 2. As an ADL registered user, please comment on your own patterns of use in relation to what we have presented here. Do you see your own patterns of use here? We are interested in the ways in which you would like to see ADL develop. 5. What changes would you like to see in the user interface? For example, comment on the usability of the graphical interface; the map browser; the search buckets; query tracking; etc. 6. What collections are most important to you and how would you like to see them develop? Are there new collections that you would like to see added? We are interested in the purposes for which you find ADL useful. 3. What have been your primary reasons for using ADL up to this point? 4. How would you like to be able to use ADL in the future? User Feedback to Descriptive Statistics

26. Alexandria Digital Library 2626 Hill/Sallis - ASIS Mid-Year, 1999 Neural Net Evaluation of ADL Use o The Project: to build system use profiles employing Neural Network techniques; in particular data mining with self-organising maps (SOMs), Tuvi Kohonen, IEEE,1990. o The Method: capture and analyse user registration data & session log data, then mine for meaningful relationships using SOMs and statistical methods. Apply a priori fuzzy labels to derived use patterns such as frequency of sessions and their duration.

27. Alexandria Digital Library 2727 Hill/Sallis - ASIS Mid-Year, 1999 Why SOMs for this research? o SOMs are proposed in the research literature as alternative (supplemental) data mining tools that generate self-modifying illustrations of data relationships when new data is input over time. o They provide a pictorial output of data clustering…fun but not always easy to quantify! o They also provide a visual (easy to comprehend) way for a manager to see what data sources are being used in a system…and what isn’t! o They provide a multi-dimensional mental map compared with standard statistical two-dimensional tables.

28. SOMs self-modify as new node relationships are identified

29. Cascade of cluster maps for individual attribute groups within the application data space domain

30. User self-reported competency @ Intermediate-Expert Level Computers WWW

31. Frequency of Sessions for 30 users The ‘jigsaw’!!! seventeen users have >10 sessions. two users have 6-10 sessions. ten have 2-5 sessions. one has 1 session.

32. ‘Jigsaw’ for clusters of mean duration per session. Each user has one mean time duration, which is classified into one of the eight duration categories.

33. Alexandria Digital Library 3 Hill/Sallis - ASIS Mid-Year, 1999 Concluding Remarks on NN Analysis o SOMs add value to data mining in that they: –are pictorial and thus illustrate clearly the clustering of data. Good for seeing the ‘shape’ of data clusters. –provide visual clues to managers who need to know what datasets are being used and how. –the network reflects its ‘learning’ about new data relationships as the use of the system changes. o Somewhat frustrating when seeking familiar quantitative inferences from statistical analysis, which must remain part of the hybrid approach.

34. Alexandria Digital Library 3434 Hill/Sallis - ASIS Mid-Year, 1999 Concluding Remarks o ADL Policy on Permitting/Supporting Evaluation Studies of the Alexandria Digital Library  To encourage the development of methodologies of data mining of registration and session log data o We would like to see both the descriptive, neural net, and other analyses further explored o We intend to establish automatic/callable scripts to perform these analyses as needed o The future of user registration: will not be required for the operational ADL system but can be required for targeted groups such as the instructors and students in test classrooms.

35. Alexandria Digital Library 3535 Hill/Sallis - ASIS Mid-Year, 1999 Thank You