1. End User Feedback Cornell Institute for Digital Collections Gale Halpern February 9, 2000

2. Why Evaluate? Identify problems with the implemented system: Hardware/connectivity problems. Software/usability problems. Collection deficiencies. Searching problems. Inadequate training/support. Un-met expectations (both instructors and students). Gather constructive feedback for improving the existing implementation. Guide directions for new development of collections and software.

3. Evaluation Instruments Student survey. Population demographics: Skill set Training session Hardware/connectivity Factual questions about search terms, # of images used, etc. General satisfaction ratings. Open-ended comments. Instructor feedback (informal discussion with TA and instructors).

4. Fall 2000 – Insight at Cornell Three undergraduate classes used Luna Insight and the Johnson Museum Digital collection to support teaching: Revelry, Rivalry and Consumption: Nineteenth Century American Holidays and Civic Life (Freshman Writing Seminar for History students) – 16 students Mapping America (Art History survey – from colonial mercantilism through the Great Depression) – 30 students Friends of the Cold Season: Pine, Bamboo and Plum (Seminar to build an exhibition and catalogue of East Asian decorative arts at the Johnson Museum) – 8 students

5. Use of Insight Freshman Writing Seminar – The digitized Johnson collection provided one primary source of material for students completing essay-writing assignments. The instructor worked with Museum staff to create a pre-defined group of images in Insight. Freshman students used the web browser from their on-campus dorm rooms to view the images. Art History Survey – Johnson Museum digitized images were one of several sources available to students for creating web-based, virtual thematic exhibitions. Other sources included David Rumsey Map Collection (using Insight), The Fine Arts Museum of San Francisco Thinker ImageBase, and the Library of Congress American Memory site. Exhibition Seminar – Students used Insight Presentation tools and the digitized Johnson Digital collection to plan and design a live exhibit of East Asian decorative arts which is currently on display at the Johnson Art Museum.

6. User Training: Freshman Writing Seminar -- CIDC and the Museum trained the instructor only. No help was offered to the students using the web browser. Art History Survey – CIDC offered a two-hour hands-on Insight training session at the beginning of the semester. Handouts including a Luna Information sheet, a Quick start Guide and other supporting documents. Contact email addresses were provided for questions. Students were shown how to download the Java client at home and were free to use it at any time in the library computer lab. They could also reserve a space at one of six specially-equipped, library supported and under-utilized Creation stations. Exhibition Seminar – same as the Art History Survey plus information about the Insight Presentation module.

7. Outcome: How many images (approximately) did each class use? Freshman writing seminar– A group of 26 view-only images were used to inspire creative discussion. Art history survey -- know that 8 images out of a possible 130 were exported from insight and used on student web pages. Exhibition class -- 160 images (some duplicates) were used in insight student presentations.

8. What Did We Discover? 1.These three classes had quite different expectations from the Johnson Museum collection and the Insight software. Two out of three classes successfully focused on use of instructor-selected images or an established thematic collection such as the Asian Art collection. The Art History survey class proved the least satisfied because they needed a broader base of well-known artworks to complete their assignments. They found these at the large American Memory and Thinker ImageBase sites.

9. What Did We Discover? 2.Dissatisfaction with access or difficulty viewing images from home. User functionality in Luna Insight was generally not the main problem. At-home installations proved to be slow and not productive. On-campus Java Client library installations were fast enough for the Exhibition Class, inadequate for the Art History Survey class. The most satisfied students were the freshman in the Writing seminar who used the web browser from their on- campus dorm rooms.

10. What Did We Discover? 3.Inadequate search terms, a high percentage of missing images from the permanent collection and incomplete descriptive data were the top complaints. We knew from the beginning that the 15,000 image + Johnson Museum digital collection needed better cataloguing for electronic searching, and represents only 60% of the complete museum collection. The local Johnson Museum collection is relatively small and eclectic. It will never provide the broad-based source for teaching an Art History survey class. Lack of a thematic organization of the on-line collection made searching even more difficult.

11. What Did We Discover? 4.Research and discovery of artworks, that would not otherwise have been found was the most positive outcome for students. Despite problems in data description and incomplete works, students in the Asian Art exhibition seminar unanimously felt that they were better off using Insight as a research and discovery tool than not using it.

12. Observations: Once students were unhappy with the at-home access or unsuccessful searches, they gave up on Insight. It would have been more effective to offer the hands-on training at the point when students needed it (not necessarily at the beginning of the semester). Providing handouts and more extensive support on search strategies may have encouraged students to try a little harder to find images.

13. Recommendations for Future Development: Extend our local collections by offering larger and broader collections (e.g. AMICO) under the umbrella of Luna Insight. Ensure that future collections are complete with high quality cataloguing and descriptive metadata. Organize broad collections thematically to make searching easier.

14. Recommendations for Future Development: Develop private collections, which serve the needs of specific instructors teaching from their personal slide collections. Improve the Insight browser to serve instructors and students (i.e. allow the Insight browser to play presentations). Incorporate more efficient copyright restrictions to improve off-campus access to large images.