1. Magia G. Krause Ph.D. Candidate School of Information University of Michigan Society of American Archivists Annual Meeting August 14, 2009 Undergraduates in the Archives : Measuring the Impact of Archival Instruction Through Rubric-based Assessment

2. Instruction and Assessment in Academic Archives Krause, Magia G.(2008). Learning in the archives: A report on instructional practices, Journal of Archival Organization, 6(4) 233-268.

3. Overview of Teaching Experiment Research Questions Do undergraduate students learn from archival instruction? What do undergraduate students learn from archival instruction? Goals of study Introduction of a reliable assessment tool to measure student learning Introduction of archival literacy scale that might compliment a curriculum of undergraduate research education in archives

4. Methodology Quasi-experiment in the field Background US history course in a large state university 82 students total Control and treatment groups To compare students that did not receive archival instruction with students that did Pre- and Post- Document Analysis Exercise To provide a baseline comparison of the groups and look for an improvement in treatment group

5. Archival Instruction Description of treatment (aka archival instruction) Lecture/presentation to entire class by archivist Hour-long session in the archives with 4 stations Bibliographic Instruction Critical Thinking Photograph Analysis Footnote/Citation Analysis

6. Photographs from the Instruction Session

7. Document Analysis Exercise Divided into 3 sections Adapted from NARA’s Teaching with Documents Written Document Analysis Photograph Analysis Finding Aid Analysis Difference between Pre- and Post- Tests Alternative primary sources and finding aid

8. Assessing Student Learning Rubric Basic definition Scoring guide that includes criteria and levels of performance Types of rubrics Holistic and analytic Steps in creating a rubric Criteria : Archival Literacy Skills Observation Interpretation/Historical Context Evaluation/Critical Thinking Research Skills

9. Rubric for this study

10. Using the Rubric 3 raters Two professional archivists with at least five years of experience teaching undergraduates and me Important for increasing the reliability of the scores Achieving kappa Final results Our Fleiss Kappa for the pretest was.80218 and.788 for the post-test, both of which are considered excellent strengths of agreement.

11. Statistical Tests Non-parametric tests Useful for analyzing ordinal and nominal data Great for small samples Tests used: Mann Whitney U Test for Independent Samples Results The tests confirmed that the null hypothesis could be rejected with 95% confidence Students in the treatment group did better on the post-test after participating in archival instruction. In fact, there is a statistically significant difference between the two groups’ scores for each of the criteria.

14. Discussion The Research Skills Problem Disappointing results Possible factors Students not actively engaged during instruction Follow-up assignment involved pre-selected documents – no chance to practice searching for and identifying documents Literature on undergraduate information-seeking behavior

15. Archival Literacy Scale Elements 4 categories of the rubric: Observation Interpretation Evaluation Research Skills Testing for reliability Results

16. Conclusions Limitations to the generalizability of results – Need to replicate the study to increase the reliability of the instruments Wish for better instructional design for instruction component Need for more communication and collaboration between archivists to develop and share curricula and standardized tools to assist in instruction and assessment

17. Thank You For more information about the study described in this presentation, please contact: Magia Krause mghetu@umich.edu.