1. VT University Libraries: Identifying, Teaching, and Assessing What Matters Most Office of Academic Assessment Ray Van Dyke, rvandyke@vt.edurvandyke@vt.edu Steve Culver, sculver@vt.edusculver@vt.edu 231-6003 November 13, 2008

2. Think about it… What is the most important indicator in determining the effectiveness of the library’s programs?

3. What is Assessment of Student Learning Outcomes? “Assessment of student learning is the systematic gathering of information about student learning, using the time, resources, and expertise available, in order to improve the learning.” – Walvoord “Assessment is a systematic way of paying attention to our curriculum.” – Nancy Metz

4. What is a student learning outcome? A student learning outcome states a specific skill/ability, knowledge, or belief/attitude students are expected to achieve through a program. Example: Students will be able to match information needs to information resources and organize an effective search strategy.

5. Other outcome examples Students will be able to distinguish between catalogs, databases, indexes, and the internet as examples of the complex nature of the information environment. Students will be able to conduct an advanced search on the WWW.

6. Setting the Context for Outcomes Assessment at Virginia Tech Why are we talking so much about assessment of learning outcomes right now?

7. Setting the Context Federal Government -- Spellings Commission Report – “Institutions are asked, ‘Are you measuring student learning?’ And they check yes or no. That must change. Whether students are learning is not a yes-or-no question. It’s how? How much? And to what effect?” (Chronicle, 11-4-06) SACS – The institution identifies Student Learning outcomes in its educational programs, assesses the extent to which it achieves these outcomes, and provides evidence of improvement based on analysis of the results. (SACS Standard 3.3.1.1)

8. Setting the Context Professional Accrediting Agencies (Example: ABET) – “…describe what students are expected to know and be able to do by the time of graduation, – …identify, collect, and prepare data to evaluate the achievement of program outcomes, – …results in decisions and actions to improve the program.” (ABET website) SCHEV – “Ensure that the institution’s academic programs and course offerings maintain high academic standards by undertaking a continuous review and improvement of academic programs…” (Institutional Performance Standards – IPS)

9. Setting the context VT Strategic Plan -- “Establishing a culture of continuous improvement -- Through benchmarking, assessment, evaluation of academic and administrative goals, and a commitment to process improvement and change, Virginia Tech will achieve greater national and international prominence.” The Most Important Reason for Assessing student learning outcomes?

10. Most Important Reason for Assessing Student Learning Outcomes… What is the most important indicator in determining the effectiveness of the libraries’ programs? Could it be, should it be what students have learned through their experiences with the libraries’ programs? Our focus then becomes our students’ learning and our teaching. The goal then becomes to find direct evidence that shows us what our students have learned. Thus, The process of learning outcomes assessment provides us with meaningful information for improving teaching and learning.

11. What is The Process for Assessing Student Learning Outcomes? 2. Gather and Analyze Information About Student Achievement Of Outcomes 3. Use Information Gathered To Improve Student Learning 1.Identify And Articulate Student Learning Outcomes

12. Step 1: Identifying and Articulating Student Learning Outcomes Owned by Library faculty What would an ideal graduate of VT know, understand, and be able to do because of participation in our programs? Identify a list of possible outcomes, knowing the list will change. Narrow to 3 - 5 key outcomes to consider first. Understand that you can look at others next year or the year after…

13. Step 2: Gathering and Analyzing Information About Student Achievement of Outcomes Direct and Indirect Measures Most effective way to assess: Commonly Used Student Assessments That May Be Used for Program Assessment (Course- embedded) – Direct assessments such as Comprehensive Exams, Internships, Essays, Portfolios, etc. – Capstone Courses – Aggregate data ePortfolio as a tool for gathering data

14. Step 3: Using Information Gathered to Improve Learning What recommendations would you make based on the data from the analysis of the assessment? – Alignment of student learning outcomes with course content with assessment tool – Course changes Introduction/emphasis/reinforcement of outcomes Duplication/repetition – Faculty development about a topic – Rethink Learning outcomes Develop and implement plans to improve

15. Indirect measures of student outcomes Interviews with students (either exit as graduates or during as continuing student) Alumni, current student, employer surveys Focus groups Percentage of students who go on to graduate school (and other such numbers related to graduates) Job placement numbers Enrollment trends Magazine ratings Grades

16. Direct evidence measures of student outcomes Student work samples (ideally embedded in course work) Capstone assignments Search log Observation of student behaviors Performance on national licensure examinations Externally reviewed internship Standardized tests Collections of student work (portfolios, eportfolios*)

17. Key elements in measuring these outcomes Need to provide high-quality, timely feedback that gives clear and meaningful directives for improvement Clear criteria before measurement will guide expectations Multiple measures of an outcome are ideal

18. Key questions to consider when choosing a measurement approach How much money do I have? How much time do I have? Is the measure important or just interesting? What are my analysis capabilities? How easily can I fit this into my regular work? Who needs to make decisions with the data? How will I document the evidence and any decisions made from the evidence?

19. Practical tips Don’t have one person do all the work. Cooperate with other departments, even at different schools. Borrow methods and instruments (if they fit). Use existing data if possible (IR, Registrar’s Office, etc.) Do as much as possible in the context of what you are already doing. Make your instrument, assignment, observation as short as it can be and still provide the information you want.

20. A data collection plan From whom can we get this information? (sampling issues are part of this) How will it be collected? When is the best time to collect the data? How will the data be evaluated? How can the data be used? How can the data be reported?

21. WEAVE Online & Evidence WEAVEonline helps us provide evidence of our students’ achievement of student learning outcomes. documents the continuous efforts to establish a culture of evidence. Provides a concrete record of systematically “paying attention to our curriculum.” (Metz) http://www.weaveonline.net/subscriber/vt/

22. In summary Think of data collection as circumstantial evidence in a court case. Though each source of data may have questions (reliability and validity) associated with it, the preponderance of evidence indicates that your students can do (or can’t do) what you say they can.

23. Sources Hatfield, S. R. (1999). Department level assessment: Promoting continuous improvement. Idea paper #35. Manhattan, KS: Idea Center. Hawthorne, J. (2004). General education outcomes scoring session report. http://www.und.nodak.edu/dept/oid/Doc uments/Gen%20Ed%20Outcomes%20Sco ring.pdf http://www.und.nodak.edu/dept/oid/Doc uments/Gen%20Ed%20Outcomes%20Sco ring.pdf

24. Links of Interest Spellings Report – http://www.ed.gov/about/bdscomm/li st/hiedfuture/reports.html http://www.ed.gov/about/bdscomm/li st/hiedfuture/reports.html Virginia Tech’s Scholar Site: – http://scholar.vt.edu http://scholar.vt.edu Home of OSP (Open Source Portfolio): – http://www.osportfolio.org/ http://www.osportfolio.org/

25. Reflection… “We are being pummeled by a deluge of data and unless we create time and spaces in which to reflect, we will be left with only our reactions.” --Rebecca Blood

26. Now for your area : A way to begin… 1. brainstorm at least three student learning outcomes for your area. 2. Take an inventory of measures already used in your program (in courses, at entry, at exit, etc.) that might relate to these outcomes. 3. Think of other measures you might use.