Direct vs Indirect Assessment of Student Learning: An Introduction Dr. Sheila Handy, Chair Business Management and Co-Chair University Assessment Committee Sponsored by University Assessment Committee February 24, 2014

Assessment “The most important purpose of assessment should be not improvement or accountability but their common aim: Everyone wants students to get the best possible education. Everyone wants them to learn what’s most important. A college’s mission statement and goals are essentially promises that the college is making to its students, their families, employers, and society. Today’s world needs people with the attributes we promise. We need skilled writers, thinkers, problem-solvers and leaders. We need people who are prepared to act ethically, to help those in need, and to participate meaningfully in an increasingly diverse and global society. Imagine what the world would be like if every one of our graduates achieved the goals we promise them! We need people with those traits, and we need them now. Assessment is simply a vital tool to help us make sure we fulfill the crucial promises we make to our students and society. “ Linda Suskie, 2010

Continuous Improvement of Student Learning Articulate Goals Offer Programs, Services and Initiatives Collect Assessment Information Use Results for Improvement

Setting the Stage Middle States Commission on Higher Education “…each program should employ direct measures of student learning…” “…each stated learning outcome should be aligned to a measure, a performance goal, and a finding. Measures can be used to evaluate more than a single learning outcome.”

Assessment Measures - Definitions DirectIndirect Direct examination or observation of student knowledge, skills, attitudes or behaviors to provide evidence of learning outcomes. Perceived extent or value of learning experiences

Assessment Measures - Examples DirectIndirect Exam ▫ Standardized ▫ Locally developed ▫ Embedded questions Juried review Portfolios with rubrics Surveys ▫ Students ▫ Alumni Focus Groups Student Records ▫ Class Attendance

Short Table Activity Introductions Envelopes ▫ Match Assessment Measure and Description ▫ Sort into Direct and Indirect Measures

Standardized Tests Licensing Exams ▫ Education - Praxis, PAPA ▫ Nursing - NCLEX ▫ Accounting - CPA Exam Major Field Test ▫ Developed by ETS ▫ Administered in a proctored environment as paper and pencil or online

MFT subjects ▫ Biology ▫ Business ▫ Chemistry ▫ Computer Science ▫ Criminal Justice ▫ Literature in English ▫ Math ▫ Physics ▫ Political Science ▫ Psychology ▫ Sociology

Standardized Tests AdvantagesDisadvantages  Carefully developed  Highly reliable  Professionally scored  Nationally normed ▫ If tests don’t match LOs of program, scores will be low ▫ MC questions that test facts – faculty may focus on higher order skills ▫ May be expensive ▫ Students may not take seriously

Locally Developed Exam AdvantagesDisadvantages Can be tied-in to specific LOs Can be administered as part of regular semester testing Can use publisher developed questions Reliability and generalizability not known Time required to prepare questions Scoring may take a long time Norms not available

Embedded Questions AdvantagesDisadvantages Can be conducted as part of regular student testing Does not require collection of additional data Can be used for both grading and assessment Faculty may think they are being assessed Takes time to develop and grade

Embedded Questions One way to assess learning without faculty “angst” ▫ Add assessment questions as a quiz administered at the beginning of the next course in the sequence ▫ Effective if different faculty teach multiple sections of an introductory course

Juried Review Suitable for assessing ▫ Performing arts projects ▫ Written assignments ▫ Oral presentations

Juried Review AdvantagesDisadvantages Provides experience and practice for student performers Can be formative and summative – indicating “value added” Can also introduce students to the standards they will be expected to achieve Achieving consensus with regard to the evaluation rubric May not be sufficient to determine if students have achieved the LOs set by the faculty

Portfolios With Rubrics Suitable for many majors ▫ Art ▫ English ▫ Computer Science ▫ Business ▫ Education

Portfolios With Rubrics AdvantagesDisadvantages Provides students with evidence to submit to potential employers Tracks student work over time Can assess higher levels of learning Allows students to reflect on their learning Faculty must agree on assessment rubric Time consuming Storage of evidence

Case Study Your goal is to develop a plan to assess this student learning outcome for the Engineering Program for the upcoming academic year. This plan should include two direct measures.

Wrap Up Resources ▫ Materials in Folders ▫ University Assessment Committee ▫ Assessment Consulting Team ▫ Website