Assessment Workshop Presented by: The Office of Institutional Research and Assessment

Assessment Workshop Why do assessment?Why do assessment? What is assessment?What is assessment? When to assess?When to assess? How to assess?How to assess?

Why do assessment? ImprovementAccountabilityAccreditation

Improvement CurriculumCurriculum Instructional methodology and practiceInstructional methodology and practice Student servicesStudent services

“Nothing is so perfect that it cannot be improved upon.” Trudy Banta

Accountability State Board of Education/ Board of RegentsState Board of Education/ Board of Regents Public accountabilityPublic accountability Competition for limited resourcesCompetition for limited resources

“Every publicly supported social services agency now has an outcome-based agenda.” Trudy Banta

Accreditation Northwest Association of Schools and Colleges (NASC)Northwest Association of Schools and Colleges (NASC) –We are to demonstrate through regular and systematic assessment that students who complete their programs, no matter where or how they are offered, have achieved a specified set of learner goals established for each program.

“There must be evidence that students are mastering the course and curriculum objectives. Given the ferocious competition developing among learning organizations worldwide, these [assessments] are necessary steps.” Trudy Banta

What is assessment? Assessment is a systematic process of looking at student achievement within and across courses by gathering, interpreting and using information about student learning for educational improvement. Assessment is a systematic process of looking at student achievement within and across courses by gathering, interpreting and using information about student learning for educational improvement. American Association of Higher Education (AAHE) American Association of Higher Education (AAHE)

Characteristics of good assessment comprehensive, ongoing and evolutionarycomprehensive, ongoing and evolutionary broad involvement from facultybroad involvement from faculty clear, assessable educational goals and objectivesclear, assessable educational goals and objectives uses a variety of assessment and evaluation methodologiesuses a variety of assessment and evaluation methodologies collects meaningful and accurate datacollects meaningful and accurate data primary emphasis is on improvement of teaching and learningprimary emphasis is on improvement of teaching and learning (adapted from Seybert, 1998) (adapted from Seybert, 1998)

Why aren’t grades enough? grading practices are not standardgrading practices are not standard need different ways of structuring program assessmentneed different ways of structuring program assessment grades reflect many things other than course content and masterygrades reflect many things other than course content and mastery objectives differobjectives differ good assessment requires multiple ways of measuring goal achievementgood assessment requires multiple ways of measuring goal achievement

When to Assess Can be Decided According to When Critical Decision-Making and Communication Occurs Entry assessment helps determine who should be admitted and who is prepared to benefit from which programs and courses.Entry assessment helps determine who should be admitted and who is prepared to benefit from which programs and courses. Midpoint assessment occurs when students reach a crucial decision point or level of attainment in their program of studies.Midpoint assessment occurs when students reach a crucial decision point or level of attainment in their program of studies. Exit assessment helps determine which students have attained the prerequisite knowledge, skills, and abilities associated with the program goals.Exit assessment helps determine which students have attained the prerequisite knowledge, skills, and abilities associated with the program goals. Follow-up (Post Completion) assessment helps determine the effectiveness of the educational programs in preparing students for further education, transfer, entry or reentry into the workforce, or the students’ personal goals.Follow-up (Post Completion) assessment helps determine the effectiveness of the educational programs in preparing students for further education, transfer, entry or reentry into the workforce, or the students’ personal goals.NCTLA

How to assess? Identify each degree and certificate program to be assessed.Identify each degree and certificate program to be assessed. Identify student learning goals and the educational criteria and experiences for each goal.Identify student learning goals and the educational criteria and experiences for each goal. Identify appropriate assessment methods and strategies.Identify appropriate assessment methods and strategies. Collect, analyze and interpret data.Collect, analyze and interpret data. Specify program improvements.Specify program improvements.

Identify each degree and certificate program to be assessed. All certificate programsAll certificate programs All undergraduate programsAll undergraduate programs All graduate programsAll graduate programs All off-campus programsAll off-campus programs

Identify student learning goals establish three to six goals for each program, both graduate and undergraduateestablish three to six goals for each program, both graduate and undergraduate identify education criteria and experiences for each goalidentify education criteria and experiences for each goal –What is to be learned? –What is the level of learning? –What is the learning applied to?

Identify Key Components of Each Goal Identify Key Skills for Each Goal Goal: Students will understand and apply logical and ethical principles to personal and social situations. What logical and ethical principles are learned regardless of specific coursework taken?What logical and ethical principles are learned regardless of specific coursework taken? How do students show their understanding and ability to apply these principles?How do students show their understanding and ability to apply these principles? How do we see students apply principles to their personal lives and development?How do we see students apply principles to their personal lives and development? How do we see students applying principles to social settings and circumstances?How do we see students applying principles to social settings and circumstances?NCTLA

What if you do not have assessable goals? 1. Examine the set of required courses. 2. Ask, “What have we been trying to teach?” (Outcomes) –Content knowledge? –Cognitive skills? –Values and attitudes? 3. Ask, “What should students know before they enter the curriculum in order to succeed?’ (Entrance Criteria) 4. Ask, “What should students know when they complete the curriculum in order to graduate?” (Exit Criteria) 5. Ask, “At what points in the curriculum are students doing well or having difficulty?” (Midpoint Criteria) 6. Ask, “Are our alumni successful in the field?” (Post Completion Criteria) Adapted from NCTLA

What if you have assessable goals, but no specific curriculum to support them? 1. Ask, “Do we really teach students (the goal)?” 2. If “Yes,” then identify if the goal is: –embedded throughout coursework, or –achieved through an identifiable pattern of coursework 3. State the specific coursework pattern required to attain the general education goal. 4. Identify entry ability required for students to succeed at the collegiate level. 5. Identify key midpoints in the development of student abilities along the general education goal. 6. Identify levels of attainment or performance required for graduation. 7. Identify levels of attainment or performance in an employment setting. NCTLA

What to Assess: Knowledge outcomes - core of concepts and material knowledgeKnowledge outcomes - core of concepts and material knowledge Skills outcomes - what a student can doSkills outcomes - what a student can do Attitudes and values outcomes - those faculty believe to be importantAttitudes and values outcomes - those faculty believe to be important Behavioral outcomes - behaviors crucial to the curriculum’s impactBehavioral outcomes - behaviors crucial to the curriculum’s impact

Bloom’s Classification of Cognitive Skills KnowledgeKnowledge ComprehensionComprehension ApplicationApplication AnalysisAnalysis SynthesisSynthesis EvaluationEvaluation

Knowledge Outcomes Describe the basic components of empirical research.Describe the basic components of empirical research. Give examples of major themes or styles in Music, Art, or Theatre.Give examples of major themes or styles in Music, Art, or Theatre. Recognize in complex text logical, rhetorical, and metaphorical patterns.Recognize in complex text logical, rhetorical, and metaphorical patterns.NCTLA

Comprehension Outcomes Correctly classify a variety of plant specimens.Correctly classify a variety of plant specimens. Explain the scientific method of inquiry.Explain the scientific method of inquiry. Summarize the important intellectual, historical, and cultural traditions in Music, Art, or Theatre from the Renaissance to Modern times.Summarize the important intellectual, historical, and cultural traditions in Music, Art, or Theatre from the Renaissance to Modern times.NCTLA

Application Outcomes Demonstrate in the laboratory a working knowledge of lab safety procedures.Demonstrate in the laboratory a working knowledge of lab safety procedures. Apply oral communication principles in making a speech.Apply oral communication principles in making a speech. Compute the area of a room.Compute the area of a room. Use editing symbols and printers’ marks.Use editing symbols and printers’ marks.NCTLA

Analysis Outcomes Distinguish between primary and secondary literature.Distinguish between primary and secondary literature. Diagram a sentence.Diagram a sentence. Listen to others and analyze their presentations.Listen to others and analyze their presentations. Differentiate between historical facts and trivia.Differentiate between historical facts and trivia.NCTLA

Synthesis Outcomes Revise faculty copy for a news story.Revise faculty copy for a news story. Formulate hypotheses to guide a research study.Formulate hypotheses to guide a research study. Create a poem, painting, design for a building.Create a poem, painting, design for a building.NCTLA

Evaluation Outcomes Compare art forms of two diverse cultures.Compare art forms of two diverse cultures. Critically assess an oral presentation.Critically assess an oral presentation. State traditional and personal criteria for evaluating works of art.State traditional and personal criteria for evaluating works of art. Draw conclusions from experimental results.Draw conclusions from experimental results.NCTLA

Identify the educational experiences for each goal. List actions intended to enable the students to achieve these goals.

Identify appropriate assessment methods and strategies Choose one or two goals from each programChoose one or two goals from each program Identify appropriate measures of goal attainmentIdentify appropriate measures of goal attainment Identify the appropriate point of measurementIdentify the appropriate point of measurement Use multiple methods of assessmentUse multiple methods of assessment

Characteristics of Effective Performance Measures Relate to goalsRelate to goals Focus on the vital few elements to measureFocus on the vital few elements to measure Foster improvementFoster improvement Are well communicatedAre well communicated Are reviewed as often as appropriateAre reviewed as often as appropriate Provide information on level, trend and comparative/ competitive dataProvide information on level, trend and comparative/ competitive data Focus on the long-term well-being of the student and the programFocus on the long-term well-being of the student and the program Adapted from Engelkemeyer, 1998

Measurement Blockers Fuzzy goals or action strategiesFuzzy goals or action strategies Unjustified trust in informal feedback systemsUnjustified trust in informal feedback systems Entrenched measurement systemsEntrenched measurement systems Incorrect focusIncorrect focus No agreement on prioritiesNo agreement on priorities Adapted from Engelkemeyer 1998

Collect, analyze and interpret data What did you find?What did you find? What were the program’s strengths and weaknesses?What were the program’s strengths and weaknesses? How well are the students learning the concepts?How well are the students learning the concepts? Is the learning achieved appropriate for the level?Is the learning achieved appropriate for the level? Is the learning being appropriately applied?Is the learning being appropriately applied?

National Center on Postsecondary Teaching, Learning, & Assessment

Identify Strategies for Change What will you do to improve student learning?What will you do to improve student learning? Which program elements should be reinforced?Which program elements should be reinforced? Which program elements should be maintained?Which program elements should be maintained? Which program elements should be strengthened?Which program elements should be strengthened? Which program elements should be modified?Which program elements should be modified? –at the undergraduate level? –at the graduate level? –off campus?

Be flexible, adaptive and prepared to change. There will always be problems.There will always be problems. Things always change (mandates, circumstance, personnel, priorities.)Things always change (mandates, circumstance, personnel, priorities.) View assessment as an evolutionary process.View assessment as an evolutionary process. (Seybert, 1989)

References The Assessment Institute Resource Book. Assessment Institute. National Center for Postsecondary Teaching, Learning and Assessment, Burlington, VT. October 15-17, Ball State University. Assessment Workbook. Munci: Ball State University Offices of Academic Assessment, Black, Karen E., Trudy W. Banta, and Jane L. Lambert. “Best Practices in Program Review.” AAHE Conference on Assessment. American Association of Higher Education, Cincinnati, OH. June 14, Bloom, Benjamin, et al. (ed) Taxonomy of Educational Objectives: Handbook I. Cognitive Domain. NY: David McKay, 1956 Ewell, Peter T., ed. “Assessing Educational Outcomes.” New Directions for Institutional Research 47 San Fransisco: Jossey-Bass, Sept, Halpern, Diane F., ed. “Student Outcomes Assessment: What Institutions Stand to Gain.” New Directions for Higher Education 59 San Fransisco: Jossey-Bass, Fall, 1987.

References Cont. Idaho State Board of Education. Governing Policies and Procedures. Boise: Idaho State Board of Education, Engelkemeyer, Susan West. “Key Measures in Organizational Performance.” AAHE Conference on Assessment. American Association of Higher Education, Cincinnati, OH. June 13, Wright, Barbara D. “Assessment for Beginners: Getting Started.” AAHE Conference on Assessment. American Association of Higher Education, Miami Beach, FL. June 11, Seybert, Jeffrey A. “Community College Assessment Strategies.” AAHE Conference on Assessment. American Association of Higher Education, Cincinnati, OH. June 14, Sims, Serbrenia J. Student Outcomes Assessment: A Historical Review and Guide to Program Development. Connecticut; Greenwood Press, Sims, Serbrenia J. Student Outcomes Assessment: A Historical Review and Guide to Program Development. Connecticut; Greenwood Press, 1992.

NEED MORE INFORMATION? Archie GeorgeJane Baillargeon