1. Effective Teaching: Documenting What Works March 2013 Part I: Measuring Student Learning Outcomes: 4 Easy Steps “Consequently, lectures simply reinforce students' feelings that the most important step in mastering the material is memorizing a zoo of apparently unrelated examples.” Eric Mazur Physics, Harvard University Facilitated by George Rehrey Indiana University Bloomington Center for Innovative Teaching and Learning grehrey@indiana.edu

2. By the end of this workshop you will be able to use the backward course design model to clearly articulate and measure the student learning outcomes for one of your courses. Today’s Goal

3. Acknowledgements The workshop is adopted from Indiana University’s Course Development Institute, designed, refined and implemented by these individuals: Eric Metzler (Kelley School of Business) Lisa Kurz (Center for Innovative Teaching and Learning) Ross Peterson-Veach (Goshen College)

4. Acknowledgements Indiana University's Course Development Institute builds upon the work of these authors: – Wiggins, G. P., & McTighe, J. – Fink, L. D. (2003) – Walvoord, B. E. F., & Anderson, V. J. (1998)

5. Agenda Write Course Goals in Measurable Language Create Authentic Assessments Articulate Learning Outcomes Break at 10:30

6. Knowledge, Skills and Attitudes Please take a few moments to complete the Knowledge, Skills and Attitudes Worksheet.

7. The Responsive Teaching and Learning Cycle Align course goals, assessments and learning outcomes Make visible the learning outcomes and grading criteria Collect evidence of student learning Reflect upon and analyze the results Share results of the reflective activity

8. Course Goals Final Assessment Learning Outcome Lectures, Class Activities, Readings, Homework Learning Outcome Lectures, Class Activities, Readings, Homework Learning Outcome Lectures, Class Activities, Readings, Homework Learning Outcome Lectures, Class Activities, Readings, Homework What must students be able to think and do to successfully complete the assessment? How will I know they have changed? How will my students be different? What must students be able to think and do to successfully complete the outcomes? Backward Course Design Model

9. Another way to look at it Traditional Method Create Final Assessment Identify Course Goals Create Final Assessment Identify Learning Outcomes Backward Model Create Syllabus and Lectures Identify Topics to cover in the course

10. Writing Internationalized Goals Using Bloom’s Revised Taxonomy Knowledge Dimension Anderson, Lorin W., David R. Krathwohl, and Benjamin Samuel Bloom. A Taxonomy for Learning, Teaching, and Assessing : A Revision of Bloom's Taxonomy of Educational Objectives. Abridged ed. New York: Longman 2001. Cognitive Process Dimension 1. Remember 2. Understand 3. Apply 4. Analyze 5. Evaluate 6. Create Knowledge Dimension A. Factual B. Conceptual C. Procedural D. Meta- Cognitive

11. Process CategoryDefinitionOther example verbs RememberRetrieve relevant knowledge from long term memory Retain, recall, recognize, memorize, recollect, think of, identify UnderstandConstruct meaning from instructional messages, oral, written, or graphic Interpret, exemplify, classify, summarize, infer, compare, explain, describe ApplyCarry out proceduresExecute, implement AnalyzeBreak material into parts and determine relationships Differentiate, organize, attribute EvaluateMake judgments based upon criteria or standards Check, critique, assess, judge, appraise CreatePut elements together into a coherent or functional whole, reorganize into new patterns Generate, plan, produce, invent Bloom’s Revised Taxonomy Process Definitions

12. Knowledge DimensionDefinitionExamples Factual KnowledgeBasic elements to be acquainted with the discipline and solve problems Terminology, vocabulary, elements and details Conceptual KnowledgeInterrelationship among basic elements Categories, principles, theories, models and structures Procedural KnowledgeHow to do something; methods of inquiry and criteria for using skills, techniques and methods Subject-specific skills, methods of inquiry and criteria for using, techniques and procedures Meta-cognitive Knowledge Knowing about knowingStrategic knowledge, reflections and self- knowledge Bloom’s Revised Taxonomy Knowledge Definitions

13. Human Anatomy Goal: “Students will be able to identify any muscle, bone, and organ of the human body as well as their constituent parts by their Latin names.” Cognitive Process Dimension 1. Remember 2. Understand 3. Apply 4. Analyze 5. Evaluate 6. Create Knowledge Dimension A. Factual X B. Conceptual C. Procedural D. Meta- Cognitive Course Goals in Measurable Language

14. Literary Theory Goal: “Students will be able to describe each of the five literary theories studied in this course (their major tenets and ideas) in a succinct paragraph.” Cognitive Process Dimension 1. Remember 2. Understand 3. Apply 4. Analyze 5. Evaluate 6. Create Knowledge Dimension A. Factual B. Conceptual X C. Procedural D. Meta- Cognitive Course Goals in Measurable Language

15. Business Goal: “Given a financial dilemma and an assortment of financial documents, students will be able to solve the dilemma and recommend the soundest financial decision to their superior.” Cognitive Process Dimension 1. Remember 2. Understand 3. Apply 4. Analyze 5. Evaluate 6. Create Knowledge Dimension A. Factual B. Conceptual X C. Procedural D. Meta- Cognitive Course Goals in Measurable Language

16. Determining learning outcomes  Know  Understand  Be exposed to  Learn  Appreciate By the end of this session, students should be able to... Measurable outcomes point to assessment and suggest classroom activities.  Explain  Articulate  Relate  Compare  Differentiate  Implement  Judge

17. Course Goals in Measurable Language Humanities: Hard to Measure: I want students to be exposed to the folklore of Indiana. Measurable: Students will be able to situate a given artifact of Indiana folklore in its historical, social, and economic context. Business: Hard to Measure: I want students to know how to complete a finance-related project efficiently when presented with a set of financial reports from their boss. Measurable: Given a financial dilemma and a sundry assortment of financial documents, students will be able to solve the dilemma and recommend the soundest financial decision to their superior.

18. Course Goals in Measurable Language Science: Hard to Measure: I want students to appreciate how species are related to each other and the implications of that for biology. Measurable: Students should be able to explain in their own words: how speciation gives rise to a branching pattern of relatedness among taxa the relationship between similarity and relatedness and why they are not all the same Science: Hard to Measure: I want students to understand how species are related to each other and the implications of that for biology. Measurable: Given a set of living organisms, students will be able to construct a Linnaean taxonomy by categorizing the characteristic traits of those organisms.

19. Please complete Part I of the Course Design Worksheet, describing KSAs as course goals written in measureable language.

20. What makes assessment authentic? An authentic assessment requires that students perform some real task, similar to what practitioners in the discipline perform.

21. Forward versus Backward (authentic) (inauthentic) An inauthentic assessment requires that students look back at what they may have learned. Arteries a. Are more elastic than veins b. Carry blood that is pumped from the heart c. Are less elastic than veins d. Both a and b e. Both b and c

22. Forward versus Backward (authentic) (inauthentic) An authentic assessment requires that students perform some real task, similar to what practitioners in the discipline perform. Imagine you are being asked to design an artificial artery: 1.Would it have to be elastic? 2.Why or why not ?

23. Course Goals: By the end of this class students will be able to apply theoretical principles to the stages of human development from conception to death, including physical, emotional, psychosocial, and cognitive influences. Students will be able to evaluate how cultural (internationalization) factors influence human development. Final Assessment: Students undertake a semester long project, writing 5 papers that discuss important aspects of the development of a “fictional person” in detail. One specific focus of the project will be on cultural influences on development. PSYCH 201

24. Please complete Part II of the Course Design Worksheet, describing an assessment you might use in order to have evidence that students have successfully achieved the course goals. At this point, think of projects and activities other than test questions.

25. Think Pair Share Please take one minute to share your course goals and final assessment.

26. Learning outcomes are the observable products and/or behaviors that demonstrate student learning. They are the knowledge, skills and attitudes students need to successfully complete the final assessment. Learning Outcomes:  Provide evidence of student learning  Clearly communicate teacher’s expectations in students’ terms  Allow students to monitor the progress throughout the semester  Provide a framework for course assignments  Serve as metaphorical buoys for the journey (your course). Learning Outcomes

27. Course Goals: Provided with an authentic financial problem (one that a business might typically face) and various financial documents, students will solve the problem and recommend the best course of action to their superior. Assessment: Students will write a professional memo, which is supported by appropriate financial documents, that recommends the best course of action to solve a given financial problem to a non- specialist.. Course: FIN 301 Working Capital Management

28. Learning Outcomes (in measurable terms): What must students be able to think and do to successfully complete the final assessment? Student Activities: What must students be able to think and do to successfully complete this learning outcome? Determine the relevance of the documents provided with respect to the given problem. Identify what information in the relevant documents pertains to the given problem. Research additional financial information to solve the given problem. With relevant financial information assembled, analyze numerical data by applying the appropriate metrics to them. Make a recommendation to their superior by writing a competent professional memo with appropriate supporting information. (Final Assessment)

29. Goals: By the end of this class students will be able to evaluate a country’s textile and apparel industry’s role in a globalized industry. Assessment: Students work in groups to create an Apparel Product Sourcing Proposal. Students will be assigned a region of the world and will develop an apparel product to be sourced to sell in the United States. Course: Textile and Apparel Trade

30. Course: AMID R404 International Textile and Apparel Trade Learning Outcomes (in measurable terms): What must students be able to think and do to successfully complete the final assessment? Student Learning: What must students be able to think and do to successfully complete this learning outcome? Compare and contrast where apparel and textile product is made and consumed and why that is the case. Use appropriate descriptors to characterize a country socially, economically, and politically. Identify and use appropriate data resources to compare apparel industries. Identify and compare national, regional and international structures that are involved in global trade in textiles and apparel. Argue the competing views of impact that textiles and apparel manufacturing and consumption has socially and economically.

31. Please complete Learning Outcomes column in Part III of the Course Design Worksheet, describing what students will have to think and do in order to successfully complete the assessment.

32. Think Pair Share Please take one minute to share your student learning outcomes.

33. Up the Mountain, Down the Mountain Before today’s workshop I used to think…. Now I think......

34. Bibliography for Course Design Angelo, T. A., & Cross P. K.(1993). Classroom assessment techniques: A handbook for college teachers. San Francisco, Calif.: Jossey-Bass. Fink, L. D. (2003). Designing significant learning experiences. San Francisco, Calif.: Jossey-Bass. Fink, L. D. (2005). Integrated Course Design. IDEA Paper #42 (March 2005). Manhattan, KS: The IDEA Center. Wiggins, G. P. (1998). Educative Assessment. San Francisco, Calif.: Jossey-Bass. Wiggins, G. P., & McTighe, J. (2005). Understanding by Design (Expanded 2nd ed.). Alexandria, VA: Association for Supervision and Curriculum Development Walvoord, B. E. F., & Anderson, V. J. (1998). Effective Grading: A Tool for Learning and Assessment (1st ed.). San Francisco, Calif.: Jossey- Bass Publishers.