1. MAINTAINING QUALITY IN BLENDED LEARNING: FROM CLASSROOM ASSESSMENT TO IMPACT EVALUATION PART I: DESIGNING AND EVALUATING LEARNING Suzanne Weinstein, Ph.D. Director of Instructional Consulting, Assessment, and Research The Schreyer Institute for Teaching Excellence The Pennsylvania State University

2. Part I Session Objectives As a result of this workshop, participants will be able to…  …write a student-centered, measureable learning objective  …map learning objectives to appropriate assessments  …use course-level evaluation data for program assessment

3. But first, some definitions

4. Assessment “Assessment is the ongoing process of establishing  clear, measurable expected outcomes of student learning, Suskie, L. (2009) Assessing student learning: A common sense guide. San Francisco: Jossey-Bass.

5. Assessment “Assessment is the ongoing process of establishing  clear, measurable expected outcomes of student learning,  ensuring that students have sufficient opportunities to achieve those outcomes, Suskie, L. (2009) Assessing student learning: A common sense guide. San Francisco: Jossey-Bass.

6. Assessment “Assessment is the ongoing process of establishing  clear, measurable expected outcomes of student learning,  ensuring that students have sufficient opportunities to achieve those outcomes,  systematically gathering, analyzing, and interpreting evidence to determine how well students learning matches the expectations and Suskie, L. (2009) Assessing student learning: A common sense guide. San Francisco: Jossey-Bass.

7. Assessment “Assessment is the ongoing process of establishing  clear, measurable expected outcomes of student learning,  ensuring that students have sufficient opportunities to achieve those outcomes,  systematically gathering, analyzing, and interpreting evidence to determine how well students learning matches the expectations and  using the resulting information to understand and IMPROVE student learning.” Suskie, L. (2009) Assessing student learning: A common sense guide. San Francisco: Jossey-Bass.

8. Evaluation “Evaluation is the analysis and use of data by faculty to make judgments about student performance. Evaluation includes the determination of a grade or a decision regarding pass/fail for an individual assignment or for a course.” Goldman and Zakel (2009)

9. A blended or hybrid course is one in which some face-to-face class periods are replaced with online activities +

10. Designing learning in a blended course

11. Designing learning in a (blended) course 1. Learning Objective 2. Assignment 3. Learning Activities Wiggins and McTighe (1998), Understanding by Design http://pixel.fhda.edu/id/six_facets.htmlhttp://pixel.fhda.edu/id/six_facets.html

12. Designing learning in a (blended) course 1. Learning Objective 2. Assignment 3. Learning Activities Face-to -face or Online? Face-to -face or Online?

13. Designing learning in a (blended) course 1. Learning Objective 2. Assignment 3. Learning Activities Face-to -face or Online? Face-to -face or Online? Integrate face-to-face and online elements

14. Designing learning in a (blended) course Learning Objective At the end of this course, students will be able to evaluate an empirical research article Assignment Team project: Find an empirical article on a topic of personal interest and collaboratively evaluate the article using a wiki Learning Activities a. Students will read an article b. Instructor will model an article critique in class http://pixel.fhda.edu/id/six_facets.html

15. Learning Objective Write a specific, learner-centered, measureable objective Taxonomy Match the objective to the closest equivalent of Bloom’s taxonomy Evaluation How you will evaluate the students on this objective? What is the best format for the evaluation? Teaching Strategies What teaching activity you will use to prepare students to meet objective? What is the best format for the teaching strategies? Face-to-face/Online connection How are the face-to- face activities connected to the online activities? At the end of this course, students will be able to evaluate an empirical research article. EvaluatingTeam project: Find an empirical article on a topic of interest and collaborate to evaluate the article using a wiki. Instructor will model an article critique (in class). Instructor modeling the critique links to students practicing the critique in the online environment. Adapted from Synthesis Worksheet: http://www.schreyerinstitute.psu.edu/pdf/Synthesis_Worksheet.pdfhttp://www.schreyerinstitute.psu.edu/pdf/Synthesis_Worksheet.pdf

16. Creating Learning Objectives

17. Identify what the learner should know or be able to do by the end of the course A learning objective should…  …focus on the learner  …be observable (i.e. measurable) http://www.schreyerinstitute.psu.edu/Tools/LearningObj/

18. Adapted from Eileen Herteis, The Gwenna Moss Teaching & Learning Centre Use Bloom’s taxonomy (1956) to develop learning objectives at various levels of learning Evaluation Synthesis Analysis Application Comprehension Knowledge

19. Adapted from Eileen Herteis, The Gwenna Moss Teaching & Learning Centre If you prefer, you can also use Krathwohl and Anderson version of the taxonomy Creating Evaluating Analyzing Applying Understanding Remembering

20. Chemistry Learning Objectives Students will be able to:  Identify the chemical deficiencies of the soil for a crop.  Distinguish between different types of chemicals based on the chemical deficiencies of the soil for a crop.  Develop a solution for the soil deficiencies in a proposal geared towards a business audience.

21. Evaluate the following learning objective 1. In this course you will understand the 3 major communication disorders.

22. Evaluate the following learning objective 2. In this course we will cover the religion of ancient Egypt.

23. Mapping Objectives to Assessment Strategies (mere examples) http://edorigami.wikispaces.com/Bloom%27s+-+Introduction TraditionalElectronic Remember Multiple choice or matching questions Online quiz or fact-based game Understand Multiple choice or Matching questions Essay summary Online quiz or fact-based game Flow chart Concept map ApplyMultiple choice questions Calculate the answer to a problem Demonstrate a procedure Online quiz Do an online search Make a podcast that examines a theory

24. Mapping Objectives to Assessment Strategies (mere examples) http://edorigami.wikispaces.com/Bloom%27s+-+Introduction TraditionalElectronic Analyze Distinguish between 2 theories Analyze data Reverse engineer a device Conduct and report on a web survey Collect and analyze media clips Contrast 2 blog posts Evaluate Evaluate an writing sample Appraise a research article Judge a marketing plan Moderate a discussion forum Evaluate a video or debate Peer evaluation CreateGenerate an expository essay Formulate a research plan Produce and deliver a presentation Generate an e- Portfolio Create a video Build a website

25. One last, but not least, important consideration: Rubrics http://edorigami.wikispaces.com/Rubrics+-+Bloom%27s+Digital+Taxonomy Rubrics for digital assignments

26. Electronic assessment examples History course objective Students will be able to create personal histories that link to specific historical times. Family e-Album

27. Electronic assessment examples Information Science and Technology Students will be able to collaboratively develop summaries of research related to online games and virtual worlds. IST 440 Research Project

28. Give it a try Go to the following link. It will take you to a google form. The questions will walk you through the process of creating the appropriate assignment for a particular objective. https://spreadsheets.google.com/viewform?formkey =dDZSbS1XOHI4ZngtMzZLNno4ZkVnWHc6MQ

29. Let’s see the results  Blended Learning Workshop Activity - Spreadsheet Blended Learning Workshop Activity - Spreadsheet

30. Use course-level assignments to measure program goals Psychology Program Goal Students will be able to understand and interpret psychological data, graphs and statistics. Course Objective Students will be able to correctly interpret a correlation coefficient.

31. Performance by student

32. Performance by learning objective

33. Resources  Hybrid Courses, University of Wisconsin at Milwaukee: http://www4.uwm.edu/ltc/hybrid/ http://www4.uwm.edu/ltc/hybrid/  Schreyer Institute for Teaching Excellence, Penn State: http://www.schreyerinstitute.psu.edu/ http://www.schreyerinstitute.psu.edu/  Bloom’s Digital Taxonomy: http://edorigami.wikispaces.com/Bloom%27s+-+Introduction  Bloom’s Digital Taxonomy- Rubrics http://edorigami.wikispaces.com/Rubrics+-Bloom%27s+Digital+Taxonomy  Evaluating Media Characteristics http://www.elearnspace.org/Articles/mediacharacteristics.htm

34. CONTACT INFORMATION Suzanne Weinstein sweinstein@psu.edu 814-863-0689 http://www.schreyerinstitute.psu.edu http://www.assess.psu.edu