1. UMCP Educational Technology Outreach On-line Course Assessment and Evaluation Model 2006 MICCA Conference Baltimore, MD May 4, 2006 University of Maryland Educational Technology Policy, Research and Outreach Director: Davina Pruitt-Mentle Copyright Davina Pruitt-Mentle 2006. This work is the intellectual property of the author. Permission is granted for this material to be shared for non-commercial, educational purposes, provided that this copyright statement appears on the reproduced materials and notice is given that the copying is by permission of the author. To disseminate otherwise or to republish requires written permission from the author.

2. MICCA 2006 (c) Davina Pruitt-MentleMay 4, 2006 2 Statement of the Problem Technologically mediated instruction offered at a distance has rapidly become an important component of higher education. However, faculty/instructors often find it difficult to assess student learning and outcomes when they have little, if any face to face interaction with participants.

3. MICCA 2006 (c) Davina Pruitt-MentleMay 4, 2006 3 Statement of the Problem While student assessment through distance learning poses a challenge, a focus on interactivity, design strategies that help decrease student anxiety level, and multi-pronged activities and assessment measures can provide unique opportunities for detailed feedback to learners.

4. MICCA 2006 (c) Davina Pruitt-MentleMay 4, 2006 4 Objectives Discussion of lessons learned from the evolution of the development and delivery of the University of Maryland, College of Education, Educational Technology Outreach On-line Course Assessment and Evaluation Model.

5. MICCA 2006 (c) Davina Pruitt-MentleMay 4, 2006 5 ETO Online Course Assessment and Evaluation Model 1.How will our courses provide evidence of student content knowledge and skill competency to meet the NCATE review board requirements? and 2.Have the course content and activities actually changed participant behaviors in the classroom/workplace? Resulted from two overarching questions: NCATENCATE=National Council on Accreditation of Teacher Education

6. MICCA 2006 (c) Davina Pruitt-MentleMay 4, 2006 6 Evaluation Strategy Conceptual framework based on Kirkpatrick's Training Model (level III) Assessment and evaluations of the on-line courses are systematic and ongoing Are used to improve the content and delivery of the on-going and future courses Both formative and summative evaluation protocols and instruments are used to guide the content and delivery

7. On-line Course Assessment and Evaluation Model

8. MICCA 2006 (c) Davina Pruitt-MentleMay 4, 2006 8 All Courses Include An on-line pre-profile “assessment” Focus on Content “knowledge” Also includes “technology knowledge” Done within WebCT

9. MICCA 2006 (c) Davina Pruitt-MentleMay 4, 2006 9 PT3 Pre-Assessment Survey PT3 Pre-Assessment Survey Web-based also available

10. MICCA 2006 (c) Davina Pruitt-MentleMay 4, 2006 10 All Courses Include An introduction discussion thread - helps the instructor find out more about the particular interest and background of the participants While this is common in on-line courses-the instructors do use this information to help assign “groups” and to modify the content

11. MICCA 2006 (c) Davina Pruitt-MentleMay 4, 2006 11 All Courses Include Throughout the course real time chat room dialogue allows participants to express comments and concerns regarding the course First Chat scheduled in Week 3-5 Supports research (Carnegie-Mellon, 2006) which indicates the best time for early course evaluations should be done in week 3-5. http://www.cmu.edu/teaching/assessment/earlycou rseeval.html http://www.cmu.edu/teaching/assessment/earlycou rseeval.html

12. MICCA 2006 (c) Davina Pruitt-MentleMay 4, 2006 12 Online Chats

13. MICCA 2006 (c) Davina Pruitt-MentleMay 4, 2006 13 Formative Content Knowledge Assessments On-going Include a variety of formats –on-line discussion threads –mini-assignments –group work –chat room discussions –projects and papers –on-line self tests and quizzes –graded quizzes and tests

14. MICCA 2006 (c) Davina Pruitt-MentleMay 4, 2006 14 Formative Content Knowledge Assessments

15. MICCA 2006 (c) Davina Pruitt-MentleMay 4, 2006 15 Importance of Feedback Feedback gathered from assignments has helped in the redesign of course activities (both in the on-going course and in redesign of the same course offered later)

16. MICCA 2006 (c) Davina Pruitt-MentleMay 4, 2006 16 Multiple Summative Evaluations Multiple summative evaluations are also used. –the post-profile assessment, –the University’s evaluation instrument which has been recreated in the online format and can be submitted anonymously, –Two (now one) open ended discussion questions and –Completion of the How I as a Student Meet Standards MatrixHow I as a Student Meet Standards Matrix

17. MICCA 2006 (c) Davina Pruitt-MentleMay 4, 2006 17 Post-Profile Assessment

18. MICCA 2006 (c) Davina Pruitt-MentleMay 4, 2006 18 Post-Profile Assessment

19. MICCA 2006 (c) Davina Pruitt-MentleMay 4, 2006 19 Post-Profile Assessment

20. MICCA 2006 (c) Davina Pruitt-MentleMay 4, 2006 20 University’s Evaluation Instrument University’s Evaluation Instrument

21. MICCA 2006 (c) Davina Pruitt-MentleMay 4, 2006 21 University Based Course Evaluation

22. MICCA 2006 (c) Davina Pruitt-MentleMay 4, 2006 22 Open Ended Discussion Questions Subject: Experience Reflection Message no. 378 Author: Davina S. Pruitt-Mentle (EDUC477)Date: Sunday, November 30, 2003 9:50pm Please share your experiences with this course this semester. What did you expect? What did you get out of the course? What will you remember months from now? (you can either post to the discussion thread, or send an email to dp151@umail.umd.edu). dp151@umail.umd.edu Please give a critical analysis of this course, course content, course experience (including the online format). What formatting features in the instructional design would make it easier to navigate (given the WebCT format)? How many "discussions, mini exercises" etc… do you feel are appropriate each week? (remember this is a graduate course and we do not meet the 3 full hours F2F or online each week). Did you prefer the assignments separate from the discussion or embedded within the discussion section? Please indicate anything else that would be helpful towards presenting this information to others. It is understood that everyone here is an adult and professional and therefore critical analysis is appreciated and in no means will be reflected in the grade. (You can either post to the discussion thread, or send an email to dp151@umail.umd.edu) dp151@umail.umd.edu

23. MICCA 2006 (c) Davina Pruitt-MentleMay 4, 2006 23 Open Ended Discussion Questions

24. MICCA 2006 (c) Davina Pruitt-MentleMay 4, 2006 24 How I as a Student Meet Standards Matrix How I as a Student Meet Standards Matrix

25. MICCA 2006 (c) Davina Pruitt-MentleMay 4, 2006 25 Changed Behaviors? To answer the second question, “Have the course content and activities actually changed participant behaviors in the classroom/workplace”? ETO uses follow-up surveys to see if and how students have continued to apply the knowledge/skills after course conclusion. This is done 1-2 semesters after completion of the course (via email).

26. MICCA 2006 (c) Davina Pruitt-MentleMay 4, 2006 26 Pre and Post Assessment Profiles of Two Selected Courses

27. MICCA 2006 (c) Davina Pruitt-MentleMay 4, 2006 27 Post-Profile Assessment Percent of teachers using Excel functions before and after course content

28. MICCA 2006 (c) Davina Pruitt-MentleMay 4, 2006 28 Post-Profile Assessment Sample comments from participants asked, “Have you utilized what you learned in EDUC 698V and if so how. Please give examples…” –I presented a workshop within my school on how to use the Data filtering feature –I taught my mentor teacher how to use the red flagging feature –I set up my grade book downloading the MSDE indicators… –I used the Chart Wizard to create a visual for a parent conference –I taught my students how to use Excel to sum/ average Artifacts were also collected

29. MICCA 2006 (c) Davina Pruitt-MentleMay 4, 2006 29 Comparison of Evaluation Models Pre-ETO Model Pre-Profile Introduction: Ice Breaker Chat-week 3- 5/Content Formative Assessments/ Assignments Post-Profile University Evaluation ETO Model Pre-Profile Introduction: Ice Breaker + Student Interest & Expertise Chat-week 3-5/Course Evaluation/Concerns/ Suggestions + Content Formative Assessments/ Assignments –flexible based on students background and interest Post-Profile University Evaluation + Course Evaluation/open ended discussion Post Course Follow-Up

30. MICCA 2006 (c) Davina Pruitt-MentleMay 4, 2006 30 End slide Contact Information: Davina Pruitt-Mentle Director: Educational Technology Policy, Research and OutreachEducational Technology Policy, Research and Outreach University of Maryland 2127 TAWES College Park, MD 20742 (301) 405-8202 dpruitt@umd.eduQuestions?