1. Assessment Techniques for Online Courses Jean A. Derco, Ed.D. Julie K. Little, Ed. D. Division of Educational Technology and The Innovative Technology Center The University of Tennessee 5 November 2003 Jean A. Derco, Ed.D. Julie K. Little, Ed. D. Division of Educational Technology and The Innovative Technology Center The University of Tennessee 5 November 2003 Copyright Derco & Little, 2003. This work is the intellectual property of the author. Permission is granted for this material to be shared for non- commercial, educational purposes, provided that this copyright statement appears on the reproduced materials and notice is given that the copying is by permission of the author. To disseminate otherwise or to republish requires written permission from the author.

2. ©2003, Derco & Little. The University of Tennessee. What is Assessment? Assessment’s purpose is to improve the quality of student learning, not to provide evidence for evaluating or grading students. Consequently, many of the concerns that constrain testing do not apply. Assessment’s purpose is to improve the quality of student learning, not to provide evidence for evaluating or grading students. Consequently, many of the concerns that constrain testing do not apply.

3. ©2003, Derco & Little. The University of Tennessee. What is Assessment? (con’t.) Assessment’s aim is to provide faculty with information on what, how much, and how well students are learning, in order to help them better prepare to succeed - both on the subsequent graded evaluations and in the world beyond the classroom.

4. ©2003, Derco & Little. The University of Tennessee. Assessment vs. Evaluation Assessment is used to determine current understanding and improve subsequent learning. Evaluation is used to determine mastery of goals and objectives. Assessment is used to determine current understanding and improve subsequent learning. Evaluation is used to determine mastery of goals and objectives.

5. Taking Assessment Online Novice vs. Expert Students

6. ©2003, Derco & Little. The University of Tennessee. An Online Course with the Content and Technology Novice Course Level: Post-Masters Course Course Content: Using technology in education –1st half of semester dealt with skill building –2nd half dealt with technology application Participants: 18 K-12 teachers Most were novice technology users –Familiar with email Course Level: Post-Masters Course Course Content: Using technology in education –1st half of semester dealt with skill building –2nd half dealt with technology application Participants: 18 K-12 teachers Most were novice technology users –Familiar with email

7. ©2003, Derco & Little. The University of Tennessee. An Online Course with the Content and Technology Expert Course Level: Doctoral and Post-Masters Course Content: Principles of effective online teaching and learning 15 Participants: –12 Mid-program or ABD Doctoral candidates –3 Post-Masters K-12 and higher ed practitioners All were expert technology users Most were experienced with the content –2/3 had prior experience developing online course materials Course Level: Doctoral and Post-Masters Course Content: Principles of effective online teaching and learning 15 Participants: –12 Mid-program or ABD Doctoral candidates –3 Post-Masters K-12 and higher ed practitioners All were expert technology users Most were experienced with the content –2/3 had prior experience developing online course materials

8. ©2003, Derco & Little. The University of Tennessee. The “Novice/Expert” Matrix Novice Content Novice Technology Novice Content Expert Technology Expert Content Novice Technology Expert Content Expert Technology

9. Assessment Examples

10. ©2003, Derco & Little. The University of Tennessee. Background Knowledge Probes Short, simple questionnaires prepared by instructors for use at the beginning of a course, at the start of a new unit or lesson, or prior to introducing and important new topic. The purpose is to determine what students may already know about a new concept, subject or topic in the course syllabus. Short, simple questionnaires prepared by instructors for use at the beginning of a course, at the start of a new unit or lesson, or prior to introducing and important new topic. The purpose is to determine what students may already know about a new concept, subject or topic in the course syllabus.

11. ©2003, Derco & Little. The University of Tennessee. Background Knowledge Probes (assessing skill in knowledge, recall & understanding) Novice Content Instructor prepares a few short answer or 10–20 low- level multiple choice questions that will probe prior knowledge of a topic. Novice Content Instructor prepares a few short answer or 10–20 low- level multiple choice questions that will probe prior knowledge of a topic. ©2003, Derco & Little. The University of Tennessee. Novice Technology Students reply to the instructor by completing the BKP posted in the body of an email. Novice Technology Students reply to the instructor by completing the BKP posted in the body of an email. Expert Content Instructor prepares two or three open-ended questions, a few short answer, or 10–20 high-level multiple choice questions that will probe prior knowledge of a topic. Expert Content Instructor prepares two or three open-ended questions, a few short answer, or 10–20 high-level multiple choice questions that will probe prior knowledge of a topic. Expert Technology Students reply to the instructor by completing an attached word-processed BKP or complete via an online survey (html-, cms-, or stand alone tool- generated). Expert Technology Students reply to the instructor by completing an attached word-processed BKP or complete via an online survey (html-, cms-, or stand alone tool- generated).

12. ©2003, Derco & Little. The University of Tennessee. Analytic Memo Basically, this is a simulation exercise. It requires students to write a one- or two page analysis of a specific problem or issue. This short, structured writing assignment provides high-quality feedback on students’ analytic and writing skills as by- product of an intellectually challenging and realistic skill-building exercise. Basically, this is a simulation exercise. It requires students to write a one- or two page analysis of a specific problem or issue. This short, structured writing assignment provides high-quality feedback on students’ analytic and writing skills as by- product of an intellectually challenging and realistic skill-building exercise.

13. ©2003, Derco & Little. The University of Tennessee. Analytic Memo (assessing skill in analysis & critical thinking) Novice Content Instructor provides a problem or situation for students to analyze. Working in pairs, students write a memo to a specified audience that describes the problem, its probable cause, and an analysis of the implications of the problem. Novice Content Instructor provides a problem or situation for students to analyze. Working in pairs, students write a memo to a specified audience that describes the problem, its probable cause, and an analysis of the implications of the problem. ©2003, Derco & Little. The University of Tennessee. Novice Technology Students construct the analytic memo, post it to the instructor in the body of an email with a cc: to the other partner. Novice Technology Students construct the analytic memo, post it in the body of an email to a class distribution list, and conduct peer critiques through email replies. Expert Content Instructor provides a problem or situation for students to analyze. Students individually write a memo to a specified audience that describes the problem, its probable cause, and analysis of the implications of the problem. Expert Content Instructor provides a problem or situation for students to analyze. Students individually write a memo to a specified audience that describes the problem, its probable cause, and analysis of the implications of the problem. Expert Technology Students construct an analytic memo using a word- processor, post to the instructor as an email attachment, and cc: the other partner. Expert Technology Students construct an analytic memo using a word- processor, post it to a class discussion forum, and conduct peer critiques in the forum.

14. ©2003, Derco & Little. The University of Tennessee. Word Journal Prompts a two-part response. –First, the student summarizes a short text in a single word. –Second, the student writes a paragraph or two explaining why he or she chose that particular word to summarize the text. The completed response is an abstract or a synopsis of the focus text. Prompts a two-part response. –First, the student summarizes a short text in a single word. –Second, the student writes a paragraph or two explaining why he or she chose that particular word to summarize the text. The completed response is an abstract or a synopsis of the focus text.

15. ©2003, Derco & Little. The University of Tennessee. Word Journal (assessing skill in synthesis & creative thinking) Novice Content Students read an article provided by the instructor and create a word journal. Novice Content Students read an article provided by the instructor and create a word journal. ©2003, Derco & Little. The University of Tennessee. Novice Technology Students submit the word journal in an email to the instructor. Novice Technology Students submit the word journal in an email to a class distribution list. Expert Content Students select an article to read and create a word journal. Expert Content Students select an article to read and create the word journal. Expert Technology Students complete the word journal as a word- processed document and email as an attachment to the instructor or post to a web page and submit the url in an email to the instructor. Expert Technology Students complete the word journal as a word- processed document and email as an attachment to the instructor; post to a web page and submit the url to a class distribution list; post to a class discussion forum, or present in a shared whitespace.

16. ©2003, Derco & Little. The University of Tennessee. Annotated Portfolio Choose a central topic, question, or problem dealt with in the course and invite students to respond to it with two or three samples of their work that demonstrate creativity. Ask students to briefly explain how the pieces in their portfolio respond to the topic, question, or problem posed. – Provide several example annotations. Choose a central topic, question, or problem dealt with in the course and invite students to respond to it with two or three samples of their work that demonstrate creativity. Ask students to briefly explain how the pieces in their portfolio respond to the topic, question, or problem posed. – Provide several example annotations.

17. ©2003, Derco & Little. The University of Tennessee. Annotated Portfolio (assessing skill in synthesis & creative thinking) Novice Content Students select two digital or non-digital samples of their work and briefly explain how each addresses the problem posed. Novice Content Students select two digital samples of their work and briefly explain how each addresses the problem posed. ©2003, Derco & Little. The University of Tennessee. Novice Technology Students word-process their explanations and submit their work and annotations to the instructor in either hard copy format or on portable media via US mail. Novice Technology Students word-process their explanations and submit their work and annotations to the instructor in either hard copy format or on portable media via US mail. Expert Content Students select three digital or non-digital samples of their work and explain how each addresses the problem posed and demonstrates creativity. Expert Content Students select three digital samples of their work and explain how each addresses the problem posed and demonstrates creativity. Expert Technology Students create a web site with navigation to display their examples and annotations, and email the url to the instructor. Expert Technology Students create a web site with navigation to display their examples and annotations and email the url to the class distribution list.

18. ©2003, Derco & Little. The University of Tennessee. Problem Recognition Tasks Choose examples of several different, but related, problem types that students find difficult to distinguish. –Make sure that each example illustrates one, and only one, type of problem. Choose examples of several different, but related, problem types that students find difficult to distinguish. –Make sure that each example illustrates one, and only one, type of problem.

19. ©2003, Derco & Little. The University of Tennessee. Problem Recognition Tasks (assessing skill in problem solving) Novice Content Instructor provides students with mini-case examples and problem types and have them match types with examples. Novice Content Instructor provides students with mini-case examples and problem types to match types with examples. ©2003, Derco & Little. The University of Tennessee. Novice Technology Students email instructor with responses. Novice Technology Students email instructor with responses. Expert Content Instructor provides students with case examples and have them name the problem type for each example. Expert Content Instructor provides students with case examples and have them name the problem type for each example. Expert Technology Students complete this exercise using the matching or multiple choice options in an online quiz tool. Expert Technology Students complete this exercise using the fill-in-the- blank or short essay options in an online quiz tool.

20. ©2003, Derco & Little. The University of Tennessee. Student-generated Test Questions Allow faculty to assess at least three aspects of student learning: 1.what students consider the most important or memorable content, 2.what they understand as fair and useful test questions, and 3.how well they can answer the questions they have posed. Allow faculty to assess at least three aspects of student learning: 1.what students consider the most important or memorable content, 2.what they understand as fair and useful test questions, and 3.how well they can answer the questions they have posed.

21. ©2003, Derco & Little. The University of Tennessee. Student-generated Test Questions (assessing skill in application & performance) Novice Content Students create three multiple-choice questions with feedback for each possible answer. The feedback should include their rationale for why each incorrect response might be chosen as correct. Novice Content Students create three multiple-choice questions with feedback for each possible answer. The feedback should include their rationale for why each incorrect response might be chosen as correct. ©2003, Derco & Little. The University of Tennessee. Novice Technology Students email instructor with questions and feedback. Novice Technology Students email instructor with questions and feedback. Expert Content Students create three essay questions with the accompanying “A+” responses. Expert Content Students create three essay questions with the accompanying “A+” responses. Expert Technology Students use an online quiz tool (ie, Hot Potatoes) to generate their questions and feedback, and email the link to their instructor along with a word-processed file of the questions and feedback. Expert Technology Students use an online quiz tool (ie, Hot Potatoes) to generate their questions and feedback, and email the link to their instructor along with a word-processed file of the questions and feedback.

22. ©2003, Derco & Little. The University of Tennessee. Classroom Opinion Polls Help faculty discover student opinions about course-related issues –Uncovering student opinions on specific issues help faculty to better gauge where and how to begin teaching about those issues and potential roadblocks. Help faculty discover student opinions about course-related issues –Uncovering student opinions on specific issues help faculty to better gauge where and how to begin teaching about those issues and potential roadblocks.

23. ©2003, Derco & Little. The University of Tennessee. Classroom Opinion Polls (assessing student openness to new ideas) Novice Content Instructor posts a few problems or questions related to the course via email. Novice Content Instructor posts a few problems or questions related to the course in a class web-based discussion forum configured for anonymous postings. ©2003, Derco & Little. The University of Tennessee. Novice Technology Students reply to the instructor’s email with their opinions. Novice Technology Students reply to the instructor’s email with their opinions and react to other’s opinions via email reply. Expert Content Instructor emails class distribution list a few problems or questions related to the course. Expert Content Instructor posts a few problems or questions related to the course in a class web-based discussion forum configured for public postings. Expert Technology Students post their opinions to the anonymous discussion forum. Expert Technology Students post their opinions to the public discussion forum.

24. ©2003, Derco & Little. The University of Tennessee. We welcome your further discussion… Jean A. Derco [derco@utk.edu] Julie K. Little [jklittle@utk.edu] This presentation and the ideas generated today will be available: http://itc.utk.edu/assessment/educause03 after 10 November 2003. Jean A. Derco [derco@utk.edu] Julie K. Little [jklittle@utk.edu] This presentation and the ideas generated today will be available: http://itc.utk.edu/assessment/educause03 after 10 November 2003.