1. © E. Kowch 2002 1 iD Instructional Design Evaluation, Assessment & Design: A Discussion (EDER 673 L.91 ) From Calgary With Asst. Professor Eugene G. Kowch march 27 (An Synchronous Meeting using WebCT discussion Thread and WWW Course Home Page Material)

2. © E. Kowch 2002 2 iD Agenda 1. Housekeeping 1. Timetable 2. Course Objectives 3. Blueprint Guidelines (online) 4. ID Model Guidelines (online) 2. Assessment and Evaluation - a Quick Overview 3. Discussion

3. © E. Kowch 2002 3 iD Timetable Keep an eye on the home page :-) Date (Class)TopicReadings/ Assts. Due March 27 (Vclass)Evaluation Rubrics Evaluation Review Lickona / nothing due April 03 (WebCT)Ethics & Values, Leadership Issues Optional online article / Blueprint Due April 10 (Vclass)Student Instruction Blueprint Presentations None / Present your Instruction Blueprint per guide April 17 (Vclass)Student ID Model Presentations None / Present your ID Model per guide

4. © E. Kowch 2002 4 iD Update: EDER 673 History of ID ID Terminology Instructional Design Philosophies Learners and Learning Theories Context based designs ID Models: A peek Needs Analysis Task Analysis Ordering Content (elaboration) Media Selection Motivation Ethics/ Ldrship SMCR/Feedback Communication Model Evaluation

5. © E. Kowch 2002 5 iD Course Objectives: Harvest Time 1. Explore theories of learning and instruction, exploring implications and possible applications for their own practice, 2. Explore, develop and articulate their own instructional design models, to see how theory and practice can be articulated, 3. Design and present efficient, effective and appealing instructional interventions informed and referenced to theory, 4. Participate and present in a collaborative learning community to exchange and consider developing design ideas as instructional designers, 5. Examine the potential and limitations of media and technology use 6. Analyze some models critically and instructional interventions associated with the selection and sequencing of content across the instructional spectrum and 7. Explore and analyze the matching of instructional strategies to characteristics of learners and content.

6. © E. Kowch 2002 6 iD Blueprint Guidelines I. Approach: ism II. Introduction: State the purpose of this instruction. III. Audience: State the intended audience for your report IV. The Performance Problem: Explain the gap you will fill by your designed instruction. V. Learners: Provide a description of the learners for whom this instruction is intended. VI. Objectives: State the learning outcomes that you desire from this 30 minute instructional module. VII. Scope and Sequence: Describe the decision making process you went through VIII. Indicate optimal and minimal requirements for media and technology IX. Indicate the instructional flow / process X. Conclusion

7. © E. Kowch 2002 7 iD Blueprint Guidelines Grading Method for this project: Format: did the student follow the requested blueprint format? 10% Content: ID Model Application: did the student include and explain how the instructional blueprint uses the student's personal ID model? 30% Is the performance problem clearly identified? 10% Clarity & Coherence: is the report clear and legible/logical? 20% Scholarly / Theoretical basis: Are key models and theories cited? 10% Conclusion: Does the conclusion explain the blueprint to someone who might contract the designer for exactly this instruction? 20% total: 100%

8. © E. Kowch 2002 8 iD Blueprint Presentation Guidelines: 6 elements

9. © E. Kowch 2002 9 iD Blueprint Presentation Guidelines: 6 elements ElementsDoes not meet requirements (B-) Meets Partial Requirements (B to B+) Meets Full Requirements (A- to A+) 1. Model application Does not present model or presentation is incomplete. Presentation and explanation of the model are satisfactory or adequate. Presentation and explanation of the model are superior.

10. © E. Kowch 2002 10 iD Blueprint Presentation Guidelines: 6 elements

11. © E. Kowch 2002 11 iD Blueprint Presentation Guidelines: 6 elements IntegrationPresentation does not include integration of model, instruction, and/or theories. Presentation includes integration of model, and instructional unit Presentation includes integration of model, instructional unit, and theories.

12. © E. Kowch 2002 12 iD Blueprint Presentation Guidelines: 6 elements

13. © E. Kowch 2002 13 iD Blueprint Presentation Guidelines: 6 elements

14. © E. Kowch 2002 14 iD Blueprint Presentation Guidelines: 6 elements See the bottom of the home page for guidelines

15. © E. Kowch 2002 15 iD ID Model Guidelines 1. Informed: Do theories underpin the model? 2. Referenced: Are theorists referenced/understood? (APA not needed) 3. Relationships: Between Elements are indicated/explained 4. Granularity: Visual Model addresses most elements of models covered in EDER 673… 5. Missing Elements: A rationale is provided for element exclusion in your model 6. Limitations of the ID Model: Are defined 7. Generalizability: Evidence or explanation of use of this model in 1. 2 different content areas 8. Scalability: Can I understand the lesson and the program design via your model? 9. Innovation: Uniqueness - do you explain a new approach?

16. © E. Kowch 2002 16 iD ID Model Guidelines

17. © E. Kowch 2002 17 iD ID Model Guidelines

18. © E. Kowch 2002 18 iD ID Model Guidelines

19. © E. Kowch 2002 19 iD ID Model Guidelines

20. © E. Kowch 2002 20 iD ID Model Guidelines

21. © E. Kowch 2002 21 iD ID Model Guidelines

22. © E. Kowch 2002 22 iD ID Model Guidelines

23. © E. Kowch 2002 23 iD ID Model Guidelines

24. © E. Kowch 2002 24 iD ID Model Guidelines

25. © E. Kowch 2002 25 iD ID model Presentation (April 17)Guidelines. ElementsDoes not meet requirements (B-) Meets Partial Requirements (B to B+) Meets Full Requirements (A- to A+) ModelDoes not present model in Power Point or presentation is incomplete for VClass Presentation and explanation of the model are satisfactory or adequate, Powerpoints are clear and file is smaller than 300 KB Presentation and explanation of the model are superior, and meets Vclass Time Management Is over 5 minutes5 minutes in length but file upload is not ready/prepared 5 minutes in length

26. © E. Kowch 2002 26 iD ID model Presentation (April 17)Guidelines.

27. © E. Kowch 2002 27 iD ID model Presentation (April 17)Guidelines. Load file B here

28. © E. Kowch 2002 28 iD Instructional Design Evaluation, Assessment & Design: A Discussion (EDER 673 L.91 ) From Calgary With Asst. Professor Eugene G. Kowch march 27 (An Synchronous Meeting using WebCT discussion Thread and WWW Course Home Page Material)

29. © E. Kowch 2002 29 iD Assessment Instruments: Assessing learners Criterion Referenced Pretest: Placement & entry level skills Readiness Questions Placement Questions Posttest: helps us identify what did not work for this learner --------------------- To make these tests, we need a lot of info: List of task based behavior objectives or Type of learning (verbal, etc, ) and Type of conditions for learning (Gagne) and events for learning Bad criteria test subjects: Writing a paragraph…affective evaluations…..

30. © E. Kowch 2002 30 iD Placement Learner Readiness Assessment (to determine entry performance) Readiness Concern Do the students have the Prerequisite skills? Pretest: Method of Inquiry for Placement no yes Provide readiness experiences Proceed with The instruction Placement Concern Have the students already Achieved the intended Outcomes? Advance Students To a higher level no yes Discuss: Math vs. Social Studies (criterion?) From Driscoll, M. (2000). Psychology for Instruction, 2nd Ed.

31. © E. Kowch 2002 31 iD Formative and Summative Assessments: ( of your Design) Formative methodology (can be done in order): 1. Education Context: 1 to 1 evaluation (designer & learner) 1. Questionnaires, interviews - check learning time, outcomes 2. Focus group 1. Nets out the instructor 2. Check the effectiveness of instruction (viability, attitudes of instr.) 3. Field Trial 1. Observe a full instruction event with the 2x modified course materials and design… test for achievement.

32. © E. Kowch 2002 32 iD Formative and Summative Assessments: ( of your Design) Summative Assessment methodology 1. Expert Judgment (assess the instructor) 1. Congruence; Can the instructor meet the needs of the organization? 2. Content: Are the materials complete, updated? 3. Instructor followed the design? 4. Utility of instructor? 5. Design: Are the principles of learning, instruction and motivation in the materials? Instruction? 2. Field Trial 1. Outcomes Analysis: impact to learner on job, in school 2. Management: Instructor, supervisor performance test

33. © E. Kowch 2002 33 iD Summative Assessment (to monitor learning progress) Provide additional learning experiences no yes Assign grades to certify mastery Discuss Have the students achieved The intended outcomes Of instruction? Summative Learner Assmt. Method Evaluate the effectiveness of the instruction

34. © E. Kowch 2002 34 iD Assessment Planning (Magliaro from notes) Assessment: Learner performance Evaluation: judging learner decisions Program Evaluation: Appraising program success Assessment Possibilities Portfolios Physical Electronic Communicate task and knowledge performance Time consuming for all! Projects Demonstrate action / processes / competencies (training) Address problem solving (education) Require relevant subject matter!

35. © E. Kowch 2002 35 iD Assessment Planning (Magliaro from notes) 5 Step assessment process 1. Identify goals and objectives from Needs Assmt 2. Match goals & objectives with assmt tools 3. Choose instructional method 4. Design tasks and scoring methods (match) 5. Think through the implications…. Guidelines: 1. Start small 2. Develop rubrics 3. Communicate purpose of assessment 4. Involve participants 5. Keep going - recursive design 6. Look for blips - unfairness, odd results

36. © E. Kowch 2002 36 iD PostTest: Method of Inquiry for Placement The instructional role of Formative Assessment (to monitor learning progress) Provide Group or Individual remediation no yes DIAGNOSTIC ASSESSMENT (to study persistent difficulties Provide feedback to reinforce learning Proceed with The planned instruction Discuss During Instruction Are the students achieving the intended Learning outcomes?

37. © E. Kowch 2002 37 iD Recall: The “one” and the “many”

38. © E. Kowch 2002 38 iD The Distribution Curve: Defining Normal Norm Relative ranking, describes what cannot be done And what can be done..

39. © E. Kowch 2002 39 iD Great questions: "Can and should instructional designers place importance on process instead of product? If yes, is this desirable and reversible? Discuss" Designers Ask..

40. © E. Kowch 2002 40 iD Great questions: Should and can evaluation be both internal and external? Should and can the evaluator personalize his or her findings and put those results into a cultural perspective? Discuss." Designers Ask..

41. © E. Kowch 2002 41 iD 1) There is the sense that "outside the company" designers are not as biased as "backyard" designers. I wonder whether this is in fact true, or whether their biases are simply different. The "outside" designers could have biases that are harmful or just as blinding as "in house" people's. Designers Ask..

42. © E. Kowch 2002 42 iD Evaluation/Assessment There seems to be a belief that objectivity is somehow better than subjectivity, and I wonder about that too. Sometimes, a subjective approach may be called for. For example, in evaluating a Native education program, would it be effective to use a white person? Maybe. Then again, maybe not. Maybe the white person would have to spend so much time and effort learning first about the culture that her work on actual evaluation would become miniscule in comparison. Designers Ask..