© E. Kowch iD Instructional Design Evaluation, Assessment & Design: A Discussion (EDER 673 L.91 ) From Calgary With Asst. Professor Eugene G. Kowch march 27 (An Synchronous Meeting using WebCT discussion Thread and WWW Course Home Page Material)

© E. Kowch iD Agenda 1. Housekeeping 1. Timetable 2. Course Objectives 3. Blueprint Guidelines (online) 4. ID Model Guidelines (online) 2. Assessment and Evaluation - a Quick Overview 3. Discussion

© E. Kowch iD Timetable Keep an eye on the home page :-) Date (Class)TopicReadings/ Assts. Due March 27 (Vclass)Evaluation Rubrics Evaluation Review Lickona / nothing due April 03 (WebCT)Ethics & Values, Leadership Issues Optional online article / Blueprint Due April 10 (Vclass)Student Instruction Blueprint Presentations None / Present your Instruction Blueprint per guide April 17 (Vclass)Student ID Model Presentations None / Present your ID Model per guide

© E. Kowch iD Update: EDER 673 History of ID ID Terminology Instructional Design Philosophies Learners and Learning Theories Context based designs ID Models: A peek Needs Analysis Task Analysis Ordering Content (elaboration) Media Selection Motivation Ethics/ Ldrship SMCR/Feedback Communication Model Evaluation

© E. Kowch iD Course Objectives: Harvest Time 1. Explore theories of learning and instruction, exploring implications and possible applications for their own practice, 2. Explore, develop and articulate their own instructional design models, to see how theory and practice can be articulated, 3. Design and present efficient, effective and appealing instructional interventions informed and referenced to theory, 4. Participate and present in a collaborative learning community to exchange and consider developing design ideas as instructional designers, 5. Examine the potential and limitations of media and technology use 6. Analyze some models critically and instructional interventions associated with the selection and sequencing of content across the instructional spectrum and 7. Explore and analyze the matching of instructional strategies to characteristics of learners and content.

© E. Kowch iD Blueprint Guidelines I. Approach: ism II. Introduction: State the purpose of this instruction. III. Audience: State the intended audience for your report IV. The Performance Problem: Explain the gap you will fill by your designed instruction. V. Learners: Provide a description of the learners for whom this instruction is intended. VI. Objectives: State the learning outcomes that you desire from this 30 minute instructional module. VII. Scope and Sequence: Describe the decision making process you went through VIII. Indicate optimal and minimal requirements for media and technology IX. Indicate the instructional flow / process X. Conclusion

© E. Kowch iD Blueprint Guidelines Grading Method for this project: Format: did the student follow the requested blueprint format? 10% Content: ID Model Application: did the student include and explain how the instructional blueprint uses the student's personal ID model? 30% Is the performance problem clearly identified? 10% Clarity & Coherence: is the report clear and legible/logical? 20% Scholarly / Theoretical basis: Are key models and theories cited? 10% Conclusion: Does the conclusion explain the blueprint to someone who might contract the designer for exactly this instruction? 20% total: 100%

© E. Kowch iD Blueprint Presentation Guidelines: 6 elements

© E. Kowch iD Blueprint Presentation Guidelines: 6 elements ElementsDoes not meet requirements (B-) Meets Partial Requirements (B to B+) Meets Full Requirements (A- to A+) 1. Model application Does not present model or presentation is incomplete. Presentation and explanation of the model are satisfactory or adequate. Presentation and explanation of the model are superior.

© E. Kowch iD Blueprint Presentation Guidelines: 6 elements

© E. Kowch iD Blueprint Presentation Guidelines: 6 elements IntegrationPresentation does not include integration of model, instruction, and/or theories. Presentation includes integration of model, and instructional unit Presentation includes integration of model, instructional unit, and theories.

© E. Kowch iD Blueprint Presentation Guidelines: 6 elements

© E. Kowch iD Blueprint Presentation Guidelines: 6 elements

© E. Kowch iD Blueprint Presentation Guidelines: 6 elements See the bottom of the home page for guidelines

© E. Kowch iD ID Model Guidelines 1. Informed: Do theories underpin the model? 2. Referenced: Are theorists referenced/understood? (APA not needed) 3. Relationships: Between Elements are indicated/explained 4. Granularity: Visual Model addresses most elements of models covered in EDER 673… 5. Missing Elements: A rationale is provided for element exclusion in your model 6. Limitations of the ID Model: Are defined 7. Generalizability: Evidence or explanation of use of this model in 1. 2 different content areas 8. Scalability: Can I understand the lesson and the program design via your model? 9. Innovation: Uniqueness - do you explain a new approach?

© E. Kowch iD ID Model Guidelines

© E. Kowch iD ID Model Guidelines

© E. Kowch iD ID Model Guidelines

© E. Kowch iD ID Model Guidelines

© E. Kowch iD ID Model Guidelines

© E. Kowch iD ID Model Guidelines

© E. Kowch iD ID Model Guidelines

© E. Kowch iD ID Model Guidelines

© E. Kowch iD ID Model Guidelines

© E. Kowch iD ID model Presentation (April 17)Guidelines. ElementsDoes not meet requirements (B-) Meets Partial Requirements (B to B+) Meets Full Requirements (A- to A+) ModelDoes not present model in Power Point or presentation is incomplete for VClass Presentation and explanation of the model are satisfactory or adequate, Powerpoints are clear and file is smaller than 300 KB Presentation and explanation of the model are superior, and meets Vclass Time Management Is over 5 minutes5 minutes in length but file upload is not ready/prepared 5 minutes in length

© E. Kowch iD ID model Presentation (April 17)Guidelines.

© E. Kowch iD ID model Presentation (April 17)Guidelines. Load file B here

© E. Kowch iD Instructional Design Evaluation, Assessment & Design: A Discussion (EDER 673 L.91 ) From Calgary With Asst. Professor Eugene G. Kowch march 27 (An Synchronous Meeting using WebCT discussion Thread and WWW Course Home Page Material)

© E. Kowch iD Assessment Instruments: Assessing learners Criterion Referenced Pretest: Placement & entry level skills Readiness Questions Placement Questions Posttest: helps us identify what did not work for this learner To make these tests, we need a lot of info: List of task based behavior objectives or Type of learning (verbal, etc, ) and Type of conditions for learning (Gagne) and events for learning Bad criteria test subjects: Writing a paragraph…affective evaluations…..

© E. Kowch iD Placement Learner Readiness Assessment (to determine entry performance) Readiness Concern Do the students have the Prerequisite skills? Pretest: Method of Inquiry for Placement no yes Provide readiness experiences Proceed with The instruction Placement Concern Have the students already Achieved the intended Outcomes? Advance Students To a higher level no yes Discuss: Math vs. Social Studies (criterion?) From Driscoll, M. (2000). Psychology for Instruction, 2nd Ed.

© E. Kowch iD Formative and Summative Assessments: ( of your Design) Formative methodology (can be done in order): 1. Education Context: 1 to 1 evaluation (designer & learner) 1. Questionnaires, interviews - check learning time, outcomes 2. Focus group 1. Nets out the instructor 2. Check the effectiveness of instruction (viability, attitudes of instr.) 3. Field Trial 1. Observe a full instruction event with the 2x modified course materials and design… test for achievement.

© E. Kowch iD Formative and Summative Assessments: ( of your Design) Summative Assessment methodology 1. Expert Judgment (assess the instructor) 1. Congruence; Can the instructor meet the needs of the organization? 2. Content: Are the materials complete, updated? 3. Instructor followed the design? 4. Utility of instructor? 5. Design: Are the principles of learning, instruction and motivation in the materials? Instruction? 2. Field Trial 1. Outcomes Analysis: impact to learner on job, in school 2. Management: Instructor, supervisor performance test

© E. Kowch iD Summative Assessment (to monitor learning progress) Provide additional learning experiences no yes Assign grades to certify mastery Discuss Have the students achieved The intended outcomes Of instruction? Summative Learner Assmt. Method Evaluate the effectiveness of the instruction

© E. Kowch iD Assessment Planning (Magliaro from notes) Assessment: Learner performance Evaluation: judging learner decisions Program Evaluation: Appraising program success Assessment Possibilities Portfolios Physical Electronic Communicate task and knowledge performance Time consuming for all! Projects Demonstrate action / processes / competencies (training) Address problem solving (education) Require relevant subject matter!

© E. Kowch iD Assessment Planning (Magliaro from notes) 5 Step assessment process 1. Identify goals and objectives from Needs Assmt 2. Match goals & objectives with assmt tools 3. Choose instructional method 4. Design tasks and scoring methods (match) 5. Think through the implications…. Guidelines: 1. Start small 2. Develop rubrics 3. Communicate purpose of assessment 4. Involve participants 5. Keep going - recursive design 6. Look for blips - unfairness, odd results

© E. Kowch iD PostTest: Method of Inquiry for Placement The instructional role of Formative Assessment (to monitor learning progress) Provide Group or Individual remediation no yes DIAGNOSTIC ASSESSMENT (to study persistent difficulties Provide feedback to reinforce learning Proceed with The planned instruction Discuss During Instruction Are the students achieving the intended Learning outcomes?

© E. Kowch iD Recall: The “one” and the “many”

© E. Kowch iD The Distribution Curve: Defining Normal Norm Relative ranking, describes what cannot be done And what can be done..

© E. Kowch iD Great questions: "Can and should instructional designers place importance on process instead of product? If yes, is this desirable and reversible? Discuss" Designers Ask..

© E. Kowch iD Great questions: Should and can evaluation be both internal and external? Should and can the evaluator personalize his or her findings and put those results into a cultural perspective? Discuss." Designers Ask..

© E. Kowch iD 1) There is the sense that "outside the company" designers are not as biased as "backyard" designers. I wonder whether this is in fact true, or whether their biases are simply different. The "outside" designers could have biases that are harmful or just as blinding as "in house" people's. Designers Ask..

© E. Kowch iD Evaluation/Assessment There seems to be a belief that objectivity is somehow better than subjectivity, and I wonder about that too. Sometimes, a subjective approach may be called for. For example, in evaluating a Native education program, would it be effective to use a white person? Maybe. Then again, maybe not. Maybe the white person would have to spend so much time and effort learning first about the culture that her work on actual evaluation would become miniscule in comparison. Designers Ask..