1. Evaluating the New Technologies Ann Sefton Faculties of Medicine and Dentistry University of Sydney

2. Outline what is our educational purpose? levels of decision-making evaluating local strategies evaluating others’ “solutions” designing evaluations using IT for program evaluation teaching students to evaluate conclusions

3. Levels of decision-making have different implications institution faculty department unit of study individual staff member –whole program, unit of study, “lesson”? –undergrad, postgrad, continuing education?

4. Evaluation of local strategies against goals and expectations measuring learning processes comparative program improvement consider quality, costs, accessibility

5. Meeting goals: example values - degree program: student-centred, independent learning reflection and self-evaluation cooperation in groups evidence-based decision making effective skills –clinical –IT

6. Meeting goals: example goals - degree program basic and clinical science –critical reasoning for medical practice patient and doctor –effective communication, clinical skills community and doctor –community concerns and population issues personal & professional development –ethics/law, humanities, reflective practice

7. Meeting goals: learning package examples understand new concept(s) solve problems(s) learn a skill simulate an experiment access new information incl. images formative self-assessment

8. Evaluation of local strategies against goals and expectations understanding learning processes –evaluation vs research

9. Evaluation of local strategies against goals and expectations understanding learning processes –evaluation vs research comparative studies –evidence-based education, cohort studies –uncontrolled variables, ethical dilemmas

10. Evaluation of local strategies against goals and expectations understanding learning processes –evaluation vs research comparative studies –evidence-based education, cohort studies –uncontrolled variables, ethical dilemmas quality improvement –structures in place for revision

11. Evaluation of local strategies against goals and expectations understanding learning processes –evaluation vs research comparative studies –evidence-based education, cohort studies –uncontrolled variables, ethical dilemmas quality improvement –structures in place for revision consider quality, costs, accessibility

12. Evaluation of imported “solutions” to compare to adopt “as is” to adapt or modify consider quality, accessibility, costs

13. Goals and expectations - learning package: learn a new concept apply knowledge already learned offer access to information, databases solve problems rehearse a skill provide feedback on learning access/simulate/replace an experiment

14. Match: IT and educational aims encourages active learning? stimulates problem-solving? triggers “what if” speculation? stimulates student-student discussion? supports further exploration? offers quizzes and/or feedback?

15. Is the content appropriate? material relevant? important? specific skills essential? level of knowledge/skill right? enhances useful generic skills? well-matched to assessment? outcomes consistent with program goals?

16. Design considerations well constructed and structured? are screens clear/acceptable to users? easily navigated? instructions clear/intuitive/consistent? can the user exit easily? too slow/fast? does it include feedback?

17. Quality and relevance is the issue/task/learning important? is the information accurate? is the approach up-to-date? are examples appropriate/engaging? are illustrations clear and relevant? if a simulation, how "real" is it?

18. Students’ needs what are their expectations? what are their specific learning needs? what are their generic learning needs? is the IT interactive and time-effective? is the IT consistent with assessments? does it offer helpful feedback?

19. Student and staff reports Student views: –exciting, interesting? useful? boring, useless? –easy to use? impenetrable? –what aspects are valued for learning? Objective measures: –does it stimulate discussion? –do they seek it out frequently? – can improved learning be measured?

20. Is the program effective and cost-effective? is the students' learning significantly increased? do they enjoy the new learning? do they rate it as high quality? are they motivated to learn more? are there cheaper but effective alternatives?

21. Using computers for evaluation of programs and units feedback can be automated, interactive feedback can be frequent (but avoid overload) response times are shortened BUT appropriate mechanisms are needed for implementation

22. Teaching students skills of evaluation enhances collegiality stimulates evidence-based practice engaging students can provide insights for them on program expectations encouraging participation important BUT they must see results! “learning for life”: information literacy

23. Effective computer-based learning: meets a clear educational need is consistent with program goals is cost-effective and timely avoids errors, misconceptions is set at an appropriate level interests and motivates students engages students actively in learning is well-designed, easy to use encourages collaboration

24. Good evaluation is essential to meeting educational need must justify costs (time, money) is difficult, expensive: many variables depends on explicit program goals is time-consuming and may not yield conclusive results