1. Elizabeth Godfrey 1

2.  Periodic assessment of results Appropriateness, effectiveness, efficiency, impact, sustainability  Identifies intended and unintended effects  Identifies what worked and what didn’t  Provides level of judgement about the overall worth of intervention  Key to project improvement  Influence decision-making or policy formulation through the provision of empirically-driven feedback  sustainability

3.  Why is the evaluation being done?  How will the information be used?  What evaluation form and approach will be most suitable for this study? Formative – monitoring progress to improve approaches Summative – overall perspective – focus on value or worth of project and designed for accountability or continuation purposes

4. 4

5.  Evaluation and assessment have many meanings…one definition: Assessment - is gathering evidence Evaluation - is interpreting data and making value judgements  Examples of assessment and evaluation Individual’s performance (grading) Program’s effectiveness (accreditation) Project’s progress and success (monitoring and validating)

6.  Effective evaluation starts with carefully defined project goals and expected outcomes  Goals provide overarching statements of project intention What is your overall ambition? What do you hope to achieve?  Expected outcomes identify specific observable results for each goal How will achieving your “intention” reflect changes in student behavior? How will it change their learning and their attitudes?

7.  Goals → Expected outcomes → Evaluation questions  Questions form the basis of the evaluation process  Evaluation process collects and interprets data to answer evaluation questions

8.  To what extent has the project been implemented as planned?  How well has the project been co-ordinated across different institutions?  To what extent have the intended outcomes been achieved?  Were there any unintended outcomes?  How well have the needs of staff been met?  To what extent have the intended student learning outcomes been achieved?  What measures if any have been put in place to promote sustainability of the project’s focus and outcomes?  Are students better able to describe the effects of changing some feature in a simple problem as a result of the intervention?  What lessons have been learned from this project and how might these be of assistance to other institutions?

9. What type of data is most appropriate? What are the most appropriate methods of data collection? How will data be analysed and presented? What ethical issues are involved in the evaluation?

10.  Surveys Forced choice or open-ended responses  Observations Actually monitor and evaluate behavior  Interviews Structured (fixed questions) or in-depth (free flowing)  Concept Inventories Multiple-choice questions to measure conceptual understanding  Rubrics for analyzing student products Guides for scoring student reports, test, etc.  Focus groups Similar to interviews but with group interaction

11. An Intelligent Tutoring System for Engineering Mechanics Tanja Mitrovic, Charles Fleischmann, Brent Martin, Pramudi Suraweera The planned evaluation will focus on the following questions:  Is the developed ITS effective?  Does it support learning better than the traditional approach?  Does the system increase students � motivation? We will compare performances of two groups of students. One group will learn the material in the traditional way, via lectures and tutorials. The other group will attend lectures, but the tutorials would be replaced by interaction with the system. Time will be controlled. We will require students to sit pre- and post-tests, to measure their knowledge. We will also collect data about their actions while solving problems, and analyze the data.

12.  Judgements will be required for each key evaluation question  Need to develop indicators or targets - standards and levels regarded as acceptable  Could use benchmarks – location or source of best practice for comparison

13. Who? key question underpinning the budget  Should it be an individual or a team?  Insiders or outsiders?  Skills – qualitative and/or quantitative  Prepare an evaluation brief or terms of reference  When to involve the evaluator?

14.  NSF’s User Friendly Handbook for Project Evaluation http://www.nsf.gov/pubs/2002/nsf02057/start.htm  Australian Learning and Teaching Council Grants Scheme – Evaluating Projects http://www.altc.edu.au/extras/altc-gsep/evaluation_plan.pdf  Student Assessment of Their Learning Gains (SALG) http://www.salgsite.org/

15.  Think about evaluation, purpose and scope from the beginning of the project  Ensure appropriate data is collected eg pre/post intervention  Evaluation provides supporting evidence to influence decision-making or policy formulation through the provision of empirically-driven feedback  sustainability

17. Elizabeth Godfrey Engineering Education Research and Project Management lizgodfrey@gmail.com