Evaluating your teaching and learning project UQ Teaching and Learning Week, 2007 Clair Hughes

Evaluation The process of determining the merit and/or worth of things for a range of purposes. Evaluation is generally concerned with: –Appropriateness - the extent to which stated or implied project goals (or expectations) and implementation processes match the stated or implied needs of nominated target group/s (e.g. students; teachers; coordinators; tutors; program, school, faculty or institutional leaders and administrators; external agencies etc) and the context in which implementation is to be undertaken. –Effectiveness - the extent to which the project or innovation has had an impact on the knowledge, skills, attitudes and/or behaviours of nominated target group/s. –Efficiency - the amount of resources or effort needed to implement the project or innovation. Adapted from Queensland School Curriculum Council, 2000, p34

Session overview Summary of key purposes of evaluation Consideration of how purpose influences: –identification of stakeholders –framing of evaluation questions –selection of data collection methods.

Funding agency requirements Ensure that evaluation is undertaken at multiple points throughout the project and is understood and reported within an evaluation framework. Carrick Competitive Grants program: Guidelines and Supporting Information – 2008 (p.23) …………………………………………………………………………………………………… Must provide a strategy for the evaluation of the project with information on outcomes for students, whether there has been a noticeable improvement or not in the expected outcomes for students and how the project outcomes will be disseminated to others within the discipline. UQ Teaching and Learning Small and Large Grants Scheme Guidelines

The main purposes of project evaluation Formative (to inform ongoing project decision-making) Monitor project progress Revise/enhance project processes and outcomes Engage key stakeholders early and during project to: –demonstrate respect –intensify participation –collaborate in identifying key indicators of success –progressively expose to new perspectives, or information Make interim reports Summative (to determine/support project findings and outcomes) Report on success/results/value Demonstrate impact Account to sponsors Sustain project outcomes Support related submissions (eg grants for funding for further related projects; promotion; teaching and learning awards) Enlightenment (to learn) Provide opportunities for project team to maximise the learning developed through participation in the project Adapted from “The Learning Partnership” 2007 and W. K. Kellogg Foundation

Models of planning for evaluation ImplementationEvaluation Dissemination ImplementationEvaluation Dissemination Implementation Evaluation Dissemination ImplementationEvaluation Dissemination

Identifying key stakeholders Who is your project audience? Who will be interested in the outcomes now? – in the future? –project funding agency? –future funding agency? –students? –academic supervisor/s? –senior management? –teaching colleagues? (supportive and antagonistic) –teachers in related disciplines? –professional association members? –industry or employers? –government organisations? –committees (appointment, promotion, T&L awards? –reviewers (conferences, publications)? –lobby groups? –the media? –other?

Frame stakeholder questions Identify two stakeholders in a project with which you are familiar or which you have in development - in your selection try to represent diverse interests. List one or more questions each stake-holder may have in relation to the project? –Examples: What is the impact of the project on student outcomes? success? satisfaction? (Funding agency) What resources are needed to implement the project innovation more widely? Is it feasible? (Senior manager) How do students experience the project? Are there any unintended/undesirable effects? (Students) What contextual factors facilitated or hindered implementation? (Teaching peers)

Collecting evidence How will data be collected? (Methods) From whom or where (stakeholders and others) will data be collected? (Sources) What will constitute evidence? (Consider both qualitative and quantitative)

The evaluation framework Evaluation questions Sources/Methods Institutional contacts StudentsExisting records/ artefacts What is GGA policy in Australian universities? Telephone interview (to clarify, address omissions or current activity) Web search of university policy documents To what extent are students conscious of the attributes they are developing throughout the course of their programs? Focus group interviews Literature review (research demonstrating student impact) EXAMPLE

References Carrick Institute for Learning and Teaching in Higher Education. (2006). Carrick Institute Dissemination Framework Retrieved from blic/dissemination_carrickframework_2006.pdf blic/dissemination_carrickframework_2006.pdf Carrick Institute for Learning and Teaching in Higher Education. (2008). Competitive Grants program: Guidelines and Supporting Information – Retrieved from blic/grants_competitive_guidelines2008_july07.pdf blic/grants_competitive_guidelines2008_july07.pdf Felton, P., Kalish, A., Pingree, A., & Plank, K. (2007). Towards a scholarship of teaching and learning in educational development. In D. Robertson & L. Nilson (Eds.), To Improve the Academy: Resources for Faculty, Instructional, and Organisational Development (pp ). Boston: Anker. The Learning Partnership. (2007). Material distributed during Leading and Managing Projects, a workshop sponsored by the Carrick Institute for Learning and Teaching in Higher Education. Melbourne, 17 – 18 September. Queensland School Curriculum Council. (2000). Procedures for the Evaluation of Approved Council Curriculum Materials. Retrieved 29 October, 2007, from The University of Queensland. (2007). UQ Teaching and Learning Small and Large Grants Scheme Guidelines. Retrieved 29 October, 2007, from learning/download/TLStrategicGrantsGuidelines07.pdfhttp:// learning/download/TLStrategicGrantsGuidelines07.pdf W. K. Kellogg Foundation. Executive summary: Introducing Program Logic Models. Material distributed during The Professional Development Program for Medical and Healthcare Educators offered jointly by the University of Queensland and Harvard Medical International August, November, 2006.