IEN Inaugural Annual Conference September 24 th 2004 Evaluation: A tool for meaning-making in a Higher Education Quality Assurance culture

Content 1. Conceptions about how mind works 2. The OTC ‘culture’ as a context for the Quality Assurance framework 3. The OTC Quality Assurance framework 4. Evaluation as a tool for ‘meaning making’ within the OTC culture 5. How ‘meanings’ are communicated as catalysts for both individual and collective development and progression

Conceptions about how mind works Bruner (1995) Computational View Information processing Culturalism Meaning-making ’ Situated’ learning and thinking

‘Education is not simply a technical business of well managed information processing…..It is a complex pursuit of fitting a culture to the needs of its members and of fitting its members and their ways of knowing to the needs of the culture’ Bruner 1986:43

Guiding Principles Vision and Mission Ethos Pedagogical Practice O.T.C. QA Framework 1.Quality Control 2.Quality Assurance 3.Continuous Quality Improvement

Guiding principles Inclusiveness Openness Relevance Student centeredness Accountability Accessibility

College Vision and Mission ‘The Open Training College will be recognized for the major role it plays in the advancement of the inclusion of people with disabilities in Irish Life; the excellence of our training; our commitment to our students; the quality of our relationships with agencies;…………………………….

Mission/vision statement cont. our advancement of adult lifelong- learning opportunities and our contribution to change and best practice at the level of the service-user, the staff member and the agency in the disability sector’.

College ethos and values Enhancing the status of people with a disability Fundamental principles Valuing students as colleagues Close working relationships Cost effectiveness and value for money Strategic planning

Pedagogical practice Shared understanding of the adult learning process Courses designed to provide and stimulate diverse learning experiences, building of the skills and knowledge that our learners bring to their learning Clearly articulated learning outcomes and related assessment procedures

The Open Training College Quality Assurance framework Please see handout

Evaluation process model for course evaluation Please see handout

How do we evaluate? ‘Responsive Evaluation’ To covey a message rather than specific data Stake (1975) ‘Evaluation is an OBSERVED VALUE compared to some STANDARD’ Scriven (1967)

OTC standards Represented in the: Guiding principles Vision and Mission statement Ethos Pedagogical Practice

Interpreting the evaluation summaries The basic task of an evaluator is made barely tolerable by the fact that he/she does not have to solve this equation in some numerical way nor to obtain a descriptive summary grade but needs merely to make a comprehensive statement of what the programme is observed to be, with useful references to the satisfaction and dissatisfaction that appropriately selected people feel toward it’ Stake (1975)

Making meanings Meaning making involves situating encounters with the world in their appropriate cultural contexts in order to know ‘what they are about’. Although meanings are ‘in the mind’ they have their origins and their significance in the culture in which they are created. It is this cultural situatedness of meanings that assures their negotiability and ultimately, their communicability’. Bruner (1986:3)

How meanings are communicated 1. Evaluation summaries (QA coordinator) 2. Meaning-making within and as a result of the shared micro- culture of the OTC (tutors) 3. Feedback to Programme Boards through Annual Course Review Reports (tutors) 4. Recommendations made to Academic Council (Programme Boards)

5. Ratification of improvements and developments (Academic Council) 6. Implementation of the same (OTC team) 7. Dissemination to all stakeholders (QA coordinator)

Summary We make meanings of evaluation data by filtering our interpretations through our minds in order to measure findings against specified standards Macro and micro cultures shape mind and provide us with a toolkit by which we construct and communicate meanings

If meanings from evaluations procedures are to be catalysts for development and progression; evaluation procedures must be embedded within a clearly defined shared organizational culture to which its members are committed.