Presentation on theme: "National Developmental evaluation Kauko Hämälainen Heikki K. Lyytinen The Finnish Education Evaluation Council The Miracle of PISA –Conference 10. -12-Sept."— Presentation transcript:
National Developmental evaluation Kauko Hämälainen Heikki K. Lyytinen The Finnish Education Evaluation Council The Miracle of PISA –Conference Sept. In Helsinki
Content 1.Where did we come from and where are we going to in the national evaluation? 2. Within what kind of administrative and organizational structure we act? 3. What kind of demands/requirements we make for the national evaluation? 4. What are the values and principles we set on the evaluation? 5. How do we carry out the developmental evaluations in practice ? 6. What have been our evaluation themes? 7. Examples of evaluation practices (evaluation projects in practice)
Change of Educational Steering System in Finland Situation in 1970s and 1980s Situation in 1990s/2000 Centralised control and decision-making Devolution of power centralised curricula long-term plans budgeting based on expenditures controlling follow up external evaluation: inspections conducted by state authorities etc. self-governance school-based curricula distinctive educational profile of educational institutions self-direction and -regulation learning organisation as a model of functioning self-evaluation and own control performance-based funding etc. new evaluation organisations National evaluation plan
Evaluation instead of inspection no separate school inspectorate and inspection visits to schools the activitis of education providers are quided by objectives laid down in legislation and the national core curricula the system relies on the proficiency of teachers in their efforts to meet the objectives strong focus on self-evaluation and external evaluation
The Forms of Steering and Improving the Quality of Education in the Finnish School System : Focus in steering by information, monitoring and evaluation and steering by professional expertise Steering by information, monitoring and evaluation Steering by professional expertise: freeing the creative resources of the school Aim: improved quality of education Market-driven steering: competition between schools Norm- and resources-based steering
Challenges for renewing evaluation in educational services responsibility to organise educational services also includes evaluating them directing educational services requires goals and evaluating how well they have been met (so-called profit responsibility) evaluation seen as a comprehensive process (cf. e.g. quality assurance) evaluation as a tool for learning and development( developmental evaluation)
evaluation adds to the active role of actors(empowerment) knowledge of the current state of affairs essential when setting goals for development controlled change when steering towards future goals (controlled change based on facts/evidence based cahnge management)
TOWARDS NEW EVALUATION CULTURE 1 Explorative/evaluative work community – EVIDENCE BASED continuing evaluation – transparency making use of research and evaluation knowledge 2 Reflective work community - towards culture of EMPOWERMENT independence, motivation, atmosphere, appreciation, security 3 Network-oriented operation cultures 4 Making use of parallelisms peer evaluation, benchmarking … 5 From data management to knowledge management – collective awareness 6 Demands for involvement, multiple perspectives, and consensus – authentic dialog 7 Managing the whole of evaluation
Towards a new evaluation system Internal administrative evaluation system New evaluation awareness 2-3 years Transition period 2-3 years Change Continuous development (new evaluation culture) Change of evaluation culture Towards developmental evaluation External evaluation system of expert networks Administrative dependence (”internal evaluation”) Expert network –based independence (external/internal)
Benefitting from the network both in evaluation and in project planning 1. Planning group: expertise of the field and interest groups draws up a plan and schedule for the project and its different stages proposes assessors to be appointed in the evaluation group to the Council 2. Evaluation group: independent experts who work in assessment and research evaluate and draw conclusions with further recommendations write a report and are responsible for it Evaluation projects are organised in two phases to maximise best expertise commitment of the field, partners and interest groups independence and impact
National production of evaluation information to serve policy-making Parliament The Ministry of Education The Government The Education Evaluation Secreteriat The Evaluation Secreteriat The Finnish Education Evaluation Council 14 members The Finnish Higher Education Evaluation Secreteriat 12 members National evaluation and information production network - Regional Authorities - Provincial State Offices - Provincial Councils - Employment and EconomicDevelopment Centres Universities Polytechnics Statistics Finland Research Institutes in Univeristies National Board of Education Providers Private Evaluation Organisations Schools The other evaluation organisations (national and international)
EducationEvaluation council in conjunction with the Ministry of Education since April 2003: planning, co-ordinating, managing and developing the evaluation of basic education and upper secondary education and training The polytechnics and univeristies are responsible for the evaluation of their own operations and outcomes (support from the Higher Education Evaluation Council)
The Finnish Higher Education Evaluation Council (FINHEEC) (FINHEEC) is an independent expert body assisting universities, polytechnics and the Ministry of Education in matters relating to evaluation. The evaluations conducted by FINHEEC can be divided into three rough categories: –Evaluations of higher education institutions: Institutional evaluations (universities and polytechnics), audit of quality work, and evaluations for the accreditation of the polytechnics –Programme and thematic evaluations –Accreditation of professional courses offered by higher education institutions
NATIONAL EVALUATION SYSTEM Indicators -quantitative -qualitative Learning outcomes General knowledge/ School subjects Professional competence General qualifications -learning to learn Situation and system- evaluations Thematic evaluations Methods and models International cooperation PISA Copenhagen process Evaluation research Self-assessment Quality control External and internal audits Paperless enviroment Indicators
AIMS OF EVALUATION IN THE FINNISH EDUCATION SYSTEM The aims of evaluation are to gather and analyse information which serves educational decision- making at the national policy and local levels and contributes to the development of education in general. In addition, the aim is to support students’ learning and the work of educators, as well as institutional development.
Educational evaluation: Focus on many levels Individuals (pupils, students, teachers) Classrooms/courses Curriculum/programmes Organizations (schools, universities) Fields (all schools in a municipality, all programmes in a discipline) The national level (national quality development and quality assurance systems) The international level (PISA,IEA, EQUIS in the business school area)
Regional evaluation The Provincial Government Act (22/1997) obliges the provincial governments to evaluate the accessibility of basic services within their borders ○ The Ministry of the Interior and other minitstries define shared evaluation targets for a number of years (i.e. the term of the government) ○ evaluations should be carried out co-operatively, consistently, and drawing primary on existing statistics and department specific databases
DISTRIBUTION OF WORK AND CO-OPERATION IN EVALUATION Ministry of Education and the Education Evaluation Council are responsible for external evaluation and its development. Education providers are primarily responsible for local evaluation and its development. The National Board of Education participates in evaluating syllabuses and learning achievements within degree regulations as a part of educational evaluation expert network Links between external evaluation, local evaluation and evaluation of basic services should be taken into account when drawing up evaluation programmes and plans. Educational institutions and providers who have participated in the evaluation must have access to the evaluation reports on their own activities.
EVALUATION OF LEARNING ACHIEVEMENTS The National Board of Education participates in evaluating syllabuses and learning achievements within degree regulations as a part of educational evaluation expert network
Evaluation of education in The Finnish National Board of Education quantitative and qualitative indicators will be defined information in the form of indicators will be produced learning achievements will be compared learning achievements will be evaluated in different school types and levels as follows (sample-based approach): * basic education: mathematics and mother tongue, regularly * basic education: other subjects and themes (for ex. sustainable development) * methods of evaluation will be developed
REQUIREMENTS FOR EVALUATION SET BY THE PARLIAMANENT EVALUATION POLICY) strengthening the role of evaluation in developing education, adds the neutrality of national evaluation of education and promotesco-operation and communication between actors in evaluation and interest groups promotes equality main emphasis should be in supporting the development of teaching and learning at school and municipality level education providers should be given priority evaluation must concern the whole learning environment
Organising evaluation Organising evaluation evaluation adequately independent of the rest of the education administration independent and impartial evaluation actors increasing the impact of evaluation especially from the viewpoint of educational institutions a system must be established to monitor the impact of evaluation
VALUE BASIS OF EVALUATION Equality means promoting educational equality by means of evaluation. Fairness means creating an ethically sustainable foundation for evaluation and refraining from comparison which may be harmful for the target of evaluation. Truthfulness means high ethical responsibility and critical examination, as well as optimisation of the reliability of evaluation.
Principles for educational evaluation Evaluation is based on education policy decisions (development aspect) The legitimacy is founded on educational legislation Education evaluation should be transparent The essential outcomes need to be published Methods used are to be recorded and justified, as reliable, valid and comparable as possible
In this regard, evaluation must provide central and local administration with valid data on how succesfully educational objectives are being achieved. Thus, evaluation should foster self-control and self regulation at the level of central and local administration and promote accountability to the various parties involved.
From the perspective of educational policy, it is important to ensure that education providers deliver the educational services that they are intended to produce and that they do this in accordance with the goals set for these services and on the basis of professional skills of adequately high standard.
Evaluation process and decision making Decision on evaluation Appointing the planning group PLANNING GROUP Planning the evaluation Evaluation group proposal Developing data gathering and feedback systems Drawing up instructions for self- assessment Evaluation visits and panels EEC Accepting the plan Appointing the evaluation group Accepting methods and measures and self- assessment models INSTITUTIONS Self-assessment, other material EVALUATION GROUP Evaluation visits Interpreting results Writing a report Publishing a report EEC feedback Quality assurance
THE IMPLEMENTATION OF EVALUATION (PLANNING AND EVALUATION NETWORK ) Education evaluation council Collaborative seminars School X (1-6) School Y (7-9) University of Jyväskylä Education evaluation secretariat National Board of Education providers Education evaluation council University of Jyväskylä University of Helsinki University of Lapland University of Turku University of Joensuu Administrative Board of Western Finland Åbo Akademi University of Oulu Lapp district court sessions
Implemented evaluation projects Basic education 1. The implementation and quality of basic security in education 2. The implementation, availability and quality of special needs education, remedial teaching and student welfare services in basic education Upper secondary education The need for special needs education in upper secondary school education
Liberal Adult Education: Institutional Structure, Service Ability and Societal Effects Training Opportunities and Provider Network of Vocational Adult Education on the Secondary Level Both projects long lasting: three years in two phases ( , and ) Based on available data: statistics, research reports, documents, the variety of material produced in various development projects etc; no primary data gathered Process: (1) multiperspective planning group comprised of education providers and relevant interest groups, (2) independent, scientific research oriented evaluation group Adult education
Basic education Teaching methods in the comprehensive school from January 1 st 2007 to December 31 st 2007 Upper secondary education Collaboration in upper secondary school education and vocational education from October 1 st 2005 to October 10 th 2006 Vocational education Workplace learning from Sebtember 2004 to October 2006 The regional effectiveness of vocational education from January 1 st 2006 to December 31 st 2007 Key competencies in vocational education from January 1 st 2007 to December 31 st 2007 Adult education The competence-based examination system
A Common Quality Assurance Framework Model (Common Core Criteria) Methodology (Self assement/Peer Review) Monitoring (External) Measurement (Indicators) Gains Mutual Trust Trancparence Credit Transfer Quality assurance methods and process, indicators, standards and norms, approach of selfevaluation
QUALITY ASSURANCE MODEL EUROPEAN COMISSION (LISBON STRATEGY) PLANNING i.e. strategies, goals, objectives, resources, anticipation and evaluation mechanism IMPLEMENTATION i.e. action plan, resources, partnerships, leadership, process management, training of trainers, didactical materials MONITORING AND EVALUATION i.e. self evaluation, external inspection, internal quality control Results of teaching, training and learning staff-oriented results, key performance results, sosietal results REVIEW feed back and prosedures for change continue improvement development METHODOLOGY: quality management, self evaluation, stakerholders’s role management of information external evaluation
Examples on evalaution projects
EVALUATION MODEL FOR LEARNING ON THE JOB LEARNING 1.Quantitative indices as part of self- assessment 2 Self-assessment by education providers -multisectoral evaluation -consensus 4 Description of best practices - Portfolio Initial assessment by an evaluator team In-depth analysis of self-assessments, and related visits to educational institutes by the evaluator team Field-specific evaluation seminars Provider feedback Final evaluation and evaluation report Backgound data web-based evaluation environment 5. Stakeholder questionnaire education committees 5. Authority interviews
FROM GOALS TO INTERACTION The evaluation of basic education pedagogy
What was evaluated? 1.Key principles of pedagogy 2.Teachers’ goal awareness 3.Range of teaching methods 4.Attendance to the pupils’ unique characteristics 5.Equality of learning processes 6.Functionality of the learning environments’ factors 7.Teaching, studying and ???
How was information produced? 1)Survey of teachers (N = 2351) 2)Survey of rectors (N = 410) 3)Visits to schools ( N = 12) 4)Hearings ( N = 12) 5)Data and research results from previous statistics, studies and evaluations
TEACHER’S IMPORTANT PEDAGOGIC PRINCIPLES –Fairness –Student oriented approach – Goal awareness – Importance of classroom piece –Stressing action and variation –The “objectivity” of pedagogic thinking – Socio-constructivism has its “opponents”
TEACHERS’ GOAL AWARENESS The status of curriculum is significant. Goal awareness highlights many aim fields of education and upbringing. Text book still has quite a significant role. Critical approaches to curriculum Wishes for time division (lesson plan) Goal awareness given – contemplation on aims thin
TAKING STUDENTS INTO ACCOUNT Students appreciated as individuals Equity central / significant Students’ differences and needs important Student oriented approach has increased Making use of students’ activity No new remedies for bullying. Few means of managing disruptive students. Special needs students cannot always be supported. Possibilities of differentiation limited in practice
RANGE OF TEACHING METHODS The range and variety of teaching methods is highlighted. Externally most often teacher oriented teaching – with student oriented elements. Instrumental use of teaching materials is becoming more common. Information technologies have enriched working methods and materials. Examining and experimental methods of working not usual. Minor changes in teaching
PREREQUISITES FOR ACTION A positive atmosphere for development and shared perception of teaching positive (rectors’ view) The pedagogic know-how of teachers good in this country. Technical equipment and options for teaching material are usually of good quality. Great variation in the preconditions of action between schools. Large and heterogeneous teaching groups is a problem. Too much variation in the resources for complementary education. Haste and the feeling of haste in school work have increased Messages of problems with air-conditioning and schools with mold and mildew problems!
THE IMMEDIATE CHALLENGES OF PEDAGOGY? THE IMMEDIATE CHALLENGES OF PEDAGOGY? 1)Large + heterogeneous groups (taking into account the broad questions of differentiation) 2)The frequency of class-room disturbance – remedies? 3)The needs of developing self-evaluation – how? 4)The curriculum includes too much content, too demanding goals, and too high criteria for good learning outcomes, the instructions for evaluation call for clarifications. 5)Teachers’ supplementary education has to be taken more seriously: more time and funding, more equity, more long-term planning, teacher oriented approach
LONG-TERM NEEDS IN THE DEVELOPMENT OF PEDAGOGY 1.Supporting pedagogic leadership at schools 2.General education in the future – launching the discussion 1.What are the contents? 2.How do we learn? 3.What do we evaluate? 3.Enhancing goal awareness and pedagogic thinking 1.levels 2.steps of advancement 3.goals 4.Increasing the amount of the studying, evaluating and developing approach in the teachers’ work … in the work communities 5.Enhancing the understanding and encountering of the students’ differences, and the pedagogy of developing. 6.Enhancing learning as collective construction of knowledge 7.Enhancing the appropriation of media, information technologies, and information flows as part of e.g. developing the learning to learn skills 8.Increasing the amount of interaction with the surrounding community /society (e.g. children’s and young people’s lifestyles and cultures) 9.Developing the evaluation skills (of the schools / teachers /students)
DISSEMINATION OF EVALUATION INFORMATION 1)Publication / announcements 2)Report, feedback for organizers? 3)National, regional, local events 4)Articles etc. 6)Developing the overall evaluation process from the point of view of making use of information (collaboration, networking, participation, client oriented approach)
The process in short The evaluation group brings all information together brings all information together describes the state of affairs describes the state of affairs defines the criteria defines the criteria evaluates the state of affairs evaluates the state of affairs points out the needs of development points out the needs of development makes some general recommendations makes some general recommendations takes part in the distribution of the results takes part in the distribution of the results