1. CES 20th Annual Conference Keun-bok Kang (CNU) & Chan-goo Yi (ETRI) ; Taejon, Korea A DESIGN OF THE METAEVALUATION MODEL A DESIGN OF THE METAEVALUATION MODEL 16. May. 2000 Keun-bok Kang, Professor Chungnam National University (CNU), Taejon, Korea Tel & Fax : 82-42-821-5845 ; E-mail : kbkang@cnu.ac.kr Chan-goo Yi, Principal Researcher Electronics and Telecommunications Research Institute (ETRI), Taejon, Korea Tel : 82-42-860-6924 ; Fax : 82-42-861-6880 ; E-mail : yicg@etri.re.kr A DESIGN OF THE METAEVALUATION MODEL A DESIGN OF THE METAEVALUATION MODEL 16. May. 2000 Keun-bok Kang, Professor Chungnam National University (CNU), Taejon, Korea Tel & Fax : 82-42-821-5845 ; E-mail : kbkang@cnu.ac.kr Chan-goo Yi, Principal Researcher Electronics and Telecommunications Research Institute (ETRI), Taejon, Korea Tel : 82-42-860-6924 ; Fax : 82-42-861-6880 ; E-mail : yicg@etri.re.kr

2. CES 20th Annual Conference Keun-bok Kang (CNU) & Chan-goo Yi (ETRI) ; Taejon, Korea 1. Introduction 1. Introduction 2. Definition of Metaevaluation 2. Definition of Metaevaluation 3. Previous Researches on the Metaevaluation 3. Previous Researches on the Metaevaluation 4. Building the Metaevaluation Model 4. Building the Metaevaluation Model 4-1. Component Factors of Metaevaluation 4-1. Component Factors of Metaevaluation 4-2. Design of the Metaevaluation Model 4-2. Design of the Metaevaluation Model 1. Introduction 1. Introduction 2. Definition of Metaevaluation 2. Definition of Metaevaluation 3. Previous Researches on the Metaevaluation 3. Previous Researches on the Metaevaluation 4. Building the Metaevaluation Model 4. Building the Metaevaluation Model 4-1. Component Factors of Metaevaluation 4-1. Component Factors of Metaevaluation 4-2. Design of the Metaevaluation Model 4-2. Design of the Metaevaluation Model CONTENTSCONTENTS

3. CES 20th Annual Conference Keun-bok Kang (CNU) & Chan-goo Yi (ETRI) ; Taejon, Korea ▣ Purpose of this study - designing of metaevaluation model that can be used in practice - in order to achieve this : review the previous approaches to metaevaluation (especially the component factors of metaevaluation) identify the component factors of metaevaluation suggest the metaevaluation model ▣ Purpose of this study - designing of metaevaluation model that can be used in practice - in order to achieve this : review the previous approaches to metaevaluation (especially the component factors of metaevaluation) identify the component factors of metaevaluation suggest the metaevaluation model 1. Introduction - 1 -

4. CES 20th Annual Conference Keun-bok Kang (CNU) & Chan-goo Yi (ETRI) ; Taejon, Korea ▣ ▣ usually metaevaluation is understood as evaluation of evaluation ▣ ▣ according to how they understand the nature and the purpose of policy evaluation, they define the metaevaluation differently ▣ ▣ policy evaluation : - (purpose of policy evaluation) produce valid and useful policy relevant information to stakeholders, to decide in making the policy or changing the existing policy, to improve the policy implementation, to identify the responsibility of policy process participants ▣ ▣ usually metaevaluation is understood as evaluation of evaluation ▣ ▣ according to how they understand the nature and the purpose of policy evaluation, they define the metaevaluation differently ▣ ▣ policy evaluation : - (purpose of policy evaluation) produce valid and useful policy relevant information to stakeholders, to decide in making the policy or changing the existing policy, to improve the policy implementation, to identify the responsibility of policy process participants 2. Definition of Metaevaluation - 2 -

5. CES 20th Annual Conference Keun-bok Kang (CNU) & Chan-goo Yi (ETRI) ; Taejon, Korea - 3 - ▣ ▣ Definition of metaevaluation : - evaluation on the evaluation and evaluation system, during or after the evaluation, in terms of evaluation paradigm, evaluation resources, evaluation process, evaluation performance and evaluation utilization ▣ ▣ Purpose of metaevaluation : - to value the evaluation(system) : summative evaluation - to improve the evaluation(system) : formative evaluation - to promote the utilization of evaluation : summative / formative evaluation ▣ ▣ Definition of metaevaluation : - evaluation on the evaluation and evaluation system, during or after the evaluation, in terms of evaluation paradigm, evaluation resources, evaluation process, evaluation performance and evaluation utilization ▣ ▣ Purpose of metaevaluation : - to value the evaluation(system) : summative evaluation - to improve the evaluation(system) : formative evaluation - to promote the utilization of evaluation : summative / formative evaluation

6. CES 20th Annual Conference Keun-bok Kang (CNU) & Chan-goo Yi (ETRI) ; Taejon, Korea 3-1. 3-1. metaevaluation is an evaluation which examines the value and merits of policy evaluation itself (Stufflebeam, 1981) - when policy evaluation is understood as examining the value and merits of policy activities - metaevaluation is a process of classifying, acquiring and utilizing descriptive and perceptive information related effectiveness, actuality, ethicality and technical appropriateness of certain evaluation activities - the purpose of metaevaluation is to be a guide for evaluation activities, and openly report upon the merits and demerits of evaluation itself 3-1. 3-1. metaevaluation is an evaluation which examines the value and merits of policy evaluation itself (Stufflebeam, 1981) - when policy evaluation is understood as examining the value and merits of policy activities - metaevaluation is a process of classifying, acquiring and utilizing descriptive and perceptive information related effectiveness, actuality, ethicality and technical appropriateness of certain evaluation activities - the purpose of metaevaluation is to be a guide for evaluation activities, and openly report upon the merits and demerits of evaluation itself 3. Previous Researches on the Metaevaluation (expecially the components of metaevaluation) (expecially the components of metaevaluation) 3. Previous Researches on the Metaevaluation (expecially the components of metaevaluation) (expecially the components of metaevaluation) - 4 -

7. CES 20th Annual Conference Keun-bok Kang (CNU) & Chan-goo Yi (ETRI) ; Taejon, Korea 3-2. 3-2. metaevaluation is examining the validity of the evaluation design, data collection and analysis - when the purpose of evaluation is understood as producing valid information for policy results - metaevaluation is focused on the technical or structural adequacy of evaluation (for example, Cook & Gruder, 1978 ; Chelimsky, 1987 ; Smith & Hauer, 1990 ; Greene, 1992) 3-2. 3-2. metaevaluation is examining the validity of the evaluation design, data collection and analysis - when the purpose of evaluation is understood as producing valid information for policy results - metaevaluation is focused on the technical or structural adequacy of evaluation (for example, Cook & Gruder, 1978 ; Chelimsky, 1987 ; Smith & Hauer, 1990 ; Greene, 1992) - 5 -

8. CES 20th Annual Conference Keun-bok Kang (CNU) & Chan-goo Yi (ETRI) ; Taejon, Korea - 6 - 3-3. 3-3. metaevaluation is assessing the usefulness of the evaluation - when the purpose of evaluation is understood as providing useful information to stakeholders (policymakers, decision makers, program managers, clients, etc) - metaevaluation is focused on assessing the evaluation utilization (for example, Mackay, 1992) 3-3. 3-3. metaevaluation is assessing the usefulness of the evaluation - when the purpose of evaluation is understood as providing useful information to stakeholders (policymakers, decision makers, program managers, clients, etc) - metaevaluation is focused on assessing the evaluation utilization (for example, Mackay, 1992)

9. CES 20th Annual Conference Keun-bok Kang (CNU) & Chan-goo Yi (ETRI) ; Taejon, Korea - 7 - 3-4. 3-4. metaevaluation is checking the reliability of the evaluative judgements(evaluation results), and the evaluation process that ensure the reduction of the bias - when evaluation is understood as making unbiased evaluative judgements of merits or worth of the policy results - for example, Scriven suggests his ‘Metaevaluation Checklist’ (not identified, quoted from Rogers, 1995) 3-4. 3-4. metaevaluation is checking the reliability of the evaluative judgements(evaluation results), and the evaluation process that ensure the reduction of the bias - when evaluation is understood as making unbiased evaluative judgements of merits or worth of the policy results - for example, Scriven suggests his ‘Metaevaluation Checklist’ (not identified, quoted from Rogers, 1995)

10. CES 20th Annual Conference Keun-bok Kang (CNU) & Chan-goo Yi (ETRI) ; Taejon, Korea - 8 - 3-5. 3-5. Larson & Berliner's Standards ▣ ▣ Larson & Berliner(1983) classified the three component factors of metaevaluation, and further divided each factor into detailed evaluation items - factors of evaluation input resources and techniques used in evaluation, environment - factors of evaluation process actual evaluation activities carried out in accordance with evaluation plan - factors of evaluation outcome number of decisions affected by the evaluation 3-5. 3-5. Larson & Berliner's Standards ▣ ▣ Larson & Berliner(1983) classified the three component factors of metaevaluation, and further divided each factor into detailed evaluation items - factors of evaluation input resources and techniques used in evaluation, environment - factors of evaluation process actual evaluation activities carried out in accordance with evaluation plan - factors of evaluation outcome number of decisions affected by the evaluation

11. CES 20th Annual Conference Keun-bok Kang (CNU) & Chan-goo Yi (ETRI) ; Taejon, Korea 3-5. 3-5. Larson & Berliner's Standards (continued) - this research was the first systematic approach to metaevaluation - but, they did not concern the evaluation purposes and types - they suggested 23 evaluation items for metaevaluation, but some items were overlapped and the classification of items is somewhat inappropriate - they had no little concern on the rational and due process 3-5. 3-5. Larson & Berliner's Standards (continued) - this research was the first systematic approach to metaevaluation - but, they did not concern the evaluation purposes and types - they suggested 23 evaluation items for metaevaluation, but some items were overlapped and the classification of items is somewhat inappropriate - they had no little concern on the rational and due process - 9 -

12. CES 20th Annual Conference Keun-bok Kang (CNU) & Chan-goo Yi (ETRI) ; Taejon, Korea 3-6. 3-6. The Joint Committee on Standards for Educational Evaluation (1994) - developed 30 principles(standards) for metaevaluation. they were grouped as such : Utility, Feasibility, Propriety, Accuracy - these standards are a useful checklist for metaevlauation, but the committee does not suggest a satisfactory framework for metaevaluation (little guidance on how to plan evaluation in order to meets these standards) - it is needed that these standards to be checked for applicability in other countries and other types of program (Rogers, 1995) 3-6. 3-6. The Joint Committee on Standards for Educational Evaluation (1994) - developed 30 principles(standards) for metaevaluation. they were grouped as such : Utility, Feasibility, Propriety, Accuracy - these standards are a useful checklist for metaevlauation, but the committee does not suggest a satisfactory framework for metaevaluation (little guidance on how to plan evaluation in order to meets these standards) - it is needed that these standards to be checked for applicability in other countries and other types of program (Rogers, 1995) - 10 -

13. CES 20th Annual Conference Keun-bok Kang (CNU) & Chan-goo Yi (ETRI) ; Taejon, Korea 3-7. 3-7. Patricia Rogers' Metaevaluation Framework (1995) ▣ ▣ a framework for evaluating evaluation in terms of their intended and actual impact on the implementation of existing programs ▣ ▣ her metaevaluation framework consists of : - 5 evaluation criteria for intermediate outcomes of program evaluation producing valid information providing useful information to decision-makers producing unbiased judgements of merit or worth involvement & illumination of relevant stakeholders empowerment of the intended program clients 3-7. 3-7. Patricia Rogers' Metaevaluation Framework (1995) ▣ ▣ a framework for evaluating evaluation in terms of their intended and actual impact on the implementation of existing programs ▣ ▣ her metaevaluation framework consists of : - 5 evaluation criteria for intermediate outcomes of program evaluation producing valid information providing useful information to decision-makers producing unbiased judgements of merit or worth involvement & illumination of relevant stakeholders empowerment of the intended program clients - 11 -

14. CES 20th Annual Conference Keun-bok Kang (CNU) & Chan-goo Yi (ETRI) ; Taejon, Korea 3-7. 3-7. Patricia Rogers' Metaevaluation Framework (1995) (continued) - evaluation's impact on processes of implementing existing programs or replacing existing programs with alternatives - evaluation's contribution to the development of programs which meets need ▣ ▣ this framework is very systematic, but the evaluation object is limited to the existing programs which is implementing 3-7. 3-7. Patricia Rogers' Metaevaluation Framework (1995) (continued) - evaluation's impact on processes of implementing existing programs or replacing existing programs with alternatives - evaluation's contribution to the development of programs which meets need ▣ ▣ this framework is very systematic, but the evaluation object is limited to the existing programs which is implementing - 12 -

15. CES 20th Annual Conference Keun-bok Kang (CNU) & Chan-goo Yi (ETRI) ; Taejon, Korea 4-1. 4-1. Component Factors of Metaevaluation - evaluation paradigm - evaluation resources - evaluation process - evaluation performance - evaluation utilization 4-1. 4-1. Component Factors of Metaevaluation - evaluation paradigm - evaluation resources - evaluation process - evaluation performance - evaluation utilization 4. Building the Metaevaluation Model - 13 -

16. CES 20th Annual Conference Keun-bok Kang (CNU) & Chan-goo Yi (ETRI) ; Taejon, Korea ① Evaluation Paradigm - evaluation purpose : rationality - evaluation type : appropriateness - evaluation object : suitability different level and range of evaluation object may lead different evaluation results ① Evaluation Paradigm - evaluation purpose : rationality - evaluation type : appropriateness - evaluation object : suitability different level and range of evaluation object may lead different evaluation results - 14 -

17. CES 20th Annual Conference Keun-bok Kang (CNU) & Chan-goo Yi (ETRI) ; Taejon, Korea ② Evaluation Resources - evaluation manpower quality and quantity of evaluators user involvement - evaluation organization structural and functional appropriateness of evaluation organization - evaluation budget and information appropriateness of evaluation budget quality(adequacy, reliability, etc) and quantity of information ② Evaluation Resources - evaluation manpower quality and quantity of evaluators user involvement - evaluation organization structural and functional appropriateness of evaluation organization - evaluation budget and information appropriateness of evaluation budget quality(adequacy, reliability, etc) and quantity of information - 15 -

18. CES 20th Annual Conference Keun-bok Kang (CNU) & Chan-goo Yi (ETRI) ; Taejon, Korea ③ Evaluation Process - evaluation procedure : objectivity and fairness - timing of evaluation : fitness to the evaluation type etc. - evaluation methodology : accuracy and validity - evaluation criteria : appropriateness - evaluation indicators : rationality ③ Evaluation Process - evaluation procedure : objectivity and fairness - timing of evaluation : fitness to the evaluation type etc. - evaluation methodology : accuracy and validity - evaluation criteria : appropriateness - evaluation indicators : rationality - 16 -

19. CES 20th Annual Conference Keun-bok Kang (CNU) & Chan-goo Yi (ETRI) ; Taejon, Korea ④ Evaluation Performance - evaluation outcome : validity - evaluation information : usefulness ④ Evaluation Performance - evaluation outcome : validity - evaluation information : usefulness - 17 -

20. CES 20th Annual Conference Keun-bok Kang (CNU) & Chan-goo Yi (ETRI) ; Taejon, Korea ⑤ Evaluation Utilization ▣ ▣ preparation and dissemination of evaluation report - clarity and impartiality of report - timeliness and dissemination of report ▣ ▣ utilization of evaluation results - instrumental utilization contribution to improving the implementing existing program or to changing the existing program, to developing the new program - conceptual utilization enlightenment of policy ⑤ Evaluation Utilization ▣ ▣ preparation and dissemination of evaluation report - clarity and impartiality of report - timeliness and dissemination of report ▣ ▣ utilization of evaluation results - instrumental utilization contribution to improving the implementing existing program or to changing the existing program, to developing the new program - conceptual utilization enlightenment of policy - 18 -

21. CES 20th Annual Conference Keun-bok Kang (CNU) & Chan-goo Yi (ETRI) ; Taejon, Korea - 19 - 4-2. Design of the Metaevaluation Model component factors of metaevaluation items of metaevaluation & criteria purpose rationality type object appropriateness suitability (level and range) manpower quality / quantity user involvement organization structural and functional appropriateness evaluation paradigm evaluation resource budget information appropriateness quantity and quality (adequacy, reliability, etc)

22. CES 20th Annual Conference Keun-bok Kang (CNU) & Chan-goo Yi (ETRI) ; Taejon, Korea - 20 - (continued) items of metaevaluation & criteria component factors of metaevaluation objectivity and fairness fitness to evaluation type etc. accuracy and validity appropriateness rationality validity usefulness procedure timing methodology criteria indicator outcome information evaluation process evaluation performance clarity and impartiality timeliness and dissemination report Improving & changing the existing program developing the new program Instrumental utilization enlightment of policy conceptual utilization evaluation utilization