Presentation is loading. Please wait.

Presentation is loading. Please wait.

ICCS / IV Conference, Prague 2011. Franz Schott: Measuring the outcome of education MEASURING THE OUTCOME OF EDUCATION FRANZ SCHOTT UNIVERSITY OF TECHNOLOGY,

Similar presentations


Presentation on theme: "ICCS / IV Conference, Prague 2011. Franz Schott: Measuring the outcome of education MEASURING THE OUTCOME OF EDUCATION FRANZ SCHOTT UNIVERSITY OF TECHNOLOGY,"— Presentation transcript:

1 ICCS / IV Conference, Prague Franz Schott: Measuring the outcome of education MEASURING THE OUTCOME OF EDUCATION FRANZ SCHOTT UNIVERSITY OF TECHNOLOGY, DRESDEN, GERMANY

2 ICCS / IV Conference, Prague Franz Schott: Measuring the outcome of education Introduction The topic of this conference is: Employability - Mobility - Flexibility. European demands challenging Religion and Education. We read in the announcement of this conference: The aim of education and training is focused on a concept of flexible skills and competencies that serve employability, mobility and flexibility. The Conference will encompass and discuss the European perspectives of education … One central perspective on education since the last decade is the view on the efficiency of the educational system This leads to demand more output-orientation of the educational system instead of input-orientation (input is for example: curriculum, training of teachers, teaching materials) The output of an educational system is what the addressees of this system have learnt, in other words: it is the outcome of education International comparative studies of the outcome of education (e.g. Programme for International Student Assessment: PISA) brought questions about the efficiency of national educational systems and about measuring the outcome of education

3 ICCS / IV Conference, Prague Franz Schott: Measuring the outcome of education We will discuss the topic: measuring the outcome of education as follows: 1.On measuring 2.On the outcome of education 3.Purposes of measuring the outcome of education 4.Problems of measuring the outcome of education 5.Summary Literature Introduction

4 ICCS / IV Conference, Prague Franz Schott: Measuring the outcome of education We have to differentiate measuring in a narrow sense and in a broad sense: Measurement in a narrow sense This concept is used in the physical sciences and defines: Measurement is the process or the result of determining the magnitude of a quantity, such as length or mass, relative to a unit of measurement, such as a meter or a kilogram (see e.g.: Deutsche Norm DIN , 1995). Measurement in a broad sense In social sciences there are no generally binding definitions of units of measurement such as meter or kilogram Therefore we find in social sciences a broader concept of measurement: Measurement is "the assignment of numerals to objects or events according to some rule (Stevens, 1946) Stevens differentiated different levels of measurement: Ordinal scale => the numbers represent a rank order (1st, 2nd, 3rd, etc.) Interval scale => the interval between the numbers is equal Ratio scale=> like interval scale plus a well-defined zero point We do not find ratio scales in social sciences 1. On measuring

5 ICCS / IV Conference, Prague Franz Schott: Measuring the outcome of education Stevens´ broad definition of measurement is accepted in all social sciences. According to this broad definition of measurement, people use often measurements in everyday life An example of a measurement in everyday life In a conversation Elisabeth says: Peter is much more self-confident than Mary. According to the above-mentioned broad definition of measurement as the assignment of numerals to objects or events according to some rule Elizabeth has made a measurement on the level of an ordinal scale: Regarding the criterion of self-confidence, she has compared Peter and Mary: Peter got rank 1st and Maria got the rank 2nd What was Elizabeth´s rule of this measurement? Perhaps she will explain: I observed Peter and Mary in situations which demand a self-confident behavior. In most of the cases, Peter performed better than Mary. To generalize: A personal trait or ability can be measured by assessing the behavior in question in a certain situation according to some rules This measurement is possible and is actually proceeded in many domains, for example: ability in math or tennis, trust in God, etc. 1. On measuring

6 ICCS / IV Conference, Prague Franz Schott: Measuring the outcome of education Criteria for good measurement The most important criteria for good measurement are: Objectivity=> the evaluation is independent of the assessing person Reliability=> a repeated evaluation has the same result Validity=> the evaluation reflects what is the intention of measurement 1. On measuring

7 ICCS / IV Conference, Prague Franz Schott: Measuring the outcome of education A personal trait or ability can be measured by assessing the behavior in question in a certain situation according to some rules. This rules can be more or less strict: Informal measuring The rules are not well-defined, often intuitive and in some cases partially unconscious proceeded This informal measuring is ever-present when we evaluate our environment, neighbors and colleagues Usually a great variability between low and high objectivity, reliability and validity Low for example caused by prejudices High for example caused by long-term observation Semi-formal measuring There are some explicit rules but not so well-defined as in a scientific developed test, for example exams in schools Semi-informal measuring is the normal case in educational settings Usually medium objectivity, reliability and validity Formal measuring The rules are well-defined by scientific requirements This formal measuring is applied by using scientific tests Usually high objectivity, reliability and validity 1. On measuring

8 ICCS / IV Conference, Prague Franz Schott: Measuring the outcome of education The outcome of education is what is learnt in a certain educational system, for example what students learn in a school Important differentiations of outcome: Various domains of the outcome: for example outcomes concerning: Mathematics Geography Religion Employability Achieved outcome: outcome as description, what is learnt Intended outcome: outcome as norm, what should be learnt Names used for intended outcomes: Learning goal or instructional objective Intended Competence Educational standard Short-term outcome: the outcome immediately after the lesson Long-term outcome: the sustainable outcome, which outlasts the short- term outcome 2. On the outcome of education Increasing generality

9 ICCS / IV Conference, Prague Franz Schott: Measuring the outcome of education There a several purposes of measuring the outcome of education, for example: International comparison of the quality of schools (e.g. PISA: Programme for International Student Assessment) Certification of the learner (e.g. diploma, university degree) Feedback to the learner Feedback for the teacher Maintaining and improving the quality of instruction The measurement of the outcome is essential to control and improve education 3. Purposes of measuring the outcome of education

10 ICCS / IV Conference, Prague Franz Schott: Measuring the outcome of education To evaluate the success of instruction, it is necessary to compare the intended outcome with the achieved outcome The quality of this comparison depends on how precise the achieved outcome and the intended outcome can be analyzed Therefore the assessment quality of the outcome depends on how precise the description of the intended outcome is There is no chance to improve instruction without a sufficient precise description and measurement of the intended outcome! 3. Purposes of measuring the outcome of education The measurement of the outcome is essential to maintain quality control and improve education INSTRUCTION MEASUREMENT OF THE ACHIEVED OUTCOME COMPARISON OF THE INTENDED OUTCOME WITH THE ACHIEVED OUTCOME COMPARISON OF THE INTENDED OUTCOME WITH THE ACHIEVED OUTCOME FEEDBACK

11 ICCS / IV Conference, Prague Franz Schott: Measuring the outcome of education We stated: There is no chance to improve instruction without a sufficient precise description and measurement of the intended outcome! Here an example: We asked: How can teachers ensure that their students have a good chance to learn during class the intended outcome of education? To answer this question, we developed approaches (Schott & Seidl, 1994; Schott & Azizi Ghanbari, 2008, 2010, 2011) called PLANA and GovI (Goal-valid Instruction) Here some aspects of these approaches: Step 1: Describe the indented outcome of education (that is an instructional goal) as accurately as necessary by specifying the sets of tasks that can be solved by the learners if they have reached the instructional goal. To specify the sets of task we have developed a special method of task analysis. Step 2: These task sets must occur in an appropriate manner during class Step 3: These task sets must occur in an appropriate manner when measuring the outcome of education These three steps ensure that both the lesson and the control of the learning success is content-valid to the intended outcome of education We speak of parallel content validity 3. Purposes of measuring the outcome of education

12 ICCS / IV Conference, Prague Franz Schott: Measuring the outcome of education GovI a proposal on how to teach and measure the intended outcome of education: => 3 Steps of an outcome-oriented instruction and measurement 3. Purposes of measuring the outcome of education The three steps ensure that both the lesson and the control of the learning success is content-valid to the intended outcome of education Step 1 PRECISE DEFINITION OF THE INTENTED OUTCOME Step 2 OUTCOME-ORIENTED INSTRUCTION Specifying sets of tasks describing the intended outcome of educationt The task sets describing the intended outcome of education must occur in an appropriate manner during class Step 3 OUTCOME-ORIENTED MEASUREMENT The task sets describing the intended outcome of education must occur in an appropriate manner in the exam PARALLEL CONTENT VALIDITY

13 ICCS / IV Conference, Prague Franz Schott: Measuring the outcome of education Arguments against the measuring of the outcome of education: Objection: Measuring the outcome of education ins incompatible with human dignity In the humanities there is a widespread mistrust of measurement and empirical research Of course, we find also misuses and poor quality of measurements, but as we saw in the example of Elizabeth, we measure continuously every day Measuring is a human characteristic Measuring is ubiquitous not only in science and education but also in everyday life Objection: Measuring the outcome of education is only possible in the case of basic skills such as factual knowledge but it is not possible in the case of advanced skills such as understanding and problem solving Advanced skills like flying an airplane or playing Bach can be tested Objection: Measurable are only cognitive abilities, emotions, attitudes and values are not measurable In the case of measuring emotions, attitudes and values, we find also misuses and poor quality of measurements, but as we saw in the example of Elizabeth: we measure continuously every day – also emotions, attitudes and values Nevertheless it is often a hard challenge to measure advanced skills, emotions, attitudes and values 4. Problems of measuring the outcome of education

14 ICCS / IV Conference, Prague Franz Schott: Measuring the outcome of education Arguments against the measuring of the output of education: Objection: Measuring the outcome of education prevents demanding lessons and sophisticated teaching, it leads to teaching to the test This argument must be taken seriously. The teacher must make sure that students are not prepared only for the test Objection: If at all, only the short-term outcome is measurable. Long-term outcomes are not measurable This argument must be taken seriously too. We usually cannot examine how the learners will behave after graduation But we can do something to promote sustainability of the outcome of education 4. Problems of measuring the outcome of education

15 ICCS / IV Conference, Prague Franz Schott: Measuring the outcome of education A careful task analysis can help to promote sustainability and measurability of the outcome of education An example: In a research project concerning outcome-oriented instruction we developed with university students a lesson for ethics education at school. The topics of the lesson were ethical concepts like Kant's categorical imperative or concepts of utilitarianism. When we began to specify the intended outcome of this lesson, my students insists that one of the most relevant outcomes of this lesson should be: The learners should recognize the importance of the topics The learners should see the sense of the lesson The challenging question was: How can one distinguish a learner who has achieved this outcome from a learner who did not achieve it? Which set of tasks can measure that? 4. Problems of measuring the outcome of education

16 ICCS / IV Conference, Prague Franz Schott: Measuring the outcome of education A careful task analysis can help to promote sustainability and measurability of the outcome of education Example continued Very relevant outcome of the lesson: The learners should recognize the importance of the topics; they should see the sense of the lesson. Which set of tasks can measure this outcome? After a long discussion we came to the following solution: A learner who has achieved this outcome is able to solve the following 3 tasks: 1.Explain: If there is no controversy on moral norms, there are no ethical problems to solve. 2.Explain: If there is a disagreement concerning a certain moral norm, then ethical rules are necessary to solve this ethical problem 3.Explain: If there are various ethical rules (such as Kant's categorical imperative and utilitarianism), then philosophical methods are necessary to compare these different rules. We decided: If a learner can solve the set of tasks consisting of these 3 tasks he or she has understood the sense of the lesson. Specifying in this way may help to make an outcome of education, which is challenging because of its compexity, measurable and sustainable. 4. Problems of measuring the outcome of education

17 ICCS / IV Conference, Prague Franz Schott: Measuring the outcome of education The concept measurement Measurement in a narrow sense is used in the physical sciences and defined as the process or the result of determining the magnitude of a quantity relative to a unit of measurement is not applicable to the outcome of education Measurement in a broad sense is defined as the assignment of numerals to objects or events according to some rule is applicable to the outcome of education Important criteria for good measurement Objectivity Reliability Validity According to the rules of assignment of numerals to objects or events we differentiate: Informal measuring Semi-formal measuring Formal measuring 5. Summary

18 ICCS / IV Conference, Prague Franz Schott: Measuring the outcome of education The concept outcome of education Different types of outcome of education Various domains of the outcome Achieved outcome: what is learnt Intended outcome: what should be learnt Short-term outcome Long-term outcome 5. Summary

19 ICCS / IV Conference, Prague Franz Schott: Measuring the outcome of education Purposes of measuring the outcome of education International comparison of the quality of schools (e.g. PISA) Certification of the learner (e.g. diploma, university degree) Feedback to the learner Feedback for the teacher Maintaining and improving the quality of instruction Problems of measuring the outcome of education We discussed arguments against the measuring of the output of education Measuring the output of education is incompatible with human dignity is not possible in the case of advanced skills emotions, attitudes and values long-term outcomes of education leads to teaching to the test We suggest that appropriate procedures of outcome-oriented instruction can help to teach and test the outcome of education A careful task analysis supports measuring the outcome of education => Measuring the outcome of education is in many cases useful and humanitarian 5. Summary

20 ICCS / IV Conference, Prague Franz Schott: Measuring the outcome of education Schott, F. & Azizi Ghanbari, S. (2008). Kompetenzdiagnostik, Kompetenzmodelle, kompetenzorientierter Unterricht. Zur Theorie und Praxis überprüfbarer Bildungsstandards. Münster: Waxmann Verlag. Schott, F & Azizi Ghanbari, S. (2010). Zur Theorie und Praxis kompetenzorientierten Lehrens und Lernens: Probleme und Lösungsmöglichkeiten. Report Psychologie. 11/ , Schott, F & Azizi Ghanbari, S. (2011, in Vorbereitung). Der Beitrag von Bildungsstandards, kompetenzorientiertem Unterricht und Kompetenzdiagnostik zur Qualitätssicherung des Bildungswesens in der Schule: Eine problemorientierte Einführung. Münster: Waxmann Verlag. Schott, F. & Seidl, P. (1994): PLANA - an ID model focussing on instructional task analysis. In: R. Tennyson, F. Schott, N. Seel & S. Dijkstra (Eds.) (1997): Instructional Design: International Perspectives. Volume I: Theory, Research, and Models. pp. 395 – 413. Lawrence Erlbaum Associates, Inc., Publishers: Hillsdale, New Jersey. Stevens, S. S. (1946). On the theory of scales of measurement. Science, 103 (2684): 677–80 Literature


Download ppt "ICCS / IV Conference, Prague 2011. Franz Schott: Measuring the outcome of education MEASURING THE OUTCOME OF EDUCATION FRANZ SCHOTT UNIVERSITY OF TECHNOLOGY,"

Similar presentations


Ads by Google