Presentation is loading. Please wait.

Presentation is loading. Please wait.

The Implementation of National Educational Standards in Austria, 18/05/2013 © BIFIE Salzburg The Implementation of National Educational Standards in Austria.

Similar presentations


Presentation on theme: "The Implementation of National Educational Standards in Austria, 18/05/2013 © BIFIE Salzburg The Implementation of National Educational Standards in Austria."— Presentation transcript:

1 The Implementation of National Educational Standards in Austria, 18/05/2013 © BIFIE Salzburg The Implementation of National Educational Standards in Austria Michael Bruneforth BIFIE - Austrian Federal Institute for Education Research, Innovation and Development of the Austrian School System EURASIAN EDUCATIONAL DIALOGUE April, Yaroslavl

2 The Implementation of National Educational Standards in Austria, 18/05/2013 © BIFIE Salzburg BIFIE - Austrian Federal Institute for Education Research, Innovation and Development of the Austrian School System Areas of work International Studies (PISA, TIMSS, PIRLS, TALIS) Implementation of national education standards Formative and summative Evaluation of standards Central examination for upper secondary education preparing for tertiary education National Education Reports

3 The Implementation of National Educational Standards in Austria, 18/05/2013 © BIFIE Salzburg Spotlight on Austrian education Very rich country (4 th highest GDP per Capita in EU) High expenditure per student (140% of EU Average) in primary and secondary education Low student teacher ratio and class size But PISA shock in 2003 and 2009: Very low PISA results in reading (470, similar to Russia 459), no EU country scores less Strong relation between education outcomes and family background

4 The Implementation of National Educational Standards in Austria, 18/05/2013 © BIFIE Salzburg Changing in paradigm in educational governance Since more than a decade there is a change in governance of schools and education in German speaking countries from input oriented control to evidence based output oriented governance and extension of school autonomy In Austria at the centre is shift towards competence oriented teaching based on the definition of nationwide compulsory education standards and a mechanism of formative and summative evaluation

5 The Implementation of National Educational Standards in Austria, 18/05/2013 © BIFIE Salzburg Legal Basis and Conditions From 2004: pilot phase with selected pilot schools 2008: alteration of the law of teaching in schools to allow for introduction of educational standards 2009: educational standards become compulsory for German, Math and English 2009 (grade 8) and 2010 (grade 4): baseline-tests in samples of about students each From 2012 onwards yearly assessment of national standards (one subject per year)

6 The Implementation of National Educational Standards in Austria, 18/05/2013 © BIFIE Salzburg Annex of the legal regulation Competency Models and descriptors are formulated as Can-Do-Statements Competency Models and descriptors are legally established Educational Standards can be interpreted as mandatory and agreed upon social objectives of instruction Educational Standards are derived from national curricula and clearly define learning outcome.

7 The Implementation of National Educational Standards in Austria, 18/05/2013 © BIFIE Salzburg Areas covered German (language of instruction) – grades 4 & 8 -Listening & Speaking -Reading -Writing texts -Spelling/Grammar Math – grades 4 & 8 4 content dimensions 4 process dimensions English, grade 8 -Listening -Reading -Speaking -Writing English Assessment: Common European Framework of Reference for Languages (CEF)

8 The Implementation of National Educational Standards in Austria, 18/05/2013 © BIFIE Salzburg Competencies are … fashionable term with a vague meaning (Weinert 2001) cognitive prerequisites to coping with a specific range of situations (Klieme, Hartig & Rauch, 2008) cognitive abilities and skills available to/acquired by individuals that enable them to solve specific problems, as well as the related motivational, volitional and social readiness and skills to utilize the solutions successfully and responsibly in variable situations (Weinert, 2002) BIFIE use: subject (domain) specific competency models, focusing on cognitive skills and proficiencies

9 The Implementation of National Educational Standards in Austria, 18/05/2013 © BIFIE Salzburg Competency Models Models of competence structures Dimensions of competence: E.g. TIMSS, grade 8: 4 content domains (number, algebra, geometry, data and chance); 3 cognitive domains (knowing, applying, reasoning ) Models of competence levels Describe specific skills and proficiencies usually shown by individuals located at specific points of the ability continuum Used to provide a criterion-referenced interpretation of measurement results

10 The Implementation of National Educational Standards in Austria, 18/05/2013 © BIFIE Salzburg Competency Models Models of competence structures Models of competence levels Development models Models describing the acquisition of skills and proficiencies May be linked to neurological/psychological theories of development Only a few exist Would be great to have a theoretical foundation for tests in different grades! (IKM – BIST)

11 The Implementation of National Educational Standards in Austria, 18/05/2013 © BIFIE Salzburg EXAMPLE MATHEMATICS, GRADE 8

12 The Implementation of National Educational Standards in Austria, 18/05/2013 © BIFIE Salzburg M8 – Competency Model Process domains (Handlungsbereiche) Content domains (Inhaltsbereiche) Level of complexity (Komplexitätsbereiche) demonstrating, modellingnumbers and unitsuse of basic skills and knowledge calculating, operatingvariable, functional dependency build connections/connect interpretinggeometric figures and shapes reflect, use knowledge of reflection explaining, reasoningstatistical representations, measures of central tendency and variance

13 The Implementation of National Educational Standards in Austria, 18/05/2013 © BIFIE Salzburg M8 – Competency Model The 4x4 grid leads to 16 nodes, using complexity as a 3rd dimension = 48 nodes Not very handy and the theoretical concept of complexity does not lend itself to assessment Content Process Complexity Competency (H3, I2, K2)

14 The Implementation of National Educational Standards in Austria, 18/05/2013 © BIFIE Salzburg Example: Competency Model M8 P: demonstrating, modelling C: numbers and measures Formulated as can-do-statements: The students are able to transform given arithmetical data into (another) mathematical representation. In doing so, direct use of basic skills is necessary transform given arithmetical data into (another) mathematical representation. In doing so, connections to other mathematical contents (terms, definitions, representations) or activities have to be established make and evaluate statements about the appropriateness as well as weaknesses and strengths of different mathematical representations (models) of arithmetical data P1/C1/K1 P1/C1/K2 P1/C1/K3

15 The Implementation of National Educational Standards in Austria, 18/05/2013 © BIFIE Salzburg Educational Standards – The Austrian Version Three areas of activities Implementation of standards at school and capacity building for teacher: Support of teachers and school heads in implementation Formative evaluation to support teachers diagnostic: On-line evaluation system to be used by teachers Census to monitor and summative evaluation at system, sub- system, school and classroom level

16 The Implementation of National Educational Standards in Austria, 18/05/2013 © BIFIE Salzburg Educational Standards – The Austrian Version School-specific development of instruction Mandatory educational standards Census-survey of all students and schools Feedback to all students, teachers, schools Feedback moderation (Rückmeldemoderation) Stimulation of quality development and support by MoE Strong linkage of monitoring and practical school and instructional development Strengthening of schools, school management and regional quality development

17 The Implementation of National Educational Standards in Austria, 18/05/2013 © BIFIE Salzburg Main functions of standards according to the legal regulation Output orientation: Sustainable output orientation in planning and implementation of instruction Promotion (Förderfunktion): Diagnosis and specific individual remediation and challenge Evaluation: reliable information about the output of instruction, schools and education system

18 The Implementation of National Educational Standards in Austria, 18/05/2013 © BIFIE Salzburg Implementation of Standards Supporting materials for instruction Praxishandbücher Themenhefte Aufgabenbeispiele, Interaktive Beispiele Best-Practice-Beispiele Integration with teacher training institutions In-service teacher training (partially mandatory) Pre-Service teacher trainin Research and development work by members of teacher training institutions

19 The Implementation of National Educational Standards in Austria, 18/05/2013 © BIFIE Salzburg Diagnosis tool for Informal Monitoring of Competencies Teacher administered on-line system Grade 3 and 7, one year before Assessment All Areas of standards Only teachers themselves have access to results

20 The Implementation of National Educational Standards in Austria, 18/05/2013 © BIFIE Salzburg Assessment of Educational Standards Legal assessment cycle: three (school) years In each cycle, the whole target population has to be assessed in all prescribed competency subdomains Grade 4: German, Mathematics Grade 8: German, Mathematics, English Students, teachers, schools and regional education authorities must get feedback about the results (to be used for quality improvement)

21 The Implementation of National Educational Standards in Austria, 18/05/2013 © BIFIE Salzburg Domain-orientated Test Design

22 The Implementation of National Educational Standards in Austria, 18/05/2013 © BIFIE Salzburg Pilot testing: about 10% of population, up to 9000 Students Main test: Census up to students Speaking domain only with sample for system monitoring 10% of census with external administration and items for next census to ensure linking over time Example: Test session Mathematics Grade 4

23 The Implementation of National Educational Standards in Austria, 18/05/2013 © BIFIE Salzburg This picture shows about 1/10 of the test materials

24 The Implementation of National Educational Standards in Austria, 18/05/2013 © BIFIE Salzburg Item Development Items are developed to measure sub-competencies or domains, eg. Content = numbers and units and process = demonstrating Dimension complexity is not used as such, since it cant be measured empirically We have 16 nodes, upon which an item should fall. We strive for unidimensional items, i.e. items that can be classified as belonging to one specific node only

25 The Implementation of National Educational Standards in Austria, 18/05/2013 © BIFIE Salzburg Item Development Items are developed to measure sub-competencies or domains, eg. Content = numbers and units and process = demonstrating Dimension complexity is not used as such, since it cant be measured empirically We have 16 nodes, upon which an item should fall. We strive for unidimensional items, i.e. items that can be classified as belonging to one specific node only

26 The Implementation of National Educational Standards in Austria, 18/05/2013 © BIFIE Salzburg Item Development

27 The Implementation of National Educational Standards in Austria, 18/05/2013 © BIFIE Salzburg Item Development Test blueprint specified the items to be developed: Mathematical node Item format: Multiple Choice, Constructed response, … Estimated difficulty of item Further requirements: adequat for age group, no biased towards gender, race, religion, also taking into account rural/urban differences. does not require other knowledge than maths contents. Items should be uni-dimensional

28 The Implementation of National Educational Standards in Austria, 18/05/2013 © BIFIE Salzburg Validation In order to secure validity of the test items, they were … written by school and university teachers reviewed by teachers (peer review and outsider review) trialled on a sample of grade 8 students – each item was answered by at least 200 students item properties were reviewed by a team of psychometricians and school teachers items with less than desirable or surprising properties were retired and not used in the main test

29 The Implementation of National Educational Standards in Austria, 18/05/2013 © BIFIE Salzburg Test Design Multiple matrix sampling Two-step approach for the test design Allocation of empty blocks to forms that satify several global prerequisites for the design: positional and contextual balance, item security (S90 vs. S10), longitudinal link (BL and BIST-UE 2015) Allocation of items to blocks that satisfy several local prerequisites: Balanced difficulty, format, representative for the competence construct, dependency (enemy items), test motivation Test information, especially along cutoff-scores

30 The Implementation of National Educational Standards in Austria, 18/05/2013 © BIFIE Salzburg Automated Test Assembly Computer-based via automated test assembly methods 1.Simulated annealing approach for the allocation of blocks to forms 2.Big-shadow-test approach for the allocation of items to blocks Sequentially filling blocks via combinatorial optimization Therein, an objective function is the model for the quality of the design Here, the optimization of ist value is bound to the optimization of the test information along the cutoff-scores Also, the objective is constraint to a space of feasible values by the prerequisites

31 The Implementation of National Educational Standards in Austria, 18/05/2013 © BIFIE Salzburg Scaling Using Rasch model Allows to express difficulties of items and students ability on one common scale Allows to match test cycles over time Allows for complex test design

32 The Implementation of National Educational Standards in Austria, 18/05/2013 © BIFIE Salzburg Standard Setting Standard setting goal: meaningfully described levels criterion-referenced Phases: Determination of number of levels Labelling of levels (PLLs) Description of levels (PLDs) Establishing the cut scores between levels 4 levels in Austria in most domains 0: standards not reached 1: partially reached standards 2: reached standards 3: exceeded standards Easy to understand for laymen

33 The Implementation of National Educational Standards in Austria, 18/05/2013 © BIFIE Salzburg Proficiency Level Descriptions M8 3 – standards exceeded Students possess fundamental knowledge and skills in all parts of the mathematics curriculum and advanced knowledge structures, which exceed the requirements of level 2, specifically more pronounced abilities of abstraction and higher proficiency in combining parts of knowledge, methods or rules. They are able to apply these independently in novel situations in a flexible way. 2 – standards reached Students posess fundamental knowledge and skills in all parts of the mathematics curriculum and can use these in a flexible way. They are able to find and apply problem solving strategies, to describe and reason an approach. They are able to handle verbal, graphical and formal representations of mathematical facts in a flexible way and can apply these appropriately. Theay are able to extract relevant information from a differently represented facts (e.g. texts, data material, graphics) and can interpret them in the respective context. They are able to relate their mathematical knowledge and can check, evaluate and/or reason mathematical statements. 1 – standards partially reached Students possess fundamental knowledge and skills in all parts of the mathematics curriculum and can master reproductive tasks and carry out routine procedures.

34 The Implementation of National Educational Standards in Austria, 18/05/2013 © BIFIE Salzburg Standard Setting A standard setting method is a mechanism for helping panelists translate their intentions to the score scale. (Reckase, 2006) Goal: Match predetermined descriptors of competency levels to an empiric scale defined by test items Determine cut scores between levels of competence Bifie invited a group of stakeholders to set the cut scores: teachers (lower and upper secondary), teachers of didactics of math; representatives of parents associations; representatives of the MoE; representatives of chamber of commerce; bifie staff Method used: Item descriptor matching method

35 The Implementation of National Educational Standards in Austria, 18/05/2013 © BIFIE Salzburg Ordered Item Booklet Easy Item Difficult Item Level 2 Level 3 Level 1 Item-descriptor matches Level 2 Level 3 Level 1 (Modified) Item descriptor matching method Freunberger & Yanagida, in press. Difficulty-ordered items Cut Score 1 Cut Score 2

36 The Implementation of National Educational Standards in Austria, 18/05/2013 © BIFIE Salzburg Standard Setting Standard setting is more about finding a societal consensus than finding a constant The goal should be clear before test development etc starts

37 The Implementation of National Educational Standards in Austria, 18/05/2013 © BIFIE Salzburg Feedback and Reports Information about results of Educational Standards SCHOOL LEVEL School report (for school management) per school, online Feedback to teachers per teacher, per class/group; online students about in grade 4 and 8 each classes about 6000 primary and 4000 secondary I School and instruction development 4600 schools about 3200 primary and 1400 secondary I Feedback to students per participant, online

38 The Implementation of National Educational Standards in Austria, 18/05/2013 © BIFIE Salzburg Province reports (pres & vice-) for each procince and grade SYSTEM LEVEL 99 school districts 15 Cities – 84 districts National report (Minister for Education) aggregated 9 provinces Primary/4 Lower sec/8, academic sec/8 9 provinces MoE School development System development Supervisory reports (federal superv. authorities for PS & LS/AS) aggregated on fed. state and supervised-school type Feedback and Reports Information about results of Educational Standards Supervisory reports (district superv. authorities for PS and LS) per district and grade

39 The Implementation of National Educational Standards in Austria, 18/05/2013 © BIFIE Salzburg School report: overview of content Information about school results: compared to AUT compared to the defined goals (in the regulation) compared to other schools with similar basic conditions (fair comparison) Areas of conflict: Educational Standards apply to all students – and therefore to all schools Conditions of schools vary widely

40 The Implementation of National Educational Standards in Austria, 18/05/2013 © BIFIE Salzburg FEEDBACK The school takes centre stage

41 The Implementation of National Educational Standards in Austria, 18/05/2013 © BIFIE Salzburg Feedback – example (school report) Competency levels

42 The Implementation of National Educational Standards in Austria, 18/05/2013 © BIFIE Salzburg Feedback – example (teacher feedback)

43 The Implementation of National Educational Standards in Austria, 18/05/2013 © BIFIE Salzburg Feedback – example (school report)

44 The Implementation of National Educational Standards in Austria, 18/05/2013 © BIFIE Salzburg Support system for schools A SCHOOL School feedback Teacher feedback School authorities Feedback Moderation Quality development- process supervision IKM (informal self evaluation)

45 The Implementation of National Educational Standards in Austria, 18/05/2013 © BIFIE Salzburg Outlook Bringing IKM (grade(s) before BIST) and BIST assessments closer together. Using items in both assessments could lead towards a better understanding of learning Backwash from doing the assessments towards research in the areas of item development, standard setting, feedback … through evaluation of results Outcomes of the assessments should not only allow us to give feedback to schools, teachers and students but also to improve the competence models we use for the assessment Prepare for computer based testing Open databases for external researchers and improve system level monitoring

46 The Implementation of National Educational Standards in Austria, 18/05/2013 © BIFIE Salzburg Thank you for your attention! MichaeI Bruneforth Bundesinstitut BIFIE Salzburg | Zentrum für Bildungsmonitoring & Bildungsstandards


Download ppt "The Implementation of National Educational Standards in Austria, 18/05/2013 © BIFIE Salzburg The Implementation of National Educational Standards in Austria."

Similar presentations


Ads by Google