Presentation is loading. Please wait.

Presentation is loading. Please wait.

Advancements in Large-scale Assessment Heiko Rölke & Krisztina Toth DIPF - Deutsches Institut für Internationale Pädagogische Forschung Frankfurt am Main.

Similar presentations


Presentation on theme: "Advancements in Large-scale Assessment Heiko Rölke & Krisztina Toth DIPF - Deutsches Institut für Internationale Pädagogische Forschung Frankfurt am Main."— Presentation transcript:

1 Advancements in Large-scale Assessment Heiko Rölke & Krisztina Toth DIPF - Deutsches Institut für Internationale Pädagogische Forschung Frankfurt am Main TAO Days | 10.09.2012 Seite 1

2 Overview Trends in LSA Closer Look at Trends Example: Making Use of Data Challenges in LSA TAO Days| 10.09.2012 Seite 2

3 Trends Main trend: CBA Sub-trends/effects of CBA: CAT towards formative assessment Complex items Simulation, interconnection Interweaving different tests, questionnaires, etc. Big data TAO Days | 10.09.2012 Seite 3

4 Trend: CBA Computer-based Assessment has to serve a purpose Costs? Time! Validity! TAO Days | 10.09.2012 Seite 4

5 CAT Also not practiced for its own sake Not as important in pure summative assessment But: Strict time limits (e.g. PIAAC) Growing demand for formative aspects in summative assessment TAO Days | 10.09.2012 Seite 5

6 Complex Items Closer at reality Simulation Most important: Ability to assess new domains TAO Days | 10.09.2012 Seite 6

7 Examples of Complex Items Simulation of an Email / Web Scenario The test person receives an email and should book cinema tickets online. Dynamic Model (MicroDYN) The test person should explore and master a dynamic system with input (exogenous) variables influencing output (endogenous) variables. Automaton with Finite State Machine (MicroFIN) The test person should interact with a mobile phone and set time to summertime.

8 Interweaving of Tests and Questionnaires Framing test only if preconditions are fulfilled Double-check findings of questionnaire TAO Days | 10.09.2012 Seite 8

9 Big Data Log data, not only results Find out what is going on Make use of data E.g. partial scoring -> Examples from PIAAC PS-TRE TAO Days | 10.09.2012 Seite 9

10 Simulation-based assessment Modern assessments: real life situations: e.g. web environments complexity of instrument – various ways Monitoring students actions - log files Analysis of test-taking paths is a relatively new field in education (reason: printed vs. computer-based data collection) Process data requires to consider and evaluate methods Berlin, 10.9.2012 | H. Rölke & K. Tóth | TAO Days 2012 | Advancements in Large-Scale Assessments 10

11 Aim to investigate how log data can be integrated into the process of educational assessment and evaluation to support researchers and practitioners in making use of the data assembled in computer-based test delivery Berlin, 10.9.2012 | H. Rölke & K. Tóth | TAO Days 2012 | Advancements in Large-Scale Assessments 11

12 Aim to investigate how log data can be integrated into the process of educational assessment and evaluation to support researchers and practitioners in making use of the data assembled in computer-based test delivery Berlin, 10.9.2012 | H. Rölke & K. Tóth | TAO Days 2012 | Advancements in Large-Scale Assessments 12

13 Hypertext Item 13

14 14

15 15 82970915{"sender":"15","type":"loading","data":""} 83346315{"sender":"15","type":"loaded","data":""} 84815315{"sender":"15","type":"user_interaction","":" \u000d\u000a \u000d\u000a"} 84909415{"sender":"15","type":"variable_change","data":{"name": "snapshot_url,"value":"http://localhost:8101/cba-runtime/itemjsessionid= LB1?custom_servicehandler=downloadService&file= C:\\...snapshot5537437125227002523.xml"}} 85585215{"sender":"15","type":"user_interaction","data":"\u000d\u000a \u000d\u000a}":"15","type":"unloaded","data":""} Example log

16 Promising methods Statistics and visualisation Clustering Classification Berlin, 10.9.2012 | H. Rölke & K. Tóth | TAO Days 2012 | Advancements in Large-Scale Assessments 16

17 Process Measures Berlin, 10.9.2012 | H. Rölke & K. Tóth | TAO Days 2012 | Advancements in Large-Scale Assessments 17 1. Number of page visits 2. Number of different page visits 3. Visit of relevant page 4. Time spent on the relevant page 5. Ratio of time spent on the relevant page 6. Ratio of time spent on the opening screen 7. Completion time

18 Statistics Process measuresMean Standard Deviation Number of page visits5.183.13 Number of different page visits2.381.38 Time spent on the relevant page10.879.77 Ratio of time spent on the relevant page.14.11 Ratio of time spent on the opening screen.64.18 Completion time67.7427.49 Berlin, 10.9.2012 | H. Rölke & K. Tóth | TAO Days 2012 | Advancements in Large-Scale Assessments 18

19 Distribution of different page visits Berlin, 10.9.2012 | H. Rölke & K. Tóth | TAO Days 2012 | Advancements in Large-Scale Assessments 19

20 Distribution of different page visits Berlin, 10.9.2012 | H. Rölke & K. Tóth | TAO Days 2012 | Advancements in Large-Scale Assessments 20

21 Visualisation 21 Berlin, 10.9.2012 | H. Rölke & K. Tóth | TAO Days 2012 | Advancements in Large-Scale Assessments

22 Visualisation 22 Berlin, 10.9.2012 | H. Rölke & K. Tóth | TAO Days 2012 | Advancements in Large-Scale Assessments

23 Cluster Analysis FeaturesCluster 1Cluster 2Cluster 3Cluster 4 Nr. of page visits Nr. of different page visits Completion time Relevant page visited (Y/N) Ratio of time on relevant page Ratio of time on starting page Distribution of sequences (%) 23%16%34%27% Ratio of correct responses 23 Berlin, 10.9.2012 | H. Rölke & K. Tóth | TAO Days 2012 | Advancements in Large-Scale Assessments

24 Cluster Analysis FeaturesCluster 1Cluster 2Cluster 3Cluster 4 Nr. of page visits Nr. of different page visits Completion time Relevant page visited (Y/N) Ratio of time on relevant page Ratio of time on starting page Distribution of sequences (%) 23%16%34%27% Ratio of correct responses25.0%1.7%92.6%84.4% 24 Berlin, 10.9.2012 | H. Rölke & K. Tóth | TAO Days 2012 | Advancements in Large-Scale Assessments

25 Cluster Analysis FeaturesCluster 1Cluster 2Cluster 3Cluster 4 Nr. of page visits.725.41 Nr. of different page visits.362.54 Completion time28.3862.08 Relevant page visited (Y/N)No Ratio of time on relevant page00 Ratio of time on starting page.93.60 Distribution of sequences (%) 23%16% Ratio of correct responses25.0%1.7% 25 Berlin, 10.9.2012 | H. Rölke & K. Tóth | TAO Days 2012 | Advancements in Large-Scale Assessments

26 Cluster Analysis FeaturesCluster 1Cluster 2Cluster 3Cluster 4 Nr. of page visits4.739.57 Nr. of different page visits2.294.53 Completion time59.6091.60 Relevant page visited (Y/N)Yes Ratio of time on relevant page.20.17 Ratio of time on starting page.62.45 Distribution of sequences (%) 34%27% Ratio of correct responses92.6%84.4% 26 Berlin, 10.9.2012 | H. Rölke & K. Tóth | TAO Days 2012 | Advancements in Large-Scale Assessments

27 Cluster Analysis FeaturesCluster 1Cluster 2Cluster 3Cluster 4 Nr. of page visits5.414.73 Nr. of different page visits2.542.29 Completion time62.0859.60 Relevant page visited (Y/N)NoYes Ratio of time on relevant page0.20 Ratio of time on starting page.60.62 Distribution of sequences (%) 16%34% Ratio of correct responses1.7%92.6% 27 Berlin, 10.9.2012 | H. Rölke & K. Tóth | TAO Days 2012 | Advancements in Large-Scale Assessments

28 Cluster Analysis FeaturesCluster 1Cluster 2Cluster 3Cluster 4 Nr. of page visits.725.414.739.57 Nr. of different page visits.362.542.294.53 Completion time28.3862.0859.6091.60 Relevant page visited (Y/N)No Yes Ratio of time on relevant page00.20.17 Ratio of time on starting page.93.60.62.45 Distribution of sequences (%) 23%16%34%27% Ratio of correct responses25.0%1.7%92.6%84.4% 28 Berlin, 10.9.2012 | H. Rölke & K. Tóth | TAO Days 2012 | Advancements in Large-Scale Assessments

29 Summary and Conclusions Berlin, 10.9.2012 | H. Rölke & K. Tóth | TAO Days 2012 | Advancements in Large-Scale Assessments 29

30 Summary and Conclusions FeaturesCluster 1Cluster 2Cluster 3Cluster 4 Nr. of page visits.725.414.739.57 Nr. of different page visits.362.542.294.53 Completion time28.3862.0859.6091.60 Relevant page visited (Y/N)No Yes Ratio of time on relevant page00.20.17 Ratio of time on starting page.93.60.62.45 Distribution of sequences (%) 23%16%34%27% Ratio of correct responses25.0%1.7%92.6%84.4% 30 Berlin, 10.9.2012 | H. Rölke & K. Tóth | TAO Days 2012 | Advancements in Large-Scale Assessments

31 Future work Pilot study – sample size Validation Other types of items require new process measures Berlin, 10.9.2012 | H. Rölke & K. Tóth | TAO Days 2012 | Advancements in Large-Scale Assessments 31

32 32

33 Future work Pilot study – sample size Validation Other types of items require new process measures We have a lot of work to do: Software developer Test developer Psychometricans Berlin, 10.9.2012 | H. Rölke & K. Tóth | TAO Days 2012 | Advancements in Large-Scale Assessments 33

34 (Selected) Challenges Authoring and Management Delivery Re-use and exchange TAO Days | 10.09.2012 Seite 34

35 (Selected) Challenges Authoring and Management Delivery Re-use and exchange TAO Days | 10.09.2012 Seite 35

36 (Selected) Challenges Authoring and Management Delivery Re-use and exchange TAO Days | 10.09.2012 Seite 36

37 (Selected) Challenges Authoring and Management Delivery Re-use and exchange TAO Days | 10.09.2012 Seite 37


Download ppt "Advancements in Large-scale Assessment Heiko Rölke & Krisztina Toth DIPF - Deutsches Institut für Internationale Pädagogische Forschung Frankfurt am Main."

Similar presentations


Ads by Google