1. Metadata in the Educational Sciences
Dr. Ingo Barkow, Associate Professor for Operational Data Management

2. Agenda Introduction Metadata in the educational sciences
The Swiss National School Monitoring (ÜGK / COFO)
An example process for metadata management in ÜGK
Where DDI can help
Where DDI (currently) cannot help
Q&A

3. Obligatory Tourism Slide
University of Applied Sciences HTW Chur
1.600 Students
3 Departments (Tourism, Management, Information Science)
Swiss Institute for Information Science
Master of Science (MSc) in Information Science and Data Management
Project Team for Research Data Management

4. Metadata in the educational sciences
Research process in the educational sciences is very similar to the social sciences
Models like the Generic Longitudinal Business Process Model (GLBPM) can also be applied here
Most large-scale studies (e.g. PISA, PIAAC, PIRLS, TIMMS) use a mix of questionnaires and cognitive items
Shift from paper-based assessment towards computer-based assessment
Metadata standards (e.g. QTI, SCORM) exist, but only for very specific parts of the research process

5. Metadata in the educational sciences
Specific challenges in computer-based assessment
Layout and screen size must be fixed to avoid changes in item difficulty
Scoring rules have to be implemented (and documented)
Very complex item types (e.g. simulations)
Mode effects (e.g. paper to computer, computer to tablet)
Use of logfile data analysis e.g. for diagnostic assessment
Statistical parameters (e.g. Cronbach‘s Alpha) have to be stored to support adaptive testing
Items are usually highly confidential (e.g. SAT, GMAT)

6. Other metadata standards in ÜGK
Some interactions from IMS Questionnaire and Test Interoperability (QTI)
ChoiceInteraction (multiple choice items with one or more correct answers)
OrderInteraction (sorting answers in a list)
AssociateInteraction (building pairs of answers)
MatchInteraction (assigning correct responses to a matrix)
GapMatchInteraction (matrix with open answers)
InlineChoiceInteraction (choice of different answers within a text)
TextEntryInteraction (open text boxes within different media)
ExtendedTextInteraction (longer text box for free text passages)
HotTextInteraction (choices of several text answers within a text)
HotSpotInteraction (clickable areas in a graphic)
SelectPointInteraction (marking an area in a graphic)
GraphicOrderInteraction (marking hotspots with numbers)
GraphicAssociateInteraction (building relations between hotspots in graphics)
GraphicGapMatchInteraction (graphics with text boxes to enter data)
PositionObjectInteraction (similar to HotSpot, but a predefined graphic can be used)
SliderInteraction (offers a slider bar for a numeric answer)
DrawingInteraction (offers a sketchboard for drawing)
UploadInteraction (data created by another tool can be attached as an answer)

7. ÜGK Mathematics Item
DIPF PowerPoint-Präsentation

8. PIAAC Problem Solving Item
DIPF PowerPoint-Präsentation

9. CBA Itembuilder MicroFIN item
DIPF PowerPoint-Präsentation

10. Background: «EDK» and «HarmoS»
In Switzerland, the main responsibility for education and culture lies with the cantons. They coordinate their work at the national level.
The 26 cantonal ministers of education (called “directors of education”) together form a political body to carry out this work: the Swiss Conference of Cantonal Ministers of Education (EDK).
Legally binding, intercantonal agreements (known as concordats) form the foundation for the work of the EDK.
One of these agreements is the HarmoS concordat, the Intercantonal Agreement on the Harmonization of Compulsory Education.
It came into effect in 2009 and is aimed at harmonizing some cornerstones of the educational system nationwide.

11. The Swiss National Objectives in Education and «ÜGK»
The Swiss National Objectives in Education are some of these cornerstones of the HarmoS concordat.
They were developed until 2011 and describe which core skills students in all cantons should have obtained after 2, 6 and 9 years of school in languages, math and natural sciences.
They are now part of all curriculums in all cantons.
In 2013 it was decided how to assess these objectives ad if the harmonization has been successful.
Therefor the Swiss National Core Skills Assessment Programme (in German: ÜGK – Überprüfung der Grundkompetenzen) was initiated.
First survey in spring 2016: math, 9th school year
Second Survey in spring 2017: languages, 6th school year
The cantons are thinking about institutionalizing these assessments and coordinating them with the PISA assessments
15 Partners conducting theses surveys

12. Tasks of HTW Chur in «ÜGK»
Leading the work package IT support and data management
IT support:
Coordinating and organizing the hardware with the service provider
Coordinating the use and set up of software and servers with DIPF
Optimizing and developing the tools for the survey management
Data management:
Introducing and optimizing the data management
Introducing metadata standards for educational data
Consulting for FORS regarding the preservation of educational data and metadata

13. A Tool for Survey Management
Will be developed for PISA and ÜGK
Development started in spring 2017
The tool should support the fieldwork and data management in an adequate manner:
Support the administration and field monitoring during the data collection process
Optimize the data documentation process by generating documentation und metadata as a byproduct without additional effort during the data collection process
Tool bases on a modified version of Rogatus Survey 2.0 (specifically the case management system)
Current version only supports CAPI – will be modified to use case “data collections in schools”
Should be able to use different metadata standards for preservation

14. Metadata-driven research process
Research project information stored in survey management system
Items enriched with metadata during design process and stored in item bank
Data collection using survey / case management system thus creating paradata / log files
Storing of all metadata, paradata and data in research data facility
Metadata and data are linked to existing publication (enhanced publication)
All items and results are available for re-use in research data facility
Research data facility ensures long-time preservation by rollover processes for format changes
Here are examples for all processes with predecessors of the upcoming toolset

15. Metadata-driven Survey Management
Software: IAB Metadata Portal (TBA21)
Metadata Standard: DDI Lifecycle 3.2

16. Metadata-driven Item Design & Item Banking
Software: TIPO Item Portal (TBA21/DIPF) / TAO (o.a.t. S.A.)
Metadata Standard: QTI v2.1

17. Metadata-driven Data Collection Process (Mobile)
Software: Rogatus Survey / Aitema Mobile Client (TBA21)
Metadata Standard: DDI Lifecycle 3.2 (not for paradata)

18. Metadata-driven Data Dissemination
Software: Rogatus Survey / Administrator Panel (TBA21)
Metadata Standard: DDI Lifecycle 3.2

19. Metadata-driven Publication Process
Software: Invenio 0.99 (CERN)
Metadata Standards: MARC21 / DataCite

20. Metadata-driven Secondary Data Analysis / Retrospective
Software: FORSbase (FORS)
Metadata Standard: DDI Lifecycle 3.2

21. Where DDI can help
DDI Lifecycle v3.2 can be used in “ÜGK” to store survey information and questionnaire design including routing
It can be the standard to forward these metadata and data from the data collection towards the data archive
Currently it can be used for the questionnaires as well as very simple cognitive items (items with a simple stimulus like a picture plus a multiple choice battery)
Upcoming survey tool is DDI-based as well as the repository at FORS
New project between HTW and FORS to create common DDI Profile based on former IAB metadata system (meaning it also contains common elements from previous versions of Questasy and Colectica)

22. Where DDI (currently) cannot help
DDI Lifecycle v3.2 is lacking the following features for computer-based assessment
Layout and screen size to avoid changes in item difficulty
Scoring rules
Response domains like “point-and-click”, “hotspot” or “highlighting”
Use of paradata / logfile data analysis e.g. for diagnostic assessment
Statistical parameters (e.g. Cronbach‘s Alpha) have to be stored with the cognitive item to support adaptive testing
Support for more complex item types like simulations
Very important are also the modifications due to the shift to DDI Lifecycle 4.0
Does DDI4 contain the same routing capabilities like DDI3.2 so a questionnaire or test can be rendered directly?
Otherwise DDI will be useless to survey organisations

23. Thanks for your attention.