Ove Edvard Hatlevik and Ingrid Radtke An Application of Standard-Setting Methods in a Formative Assessment in Digital Responsibility among Norwegian Eight Graders.

Agenda Introduction Digital skills in the curriculum Digital responsibility Experience with standardsetting

Norwegian Centre for ICT in education Governmental agency Activities – Online guidelines for schools and educators – Summarizing experiences with methods and tools in the classroom – Surveys and monitoring

Assessment Summative assessment Formative assessment – For example voluntary learning supportive tests – Feedback on how to improve

What to measure? Concept – Digital competence or ICT literacy or … Framework – E.g. the DIGCOMP framework (Ferrari, 2013) has five areas with 21 dimensions and three proficiency levels – European Commission, Joint Research Centre, Institute for Prospective Technological Studies

Different concepts ICT literacy (ETS, 2006; Ainley et al., 2007) – Computer and Information literacy (Fraillon et al., 2013) – The International Computer and Information Literacy Study (ICILS) 2013 Digital literacy (Aviram & Eshet-Alkalay, 2006 ) Digital competence (Erstad, 2006; Ferrari, 2012) Internet skills (Van Deursen & Van Dijk, 2009 ) iSkills (ETS, 2008)

About the concepts Often tools or technology (e.g. ICT/Digital/Internet) combined with learning domains (e.g. skill, competence, literacy) Definitions often include modes (critically or reflectively) or purposes (work or leisure)

One example of an international used definition of CIL Computer and information literacy refers to: “individual’s ability to use computers to investigate, create, and communicate in order to participate effectively at home, at school, in the workplace, and in society” (Fraillon et al., 2013, p. 17). – The International Computer and Information Literacy Study (ICILS) 2013

International frameworks iSkills (Educational testing service, 2006) ATC21s (Binkley et al, 2012) DIGCOMP (Ferrari, 2013) Computer and information literacy (Fraillon et al, 2013) Australian Curriculum, Assessment and Reporting Authority (ACARA) (2012)

National initiatives to assess some of these concepts / frameworks Australia Chile Norway Korea U.S. International Computer and Information Literacy Study 2013 (Fraillon et al., 2014)

National Curriculum (Norway) Digital skills as a “key competence” Sources describing digital skills – A general framework of basic skills – A general description within the subjects – Explicit competence aims for the subjects The Norwegian Directorate for Education and Training

Digital skills … … involve being able to use digital tools, media and resources efficiently and responsibly, […] … include developing digital judgement by acquiring knowledge and good strategies for the use of the Internet. The Norwegian Directorate for Education and Training (2012, p. 12)

General framework of digital skills To acquire and process digital information To produce digital information Digital judgementCommunication

Example of a digital judgement task from the ICILS 2013

Example of description: Digital skills in Natural science involves … […] mastering searching and research strategies, learning to evaluate sources critically and selecting relevant information about themes within natural sciences.

Examples of one competence aims (social science): use digital tools to present work in the social sciences and comply with the rules for personal data protection and copyright.

Assessment team 7 employees at the ICT- senter Different skills and background - teaching-experience - ict-skills - research and statistics - assessment skills

Assessment in Digital Responsibility One out of 5 formative assessments in digital competence In cooperation with the Directorate of Education Available from november 2015

Assessment development 3 rounds with piloting (and visiting schools) Analyses with classical test-thoery and IRT Standard setting with Angoff- and Bookmark- method

Definition Digital Responsibility Digital judgement means being able to use digital tools, media and resources in a responsible manner, and being aware of rules for protecting privacy and ethical use of the Internet. Search and process: … interpret digital information appropriately and critically. Produce …. Be aware of copyright issues.

Digital Responsibility Items were developed in 6 content areas – critical use of digital sources – ownership of digital material – safety issues – personal integrity – cyber bullying – green data

Scoring and item-types scoring - each correct answered item gives one score. item-types - multiple choice and multiple response, knowledge matrix, matching, pull- down list and drag and drop.

Exsamples for Competence Aims Norwegian use different types of digital and paper-based dictionaries be familiar with copyright rules on the use of sources

Application of Digital Responsibility Use different types of digital dictionaries – critical use of digital sources – safety issues – personal integrity – ownership of digital material – cyber bullying – green data

Development of Assessment Autumn 2013, discussion and small pilot with 2 classes Spring 2014, 90 items with 1395 students from 36 schools Nov 2014, 60 items with 1062 students from 26 schools, representative sample

Final assessment – 50 items

Performance standards Three performance standards Low – middels- high Lowest level was divided into two

Performance standards Related to the items in the test Not the ‘ideal’ student

Standard setting workshop 10 experts Help from Cito to conduct the workshop

Expert panel Teacher with hands on experience with the use of ICT in schools Other subject-matter experts Stakeholders?

About the workshop Introduction of 3 performance standards Training session with Angoff-method Both Angoff- and Bookmark-method were used

Angoff method Suppose a group of 100 borderline students. How many of them will answer this item correctly? Cut-off points. Sum judgements. Reflections

Bookmark method List with ordered items by difficulty after IRT- analyses Experts choose the items which mark the borderline for the levels of the students. Cut-off points: average of the bookmarks set by experts.

Results About the same cutting point for the higher level, level 2-3 Higher cutting point for the lower level after Angoff-method, level 1-2 Standard-deviation

Decision-making Which factors would be important?

1. Distribution of items and students Number of items to qualify for a certain level. Number of students on that level. (As seen from the pilot-data.) Difficulty of items around the cutting-score.

2. More validity arguments Level 3 should be exclusive! Not too many students in Level 1!

Guidance material Group-level Item-level Organisation-level

Way forward! Limitations? Science versus individual judgements Validity – accept among teachers Aim is to improve teaching and learning!