ON DEMAND TESTING and the Victorian Curriculum Fran O’Mara Project Manager

Introduction Welcome to this On Demand testing Webinar. My name is Fran O’Mara and I manage On Demand Testing for the VCAA.

Introduction What we will cover Brief overview of On Demand. What tests and items are available What has changed in the tests with the transition of On Demand from AusVELS to the new Victorian Curriculum

Introduction What we will cover Uses of On Demand tests Examples of Students’ test paths with questions On Demand Reports and interpreting the On Demand Standard Score

Introduction What we will NOT cover Technical setup and hands-on management of administrative functions in On Demand (If there is a need, PD on how to navigate the program can be scheduled at a later date)

Introduction What we will NOT cover Technical questions / System problems. Please still submit these questions and if time permits, I will take some on notice at the end of the session. Otherwise, we will contact you to resolve your technical issues later.

Do you already use On Demand Testing at your school? VCAA Assessment Online - On Demand Testing Program Show of Hands Do you already use On Demand Testing at your school? Yes No

What level of expertise do you have with On Demand? VCAA Assessment Online - On Demand Testing Program Show of Hands What level of expertise do you have with On Demand? Expert Comfortable Beginner

VCAA Assessment Online - On Demand Testing Program Introduction On Demand Testing delivers online tests from the VCAA central repository to students’ computers via the Internet and a school’s local network. Teachers use the resource when, where and how they choose.

Introduction On Demand testing is optional. VCAA Assessment Online - On Demand Testing Program Introduction On Demand testing is optional. It is a time-saving resource that can be administered to a single student and/or a whole class. And it’s free.

VCAA Assessment Online - On Demand Testing Program Technical Overview

Types of Tests Available VCAA Assessment Online - On Demand Testing Program Types of Tests Available Computer Adaptive Tests are available for Mathematics and English. Linear tests are not currently available. Available tests list: http://www.vcaa.vic.edu.au/Documents/ondemand/available_odtests.pdf

Type of Tests Available VCAA Assessment Online - On Demand Testing Program Type of Tests Available Mathematics: Years 3 - 10 60 Item General Tests 30 Item Focus Tests New: Statistics and Probability Number and Algebra Measurement and Geometry

Type of Tests Available VCAA Assessment Online - On Demand Testing Program Type of Tests Available Resources On Demand tests and Victorian Curriculum coverage.pdf (for English Tests only)

Type of Tests Available VCAA Assessment Online - On Demand Testing Program Type of Tests Available English: Years 3 - 10 60 Item General Tests 30 Item Focus Tests Reading Language Skills Word Building

Show of Hands Which Tests do you use most? Mathematics English Both VCAA Assessment Online - On Demand Testing Program Show of Hands Which Tests do you use most? Mathematics English Both

Type of Tests Available VCAA Assessment Online - On Demand Testing Program Type of Tests Available Resource Transition_AusVELS to VC.pdf Reporting has changed for Levels and Domains. Scales have been maintained to generally align. However, it must be remembered that the item pool has been recalibrated. Outliers have been removed, items have been adjusted to align with the new curriculum.

What’s changed in the tests VCAA Assessment Online - On Demand Testing Program What’s changed in the tests All Tests: No Standard Level 1 Minimum Score is ‘Below 1.0’ Mathematics ends at Level 10 Adv. Maximum score is ‘Above 10.9’

What’s changed in the tests VCAA Assessment Online - On Demand Testing Program What’s changed in the tests English ends at Level 10 Maximum score is ‘Above 9.9’ English Curriculum coverage has been expanded and changed for tests.

Show of Hands Which English Tests do you use most? General Reading VCAA Assessment Online - On Demand Testing Program Show of Hands Which English Tests do you use most? General Reading Language Skills Word Knowledge

What’s changed in the tests VCAA Assessment Online - On Demand Testing Program What’s changed in the tests English – Reading These tests focus on: Language Mode: Reading and Viewing Strand: Literacy Sub-strand: Interpreting, analysing, evaluating. Some Items previously in Reading are now part of the Language Skills and Word Knowledge tests

What’s changed in the tests VCAA Assessment Online - On Demand Testing Program What’s changed in the tests English – Language Skills These tests focus on the Language Modes of Reading and Viewing, and Writing – multiple Strands Items test knowledge about language and language use, from text purpose, structures and features, to finer features at the paragraph and syntactic level.

What’s changed in the tests VCAA Assessment Online - On Demand Testing Program What’s changed in the tests English – Word Knowledge These tests focus on the Language Modes of Reading and Viewing, and Writing – multiple Strands Items test knowledge of word formation, word changes that follow function, precision in word use for shades of meaning, and spelling Some of these Items used to be part of the reading tests bank Word Knowledge Items test knowledge of word formation complete the word word changes that follow function - verb endings precision in word use for shades of meaning – choice of what best describes knowledge of the sound-letter system homonyms, synonyms, antonyms Pure spelling

Show of Hands What year level ranges are you testing? 3 – 6 7 – 10 VCAA Assessment Online - On Demand Testing Program Show of Hands What year level ranges are you testing? 3 – 6 7 – 10 Pre-Year 3

Uses of On Demand Testing VCAA Assessment Online - On Demand Testing Program Uses of On Demand Testing Tests are aimed at Years 3 – 10 and are best suited to: Determining a student’s general ability level relative to the Curriculum. Corroborating teacher judgments Identifying students’ strengths and weaknesses and assisting with forward planning of teaching programs

Uses of On Demand Testing VCAA Assessment Online - On Demand Testing Program Uses of On Demand Testing Testing new intake students or late arrivals Pre-testing students before implementing a targeted program and applying the same test to post-test following engagement with the targeted program Longitudinal tracking of students’ outcomes

Uses of On Demand Testing VCAA Assessment Online - On Demand Testing Program Uses of On Demand Testing On Demand provides automatic correction of closed response tests and generation of test reports with Standard Scores The Standard Score provides an indication of the general curriculum Standard Level range in which a student can be challenged with materials. More about scoring later under ‘Reports’

Uses of On Demand Testing VCAA Assessment Online - On Demand Testing Program Uses of On Demand Testing Which Level adaptive test should I give a student? Which Level adaptive test should I give a student? While there is no exact answer for choosing the right adaptive test level for a student to start with, choosing one that is pitched just below a student’s estimated level of ability is an approach that ensures that the student can build some initial confidence while the test quickly establishes their actual level of ability. Also, this approach ensures that students are engaged early on in the test session with questions that are challenging. If a student’s estimated level of ability is not known, then a test at the student’s curriculum year level can be administered and it will adapt up or down depending on the student’s responses. While this approach will generate a Standard Score that reflects a student’s ability level, the student may find the initial sequence of questions too easy or too difficult leading to less diagnostic information in the final report.

Show of Hands Have you seen the On Demand Testing Question Types? No VCAA Assessment Online - On Demand Testing Program Show of Hands Have you seen the On Demand Testing Question Types? No Yes Do you use the links in reports Do you sit tests with the students

Types of Test Questions Multiple Choice Key in Short Answer Drag and Drop Hot Spot Word Spot Calculator

Multiple Choice Single Answer

Multiple Choice Single Answer VCAA Assessment Online - On Demand Testing Program Multiple Choice Single Answer

VCAA Assessment Online - On Demand Testing Program Key in Short Answer

VCAA Assessment Online - On Demand Testing Program Key in Short Answer

VCAA Assessment Online - On Demand Testing Program Key in Short Answer

VCAA Assessment Online - On Demand Testing Program Key in Short Answer

VCAA Assessment Online - On Demand Testing Program Drag and Drop Answer

VCAA Assessment Online - On Demand Testing Program Drag and Drop Answer

VCAA Assessment Online - On Demand Testing Program Hotspot

VCAA Assessment Online - On Demand Testing Program Hotspot

VCAA Assessment Online - On Demand Testing Program Wordspot

VCAA Assessment Online - On Demand Testing Program Wordspot

VCAA Assessment Online - On Demand Testing Program Bubblespot

VCAA Assessment Online - On Demand Testing Program Bubblespot

VCAA Assessment Online - On Demand Testing Program Calculator Question

VCAA Assessment Online - On Demand Testing Program Calculator Question

On Demand Testing Reports VCAA Assessment Online - On Demand Testing Program On Demand Testing Reports Resources Composition of Adaptive tests.pdf Sample Student Roadmap English Reading 4.pdf Sample Student Roadmap Maths Num_Alg 6.pdf

On Demand Testing Reports

On Demand Testing Reports

On Demand Testing Reports

On Demand Testing Reports

VCAA Assessment Online - On Demand Testing Program Show of Hands How do you use the On Demand Testing Standard Scores at your school? For end-of-year student reporting For pre/post testing For tracking growth General advice The On Demand tests have been validated in a variety of ways and are extensively quality assured. Schools can be confident that the tests produce reliable results and support a range of educational purposes. It is important, however, to recognise that in any assessment situation, some limitations to the conclusions that can be drawn from a single assessment do apply. To assist On Demand users to effectively interpret and act on the data reported in the On Demand system, themes such as triangulation, aggregation and unexpected results are covered in brief here.

On Demand Testing Reports VCAA Assessment Online - On Demand Testing Program On Demand Testing Reports Using the data It is recommended that On Demand data is not used as the sole source of testing information. What should I do if a student receives an unexpected result? What is triangulation and why is it important? Triangulation involves considering and drawing a conclusion from multiple sources of evidence in combination. In an assessment context, this might include considering the results from a variety of informal or formal assessment activities, and it could also include teacher judgement information. Triangulation is a recognised element of best practice in classroom assessment as it takes into account the fact that students can perform quite differently on any given day and also that different students may perform better on some assessment types over others. What should I do if a student receives an unexpected result? Investigation of a student’s performance is advisable in instances where a student performs unexpectedly, particularly after referring to other sources of evidence about the student’s proficiency. Review what the student answered correctly and incorrectly. The reported result should make sense in terms of the question content the student engaged with and the skills the student was able to demonstrate throughout the test. If you require further information please contact the On Demand help desk.

On Demand Testing Reports VCAA Assessment Online - On Demand Testing Program On Demand Testing Reports Using the data How does the VCAA ensure the accuracy of On Demand test scores? How does the VCAA ensure the accuracy of On Demand test scores? The VCAA continually undertakes validation studies involving internal and external expert reviewers. These studies analyse large numbers of student responses to test questions to ensure that the results reported by the On Demand system accurately summarise the level of knowledge and skill demonstrated by each student. Furthermore, the validity of each test question is routinely evaluated through curriculum mapping reviews and through statistical analyses. These analyses also show that the On Demand adaptive tests produce results with a high level of reliability which are fit for a variety of purposes

On Demand Testing Reports VCAA Assessment Online - On Demand Testing Program On Demand Testing Reports The Standard Score How is the On Demand Adaptive Test Standard Score Calculated? How is the On Demand Adaptive Test Standard Score Calculated? Analyses 1.4 million and 840000 students’ test session responses over two years Item data will be analysed off-cycle again in May this year to determine how On Demand is tracking with the Victorian Curriculum. The statistically calibrated difficulty of a question determines its location relative to the Standard Score scale. The question’s content and associated content descriptor determines its location relative to the curriculum standard levels. The difficulty of each question presented to a student is used in the calculation of the student’s Adaptive Test Standard Score. The calculation is based on the Rasch model, which is also used in other assessment programs such as the Programme for International Student Assessment (PISA) and the National Assessment Program – Literacy and Numeracy (NAPLAN). Please note however that the levels and scores reported by On Demand are not associated with, nor aligned to, the ten Bands against which NAPLAN tests are reported. Direct comparisons between NAPLAN and On Demand test outcomes are not valid and not advised.

On Demand Testing Reports VCAA Assessment Online - On Demand Testing Program On Demand Testing Reports The Standard Score The Standard Score provides an indication of the general curriculum Standard Level range in which a student can be challenged with materials. Interpreting a student’s reported Standard Score outcome The On Demand adaptive tests provide teachers with an estimated ability standard score. The knowledge and skills assessed by any single test represent only a sampling of content descriptions from the Victorian Curriculum. The On Demand standard score does not represent full achievement against the Victorian Curriculum Achievement Standards. Use of the On Demand scores to represent mastery of an Achievement Standard is not an appropriate use of these scores.

On Demand Testing Reports VCAA Assessment Online - On Demand Testing Program On Demand Testing Reports The Standard Score The Standard Score is not indicative of mastery at that level and does not represent full achievement against the Victorian Curriculum Achievement Standards. A student whose Standard Score result indicates a certain curriculum standard level has demonstrated a limited range of knowledge and skills commensurate with the curriculum outcomes up to and including that standard level. The student may also have demonstrated some skills assessed at higher levels. The reported Standard Score therefore provides an indication of the curriculum standard level at which a student can engage and be challenged in terms of their knowledge and skills. The Standard Score is not indicative of mastery at that level and does not represent full achievement against the Victorian Curriculum Achievement Standards. The scores reported by the On Demand adaptive tests provide valuable information to support a range of educational purposes

Summing Up Any more questions? No Level One reporting VCAA Assessment Online - On Demand Testing Program Summing Up Any more questions? No Level One reporting New Maths Statistics and Probability Tests English Tests have a new Item mix English Curriculum Coverage Reports and Using the Data The Standard Score Survey Thank You

On Demand Contacts and Support VCAA Assessment Online - On Demand Testing Program On Demand Contacts and Support VCAA On Demand Documentation http://www.vcaa.vic.edu.au/Pages/prep10/ondemand/index.aspx Helpdesk Freecall Number 1800 827 721 Email vcaa.ondemand.support@edumail.vic.gov.au