1. ON DEMAND TESTING and the Victorian Curriculum
Fran O’Mara
Project Manager

2. Introduction Welcome to this On Demand testing Webinar.
My name is Fran O’Mara and I manage On Demand Testing for the VCAA.

3. Introduction What we will cover Brief overview of On Demand.
What tests and items are available
What has changed in the tests with the transition of On Demand from AusVELS to the new Victorian Curriculum

4. Introduction What we will cover Uses of On Demand tests
Examples of Students’ test paths with questions
On Demand Reports and interpreting the On Demand Standard Score

5. Introduction What we will NOT cover
Technical setup and hands-on management of administrative functions in On Demand
(If there is a need, PD on how to navigate the program can be scheduled at a later date)

6. Introduction What we will NOT cover
Technical questions / System problems.
Please still submit these questions and if time permits, I will take some on notice at the end of the session.
Otherwise, we will contact you to resolve your technical issues later.

7. Do you already use On Demand Testing at your school?
VCAA Assessment Online - On Demand Testing Program
Show of Hands
Do you already use On Demand Testing at your school?
Yes No

8. What level of expertise do you have with On Demand?
VCAA Assessment Online - On Demand Testing Program
Show of Hands
What level of expertise do you have with On Demand?
Expert
Comfortable
Beginner

9. VCAA Assessment Online - On Demand Testing Program
Introduction
On Demand Testing delivers online tests from the VCAA central repository to students’ computers via the Internet and a school’s local network.
Teachers use the resource when, where and how they choose.

10. Introduction On Demand testing is optional.
VCAA Assessment Online - On Demand Testing Program
Introduction
On Demand testing is optional.
It is a time-saving resource that can be administered to a single student and/or a whole class.
And it’s free.

11. VCAA Assessment Online - On Demand Testing Program
Technical Overview

12. Types of Tests Available
VCAA Assessment Online - On Demand Testing Program
Types of Tests Available
Computer Adaptive Tests are available for Mathematics and English.
Linear tests are not currently available.
Available tests list:

13. Type of Tests Available
VCAA Assessment Online - On Demand Testing Program
Type of Tests Available
Mathematics: Years
60 Item General Tests
30 Item Focus Tests
New: Statistics and Probability
Number and Algebra
Measurement and Geometry

14. Type of Tests Available
VCAA Assessment Online - On Demand Testing Program
Type of Tests Available
Resources
On Demand tests and Victorian Curriculum coverage.pdf (for English Tests only)

15. Type of Tests Available
VCAA Assessment Online - On Demand Testing Program
Type of Tests Available
English: Years
60 Item General Tests
30 Item Focus Tests
Reading
Language Skills
Word Building

16. Show of Hands Which Tests do you use most? Mathematics English Both
VCAA Assessment Online - On Demand Testing Program
Show of Hands
Which Tests do you use most?
Mathematics
English
Both

17. Type of Tests Available
VCAA Assessment Online - On Demand Testing Program
Type of Tests Available
Resource
Transition_AusVELS to VC.pdf
Reporting has changed for Levels and Domains.
Scales have been maintained to generally align. However, it must be remembered that the item pool has been recalibrated.
Outliers have been removed, items have been adjusted to align with the new curriculum.

18. What’s changed in the tests
VCAA Assessment Online - On Demand Testing Program
What’s changed in the tests
All Tests: No Standard Level 1
Minimum Score is ‘Below 1.0’
Mathematics ends at Level 10 Adv.
Maximum score is ‘Above 10.9’

19. What’s changed in the tests
VCAA Assessment Online - On Demand Testing Program
What’s changed in the tests
English ends at Level 10
Maximum score is ‘Above 9.9’
English Curriculum coverage has been expanded and changed for tests.

20. Show of Hands Which English Tests do you use most? General Reading
VCAA Assessment Online - On Demand Testing Program
Show of Hands
Which English Tests do you use most?
General
Reading
Language Skills
Word Knowledge

21. What’s changed in the tests
VCAA Assessment Online - On Demand Testing Program
What’s changed in the tests
English – Reading
These tests focus on:
Language Mode: Reading and Viewing
Strand: Literacy
Sub-strand: Interpreting, analysing, evaluating.
Some Items previously in Reading are now part of the Language Skills and Word Knowledge tests

22. What’s changed in the tests
VCAA Assessment Online - On Demand Testing Program
What’s changed in the tests
English – Language Skills
These tests focus on the Language Modes of Reading and Viewing, and Writing – multiple Strands
Items test knowledge about language and language use, from text purpose, structures and features, to finer features at the paragraph and syntactic level.

23. What’s changed in the tests
VCAA Assessment Online - On Demand Testing Program
What’s changed in the tests
English – Word Knowledge
These tests focus on the Language Modes of Reading and Viewing, and Writing – multiple Strands
Items test knowledge of word formation, word changes that follow function, precision in word use for shades of meaning, and spelling
Some of these Items used to be part of the reading tests bank
Word Knowledge Items test knowledge of word formation
complete the word
word changes that follow function - verb endings
precision in word use for shades of meaning – choice of what best describes
knowledge of the sound-letter system
homonyms, synonyms, antonyms
Pure spelling

24. Show of Hands What year level ranges are you testing? 3 – 6 7 – 10
VCAA Assessment Online - On Demand Testing Program
Show of Hands
What year level ranges are you testing?
3 – 6
7 – 10
Pre-Year 3

25. Uses of On Demand Testing
VCAA Assessment Online - On Demand Testing Program
Uses of On Demand Testing
Tests are aimed at Years 3 – 10 and are best suited to:
Determining a student’s general ability level relative to the Curriculum.
Corroborating teacher judgments
Identifying students’ strengths and weaknesses and assisting with forward planning of teaching programs

26. Uses of On Demand Testing
VCAA Assessment Online - On Demand Testing Program
Uses of On Demand Testing
Testing new intake students or late arrivals
Pre-testing students before implementing a targeted program and applying the same test to post-test following engagement with the targeted program
Longitudinal tracking of students’ outcomes

27. Uses of On Demand Testing
VCAA Assessment Online - On Demand Testing Program
Uses of On Demand Testing
On Demand provides automatic correction of closed response tests and generation of test reports with Standard Scores
The Standard Score provides an indication of the general curriculum Standard Level range in which a student can be challenged with materials.
More about scoring later under ‘Reports’

28. Uses of On Demand Testing
VCAA Assessment Online - On Demand Testing Program
Uses of On Demand Testing
Which Level adaptive test should I give a student?
Which Level adaptive test should I give a student?
While there is no exact answer for choosing the right adaptive test level for a student to start with, choosing one that is pitched just below a student’s estimated level of ability is an approach that ensures that the student can build some initial confidence while the test quickly establishes their actual level of ability. Also, this approach ensures that students are engaged early on in the test session with questions that are challenging.
If a student’s estimated level of ability is not known, then a test at the student’s curriculum year level can be administered and it will adapt up or down depending on the student’s responses. While this approach will generate a Standard Score that reflects a student’s ability level, the student may find the initial sequence of questions too easy or too difficult leading to less diagnostic information in the final report.

29. Show of Hands Have you seen the On Demand Testing Question Types? No
VCAA Assessment Online - On Demand Testing Program
Show of Hands
Have you seen the On Demand Testing Question Types? No
Yes
Do you use the links in reports
Do you sit tests with the students

30. Types of Test Questions
Multiple Choice
Key in Short Answer
Drag and Drop
Hot Spot
Word Spot
Calculator

31. Multiple Choice Single Answer

32. Multiple Choice Single Answer
VCAA Assessment Online - On Demand Testing Program
Multiple Choice Single Answer

33. VCAA Assessment Online - On Demand Testing Program
Key in Short Answer

34. VCAA Assessment Online - On Demand Testing Program
Key in Short Answer

35. VCAA Assessment Online - On Demand Testing Program
Key in Short Answer

36. VCAA Assessment Online - On Demand Testing Program
Key in Short Answer

37. VCAA Assessment Online - On Demand Testing Program
Drag and Drop Answer

38. VCAA Assessment Online - On Demand Testing Program
Drag and Drop Answer

39. VCAA Assessment Online - On Demand Testing Program
Hotspot

40. VCAA Assessment Online - On Demand Testing Program
Hotspot

41. VCAA Assessment Online - On Demand Testing Program
Wordspot

42. VCAA Assessment Online - On Demand Testing Program
Wordspot

43. VCAA Assessment Online - On Demand Testing Program
Bubblespot

44. VCAA Assessment Online - On Demand Testing Program
Bubblespot

45. VCAA Assessment Online - On Demand Testing Program
Calculator Question

46. VCAA Assessment Online - On Demand Testing Program
Calculator Question

47. On Demand Testing Reports
VCAA Assessment Online - On Demand Testing Program
On Demand Testing Reports
Resources
Composition of Adaptive tests.pdf
Sample Student Roadmap English Reading 4.pdf
Sample Student Roadmap Maths Num_Alg 6.pdf

48. On Demand Testing Reports

49. On Demand Testing Reports

50. On Demand Testing Reports

51. On Demand Testing Reports

52. VCAA Assessment Online - On Demand Testing Program
Show of Hands
How do you use the On Demand Testing Standard Scores at your school?
For end-of-year student reporting
For pre/post testing
For tracking growth
General advice
The On Demand tests have been validated in a variety of ways and are extensively quality assured. Schools can be confident that the tests produce reliable results and support a range of educational purposes. It is important, however, to recognise that in any assessment situation, some limitations to the conclusions that can be drawn from a single assessment do apply. To assist On Demand users to effectively interpret and act on the data reported in the On Demand system, themes such as triangulation, aggregation and unexpected results are covered in brief here.

53. On Demand Testing Reports
VCAA Assessment Online - On Demand Testing Program
On Demand Testing Reports
Using the data
It is recommended that On Demand data is not used as the sole source of testing information.
What should I do if a student receives an unexpected result?
What is triangulation and why is it important?
Triangulation involves considering and drawing a conclusion from multiple sources of evidence in combination. In an assessment context, this might include considering the results from a variety of informal or formal assessment activities, and it could also include teacher judgement information. Triangulation is a recognised element of best practice in classroom assessment as it takes into account the fact that students can perform quite differently on any given day and also that different students may perform better on some assessment types over others.
What should I do if a student receives an unexpected result?
Investigation of a student’s performance is advisable in instances where a student performs unexpectedly, particularly after referring to other sources of evidence about the student’s proficiency. Review what the student answered correctly and incorrectly. The reported result should make sense in terms of the question content the student engaged with and the skills the student was able to demonstrate throughout the test. If you require further information please contact the On Demand help desk.

54. On Demand Testing Reports
VCAA Assessment Online - On Demand Testing Program
On Demand Testing Reports
Using the data
How does the VCAA ensure the accuracy of On Demand test scores?
How does the VCAA ensure the accuracy of On Demand test scores?
The VCAA continually undertakes validation studies involving internal and external expert reviewers. These studies analyse large numbers of student responses to test questions to ensure that the results reported by the On Demand system accurately summarise the level of knowledge and skill demonstrated by each student. Furthermore, the validity of each test question is routinely evaluated through curriculum mapping reviews and through statistical analyses. These analyses also show that the On Demand adaptive tests produce results with a high level of reliability which are fit for a variety of purposes

55. On Demand Testing Reports
VCAA Assessment Online - On Demand Testing Program
On Demand Testing Reports
The Standard Score
How is the On Demand Adaptive Test Standard Score Calculated?
How is the On Demand Adaptive Test Standard Score Calculated?
Analyses 1.4 million and students’ test session responses over two years
Item data will be analysed off-cycle again in May this year to determine how On Demand is tracking with the Victorian Curriculum.
The statistically calibrated difficulty of a question determines its location relative to the Standard Score scale. The question’s content and associated content descriptor determines its location relative to the curriculum standard levels. The difficulty of each question presented to a student is used in the calculation of the student’s Adaptive Test Standard Score. The calculation is based on the Rasch model, which is also used in other assessment programs such as the Programme for International Student Assessment (PISA) and the National Assessment Program – Literacy and Numeracy (NAPLAN). Please note however that the levels and scores reported by On Demand are not associated with, nor aligned to, the ten Bands against which NAPLAN tests are reported. Direct comparisons between NAPLAN and On Demand test outcomes are not valid and not advised.

56. On Demand Testing Reports
VCAA Assessment Online - On Demand Testing Program
On Demand Testing Reports
The Standard Score
The Standard Score provides an indication of the general curriculum Standard Level range in which a student can be challenged with materials.
Interpreting a student’s reported Standard Score outcome
The On Demand adaptive tests provide teachers with an estimated ability standard score. The knowledge and skills assessed by any single test represent only a sampling of content descriptions from the Victorian Curriculum. The On Demand standard score does not represent full achievement against the Victorian Curriculum Achievement Standards. Use of the On Demand scores to represent mastery of an Achievement Standard is not an appropriate use of these scores.

57. On Demand Testing Reports
VCAA Assessment Online - On Demand Testing Program
On Demand Testing Reports
The Standard Score
The Standard Score is not indicative of mastery at that level and does not represent full achievement against the Victorian Curriculum Achievement Standards.
A student whose Standard Score result indicates a certain curriculum standard level has demonstrated a limited range of knowledge and skills commensurate with the curriculum outcomes up to and including that standard level. The student may also have demonstrated some skills assessed at higher levels. The reported Standard Score therefore provides an indication of the curriculum standard level at which a student can engage and be challenged in terms of their knowledge and skills. The Standard Score is not indicative of mastery at that level and does not represent full achievement against the Victorian Curriculum Achievement Standards. The scores reported by the On Demand adaptive tests provide valuable information to support a range of educational purposes

58. Summing Up Any more questions? No Level One reporting
VCAA Assessment Online - On Demand Testing Program
Summing Up
Any more questions?
No Level One reporting
New Maths Statistics and Probability Tests
English Tests have a new Item mix
English Curriculum Coverage
Reports and Using the Data
The Standard Score
Survey
Thank You

59. On Demand Contacts and Support
VCAA Assessment Online - On Demand Testing Program
On Demand Contacts and Support
VCAA On Demand Documentation
Helpdesk
Freecall Number