1. Secondary Information Systems
Introduction to CEM
Secondary Information Systems
Dr Robert Clark
ALIS Project Manager

2. The Projects

3. The Projects Alis Yellis MidYIS A / AS / Btec / IB etc Year 13 GCSE
Computer Adaptive Baseline Test
(paper test also available)
Year 13
Year 12
GCSE
Year 11
Yellis
Computer Adaptive Baseline Test
(paper test also available)
Year 10
Combines curriculum tests with developed ability
INSIGHT
Year 9
MidYIS
Computer Adaptive Baseline Test
(paper test also available)
Year 8
(+ additional)
Year 7

4. Early Testing & Predictions
Typical Timeline
Testing is available from June at end of previous academic year
i.e. Y6 / Y9 / Y11
Early Testing & Predictions
The baseline score is measured as early as possible and the information sent to the CEM Centre
The institution then receives feedback from the CEM Centre: baseline test results and predictions
The public examination results are sent to the CEM Centre when received by the institution
The value added data is available from September via the Institution’s website at the CEM Centre

5. Predictions

6. How CEM ‘Predictions’ are made….
How CEM ‘Predictions’ are made…
Subject X
A / B C
Regression Line (…Trend Line, Line of Best Fit)
Outcome = gradient x baseline + intercept
Correlation Coefficient (~ 0.7)

7. Some Subjects are More Equal than Others….D E
>1 grade

8. Some Subjects are More Equal than Others…. GCSE (MidYIS or Yellis)F E D C B A A*
Test Score
GCSE Grades
Art & Design
Biology
Chemistry
Economics
English
French
Geography
German
History
Ict
Mathematics
Media Studies
Music
Physical Education
Physics
Religious Studies
Science (Double)
Spanish
1 grade

9. Baseline Measurement

10. Problems with Key Stage Baselines
Not all students have KS baselines
No KS3
Foreign Students
Vocational Students
Adult Learners
KS exams do not always represent ‘Start of Course’ ability
Post-16 : Year(s) out or intermediate years
Prior Value-Added
Can you add value at every Key Stage ?
Under achievement leading to under expectation
One teacher’s output = another teacher’s input
Teaching to the test
Does performance represent ability, effort or exam technique ?
Aptitude & fluency vs Achievement & knowledge
Transfer of Data

11. Although Key Stage baselines can be a very good indicator of potential attainment, by themselves they are not sufficient.
Key Stage baselines are confounded by the effects of prior treatment.

12. The Computer Adaptive Test
Test performed online – results automatically transmitted to CEM.
Minimal installation / setup required - if any.
Adaptive – difficulty of questions changes in relation to ability of student.
Efficient – no time wasted answering questions that are far too easy or difficult.
Wider range of ability
Less stressful on students – more enjoyable experience than paper test.
Less demanding invigilation.
Cheaper !
In 2010/ 2011 over 200,000 students across yrs 7-13 sat this test
Try it yourself at

13. Baseline Feedback
Reports, Graphs
& Predictions

14. IPRs (Individual Pupil Record Sheets)
Look for sections that are inconsistent
Also available based on MidYIS, Alis, SOSCA & INSIGHT scores

15. Intake Profiles
Also available based on MidYIS, Yellis, SOSCA and INSIGHT scores

16. Intake Profiles (Historical)

17. Predictions – MidYIS example
Similar spreadsheets available from Yellis, SOSCA, INSIGHT

18. Predictions - Alis example

19. Chances Graphs Grade Percent Grade Percent Grade Percent Grade Percent
English Language - band D 2 10 23 31 24 9 1 20 30 40 50 U G F E D C B A A*
Grade
Percent
English Language - band C 1 2 6 20 35 29 7 10 30 40 50 U G F E D C B A A*
Grade
Percent
English Language - band B 2 7 24 40 21 5 10 20 30 50 U G F E D C B A A*
Grade
Percent
English Language - band A 1 8 26 35 23 7 10 20 30 40 50 U G F E D C B A A*
Grade
Percent

20. Predictions vs
Targets

21. What is a ‘Prediction’ ? Targets ?
NOT a forecast of the grade the student will get
An indication of the grade (points score) achieved on average by students of similar ability in the previous year
Targets ?
Minimum Targets – Round CEM prediction down ?
Realistic Targets – Use nearest CEM grade ?
Challenging Targets – 75th percentile ?
Prior Value Added ?
Arbitrary grade fraction ?

22. 75th Percentile Predictions
If all students attain 75th percentile predictions, School VA will be at top 25%
Provided by all Projects in addition to 50th percentile
Alis : Excel spreadsheet - ‘Predictions – Spreadsheet (75th Percentile)’
Pre-16 : Adjust button on predictions spreadsheet
Prior Value Added
Only where prior VA is positive ?
1 year or 3 ?
Reasonable to use raw residual figures as this is an approximate measure and raw residuals give grade fractions
Alis : Can be calculated using PARIS software
Data used to inform, not replace, professional judgement

23. 75th Percentile
Prior Value Added

24. 75th Percentile
Step 1
Prior
Value-Added

25. Value-Added Feedback
Reports & Graphs

26. Measuring Value-Added – Terminology.
Measuring Value-Added – Terminology
Exam grade
-ve VA
+ve VA
BASELINE SCORE
Raw Residual
Trend Line/Regression Line

27. The position of the national trend line is of critical importance
Measuring Value-Added – An Example
National Trend
‘Average’ Student A* U B C D E F G
Result
-ve (- 2 grades)
Alf
Bob
Subject A
Subject B
+ve (+ 2 grades)
Chris
Low Ability
Average Ability
High Ability
Baseline Score
The position of the national trend line is of critical importance

28. Standardisation of Residuals
(Raw) Residuals can be used to examine an individual’s performance
Standardised Residuals are used to compare performance of groups
Standardised Residuals are independent of year or qualification type
For a class, subject, department or whole institution the Average Standardised Residual is the ‘Value-Added Score’
Standardised Residual = Residual / Standard Deviation (National Sample)
When using Standardised Residuals then for an individual subject
where N = number of results in the group
(for combinations of subjects consult the relevant project)
95% Confidence Limit = 2.0 x Standard Error
99% Confidence Limit = 2.6 x Standard Error
99.7% Confidence Limit = 3.0 x Standard Error

29. Burning Question :
What is my Value-Added Score ?
Better Question :
Is it Important ?

30. Statistical Process Control (SPC) Chart
Value Added Feedback…
Statistical Process Control (SPC) Chart
Year

31. Subject Summary Standardised Residual Graph
-0.2
-0.1
0.1
-0.6
-0.7
-0.4
-2.2
-0.5
2.0
0.4
-4.0
-3.0
-2.0
-1.0
0.0
1.0
3.0
4.0
Art
Biology
Chemistry
Design and Technology
Drama
English Language
English Literature
French
Geography
German
History
Home Economics
Information Technology
Maths
Media Studies
Music
Physics
Religious Studies
Spanish
Average Standardised Residual

32. The Scatter Plot Grade Points Equivalent Look for Patterns…
Baseline Score
Grade Points Equivalent
Look for Patterns…
General Underachievement / over achievement ?
Do any groups of students stand out ?
– high ability vs low ability ?
– male vs female ?

33. Other things to look for…
Why did these students do so badly ?
Why did this student do so well ?
How did they do in their other subjects ?

34. PARIS Software

35. PARIS is ….. PARIS analyses ….. PARIS provides …..
Software to install and use in school
An interactive reporting tool
Included free with Alis / Yellis / MidYIS
PARIS analyses …..
Potential Performance
Intermediate Performance
Actual Performance
PARIS provides …..
Student level reports
Subject Level reports
Institution Level Reports

36. Attitudes

37. There is more to school / college than exams….
Student attitudes
Student Welfare & Safety
Non-academic activities
Support
Social and personal development
Parental Survey
Induction Survey
Self Evaluation (Every Child Matters)
Attitudinal MidYIS INSIGHT Attitudinal Yellis Full ALIS
Try it yourself at

38. Other Issues

39. Points to mull over… Independence – no agendas
Transparency of Analysis
Self Evaluation
Straightforward and standardised administration
Prompt Feedback
Full working hours phone / support
Student focus
Replacement for KS3
Innovative online adaptive testing available – student experience
Longitudinal analysis with appropriate error backgrounds
Non-curriculum embedded baselines available
Attitudinal surveys available – Every Child Matters…

40. Dr Robert Clark
Alis Project Manager