1. Secondary Information Systems
Introduction to CEM
Secondary Information Systems
Dr Robert Clark
ALIS Project Manager

2. The Analysis

3. Linear Least Squares Regression.
Linear Least Squares Regression
Subject X
-ve VA
+ve VA
Residuals
Regression Line (…Trend Line, Line of Best Fit)
Outcome = gradient x baseline + intercept
Correlation Coefficient (~ 0.7)

4. The position of the national trend line is of critical importance
Measuring Value-Added – An Example
National Trend
‘Average’ Student A* U B C D E F G
Result
-ve (- 2 grades)
Alf
Bob
Subject A
Subject B
+ve (+ 2 grades)
Chris
Low Ability
Average Ability
High Ability
Baseline Score
The position of the national trend line is of critical importance

5. Some Subjects are More Equal than Others…. A-Level (Alis)
Grade
> 1 grade
Photography
Sociology
English Lit
Psychology
Maths
Physics
Latin C D E C B A A*
Average GCSE

6. Business and Management
Some Subjects are More Equal than Others….
IB (Alis)
Biology
Business and Management
Chemistry
Design Technology
Economics
English_A1
Film
French_B
Geography
History
Mathematics
Music
Philosophy
Physics
Psychology
Spanish_B
Theatre Arts
Visual Arts 7 6
Grade 5 4 C B A A*
Average (I)GCSE Score

7. Some Subjects are More Equal than Others…. GCSE (MidYIS or Yellis)F E D C B A A*
Test Score
GCSE Grades
Art & Design
Biology
Chemistry
Economics
English
French
Geography
German
History
Ict
Mathematics
Media Studies
Music
Physical Education
Physics
Religious Studies
Science (Double)
Spanish
1 grade

8. Standardisation of Residuals
(Raw) Residuals can be used to examine an individual’s performance
Standardised Residuals are used to compare performance of groups
Standardised Residuals are independent of year or qualification type
For a class, subject, department or whole institution the Average Standardised Residual is the ‘Value-Added Score’
Standardised Residual = Residual / Standard Deviation (National Sample)
When using Standardised Residuals then for an individual subject
where N = number of results in the group
(for combinations of subjects consult the relevant project)
95% Confidence Limit = 2.0 x Standard Error
99% Confidence Limit = 2.6 x Standard Error
99.7% Confidence Limit = 3.0 x Standard Error

9. The Projects

10. Alis : yr 12/13
Yellis : yr 10/11
Secondary Age Range Projects
Insight : Yr 9
MidYIS : Yr 7/8/9
InCAS : Primary
Pips : Primary

11. Typical Timeline Autumn Term August September Measure Baseline
Prediction
Reports
Collect Results
August
Value-Added
Feedback
September

12. Baseline Measurement

13. Baselines A / AS / IB etc GCSE Year 13 Alis Paper test (TDA) or
Computer Adaptive Baseline Test
Year 12
GCSE
Year 11
Yellis Paper test or
Computer Adaptive Baseline Test
Year 10
Combines curriculum tests with developed ability (end Y9 & end Y8)
INSIGHT
Year 9
MidYIS Paper test or
Computer Adaptive Baseline Test
Year 8
(+ additional)
Year 7

14. Problems with Key Stage Baselines
Not all students have KS baselines
no KS2 / KS3
Foreign Students
(Vocational Students)
(Adult Learners)
KS exams do not always represent ‘Start of Course’ ability
Post-16 : Year(s) out or intermediate years
Prior Value-Added
Can you add value at every Key Stage ?
Under achievement leading to under expectation
One teacher’s output = another teacher’s input
A level playing field ?
Teaching to the test
Does performance represent ability, effort or exam technique ?
Aptitude & fluency vs Achievement & knowledge

15. In line with Expectation
The Effect of Prior Value Added
Beyond Expectation
+ve Value-Added
In line with Expectation
0 Value-Added
Below Expectation
-ve Value-Added
Average GCSE = 6
Do these 3 students all have the same ability ?

16. Key Stage baselines are confounded by the effects of prior treatment.
Although Key Stage baselines can be a very good indicator of potential attainment, by themselves they are not sufficient.
Key Stage baselines are confounded by the effects of prior treatment.
Need for independent, non-curriculum embedded baseline tests
yrs 7 / 8 / S1 / 9 / S2
yrs 10 / 11
yrs 12 / 13 +
Online Computer Adaptive or Traditional Paper

17. The Computer Adaptive Test
Test performed online – results automatically transmitted to CEM.
LAN version available if Web access unreliable.
Minimal installation / setup required - if any.
Adaptive – difficulty of questions changes in relation to ability of student.
Efficient – no time wasted answering questions that are far too easy or difficult.
Wider range of ability
Less stressful on students – more enjoyable experience than paper test.
Less demanding invigilation.
Cheaper !
In 2009 / 2010 over 200,000 students across yrs 7-13 sat this test
Try it yourself at

18. Baseline Feedback
Reports, Graphs
& Predictions

19. IPRs (Individual Pupil Record Sheets)
Look for sections that are inconsistent
Also available based on MidYIS, Alis & INSIGHT scores

20. Intake Profiles
Also available based on MidYIS, Yellis and INSIGHT scores

21. Intake Profiles (Historical)

22. Intake Profiles – 7 Band for Independent Schools (Midyis)

23. Predictions Average performance by similar students in past exams
Predictions are available in the following forms:
Formal Reports (Alis)
Spreadsheets (All Projects)
Paris Software (Alis / Yellis / MidYIS)
Feedback is available on Web / CD

24. Predictions – MidYIS example
Similar spreadsheets available from Yellis and INSIGHT

25. Predictions - Alis example

26. Value-Added Feedback
Reports & Graphs

27. Statistical Process Control (SPC) Chart
Value Added Feedback…
What is my score ?
does it matter ?
Statistical Process Control (SPC) Chart
Year

28. Subject Summary
Standardised Residual Graph

29. The Scatter Plot Grade Points Equivalent Look for Patterns…
Baseline Score
Grade Points Equivalent
Look for Patterns…
General Underachievement / over achievement ?
Do any groups of students stand out ?
– high ability vs low ability ?
– male vs female ?

30. Other things to look for…
Why did these students do so badly ?
Why did this student do so well ?
How did they do in their other subjects ?

31. Independent Schools
Analysis

32. A. It depends on what question you are asking…
MidYIS provides…
Independent schools baseline standardisation
Independent schools specific predictions & chances graphs
7 Band Intake profiles
Independent schools specific value-added analysis
Alis provides…
Independent schools specific value-added analysis
Q. Which one should I use – Independent only analysis or all schools analysis ?
A. It depends on what question you are asking…

33. PARIS Software

34. PARIS is ….. PARIS analyses ….. PARIS provides …..
Software to install and use in school
An interactive reporting tool
Included free with Alis / Yellis / MidYIS
PARIS analyses …..
Potential Performance
Intermediate Performance
Actual Performance
PARIS provides …..
Student level reports
Subject Level reports
Institution Level Reports

35. Attitudes

36. There is more to school / college than exams….
Student attitudes
Student Welfare & Safety
Non-academic activities
Support
Social and personal development
Parental Survey
Induction Survey
Self Evaluation (Every Child Matters)
Attitudinal MidYIS INSIGHT Attitudinal Yellis Full ALIS
Try it yourself at

37. Other Issues

38. Points to mull over… Independence – no agendas
Transparency of Analysis
Self Evaluation
Straightforward and standardised administration
Prompt Feedback
Full working hours phone / support
Student focus
Replacement for KS3
Innovative online adaptive testing available – student experience
Longitudinal analysis with appropriate error backgrounds
Non-curriculum embedded baselines available
Attitudinal surveys available – Every Child Matters…

39. Dr Robert Clark
Alis Project Manager