1. A school effectiveness study
The decline of students’ school-related attitudes during the middle years of secondary education
A school effectiveness study
Jan Van Damme
Georges Van Landeghem
K.U.Leuven
ICSEI 2003

2. Well-being at the school (WELLBS): measures to what extent a student feels well at his/her school
Academic self-concept (ACSELFC): to what degree does a student feel able to cope with the demands of the school work?
Motivation towards learning tasks (MOTLT): measures the amount of effort the student devotes to studying
Focus on three attitudinal outcomes
ICSEI 2003

3. Three of nine noncognitive outcomes studied (eight attitudes, one behavioral outcome).
Range: 1 to 5. Horizontal axis: grades.
De decrease of the means is significant.
Steepest decline during second cycle (= third and fourth grades) (see also next slide)
ICSEI 2003

4. Average change per grade
The graphs on the previous slide and the table above refer to a fairly select subset of 2207 Flemish students who finish secondary school on time.
ICSEI 2003

5. Is it possible for policy makers to remedy this decline? If so: how?
Pragmatic approach: school effectiveness research.
(1) Are schools different with regard to this decline in students’ attitudes?
(2) Can we find school characteristics correlated with those differences between the schools?
Black box approach: quick results, no detailed knowledge of underlying processes needed, results are indications only, interpretations heavily dependent on a priori assumptions.
ICSEI 2003

6. Existing database (LOSO).
Analysis of subset consisting of 3670 students.
Three career types (ASO = general track, TSO = technical, BSO = vocational).
Only students who remain long enough in the same school (therefore 3670 instead of 4435 students of the LOSO cohort).
ICSEI 2003

7. I am glad to go to this school. I think it’s nice at school.
WELLBS, 4 items,  = 0.86:
I am glad to go to this school.
I think it’s nice at school.
If the choice was mine, I would rather go to another school. (-)
If we were to move to another neighbourhood, I would prefer to stay at this school.
ACSELFC: 9 items,  = 0.80
MOTLT: 5 items,  = 0.82
ICSEI 2003

8. Net school effect measures
(First step in school effectiveness research.)
(Bivariate model with pupil and school level.)
School effects at the end of the second and the fourth grade.
Stability measure: correlation at the school level.
DUTCHg = standardised achievement score Dutch (mother tongue)
ICSEI 2003

9. Net school effect measures (continued)
Global: summarizes the school effects of both measurement occasions. (Based on generalized variance: square root of determinant of covariance matrix.)
Average: alternative global school effect measure: school effect on average of outcome (across both occasions).
Change: school effect on difference between outcomes on the two occasions.
Conditional: school effect on outcome at the end of the fourth grade, correcting for the outcome at the end of the second grade (together with the background and initial characteristics). (This is also the school effect on the change in the outcome, correcting for the outcome at the end of the second grade and the background and initial characteristics.)
ICSEI 2003

10. Background characteristics: Socio-economic status of the family (SES)
Sex (GIRL)
Language spoken at home (DUTCHHOME)
Initial characteristics:
Initial cognitive ability (COGN)
Achievement motivation (AM)
Immunity to stress (STRESSIMM)
Age at start of secondary education (NORMALAGE)
Net effects = correcting for student’s background and initial characteristics.
ICSEI 2003

11. School characteristics (higher grades), p-values:
(Second step in school effectiveness research.)
(First part: screening of school characteristics.)
23 school characteristics based on two samples of teachers per school.
This measurement: sample of teachers in higher grades (i.e.: grades 3 to 6).
Inserted one by one in bivariate net effects model. (Two measurement occasions, therefore: school characteristic represented by two coefficients.)
Resulting: 3 x 23 = 69 p-values.
(Table 15, p. 29.)
PERSDEV = Orientation towards the development of the person(ality) of the students.
PUPILACT = Amount of active student participation in the course.
JOBSAT = job satisfaction.
ICSEI 2003

12. School characteristics (lower grades), p-values:
This measurement: sample of teachers in lower grades (1 and 2).
Again two coefficients for the school characteristic.
Additional table of 69 p-values.
(Table 16, p. 31.)
ICSEI 2003

13. Measurement lower grades versus measurement higher grades (PERSDEV)
Score for lower grades and score for higher grades will be interpreted as measurements of two different underlying school characteristics.
Therefore: both scores will be inserted together in one model (two school level variables, two measurement occasions, four coefficients).
ICSEI 2003

14. A closer look at thirteen relationships:
(1) The relationship between the quality of the functioning of the school as an organization (FUNCTIORG) and the change in well-being (WELLBS). (pHIGH = en pLOW = 0.033) 
(7) The connection between the strength of personal relationships between teachers and pupils (PERSRELPUPIL) and the change in academic self-concept (ACSELFC). (pHIGH = en pLOW = 0.125) 
List of 13 relationships (selected among the 69 candidates on the basis of the p-values) that merit a closer examination (pp ).
Two examples on this slide.
The 13 relationships have been selected among the 69 candidates by using the following rule: at least one of both measurements (higher grades or lower grades) yielded a p-value smaller than 5%.
ICSEI 2003

15. To what extent is the school oriented towards the development of the personality (PERSDEV)? Items:
My primary task as a teacher is to support young people on their journey to adulthood.
The development of students’ social skills is at least as important as the development of their cognitive skills.
(Second step in school effectiveness research.)
(Second part: detailed analysis of the remaining relationships between a school characteristic and an outcome.)
(Example: relationship between PERSDEV and ACSELFC.)
PERSDEV is based on 14 items. Two items are shown on the slide. Referring back to the original items list in the course of interpretation is considered important.
ICSEI 2003

16. PERSDEV/ACSELFC
The figure is a schematic representation of the modeling of the (conditional) change of the academic self-concept (between the end of the second and the end of the fourth grade) as a function of the two PERSDEV scores.
The difference between the school with the most favourable change (open triangle) and the school with the worst change (open square) can be shown to be significantly different from zero.
The figure shows the direction of the relationship. The schools with a more favourable change (triangles) are in the part of the figure above the main diagonal. The schools with a less favourable change (squares) are below the diagonal. Consequently, “better” schools are reached by moving along a line parallel with the second diagonal, that is: towards higher values of PERSDEV in both the higher and the lower grades.
ICSEI 2003

17. Coefficients
PERSDEV/ACSELFC
Interpretation: a stronger orientation towards the development of the personality of the students (in the lower grades as well as in the higher grades) enhances the academic self-concept in the higher grades of secondary education.
The two coefficients that mark the positive dependence of the change in the academic self-concept on both PERSDEV scores (correcting for the academic self-concept at the end of the second grade and for the background and initial characteristics) are significant.
They arise from a positive relationship between both PERSDEV scores and the academic self-concept at the end of the fourth grade.
ICSEI 2003

18. When students are more actively involved in the courses of the first grades, they are better able to withstand the influences that undermine their well-being at school in subsequent years.
Satisfied teachers are better able to motivate their students into working hard for school.
Two other relationships (out of the set of thirteen candidates) satisfied the simultaneous requirements of being significant and interpretable and of yielding an interpretation that may be of use for educational policy makers.
ICSEI 2003