1. Widening Participation whilst Narrowing Attainment Gaps between Student Groups: A Realistic Objective for Higher Education?
Introduction:
How this study came about: Moira’s work on unconscious bias and assessment (presentation in Learning and Teaching conference)
An investigation into how the features of programme modules, the formats and timings of assessments and the characteristics of student cohorts interact to influence student attainment
Richard McManus
Moira Mitchell

2. The literature… National Studies:
Based on HESA data (degree classification) for example:
Broecke and Nicholls (2007)
65,000 degree classifications
Fielding, Charlton, Kounali and Leckie (2008)
66,000 degree classifications
Richardson (2008)
241,300 degree classifications
Institutional Studies:
Thinner on the ground and often qualitative:
Dhanda (2010)
(institutional attainment data by school and module, plus qualitative study with 34 students)
Richardson (2010,2011)
(surveys based on 1,146 and 929 students respectively)
The essential role of curriculum design, learning, teaching and assessment practices is recognised (ECU/HEA, 2010; Singh, 2011; Stevenson, 2012) but there is limited evidence about LTA:
“We do not know what aspects of teaching and assessment practices in higher education might be responsible for variations in the attainment gap.” Richardson (2015)
Richard McManus
Moira Mitchell

3. Our Study… Our study uses CCCU Business School data:
Including students who enrolled between 2011 and 2013 incorporating 1,291 students;
Considers specific module and assessment results:
In total, 179 separate assessments, and over 23,000 separate assessment marks.
Simultaneously considers difference across different demographic characteristics;
Identifies both the presence of attainment differences and the causes of these in a two-stage approach.
Richard McManus
Moira Mitchell

4. Our Study…

5. Our Study… How could our study enhance what is known?
Identifying areas of learning, teaching and assessment practices for adaptation
Designing adaptations
Evaluating adaptations
Richard McManus
Moira Mitchell

6. The sample and attainment gaps…
Total
BME
Cohort 2013
450
32%
Cohort 2012
395
37%
Cohort 2011
446
34%
Significance Ratio
Mean
Median
Min
Max
Std Dev
0.1
0.05
0.01
Ethnicity
-0.10
-0.09
-0.37
0.17
0.09
45%
37%
25%
Proportional attainment difference by demographic in a specific assessment

7. Results… Ethnicity attainment gaps are impacted by:
The average mark of the assessment;
Attainment gaps fall when the average mark in an assessment is higher; attainment gaps are smaller by three percentage points when the average mark is 10 percentage points higher.

8. Results… Ethnicity attainment gaps are impacted by:
The average mark of the assessment;
Whether the assessment is practical based or in class;
Practical assignments and in class tests both lead to an attainment gap 2.7 percentage points lower compared to exams.

9. Results… Ethnicity attainment gaps are impacted by:
The average mark of the assessment;
Whether the assessment is practical based or in class;
If the assessment is numerical;
Numerical tests lead to an attainment gap 7.7 percentage points higher.

10. Results… Ethnicity attainment gaps are impacted by:
The average mark of the assessment;
Whether the assessment is practical based or in class;
If the assessment is numerical;
Cohort effects.

11. Other results…
Variables which were not significant (for any demographic):
Ratio of BME, international, and disabled students to total population of students in the module;
Whether the coursework was a traditional essay style set up or something less traditional like a case study or a specific type advertising proposal (for example) but still written;
The term the assessment was performed/submitted (but could be explained by the ‘order’ variable).
Variables which were not significant (specifically to BME student):
The number of students in the module;
Anonymous marking;
Coursework;
MCQs;
Group work;
Timing of assessments.

12. Implications…
Richard McManus
Moira Mitchell

13. The Sample: Students Total BME Male International Disabled Cohort 2013
Total
BME
Male
International
Disabled
Cohort 2013
450
32%
52%
18% 7%
Cohort 2012
395
37%
23%
11%
Cohort 2011
446
34%
53%
31% 9%
Richard McManus
Moira Mitchell

14. The Sample: Assessment and Modules
No. of assessments
No. of modules
Ave. no. of students per module
% of exams
% of coursework
% of practical assessments
Year 1 57 7
288
75%
14%
11%
Year 2 89 20 70
31%
49%
20%
Year 3 33 16 63
33%
48%
18%
Richard McManus
Moira Mitchell

15. Results Stage 1: Prevalence of Attainment Gaps
Each attainment gap is presented as the proportional difference of one demographic group against the benchmark, where negative coefficients represent a negative attainment gap. For example, BME students on average score 10% less than the benchmark average.
Significance Ratio
Mean
Median
Min
Max
StdDev
10% 5% 1%
Ethnicity
-0.10
-0.09
-0.37
0.17
0.09
45%
37%
25%
Gender
-0.05
-0.54
0.62
0.12
34%
27%
15%
International
-0.06
-0.65
0.16
0.11
35%
26% 9%
Disability
-0.04
-0.02
-0.40
0.43
0.13
17% 8% 2%
The benchmark is a white female student from the UK with no disability.
Proportional attainment difference by demographic
Attainment differences are additive, such that (for example) a male BME student performs on average 15% lower than benchmark.

16. Results Stage 1: Assessment Method
Richard McManus
Moira Mitchell

17. Results Stage 1: Assessment Method
Richard McManus
Moira Mitchell

18. Results Stage 2: Determinants of Attainment Gaps
Ethnicity
Gender
International
Disability
Students(100)
0.022***
(0.000)
-0.028***
(0.005)
0.0278**
(0.018)
Male Ratio
0.121**
(0.032)
Average
Mark
0.003***
0.004***
(0.002)
Weight
0.001**
(0.048)
(0.028)
Anonymity
-0.024*
(0.080)
0.048***
(0.009)
Coursework
0.059**
(0.010)
0.081***
Practical
0.027*
(0.078)
0.050*
(0.053)
0.072***
(0.001)
In Class
0.027**
(0.019)
0.036*
(0.084)
MCQ
0.043**
Numerical
-0.077***
Group
-0.044***
Cohort2012
-0.093***
0.063***
Cohort2013
-0.028**
(0.023)
Order
-0.047**
(0.035)
0.045*
(0.070)
Year2
-0.071**
(0.011)
0.052*
(0.094)
Year3
(0.006)
0.091**
(0.015)
Constant
-0.261***
-0.322***
-0.231***
(0.003)
-0.440***
𝑹 𝟐
0.325
0.280
0.349
0.321 n
179
Module variables
Male and disabled students perform relative better in larger modules, whereas international students do not.
International students also perform relatively better in assessments with a higher male-to-female ratio

19. Results Stage 2: Determinants of Attainment Gaps
Ethnicity
Gender
International
Disability
Students(100)
0.022***
(0.000)
-0.028***
(0.005)
0.0278**
(0.018)
Male Ratio
0.121**
(0.032)
Average
Mark
0.003***
0.004***
(0.002)
Weight
0.001**
(0.048)
(0.028)
Anonymity
-0.024*
(0.080)
0.048***
(0.009)
Coursework
0.059**
(0.010)
0.081***
Practical
0.027*
(0.078)
0.050*
(0.053)
0.072***
(0.001)
In Class
0.027**
(0.019)
0.036*
(0.084)
MCQ
0.043**
Numerical
-0.077***
Group
-0.044***
Cohort2012
-0.093***
0.063***
Cohort2013
-0.028**
(0.023)
Order
-0.047**
(0.035)
0.045*
(0.070)
Year2
-0.071**
(0.011)
0.052*
(0.094)
Year3
(0.006)
0.091**
(0.015)
Constant
-0.261***
-0.322***
-0.231***
(0.003)
-0.440***
𝑹 𝟐
0.325
0.280
0.349
0.321 n
179
Assessment variables
All attainment gaps are smaller the higher the average mark in the assessment.
Male and disabled students perform relatively better in modules with more weighting to the final grade.
International students perform better in modules which are anonymously marked, whereas male students do not.
International and disabled students perform better in coursework (essay) based assignments.
BME, international and disabled students perform relatively better in practical assignments.
BME and international students perform relatively better during ‘in class’ tests compared to more formal exams.
Disabled student perform relatively better in assessments which contain MCQs.
BME students perform relatively worse in assessments with numerical based questions.
Male students perform relatively worse in group based work.

20. Results Stage 2: Determinants of Attainment Gaps
Ethnicity
Gender
International
Disability
Students(100)
0.022***
(0.000)
-0.028***
(0.005)
0.0278**
(0.018)
Male Ratio
0.121**
(0.032)
Average
Mark
0.003***
0.004***
(0.002)
Weight
0.001**
(0.048)
(0.028)
Anonymity
-0.024*
(0.080)
0.048***
(0.009)
Coursework
0.059**
(0.010)
0.081***
Practical
0.027*
(0.078)
0.050*
(0.053)
0.072***
(0.001)
In Class
0.027**
(0.019)
0.036*
(0.084)
MCQ
0.043**
Numerical
-0.077***
Group
-0.044***
Cohort2012
-0.093***
0.063***
Cohort2013
-0.028**
(0.023)
Order
-0.047**
(0.035)
0.045*
(0.070)
Year2
-0.071**
(0.011)
0.052*
(0.094)
Year3
(0.006)
0.091**
(0.015)
Constant
-0.261***
-0.322***
-0.231***
(0.003)
-0.440***
𝑹 𝟐
0.325
0.280
0.349
0.321 n
179
Cohort variables:
For all demographic groups there are specific cohort (year of intake) effects, suggesting some group dynamics.

21. Results Stage 2: Determinants of Attainment Gaps
Ethnicity
Gender
International
Disability
Students(100)
0.022***
(0.000)
-0.028***
(0.005)
0.0278**
(0.018)
Male Ratio
0.121**
(0.032)
Average
Mark
0.003***
0.004***
(0.002)
Weight
0.001**
(0.048)
(0.028)
Anonymity
-0.024*
(0.080)
0.048***
(0.009)
Coursework
0.059**
(0.010)
0.081***
Practical
0.027*
(0.078)
0.050*
(0.053)
0.072***
(0.001)
In Class
0.027**
(0.019)
0.036*
(0.084)
MCQ
0.043**
Numerical
-0.077***
Group
-0.044***
Cohort2012
-0.093***
0.063***
Cohort2013
-0.028**
(0.023)
Order
-0.047**
(0.035)
0.045*
(0.070)
Year2
-0.071**
(0.011)
0.052*
(0.094)
Year3
(0.006)
0.091**
(0.015)
Constant
-0.261***
-0.322***
-0.231***
(0.003)
-0.440***
𝑹 𝟐
0.325
0.280
0.349
0.321 n
179
Timing effects:
Male students perform relatively worse as the year progresses, whereas international students perform relatively better.
However, international students perform worse as the academic years progress (perform worse in the 2nd year and worse again the 3rd).
The opposite is true for disabled students whose relative performance improves through the academic year.

22. Our Approach: Advantages
Looks at all students who start course, not just those who complete.
Analyses final grade and therefore data is continuous rather than discrete (% vs grade boundaries).
Considers different demographic groups simultaneously:
Ethnicity
Gender
Disability
Nationality
Can look for potential causes in attainment differences between:
Module characteristics (number of students, demographics of students)
Assessment characteristics (type of assessment, type of question, group or individual, anonymously marked, etc.)
Cohort effects (defined by the year of the student intake)
Timing effects (year of study, time of year)
This can then influence decisions on assessment and curriculum.
Richard McManus
Moira Mitchell

23. Our Approach: Disadvantages
Although a rich dataset, not much scope to look at demographic characteristics interaction:
In our methodology attainment gaps are linear and additive
Potential for influencing outliers:
Only consider modules with more than 20 students;
Apply a ‘weighted least squares’ approach.
Only consider one school in one university over a three year period of time.
Restrained to only consider those things we can measure; what we did not have data for:
Attendance;
Engagement;
Whether students are working;
Attitudes;
Etc.
Richard McManus
Moira Mitchell