Presentation on theme: "Quick win or slow burn? Modelling UK HE CAA uptake Bill Warburton University of Southampton."— Presentation transcript:
Quick win or slow burn? Modelling UK HE CAA uptake Bill Warburton University of Southampton
Agenda Why isn’t CAA uptake higher? Quick win / slow burn Dual-path theory of CAA uptake Models of CAA uptake What does ‘success’ really mean? Applications of dual-path theory
Why isn’t uptake higher? Background to the research –Academics seemed to avoid CAA even where it appeared to be fit for purpose –Wanted to understand why this was –Three year research project including national survey and targeted interviews –Output was a grounded theory of CAA uptake
Quick wins? Pressure on tutors (Gibbs et al. 2000 etc.) –Worsening student/staff ratios –Pressure for more frequent monitoring of student progress –Students more demanding –RAE Productivity pressures utilitarian approach less favourable outcomes peers discouraged
Quick wins? … when the email came round about the [CAA] disaster… some of those colleagues… just went non-linear… how can we possibly have… taken on something which under the most fundamentally obvious things that it had to work under, it fails at the first hurdle? (Tutor AmO5M007)
Slow burn? Tutors aiming primarily for pedagogical improvements incurred less risk Often managed by Learning Technologists Characterised by a staged approach: –Incremental growth in stakes (and risk) –Lower cumulative risk levels –Modest productivity gains
Dual-path theory of CAA uptake Consequence: Lower-risk trajectory Conditions Tutor’s priority is for learning & teaching gains Interaction Ad-hoc practice discourages colleagues Negative feedback loop Context Tutor’s learning & teaching practice Perceived risk level Consequence: Higher-risk trajectory Interaction Structured practice encourages uptake amongst peers Conditions Tutor’s priority is for productivity gains Positive feedback loop TUTOR DECIDES TO USE CAA HIGHER-RISK ZONE LOWER-RISK ZONE Risk mitigating measures taken
Dual-path theory of CAA uptake TUTOR DECIDES TO USE CAA Strategic resourcing Physical Infrastructure Learning & teaching infrastructure Tutor’s experience mitigates risk Procedural risk mitigating measures taken by LTs Structured practice encourages uptake amongst peers Lower-risk trajectory Higher-risk trajectory Prioritise learning & teaching gains Positive feedback loop Negative feedback loop Tutor’s priority is for productivity gains Ad-hoc practice discourages peers Perceived risk level Strategic funding & commitment Physical risk mitigation measures taken by IT depts Institutional validation of existing good practice Tutor’s learning & teaching practice Strategic policy & strategy
Models of CAA uptake – Concentric shell Individual consequence 1 Tutor decides to use CAA Strategic cultural conditions Pressures for greater productivity Drive for online learning- MLEs, VLEs Effective strategy to manage change Informed long-term top-down commitment Clear institutional L&T strategy inc. QA of CAA Infrastructure cultural conditions Effective CAA Best Practice guide Central effective support for CAA Shared question banking Proven cost/benefit gains Strategic operational conditions Secure central financial resources Tutor cultural conditions Tutors not conscripted Tutors’ traditional autonomy threatened Tutors have realistic expectations of CAA Tutors believe CAA drives pedagogic development Tutor operational conditions Tutors acquired a good grasp of assessment Tutors early experience of CAA was good Tutors are innovators and early adopters Tutors acquired requisite IT skills Strategic cultural interactions with infrastructure SMT gives credit for CAA initiatives & allows time to develop CAA Strategic operational interactions with infrastructure SMT provides adequate resources for effective support & training Infrastructure interactions with tutors LTs manage complexity of CAA for tutors LTs & peers provide tutors with subject-specific exemplars LTs & tutors work together in teams to create & review tests CAA champions encourage tutors Enthusiastic students encourage tutors Infrastructure operational conditions CAA system is easily used Adequate N/W & W/S infrastructure CAA system is stable, reliable, resilient CAA system is effectively interoperable CAA system is flexible, fit-for-purpose CAA system is secure & centrally supported CAA system is accessible to all students Individual consequence 2 Tutor’s CAA trajectory is low risk Infrastructure consequence LTs & tutors reassured that CAA is ‘safe’ & fit for purpose Strategic consequence SMT demonstrates institutional commitment to tutors& LTs
Models of CAA uptake - Tutor Trajectories Piloting Low stakes, small scale Low stakes, large scale Embedded End of life Planning Linear Institutional Linear tutor Disastrous Ad hoc Cut short Twist or stick High stakes
Models of CAA uptake – ‘Arctic Roll’ Model 1 Tutor’s propensities Tutor’s experiences 7. Calls for improved assessment practice from funding bodies and other central organisations Time Tutors’ CAA trajectories L&T practices Physical infrastructure Strategy Resourcing 2. Coordinated dissemination of CAA practice facilitated by L&T specialists 1. Ad hoc dissemination of CAA practice at department level by individual tutors 5. Coordinated strategy for CAA uptake approved by SMT 3. Coordinated procedural control by LTs 4. Coordinated physical risk mitigation by central L&T specialists 6. Coordinated resourcing provided through senior management
Models of CAA uptake – ‘Arctic Roll’ Model 2 Time 4. Fragmentary approach to physical risk mitigation by L&T specialists 6. Resources withheld by senior management Tutor’s propensities Tutor’s experiences 7. Widely published concerns about ‘dumbing down’ in national press L&T practices Physical infrastructure Strategy Resourcing Tutors’ CAA trajectories 2. Ineffective dissemination of CAA practice facilitated by L&T specialists 1. Poor publicity from tutors at department level 5. Inertia of SMT in coordinating a strategy for CAA uptake 3. Ineffective procedural risk mitigation by L&T specialists
What does ‘success’ mean? Whose notion of success? Tutors? Learning technologists? QA staff? Senior management? Metrics of success on an institutional scale… ‘Embeddedness’ i.e. is taken for granted & has become ‘invisible’ and is (for some well- informed specialists) the key metric of success Level of integration with corporate MIS systems e.g. MLE ‘Width’ of practice (in terms of scale) Success on what scale? Institutional? Faculty/department? Individual tutors? Number of users Number of tests taken Range of items types Range of use (diag/form/sum) Range of subject use (Hum/Qual/Quant) ‘Width of dissemination’ Has the application been taken up by other institutions?
Applying dual-path theory At institutional level –Institution-specific factors populate models –Reveal likely patterns of institutional uptake At the level of individual tutors –risk propensity predicts CAA trajectory type –could be used to direct resources where they might be used most effectively to mitigate risky practice Clear descriptors facilitate common references for characterising uptake
Questions? That’s All, Folks! Do you see any important gaps? Does the Dual Path Risk Mitigation idea ring true?