Measures for assessing the impact of ICT use on attainment Ian Stevenson University of Leeds

Structure of talk What Impact 2 tells us Aims of the Study Focus of the Study Modelling teaching and learning Data Collection Measured developed Observations based on the study

What Impact 2 tells us ICT usage can have positive effects on high stakes examinations Children’s ICT home experience can have a significant impact on their attainment Difficult to establish a causal link between ICT experience and attainment Context of learning is significant

Aims of the Becta Study Building on ImpaCT2 to develop a set of measures capable of collecting “snapshot” data on relationship between ICT and attainment Examine aspects of resourcing, content, practice and other factors that play a part in governing the likely impact that ICT may have on attainment.

Project Team Jeremy Higham (Steering Group) Isobel Jenkins (Observer) John Monaghan (Steering Group) Ian Stevenson (Director) John Threlfall (Steering Group) Stephen Wilkinson (Observer)

Focus of the study What ICT-related factors are associated with development of: Curricular achievement? ICT skills? Personal factors Motivation? Self-confidence? Collaboration? Autonomy?

Modelling teaching and learning Purpose Outcomes Structure Conditions Resources distribution, site policy, curriculum Phase 1 Tools Roles Organisation Phase 2 Tools Roles Organisation Phase 3 Tools Roles Organisation

Tools leader using ICT learners using ICT individually, initiated by the leader learners using ICT in collaborative tasks initiated by the leader learners using ICT individually, initiated by themselves learners using ICT in collaborative tasks initiated by themselves learners interacting via ICT, initiated by the leader learners interacting via ICT, initiated by themselves learners creating using ICT (visual arts, music, etc).

Organisation of teaching and learning leaders’ lecturing/explanation leaders' teamwork learners' teamwork learners' working individually learners' reporting or presenting own material.

Roles of leaders and learners leader giving information leader directing questions and answers to reproduce facts leader directing conversation leader stimulating reflections or other critical analysis learners directing conversation with peers or leaders.

Data Collection and Organisation Focus on teaching and learning activities Observation and interview with activity leaders Survey of sites, leaders and learners Use relational database for organisation and analysis of the data collected

Data Collected Leeds “average LEA” (Ofsted) 24 sites visited between February and April 2003 selected to cover location, phase,size, and socio- economic background 61 activities observed across all KS 48 leaders and 50 groups of learners

Measures Developed Extent to which an activity met its intended outcomes (rated 1-6) Leaders’ confidence in using ICT for teaching and learning Structure of an activity Potential for attainment

Outcomes achieved? RatingDescription 1All students achieved substantial improvement 2All students achieved significant improvement 3All students achieved some improvement 4Most students achieved substantial improvement 5Most students achieved significant improvement 6Most students achieved some improvement

Leaders’ confidence with ICT Score was based on a survey of leaders’ ICT background and usage for teaching, learning, and administration Three groups of leaders’ confidence scores: Group A (11%): long experience of teaching, but who made comparatively little use of ICT for teaching or administration. Group B (47%): long teaching experience but who made extensive use of ICT for teaching or administration Group C (42%) used ICT for all of their relatively short teaching career.

Structure of teaching and learning Activities fell into two main types: Leader-centred approach: leader using the ICT asking learners questions to check facts or to confirm understanding Learner-centred approach: learners using the ICT directing the conversation as they worked on the task that they were set or chose. Leaders asked questions to encourage critical reflection or to confirm understanding

Potential for Attainment Potential for contributing to attainment, based on : Hardware configuration Leader confidence score Structure of teaching and learning Outcome rating for the activity

Structure of teaching and learning Nearly 90% of activities with highest PA score had a learner-centred structure Learners had the necessary ICT skills for the activity Learners had the opportunity for sustained and meaningful work using ICT, either on their own or with others.

Leader Confidence 75% of activities with the highest PA score were led by members of leader group B, with the rest in group C Activities had clear outcomes and were carefully planned, using ICT appropriately Leaders were skilled in engaging learners in reflection and analysing their own work with ICT, both informally and with whole groups

Further Work Expand the database to cover a larger range of sites Use the measures to monitor ICT and attainment over a period of time at a variety of sites Focus on learners: Relationship between specific types of attainment and ICT applications used in different settings Role of ICT experience in non-formal settings for developing attainment Value-added analysis of learners’ ICT use and their attainment, over a period of several years

Understanding ICT and pedagogy Mediational tool: Purpose, context, meaning Activity theory: System and structure Rules CommunityOrganisation Subject Object Tool Outcome Activity System Activity Structure

Structure of Activities SubjectObject Tool Outcome Affordance and Constraint

Community and Context SubjectObject Tool Outcome OrganisationCommunityRules Tools and Communities Rules and ontology Pedagogy and structure