DfE Data Exchange Project Launch Event: Pre-Procurement Supplier Session 6 th June 2013 2 St Pauls Place, Sheffield.

DfE Data Exchange Project Launch Event: Pre-Procurement Supplier Session 6 th June 2013 2 St Pauls Place, Sheffield

Welcome and Introductions  Domestics  The objectives of the day  Personal reflections  Outline of the session: –Analytical Review: Remit and Recommendations –The Departmental Response: The story so far –Technical Architecture Work: Presentation and challenge –Discussion topics and Q&A

DfE Data Exchange Project The Analytical Review The remit and my strategic recommendations for future data needs Roger Plant, So Direct Ltd. Co –Author, The DfE Analytical Review

The Analytical Review  Terms of Reference  Key factors  Conclusions and recommendations This review asked that we find a system where only data essential to the Department and to a devolved self-improving system is collected. Such a system needs to be efficient, trusted and must maintain appropriate standards of quality.

 The number of different systems used to collect and deliver the data with each needing specific new programming, documentation and user training whenever data is changed.  The same data is being changed and checked in parallel by different people at different processing stages without modifications feeding back to the source (the delivery organisation which generated the data).  That the Department is dealing with ever increasing numbers of delivery organisations during this checking process. As more schools convert to Academies the Department must check data with thousands of schools individually rather than through 150 local authorities.  The fact that data held by schools and other frontline providers on their systems is not the same as that held by local authorities or the Department. This means that data has to be reconciled every time there is a data collection. Analytical Review: Key Factors to address

We need to move to model where:  Data can be automatically moved from one organisation to another with no manual intervention  Data is updated and shared on a real-time basis as part of day-to-day business processes  Data is available to all of the people who need it for decision making when they need it. Analytical Review: High Level Vision

To achieve this vision the Department should put in place the standards and technology that will enable the automatic flow of data. This can be done using off-the shelf technology that is widely used commercially, that will: –Collect only the data that is needed at a given moment rather than larger set-piece collections. This will support the gathering of timely and less burdensome data exchanges. –Automate data checking and validation, with responsibility for accuracy pushed back to source. –Provide real-time feedback to delivery organisations and third parties in a format that can be used for comparative analysis. The more data is used and is useful, the greater the incentive to increase accuracy and so the system becomes self-cleaning. –Enhance the efficiency and effectiveness of the data warehouse and the School Performance Data Programme as data will be more readily available and up-to-date. –Draw on existing industry-wide standards so that data sets can be efficiently matched together, whilst also allowing new categories of data to be added for use in schools, authorities and the Department at minimum cost. –Automate data collection wherever possible for all education and children’s services. For example, through agreement with schools, attendance data or free school meals data could be regularly accessed. –Support teaching and learning directly. The system will be able to cater for broadening data demands particularly in relation to performance and pedagogical data held in systems such as learning platforms. –Reduce the costs related to the management and use of data both within the Department and throughout the education and children’s services community. Analytical Review…more specifically…

 The Department should source an interoperability system for the exchange, use and maintenance of all data.  An appropriate data model should be developed through collaborative design between users and system suppliers coordinated by the Information Standards Board.  The Star Chamber should be used to ratify the inclusion of data items in the data model. The remit and membership of Star Chamber should be reviewed as part of the changes to the system  Data should be gathered and used in real-time, as part of the day-to-day business processes. Some formal collections may still be timetabled (for example core pupil data - once a year) whilst others will be running regularly (for example attendance data gathering - weekly).  The data exchange process should include a data validation system designed to ensure that data is always validated at source.  The data validation service should be able to be used independently of data exchanges to help ensure the accuracy of source data.  The Department’s data warehouse should become the one-stop-shop for all education and children’s services data in England.  The system should provide analytical tools within a Departmental web portal offering controlled access to reports and data extractions for the community. Analytical Review: Further recommendations

Learning from International Experience: With thanks to NSIP - Australia Extracts have been taken from NSIP PROGRAM REVIEW A REVIEW FOR THE NATIONAL SCHOOLS INTEROPERABILITY PROGRAM (NSIP) STEERING GROUP 12 FEBRUARY 2013

AUSTRALIA - NATIONAL SCHOOLS INTEROPERABILITY PROGRAM (NSIP)  Achieving interoperability between education systems is increasingly crucial –The ability for education information and communication technology (ICT) systems to effectively interoperate is becoming essential to deliver national and multi-jurisdiction education policies and projects.  The evolving policy context is driving demand for further work on systems interoperability –There is agreement that demand for interoperability support is growing and is principally driven by four primary factors: a growing trend towards national education policy and projects; constrained public finances; technological advances creating new opportunities for education innovation; and desire for student-centred programs.

 Interoperability refers to the capacity for information systems to effectively exchange data using agreed standards and architectures. The concept of interoperability is not new, but there is a growing view that systems must be able to share information and resources to support common education objectives. Achieving interoperability requires a coordinated focus on technical standards, the design of systems that capture, store and transfer data, as well as the strategic decision making and business processes of system owners. Interoperability can be described through its practical applications, including:  Interoperability of systems that provide access to learning resources and teaching tools  Interoperability of systems that enable reliable and secure transfer of student data  Interoperability of systems that support online assessment and performance monitoring, including the linking of datasets required for reporting purposes  Interoperability of systems that will support online learning using products and services from multiple providers on an increasingly diverse range of devices. AUSTRALIA - NATIONAL SCHOOLS INTEROPERABILITY PROGRAM (NSIP)

NSIP Conclusions 1: Interoperability will remain a priority for national education projects, and will be difficult to achieve without a national forum bringing together the views of the full range of education stakeholders. 2: There is a strong case for a nationally representative interoperability entity to be retained, performing many or all of the functions currently undertaken by NSIP. 3: Interoperability support should apply to all national or multi-jurisdictional education projects involving technology components. 4: Understanding of the value of interoperability and underpinning issues that need to be solved should be broadened beyond those with specific ICT knowledge.

 Data standards –We must go forward beyond Common Basic Data Set  Governance –We will establish protocols to cover data exchange so ensuring “Managed Interoperability”  Usage –We will ensure that data exchange has the potential to support all aspects of education management In conclusion, we should work for a clearer articulation of the role and benefits of interoperability. This will increase understanding of its potential value among stakeholders and in turn will help to broaden support for systems interoperability in general, and potentially assist in clearing roadblocks in the delivery of relevant projects. Analytical Review – going forward…

DfE Data Exchange Project The Data Exchange Project The Department’s Response to the Analytical Review, progress to date. Iain Bradley, Data Exchange Lead. Education Data Division, DfE

Data Exchange: The Initial DfE Response & activities  Initial senior reactions  Taking stock and building knowledge  Identification of, and relationship with, key interdependencies  Very early stakeholder engagement with MIS suppliers and Star Chamber Phases we identified (…but remain subject to change): Phase One: Phase One: ‘Defining the solution requirements with confidence’ up to July 2013. Depending on outcome of that… Phase Two: Sourcing the solution: August 2013 to March 2014 Phase Three: Building the solution: April 2014 onwards

Data Exchange Phase One: Defining the solution requirements with confidence Eight Core Activities… 1. Understanding Interoperability & Learning from others 2. Stakeholder Engagement 3. Scoping of Data Exchange 4. Demonstrations of Technologies & input from implementers 5. Legislation 6. Interdependency 1: Definitions, Data Models & Standards 7. Technical Architecture: Documenting requirements and appraising high level solutions 8. Establishing the project and governance

Data Exchange Project Progress so far – critical path summary Understanding DfE needs Requirements Gathering Provider and LA engagement ‘Common Language’ Gap Analysis (i.e. standards) Proposed Scoping & Phasing Identifying High level options Technical Architecture Technical Discussions about Candidate solutions Understanding Landscape & fit with SPDP / ISB Options Appraisal Business Case & Solution Sourcing Today

Data Exchange: Current Activities Supplier and Launch Event…today! AM – ‘pre-procurement supplier engagement’ – ensuring a common and publically available communication about our response to the Analytical Review, and likely direction of travel of the project. Technical / Supplier focus PM – small group of school leads to hear about the proposals, sense check requirements captured so far during school / LA engagement sessions and consider the softer handling of a successfully managed data exchange. Demonstrations of technologies Following our request on Contracts Finder, working with organisations who are supporting the requirements and architectural work by: Demonstrating the relative quality of near real time data compared with current census processes Discussing pros and cons of where to position SPDP Data Warehouse within a solution Learning from similar exchange implementations in the public sector Focusing on detailed elements of requirements such as security to assist thinking on requirements.  Significant number of technical conversations to assess fit with requirements – more of that later.  Data Transformation Programme governance establishment and further flushing out of interdependencies & implications  Assessment of candidate solutions by Actica completes July  Drawing together knowledge captured so far to inform detailed scope and phasing proposals, likely to be based on: –Demand for regular data movement –Maturity of data standards / data model to inform ‘market readiness’ –Potential savings / benefits to sector to be made as a result.

Data Exchange Project : What a data exchange means to us Principles to note: School / Provider MIS market & relationship not affected All MIS are still free to be structured as they please (yellow, green, brown etc) All MIS need to be able to talk the same language at some point, hence the ‘common language translator’ prior to the exchange all being red. A common language is needed for all exchange approaches Movement of data can be out of, and into, a school / LA / DfE The central exchange technologies can be set up in infinite different ways…figuring out the best way is where we need most help! Data exchange

Data Exchange: Benefits in an afternoon… Children Quicker Identification of those going off role and not reappearing – supporting troubled families Interaction with school nurses would reduce ‘falling between the gap’ when moving schools Data held on the child automatically accessible should the child cross LA boundaries e.g.: statement information immediately available Quicker processes for decision making e.g. FSM Schools / Providers Reduced collection burden from DfE Greater recognition / consistency with DfE / ‘published’ data Greater access to comparative data Common Transfer File always available, not dependent on departing school sending Allow existing system compatibility Input once use many times. No need to send several consumers the same data (eg LA, DfE Ofsted) Potential to speed up processes such as HR data when teachers move to a new school LAs Reduced administrative costs for things like FSM entitlement Less time chasing schools for data More frequent attendance data = better understanding of impact of interventions Better information to support capacity knowledge and admissions process Allow existing system compatibility Cross boundary information immediately available DfE More timely evidence base to support decisions Less cost associated with obtaining data More frequent information Ability to develop new collections more quickly Future proof system easily adapted to handle academisation programme and future initiatives Together with SPDP, support greater use of data in schools and research community Increase internal monitoring. Enable schools and LAs to drive benchmarking and analytical agenda. Putting evidence in the hands of local decision makers Fast and simple data movement

Data Exchange Project: Challenges still to be resolved  Sector wide solutions Vs something we can deliver  Who would fund what?  Where does the SPDP Data Warehouse sit in the solution?  Information Standards – implementation and further developments

Data Exchange Project: Introducing Actica  Early on, we felt that to enable us to move with pace and confidence, we needed technical architecture expertise within the project.  Actica, the Departments strategic infrastructure and asset management contractor were appointed in Spring.  Engaged them early with this priority project to support us with the following:  Information and requirements gathering from across the sector  Assessment of the candidate solution options  Refine recommended solution option and reporting (focusing on a proposed implementation strategy) …i.e. getting an architectural perspective to work up requirements sufficiently well to communicate what we want/need, and to identify the most efficient technological solution to meet those requirements.

DfE Data Exchange Project Technical Architecture Work Requirements, options and assessment Andy Evason / David Robinson Actica Consulting

Data exchange – where might it go? (A non exhaustive brainstorm of interest) Data Exchange System Pupil Referral Unit Independent Schools Maintained Schools Academies / Free schools Local Authorities Educational Data Division Standards and Testing Agency Education Funding Agency National College for Teaching & Learning Awarding Bodies Ofsted Further Education Higher Education Information Authority Policy Analysts in DfE

Vision – a reminder  The vision for the ‘To Be’ state Data Exchange capability is that data is:  Moved from one organisation to another with minimal manual intervention and interoperable data exchange  Uploaded to an operational data store and shared on a near real-time basis as part of day-to-day business processes that require it  Available to all of the people who need it for decision making when they need it

Introduction to requirements  Methodology  Analysed existing information flows  Talked to a range of stakeholders within the education sector and in industry  Developed candidate requirements, and subject to review by stakeholders  Types of requirement  Functional – what data exchange has to do  Eg Allow standard information flows to be defined  Non-functional – how well it must do it  Performance (data transfer in less than 30 minutes)  Security  Requirements should be sufficiently detailed to select best option, not for procurement at this stage

High Level Functional Requirements CodeFunctional Requirements FR1 The Data Exchange Capability (DEC) shall be capable of providing dataflow services in either direction between any two end points (including statutory data exchanges and sending notifications to interested parties) An end point can be a school or a local authority or the Department. We want to be able to exchange information flexibly between any two or group of end points to future proof the solution FR2The DEC shall be able to transfer any dataflow sets within maximum volume limits We want a solution that is as flexible as possible to avoid cost and delay in setting up new data collections / exchanges

High Level Functional Requirements CodeFunctional Requirements FR3The DEC shall provide facilities for authorised users to define dataflow services in terms of: 1.Data flow contents 2.Schedule for collection 3.Performance targets 4.Source and destination end points (individual or multiple) We need to be able to set up the dataflows, including what they contain, when they need to be sent, how quickly and who to. We don’t want to have to ask the solution provider to do this for us, because that will increase cost and introduce delay FR4The DEC shall require minimal manual intervention once a dataflow service has been defined and configured We want to make the transfer of information as easy as possible. Ideally a school will only need to set up a dataflow once and it will happen automatically afterwards.

High Level Functional Requirements CodeFunctional Requirements FR5 The DEC shall provide facilities for authorised users to add, maintain or remove end points We need to be able to add in new schools or agencies that education sector users want to exchange information with quickly and easily and without having to pay the supplier to do this. FR6 The DEC shall provide facilities for authorised users to define validation rules, including: 1.What the validation rule is 2.At what points it should be applied 3.Where validation errors should be reported to Validation is important. Ideally the school management information systems will validate information as it is entered, but if any get past this stage, we want to be able to catch them as soon as possible. Over time new rules might be identified, so we want to be able to add these in as we go along, rather than needing to pay the supplier to make changes. If an error is detected, then we need to make sure that the right contact is informed, so that the data can be corrected at source, improving data quality for all improved

High Level Functional Requirements CodeFunctional Requirements FR7The DEC shall provide capabilities for authorised users to: 1.Undertake ad-hoc queries of any data that they are authorised to access 2.Create, maintain, schedule, use and delete standard reports 3.Prepare ad-hoc reports Once data has been collected then it needs to be possible to report on it. Schools need to be able to view their own data and to see how they compare with (anonymised) others. Local Authorities need to view information about schools under their control The DfE need to undertake statistical and other analysis of the data. In all cases we do not want to pay the supplier to develop reports. FR8The DEC shall provide capabilities for authorised users to write to end point MIS and other systems This is required, for example, to enable local authorities to provide information on free school meals to schools.

High Level Non Functional Requirements CodeNon Functional Requirements NFR1 The DEC shall achieve a data transfer time of: 1. 30 minutes or less (TBC) for < 10% of the defined dataflow services 2.24 hours or less (TBC) for 45% of the defined dataflow services 3.7 days or less (TBC) for 45% of the defined dataflow services We have been told that there is a requirement for data to be transferred in ‘near real time’ but it isn’t clear what this means. We think that 30 minutes is quick enough and that this would only be required for a small percentage of dataflows. Is this right? Are the other timescales reasonable? NFR2The DEC shall be scalable: 1.To manage peaks and troughs (in volume, capacity and pupil cohort sizes) 2.Such that it can, if required, take on other Department data flow sets We want the DEC to be able to cope with higher levels of data exchange in the future if required, to avoid the need to start again.

High Level Non Functional Requirements CodeNon Functional Requirements NFR3The DEC shall be able to support 1.40,000 end points (TBD) 2.5 million messages daily (TBD) The DEC needs to be able to support at least 25,000 schools and a range of other organisations. It may be that some schools have multiple end points, so this number needs confirming. As to the number of messages, it will depend on the way of working. If messages are sent after every change, then the number of messages may be higher but the messages will be smaller. At the other extreme, if schools sent a message with all changes once a week, then this would be closer to 5,000 /day. NFR4The DEC shall be: 1.Available 24 hours per day, 7 days per week (not including maintenance periods) with a 99% availability 2.Resilient to ensure no more than 30 minutes interruption of service for Data Exchange services The DEC will be a core infrastructure element within the education sector and should have a correspondingly high availability

High Level Non Functional Requirements CodeNon Functional Requirements NFR5The DEC shall only allow users to access data for which they have a business need and legal right to do so We need to ensure that access to data is controlled, so that data providers can be reassured that the data that they provide is only accessed by those authorised to do so. The DEC will need to be designed and built with security in mind. NFR6The DEC must comply with the HMG IA requirements for information up to: 1.IL3 for individual messages 2.IL4 in aggregate The requirements for security are published in the Security Policy Framework and supporting documents. The Business Impact Levels (IL) are a measure of the impact of the inappropriate release of information. This requirement defines the impact level of the data and this will affect the strength of the security controls that need to be put in place.

High Level Non Functional Requirements CodeNon Functional Requirements NFR7The DEC shall record when data was gathered, from where and who entered it (TBD) We need this audit trail to meet security requirements and for data quality management. NFR8 The DEC shall provide an intuitive user interface that satisfies the requirements of the DDA and related accessibility standards (eg W3C) We need the DEC to be easily usable by anyone that needs to use it as part of their role NFR9 The DEC shall make use of open standard technologies This is a requirement based on HMG ICT strategy. It means that it should be easier to maintain or upgrade the system in the future and avoid lock in to any particular supplier.

Options  Architectural Solution Options 1.Distributed 2.Hierarchical 3.Hub and Spoke 4.Centralised  Real world examples  SIF 3 (Option 2 or 3)  The Data Service DCFT solution (Option 3)  Development of COLLECT and existing data transfer infrastructure (Option 3)  Also considering data model and procurement aspects of options

Option 1: Distributed Every end point potentially talks to every other No central store Data pulled when needed LAs and DfE store data for trend analysis

Option 2: Hierarchical MIS data from schools pushed to LAs Data pushed from LAs to DfE for analysis Interface with others (Ofsted, STA etc) from most appropriate end-point School to school transfer between schools or common LA

Option 3: Hub and spoke All data transfer via central hub LAs can operate own systems or use portal to access info as they choose School to school or similar transfers via hub

Option 4: Centralised All schools use same cloud hosted systems, no data in schools, LAs or DfE Census, LA access, EDD and school to school all within single system

Option assessment criteria  Sector Strategic fit  Sector strategy  Business strategies  Wider HMG strategies and policies, including procurement  Cultural sensitivities  Satisfaction of education sector requirements  Functional  Non-functional  Cost  Whole life cost over 5 years  Including capital, support, maintenance, development, etc  Risk  Timescale

Way forward  Apply criteria to options  Identify recommended option  Agree best option  Refine selected option  Report  Future activities  Seek approval to proceed  Procure  Implement

Technical Architecture: Q&A

DfE Data Exchange Project Open Discussions Theme 1: Managed Interoperability – what would best practice look like and who should be involved? Theme 2: Supporting Data Quality – what services & behaviours should the project involve / provide to ensure MIS and data exchange systems maximise the potential to validate data upon entry / at source? Theme 3: Future working: What behaviours and standards would you like to see from the Data Exchange Project going forwards?

Final Panel Q&A -Roger Plant -Iain Bradley -Andy Evason -Jude Hillary

Contact Details Roger Plant: rogerplant@gmail.comrogerplant@gmail.com Iain Bradley: Iain.Bradley@education.gsi.gov.ukIain.Bradley@education.gsi.gov.uk Andy Evason: Andy.Evason@actica.co.ukAndy.Evason@actica.co.uk David Robinson: David1.Robinson@education.gsi.gov.ukDavid1.Robinson@education.gsi.gov.uk General Project Enquiries: Data.Exchange@education.gsi.gov.ukData.Exchange@education.gsi.gov.uk

DfE Data Exchange Project Launch Event: School and LA focus group 6 th June 2013 2 St Pauls Place, Sheffield

Welcome and Introductions  Domestics  The objectives of the day  Personal Reflections  Outline of the session: –Analytical Review: Remit and Recommendations –The Departmental Response: The story so far –What data exchange might mean for schools and data movements which could benefit –Supporting technologies with good governance –Capturing information about volumes, speeds and further thinking