Presentation on theme: "DfE Data Exchange Project"— Presentation transcript:
1DfE Data Exchange Project The Analytical ReviewThe remit and my strategic recommendations for future data needsRoger Plant, So Direct Ltd.Co –Author, The DfE Analytical Review
2The Analytical Review Terms of Reference Key factors Conclusions and recommendationsThis review asked that we find a system where only data essential to the Department and to a devolved self-improving system is collected. Such a system needs to be efficient, trusted and must maintain appropriate standards of quality.
3Analytical Review: Key Factors to address The number of different systems used to collect and deliver the data with each needing specific new programming, documentation and user training whenever data is changed.The same data is being changed and checked in parallel by different people at different processing stages without modifications feeding back to the source (the delivery organisation which generated the data).That the Department is dealing with ever increasing numbers of delivery organisations during this checking process. As more schools convert to Academies the Department must check data with thousands of schools individually rather than through 150 local authorities.The fact that data held by schools and other frontline providers on their systems is not the same as that held by local authorities or the Department. This means that data has to be reconciled every time there is a data collection.
4Analytical Review: High Level Vision We need to move to model where:Data can be automatically moved from one organisation to another with no manual interventionData is updated and shared on a real-time basis as part of day-to-day business processesData is available to all of the people who need it for decision making when they need it.
5Analytical Review…more specifically… To achieve this vision the Department should put in place the standards and technology that will enable the automatic flow of data. This can be done using off-the shelf technology that is widely used commercially, that will:Collect only the data that is needed at a given moment rather than larger set-piece collections. This will support the gathering of timely and less burdensome data exchanges.Automate data checking and validation, with responsibility for accuracy pushed back to source.Provide real-time feedback to delivery organisations and third parties in a format that can be used for comparative analysis. The more data is used and is useful, the greater the incentive to increase accuracy and so the system becomes self-cleaning.Enhance the efficiency and effectiveness of the data warehouse and the School Performance Data Programme as data will be more readily available and up-to-date.Draw on existing industry-wide standards so that data sets can be efficiently matched together, whilst also allowing new categories of data to be added for use in schools, authorities and the Department at minimum cost.Automate data collection wherever possible for all education and children’s services. For example, through agreement with schools, attendance data or free school meals data could be regularly accessed.Support teaching and learning directly. The system will be able to cater for broadening data demands particularly in relation to performance and pedagogical data held in systems such as learning platforms.Reduce the costs related to the management and use of data both within the Department and throughout the education and children’s services community.
6Analytical Review: Further recommendations The Department should source an interoperability system for the exchange, use and maintenance of all data.An appropriate data model should be developed through collaborative design between users and system suppliers coordinated by the Information Standards Board.The Star Chamber should be used to ratify the inclusion of data items in the data model. The remit and membership of Star Chamber should be reviewed as part of the changes to the systemData should be gathered and used in real-time, as part of the day-to-day business processes. Some formal collections may still be timetabled (for example core pupil data - once a year) whilst others will be running regularly (for example attendance data gathering - weekly).The data exchange process should include a data validation system designed to ensure that data is always validated at source.The data validation service should be able to be used independently of data exchanges to help ensure the accuracy of source data.The Department’s data warehouse should become the one-stop-shop for all education and children’s services data in England.The system should provide analytical tools within a Departmental web portal offering controlled access to reports and data extractions for the community.
7Learning from International Experience: With thanks to NSIP - Australia Extracts have been taken fromNSIP PROGRAM REVIEWA REVIEW FOR THE NATIONAL SCHOOLSINTEROPERABILITY PROGRAM (NSIP)STEERING GROUP 12 FEBRUARY 2013
8AUSTRALIA - NATIONAL SCHOOLS INTEROPERABILITY PROGRAM (NSIP) Achieving interoperability between education systems is increasingly crucialThe ability for education information and communication technology (ICT) systems to effectively interoperate is becoming essential to deliver national and multi-jurisdiction education policies and projects.The evolving policy context is driving demand for further work on systems interoperabilityThere is agreement that demand for interoperability support is growing and is principally driven by four primary factors: a growing trend towards national education policy and projects; constrained public finances; technological advances creating new opportunities for education innovation; and desire for student-centred programs.
9AUSTRALIA - NATIONAL SCHOOLS INTEROPERABILITY PROGRAM (NSIP) Interoperability refers to the capacity for information systems to effectively exchange data using agreed standards and architectures. The concept of interoperability is not new, but there is a growing view that systems must be able to share information and resources to support common education objectives. Achieving interoperability requires a coordinated focus on technical standards, the design of systems that capture, store and transfer data, as well as the strategic decision making and business processes of system owners. Interoperability can be described through its practical applications, including:Interoperability of systems that provide access to learning resources and teaching toolsInteroperability of systems that enable reliable and secure transfer of student dataInteroperability of systems that support online assessment and performance monitoring, including the linking of datasets required for reporting purposesInteroperability of systems that will support online learning using products and services from multiple providers on an increasingly diverse range of devices.
10NSIP Conclusions1: Interoperability will remain a priority for national education projects, and will be difficult to achieve without a national forum bringing together the views of the full range of education stakeholders.2: There is a strong case for a nationally representative interoperability entity to be retained, performing many or all of the functions currently undertaken by NSIP.3: Interoperability support should apply to all national or multi-jurisdictional education projects involving technology components.4: Understanding of the value of interoperability and underpinning issues that need to be solved should be broadened beyond those with specific ICT knowledge.
11Analytical Review – going forward… Data standardsWe must go forward beyond Common Basic Data SetGovernanceWe will establish protocols to cover data exchange so ensuring “Managed Interoperability”UsageWe will ensure that data exchange has the potential to support all aspects of education managementIn conclusion, we should work for a clearer articulation of the role and benefits of interoperability. This will increase understanding of its potential value among stakeholders and in turn will help to broaden support for systems interoperability in general, and potentially assist in clearing roadblocks in the delivery of relevant projects.
12Data Exchange Phase One: Defining the solution requirements with confidence Eight Core Activities…1. Understanding Interoperability& Learning from others5. Legislation2. Stakeholder Engagement6. Interdependency 1: Definitions,Data Models & Standards3. Scoping of Data Exchange7. Technical Architecture:Documenting requirementsand appraising high level solutions4. Demonstrations of Technologies &input from implementers8. Establishing the projectand governance
13Data Exchange Project : What a data exchange means to us Principles to note:School / Provider MIS market &relationship not affectedAll MIS are still free to bestructured as they please(yellow, green, brown etc)All MIS need to be able totalk the same language at somepoint, hence the ‘commonlanguage translator’ prior tothe exchange all being red. Acommon language is neededfor all exchange approachesMovement of data can beout of, and into, a school / LA/ DfEThe central exchangetechnologies can be set up ininfinite different ways…figuringout the best way is where weneed most help!Data exchange
14Data Exchange: Benefits in an afternoon… ChildrenQuicker Identification of those going off role and not reappearing – supporting troubled familiesInteraction with school nurses would reduce ‘falling between the gap’ when moving schoolsData held on the child automatically accessible should the child cross LA boundariese.g.: statement information immediately availableQuicker processes for decision making e.g. FSMDfEMore timely evidence base to support decisionsLess cost associated with obtaining dataMore frequent informationAbility to develop new collections more quicklyFuture proof system easily adapted to handleacademisation programme and future initiativesTogether with SPDP, support greater use of datain schools and research communityIncrease internal monitoring.Enable schools and LAs to drive benchmarkingand analytical agenda. Putting evidence in thehands of local decision makersSchools / ProvidersReduced collection burden from DfEGreater recognition / consistency withDfE / ‘published’ dataGreater access to comparative dataCommon Transfer File always available,not dependent on departing school sendingAllow existing system compatibilityInput once use many times. No need tosend several consumers the same data (eg LA, DfEOfsted)Potential to speed up processes such asHR data when teachers move to a new schoolFast andsimple datamovementLAsReduced administrative costs for things like FSM entitlementLess time chasing schools for dataMore frequent attendance data = better understanding of impact of interventionsBetter information to support capacity knowledge and admissions processAllow existing system compatibilityCross boundary information immediately available
15Vision – a reminderThe vision for the ‘To Be’ state Data Exchange capability is that data is:Moved from one organisation to another with minimal manual intervention and interoperable data exchangeUploaded to an operational data store and shared on a near real-time basis as part of day-to-day business processes that require itAvailable to all of the people who need it for decision making when they need it15
16High Level Functional Requirements CodeFunctional RequirementsFR1The Data Exchange Capability (DEC) shall be capable of providing dataflow services in either direction between any two end points (including statutory data exchanges and sending notifications to interested parties)An end point can be a school or a local authority or the Department.We want to be able to exchange information flexibly between any two or group of end points to future proof the solutionFR2The DEC shall be able to transfer any dataflow sets within maximum volume limitsWe want a solution that is as flexible as possible to avoid cost and delay in setting up new data collections / exchanges16
17High Level Functional Requirements CodeFunctional RequirementsFR3The DEC shall provide facilities for authorised users to define dataflow services in terms of:Data flow contentsSchedule for collectionPerformance targetsSource and destination end points (individual or multiple)We need to be able to set up the dataflows, including what they contain, when they need to be sent, how quickly and who to. We don’t want to have to ask the solution provider to do this for us, because that will increase cost and introduce delayFR4The DEC shall require minimal manual intervention once a dataflow service has been defined and configuredWe want to make the transfer of information as easy as possible. Ideally a school will only need to set up a dataflow once and it will happen automatically afterwards.17
18High Level Functional Requirements CodeFunctional RequirementsFR5The DEC shall provide facilities for authorised users to add, maintain or remove end pointsWe need to be able to add in new schools or agencies that education sector users want to exchange information with quickly and easily and without having to pay the supplier to do this.FR6The DEC shall provide facilities for authorised users to define validation rules, including:What the validation rule isAt what points it should be appliedWhere validation errors should be reported toValidation is important.Ideally the school management information systems will validate information as it is entered, but if any get past this stage, we want to be able to catch them as soon as possible.Over time new rules might be identified, so we want to be able to add these in as we go along, rather than needing to pay the supplier to make changes.If an error is detected, then we need to make sure that the right contact is informed, so that the data can be corrected at source, improving data quality for all improved18
19High Level Functional Requirements CodeFunctional RequirementsFR7The DEC shall provide capabilities for authorised users to:Undertake ad-hoc queries of any data that they are authorised to accessCreate, maintain, schedule, use and delete standard reportsPrepare ad-hoc reportsOnce data has been collected then it needs to be possible to report on it.Schools need to be able to view their own data and to see how they compare with (anonymised) others.Local Authorities need to view information about schools under their controlThe DfE need to undertake statistical and other analysis of the data.In all cases we do not want to pay the supplier to develop reports.FR8The DEC shall provide capabilities for authorised users to write to end point MIS and other systemsThis is required, for example, to enable local authorities to provide information on free school meals to schools.19
20High Level Non Functional Requirements CodeNon Functional RequirementsNFR1The DEC shall achieve a data transfer time of:30 minutes or less (TBC) for < 10% of the defined dataflow services24 hours or less (TBC) for 45% of the defined dataflow services7 days or less (TBC) for 45% of the defined dataflow servicesWe have been told that there is a requirement for data to be transferred in ‘near real time’ but it isn’t clear what this means.We think that 30 minutes is quick enough and that this would only be required for a small percentage of dataflows. Is this right? Are the other timescales reasonable?NFR2The DEC shall be scalable:To manage peaks and troughs (in volume, capacity and pupil cohort sizes)Such that it can, if required, take on other Department data flow setsWe want the DEC to be able to cope with higher levels of data exchange in the future if required, to avoid the need to start again.20
21High Level Non Functional Requirements CodeNon Functional RequirementsNFR3The DEC shall be able to support40,000 end points (TBD)5 million messages daily (TBD)The DEC needs to be able to support at least 25,000 schools and a range of other organisations. It may be that some schools have multiple end points, so this number needs confirming.As to the number of messages, it will depend on the way of working. If messages are sent after every change, then the number of messages may be higher but the messages will be smaller. At the other extreme, if schools sent a message with all changes once a week, then this would be closer to 5,000 /day.NFR4The DEC shall be:Available 24 hours per day, 7 days per week (not including maintenance periods) with a 99% availabilityResilient to ensure no more than 30 minutes interruption of service for Data Exchange servicesThe DEC will be a core infrastructure element within the education sector and should have a correspondingly high availability21
22High Level Non Functional Requirements CodeNon Functional RequirementsNFR5The DEC shall only allow users to access data for which they have a business need and legal right to do soWe need to ensure that access to data is controlled, so that data providers can be reassured that the data that they provide is only accessed by those authorised to do so. The DEC will need to be designed and built with security in mind.NFR6The DEC must comply with the HMG IA requirements for information up to:IL3 for individual messagesIL4 in aggregateThe requirements for security are published in the Security Policy Framework and supporting documents. The Business Impact Levels (IL) are a measure of the impact of the inappropriate release of information. This requirement defines the impact level of the data and this will affect the strength of the security controls that need to be put in place.22
23High Level Non Functional Requirements CodeNon Functional RequirementsNFR7The DEC shall record when data was gathered, from where and who entered it (TBD)We need this audit trail to meet security requirements and for data quality management.NFR8The DEC shall provide an intuitive user interface that satisfies the requirements of the DDA and related accessibility standards (eg W3C)We need the DEC to be easily usable by anyone that needs to use it as part of their roleNFR9The DEC shall make use of open standard technologiesThis is a requirement based on HMG ICT strategy. It means that it should be easier to maintain or upgrade the system in the future and avoid lock in to any particular supplier.23