1. tinyurl.com/ydfyav8p

2. Data Sources and Data Challenges

3. https://data.gov.uk/publisher
Data Publishers
We will be taking a look at a small number of national datasets produced by several data publishers:
For a full list see:

5. Datasets, reports and documentation
Datasets are often produced as part of a series.
For example, the children looked after statistics produced by the DfE are released in September every year.
Depending on your question, you may simply be interested in the latest release, or all entries in the series.
As a dataset is released so does a report which typically covers:
Headline statements.
Current status of each measure in the dataset.
Any changes in trend from the previous releases.
These reports are a good way for you to quickly assess how useful a dataset might be.

6. Areas and Mapping

7. Areas and Mapping Two main ways of defining a geographical area:
Administrative boundaries
Output Areas and Super Output Areas
For more information on these, as well as information on other geographical units go to:

8. Administrative Boundaries for England

9. Administrative Boundaries for England
Electoral wards/divisions are the building blocks of administrative and electoral areas.
In 2016 there were 7,445 electoral wards/divisions in England.
They can be subject to change and changes can occur every year.
Boundaries are reviewed by the Boundary Commission (LGBCE) and they try to maintain equal populations across neighbouring wards for the purposes of voting.

10. Output Areas for England
Specifically designed for analysis.
Output Areas were introduced as part of the 2001 Census for England and Wales, and last updated as part of the Census.
Built from clusters of neighbouring postcodes.
Each OA was designed to be as socially homogenous as possible based on tenure of household and dwelling type.
Mixtures of urban and rural postcodes were avoided where possible.
Output Area has an average population size of 309.
32,844 Lower-layer Super Output Areas (LSOA)
6,791 Middle-layer Super Output Areas (MSOA)
OAs and SOAs were designed so they can be aligned to local authority district (LAD) boundaries.

11. Output Areas for England

12. How to assign data to an area?
If your data has post codes, you can use the ONS Postcode Directory to lookup the Ward, Output Area, LSOA, etc.
The ONSPD provides lots of different areas, some historic.
Key columns in the ONSPD:
PCD – Postcode
DOINTR – Date of introduction
DOTERM – Date of termination
OSWARD – Electoral ward division
OSLAUA – Local authority district (LAD, LB, UA, MB)
OA01, OA11 – 2001 and 2011 Census Output Area
LSOA01, LSOA11 – 2001 and 2011 Census LSOA
MSOA01, MSOA11 – 2001 and 2011 Census MSOA
For a full definition see the User Guide that you download with the ONSPD.
[Example: ONSPD_AUG_UK_2017_UK_LA.csv]

13. Getting area boundaries
Just some tips when trying to find boundaries online.
ONS provide boundaries at each of the different area levels.
Resolution:
Full – highest resolution, used for Advanced GIS analysis, very large file sizes
(Super | Ultra) Generalised – much lower resolution, file size can be 10% of ‘Full’, used for visualisation
Extent:
Extent of the realm – used for analysis
Clipped to the coastline – used for visualisation, as maps often resemble what someone would expect the costal boundary to look like.
Map boundaries can come in a variety of formats:
Shapefile, KML, GeoJSON, WMN or WFS

14. Resolution: Full vs Generalised

15. Population Estimates and Projections

16. ONS population estimates
What are they?
They represent the number of people who usually live in an area on 30 June of each year, regardless of nationality.
ONS provide annual population estimates at all the area levels already mentioned.
Annually estimates are available from onwards.
Estimates for an area are provided by gender and year of age (capped at 90 and above)
ONS use a variety of data sources and methods to produce the estimates, information on this can be found in their “QMI report”.
They release estimates for the previous year normally on June each year.

17. ONS population estimates
What do the estimates look like?
What can population estimates be used for?
Used by local government and the health sector, where they are used for planning and monitoring service delivery, and managing resource allocation.
Often used as an input. So the population estimates by themselves are not necessarily helpful. But can be used to calculate measures such as rates.
Where can I find the data?
Two download options from ONS population estimates page:
Mid-2016 single spreadsheet – good for visual inspection
Mid-2016 detailed time series – good for loading into a database and using for analysis

18. Rates of stop and search (1)
How does the rate of stop and searches differ by age?
Define rate:
𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑝𝑒𝑜𝑝𝑙𝑒 𝑠𝑡𝑜𝑝𝑝𝑒𝑑 𝑆𝑖𝑧𝑒 𝑜𝑓 𝑝𝑜𝑝𝑢𝑙𝑎𝑡𝑖𝑜𝑛 ×10,000
Look at Lancashire, Q4 (Oct, Nov, Dec) 2016
Age
Pop. Est.
Number
Stopped
Rate
0-9
140,000
10-17
106,700
150 14
18-24
108,600
222 20
25-34
142,800
175 12

19. Rates of stop and search (2)
Table of rates in Lancashire, from Q to Q4 2016
Age
2015
2016 Q2 Q3 Q4 Q1
0-9
10-17 23 31 38 32 21 13 14
18-24 36 46 37 28 18 20
25-34 19 25 12

20. Rates of stop and search (3)
Figures of counts and rates in Lancashire, from Q to Q4 2016

21. ONS population projections
What are they?
Projections are published every 2-3 years, but there’s no set timetable.
ONS provide annual estimates of the future size, gender and age structure of the population, down to the Local Authority District level.
Latest release was published in 2016, and contains annual projections from mid-2016 up to mid-2116
Estimates are provided by gender and year of age.
Uses mid-year population estimates, assumptions about future fertility, mortality and migration.
They represent the number of people who are expected to be usually living in an area on 30 June of each year, regardless of nationality.

22. ONS population projections
What do the projections look like?

23. ONS population projections
What can population projections be used for?
Similar usage as population estimates.
Used by local government and the health sector, where they are used for planning service delivery, and managing future resource allocation.
Often used as an input, along with existing rates to calculate future numbers.
Forecasting methods can use the population projections to improve the quality of predicted values.
Where can I find the data?
Several dataset options from ONS population projections page:
Table A2-4: Principal projection - England population in age groups
Table 2: Subnational Population Projections for Local Authorities in England

24. English Indices of Deprivation

25. English Indices of Deprivation
What are they?
A measure of relative deprivation in small areas, not actual deprivation.
Good for identifying the most deprived area, not deprived people.
Good for comparing small areas across England, but not Scotland, Wales and Northern Ireland.
A general measure of deprivation is published called the Index of Multiple Deprivation (IMD).
IMD scores are provided for each LSOA, and averages are provided for the Local Authority Districts.
As well as the overall IMD measure, scores are provided across seven domains.

26. English Indices of Deprivation
What have they been used for?
Targeting resources: Use by national and local organisations to identify places for prioritising resources and more effective targeting of funding.
Example: used to distribute £448m of funding to local authorities for the Troubled Families Programme.
Example: the most deprived 15 per cent of neighbourhoods were eligible for insulation measures from energy companies.
Policy and strategy: Development of evidence base to help understand current need and demand for services.
Example: used to assess the equality of access to local health and other services.
Research: understanding challenges and performance of different areas subject to different services and policies.
Example: Relationship between pupil attainment and neighbourhood deprivation.

27. Seven domains of deprivation
Income Deprivation - proportion of the population experiencing deprivation relating to low income. Employment Deprivation – proportion of the working-age population who would like to work but are unable to do so.

28. Seven domains of deprivation
Education Deprivation – lack of attainment and skills in the local population. Health Deprivation – measures risk of premature death and the impairment of quality of like through poor health.

29. Seven domains of deprivation
Crime – risk of personal and material victimisation at a local level. Barriers to Housing and Services – physical and financial accessibility of housing and local services.

30. Seven domains of deprivation
Living Environment Deprivation – quality of the local environment: housing, air quality, road traffic accidents.

31. Association between the domains

32. English Indices of Deprivation
Where can I find the data?
The Indices have already been aggregated and summarised in lots of different ways at the LSOA and Local Authority level.
deprivation-2015
File 2: Domains of deprivation – Ranks of each LSOA regarding the Index of multiple deprivation, as well as in each of the seven domains.
File 5: Scores for the indices of deprivation – Score for each LSOA regarding IMD and each of the seven domains.
File 10: Local Authority District Summaries – Average rank for IMD and the seven domains for each Local Authority District.

33. Pupil Absence

34. Pupil Absence What is it?
DfE reports on absence levels of all pupils using two key measures:
One session is equal to half a day.
Overall absence: total of authorised and unauthorised absent sessions, including illness.
Overall absence rate:
𝑇𝑜𝑡𝑎𝑙 𝑎𝑏𝑠𝑒𝑛𝑐𝑒 𝑠𝑒𝑠𝑠𝑖𝑜𝑛𝑠 𝑇𝑜𝑡𝑎𝑙 𝑠𝑒𝑠𝑠𝑖𝑜𝑛𝑠 ×100
Persistent absence: overall absence equates to 10 percent or more of the possible sessions.
Persistent absence rate:
𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑝𝑢𝑝𝑖𝑙𝑠 𝑐𝑙𝑎𝑠𝑠 𝑎𝑠 𝑝𝑒𝑟𝑠𝑖𝑠𝑡𝑒𝑛𝑡 𝑎𝑏𝑠𝑒𝑛𝑡𝑒𝑠𝑠 𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑝𝑢𝑝𝑖𝑙𝑠 ×100

35. Pupil Absence What level is are the rates reported at?
National, local authority, and school.
What is it used for?
Specialist publications such as the Good Schools Guide.
Development of policy (Education Policy Institute).
Research: Does missing one week of school lead to lower grades?

36. Pupil Absence
What does the data look like? 2015/16 map of local authority overall absence rates. Lancashire appears to be fairly low. Liverpool, Blackpool, the Wirral fairly high.

37. Pupil Absence Where can I find the data?
Date released by Autumn term, Spring term, or as a full-year release.]
Main Text – this is the associated report.
Main Tables – pretty tables, easy to read.
Underlying Data – good for loading into stats software.

38. Limitations of area-based analyses

39. Ecological fallacy
The ecological fallacy is when someone takes inferences and associations made at an area-level and tries to make statements about individual people.
We cannot use information we have about an area to make inferences about an individual. Associations at an area-level may not exist at the individual level.
Example: over a 12 month period look at two groups of pupils in terms of:
whether or not a pupil has been absent
whether or not they have been stopped and searched.

40. Example
Absent
Stopped
Size
Absent
Stopped 10 4 3 5
Absent
Stopped

41. Example Area ID Absent Stopped 1 36 15 2 42 9 3 35 17 4 33 5 39 13 640 7 8 32 10 26
Can only make statements about the area. From the example, it’s true to say in areas of higher absence we have higher rates of children getting stopped and searched.

42. Finally

43. Two data sources I have not talked about
Understanding Society
Crime Survey for England and Wales
UK household longitudinal survey.
Monitors the extent of crime.
Interview people every year, of all ages about their lives.
Used to evaluate and develop crime deduction policies.
Publish data at the Police Force Area.
National, regional and local data.
Sample size in 2015/16 was 35,000.
Annual findings reports
Minimum of 650 per police force area.
Case studies.
Amazing data documentation.
Data for these is accessed via the UK Data Service which is beyond this session.

44. Lab exercise: Explore! Pick a data publisher or a dataset
Find the latest release and the corresponding report.
Look for the Underlying data.
What’s the lowest area-level available for the data?
How might you use it?
Are you interested in having data over time, or just one year snapshot?
TIP: If you get lost in the streams of tables that get published, go back to Google and try again.