1. Measuring Coverage: Post Enumeration Surveys Owen Abbott Office for National Statistics, UK

2. Agenda Introduction Why have a PES? Essential features of a PES
Survey Design
Fieldwork
Analysing the data
Matching
Estimation
Results from 2001 UK Census
Discussion

3. Why do we need a PES? Census won’t count every household or person
Undercount causes bias in estimates
In the UK in 2001, we estimated that 3 million persons (6%) did not fill in the form
Increasing problem from 1981 to 1991 to 2001
The undercount is not evenly spread
Inner Cities
Deprived areas
Young persons

4. Why do we need a PES? Census counts alone not good enough
UK Users demand robust census population estimates
Central Government resource allocation
Yearly demographic population estimates
Government Policy
So we need to measure how many households and persons the census misses, and work out:
where they are missed from
their characteristics

5. Basic Methodology PES - Census Coverage Survey (CCS) in UK
In the UK approx 1% population
Match the PES to the Census
Use the people the PES sees that the census didn’t to estimate how many missed
where and characteristics
Add to the Census counts (either at aggregate level or impute (UK))

6. 2001 UK ‘One Number Census’ framework

7. Post Enumeration Survey
Key features:
A - Design
Sample survey
Sample size dependent on accuracy (and geographic level) requirements
B - Fieldwork
Conducted after the census has finished
Independent re-enumeration
Area based
Door to door interview
Focused on measuring coverage

8. Post Enumeration Survey - Design
Multi-stage Stratified sample
Select a sample of (small) geographical areas that can be re-enumerated
UK uses Postcodes (about 20 hhs)
US uses blocks (about ????100 hhs)
Sample stratified by:
Geography
Area type
Demography

9. 2001 UK PES Design Geographical Strata:
Local Authorities (mean pop 120k) grouped into contiguous groups called Estimation areas (EAs), each having 500k pop
Area Type and Demographic strata:
Within every EA a sample of 1991 Enumeration Districts was selected, stratified using a hard-to-count index and the 1991 age-sex structure
(1991 EDs have about 200 households)

10. 2001 UK PES Design
Hard to count index was a national stratification using a combination of variables associated with undercount e.g:
Unemployed
Multi-occupied
Private rented
Language difficulty
3 level index, split into 40%, 40%, 20% nationally
Within each selected ED a sample of 3, 4 or 5 postcodes was selected

11. Post Enumeration Survey - Field
Aim: enumerate all the people and households in the sampled areas
Carry out the survey after the Census
Census fieldwork finished
Independence critical (see later)
Interview based
Independent re-enumeration
Separate fieldforce and management
No address list (UK have address list for Census)
Difficult if doing quality at same time, as not independent

12. Post Enumeration Survey - Field
In UK, focused on measuring coverage
Previously measured quality as well
Found that separate surveys more effective
Can focus on getting maximal response in sampled areas
UK 2001 PES used very short interview
key household and demographic questions only
Accommodation type
Tenure
Name
Gender
Date of Birth (or Age)
Student
Ethnicity
Activity last week

13. Post Enumeration Survey - Field
Other initiatives to maximise response:
Pairwork and teamwork
Refusal avoidance training
Calling strategy
Up to 10 attempts to interview
Last attempt deliver form to return in post

14. Post Enumeration Survey
Interviewer Duties:
Establish the postcode boundaries
Conduct independent listing of all residential and non-residential addresses
Seek out obscure accommodation
Deliver advance notification cards
Identify/probe for all households at an address
Make contact with householders
Conduct doorstep interviews
Persuade potential refusals
Report Progress

15. Post Enumeration Survey
Map

16. Post Enumeration Survey
Property Listing

17. Analysing the data - Matching
Match Census returns to CCS returns
Require very high quality
Minimise false negative matches (missed matches, see later)
In 2001, we used hierarchical nature of data to help match
Match within sampled areas (geographical blocking)
First match household
Then match persons within households

18. Analysing the data - Matching
Used a five stage strategy, designed to minimise false negative matches:
Exact matching
High probability matching
Clerical assisted probability matching
Clerical matching
Final expert review of non-matches
Developed our own in-house system
Allowed access to scanned form images (this was crucial)

19. PO155RR
ERIC
SMITH 13
MALE
SINGLE 29

20. Analysing the data - Matching
Output:
Match between Census and CCS
Census only
CCS only

21. Analysing the data – Estimation
Dual System Estimation (DSE)
Capture-recapture as used for wildlife
Simple example: How many fish in a lake?
Catch as many as possible on day 1
Count them (N1)
Mark with a red dot
Return them to the lake
Catch as many as possible on day 2
Count them (N2)
Count how many have red dots (N12)
Number of fish in lake= (N1 * N2)/N12

22. Analysing the data - Estimation
Use matched Census+CCS data
DSE estimates adjustment for those missed in both Census and CCS
Counted By CCS
Yes No
Counted Yes n11 n10 n1+
By Census No n01 n00 n0+
n+1 n+0 n++
DSE count (for a postcode):
n++ = n1+ x n+1  n11

23. Analysing the data - Estimation
DSE assumptions
Independence
Homogeneity of capture probabilities
Perfect matching
Closure
No list inflation
Violation of these assumptions leads to bias (in both directions)
Lots of literature on DSE

24. Analysing the data – Estimation
DSE can only be used within the sample
Need additional step to get to population totals
In 2001, we used DSE at postcode level
Then used a ratio estimator to predict for non-sampled postcodes (again lots of literature)

25. Analysis – Getting to small areas
Ratio estimator produced estimates for 500k population blocks
Needed estimates for Local Authorities (about 120k population)
Sample size not sufficient to do directly
So used small area estimation techniques
these borrow strength across areas
We used a fixed effect to model LA differences
LA population estimates from the model then constrained to EA totals

26. Quick summary of 2001 UK method
In 2001, One Number Census methodology was developed
Large CCS (320,000 households)
Matching
Capture Recapture
Modified ratio estimator
Small area estimation to get LA totals
Imputation
Estimated 1.5 million households missed
3 million persons missed (most from the missing households but some from counted households)

27. Results
England and Wales population about 50m individuals in 20m households
Estimated 1.5 million households missed
3 million persons missed (most from the missing households but some from counted households)

28. Underenumeration in 2001

29. Response Rates in 2001

30. Summary Fundamental that the census is good
This does not make a bad census good, it makes a good census better!
US, Australia, NZ, Canada, UK all measure coverage (and most use a PES)
All aim at measuring coverage for assessing census quality, most do not fully adjust the outputs
Coverage for most is around 96-98%
Increasing problems of overcoverage
The design and fieldwork of the PES are important to get right

31. More info
Brown, J.J., Diamond, I.D., Chambers, R.L., Buckner, L.J., and Teague, A.D. (1999), “A methodological strategy for a one-number census in the UK,” Journal of the Royal Statistical Society A, 162,