1. IPUMS-International Integration Process
Matt Sobek Minnesota Population Center

2. 1 2 3 4 Input material Pre-processing Standardization Integration DATA
Data files
Batch samples
Reformat data
Donation
Draw sample
Confidentiality A
Code clean-up
Verify data
Confidentiality B
Harmonize codes
Variable programming
Constructed variables
METADATA
Data dictionary
Enumeration forms
Enum. instructions
Sample information
Translate to English
Images to editable files
Ipums data dictionary
Tag enumeration text
Document unharmonized variables
Variable descriptions
Sample design

3. Matt Sobek Minnesota Population Center sobek@pop.umn.edu
End
Matt Sobek Minnesota Population Center

4. Batch Samples
In spring we identify the samples to integrate the following year.
Samples are processed as a group – one per year. The entire batch of samples is processed through each stage before we proceed to the next step.
There is little flexibility in the work process. If a sample is not available for processing during the earliest stages of integration, it cannot be included in the data release for that year.

5. Original Input Data
Some examples of differing file formats:
SPSS and SAS system files
Redatam-format
IMPS format
Records that combine household and person characteristics
Separate files for persons, households (and dwellings, buildings)
Different types of records (mortality or migration)
Separate files for different administrative units

6. Reformatting: Original Data File

7. Reformatting: Data File after Reformatting

8. Reformatting: Rectangular Sample
(Person records only; household data duplicated on person records)
geography
housing
person (head)
person (child)
person (spouse)
geography
housing
person (head)
person (child)
geography
housing
person (head)
person (spouse)
person (child)
(Brazil 1980)

9. Reformatting: Dwelling-Household-Person Sample
(Separate dwelling and household records)
dwelling
household
dwelling
household
person (head)
person (spouse)
person (child)
person (head)
person (spouse)
person (child)
dwelling
household
household
person (head)
person (child)
person (head)
person (child)
dwelling
household
person (head)
person (spouse)
dwelling
household
person (head)
person (spouse)
(Chile 1992)

10. Reformatting: Merge Household and Person Files
Household File
serial 001
geog & housing
serial 002
serial 003
serial 001
household
serial 001
head
spouse
serial 002
household
serial 002
head
child
Person File
serial 001
head
spouse
serial 002
child
serial 003
serial 003
household
serial 003
head
(Brazil 2000)

11. Reformatting: Persons not Organized in Households
(Individuals only; not organized in households)
geog
person
housing
geog
person
housing
household
person
household
person
household
person
household
person
household
person
(Mexico 1960)

12. Donation and Error Correction
Data are tested for errors that affect structural integrity, such as merged households, unmatched person and household records, corrupted records, etc. Such errors often do not affect tabulations, but create inconsistencies across records within households that affect sophisticated analyses.
Some problems can be resolved with custom programming.
Other problems are resolved by donating (substituting) a donor household for the corrupted one.
Households are divided into strata based on predictor variables. Donors are drawn from the same strata as the corrupted household, ensuring they share key characteristics.
If a sample is drawn from the full census, a substitute donor record is used; if we are already starting with a sample, the donor record is duplicated. A flag indicates that a record was duplicated.

13. Drawing a Sample
About one-third of IPUMS samples are drawn from full-count data.
After reformatting, we draw a systematic sample of every Nth dwelling to yield the desired sample density – typically 10%.
If the input data are not full-count (for example, they include only the long-form records), the sample design might have to account for differing sample densities between areas.
Very large dwelling units (over 30 persons) are sampled at the individual level – not as intact units – in order to reduce sampling error. Every Nth individual is taken.

14. Confidentiality Measures: A
Swap a small percentage of cases between geographic areas.
Reorder households within geographic areas.
Suppress low-level geographic variables.
Suppress any variable deemed too sensitive by the National Statistical Office.
Encrypt all versions of the data prior to the imposition of these confidentiality measures.

15. Code Clean-Up: Recoding Unharmonized Variables
Recode the input variables to conform to some basic standards for treatment of missing values, etc.
Recode stray values into a consolidated missing category as appropriate.
Convert non-numeric characters to numeric.
Most recoding is performed using a data translation matrix like the one below for Marital Status in 1984 Costa Rica. If the recoding requires more complex logic, use custom programming.

16. Verify Data: Unharmonized Variables
Examine the marginal frequencies of every input variable.
Analyze the data universe for each variable – the population at risk of having a response. Determine the theoretical universe from enumeration materials or other documentation, then empirically determine any discrepancies from that universe.
Document the universe for each variable and any other observations.

17. Confidentiality Measures: B
Recode geographic units to ensure small localities cannot be identified (typically those with fewer than 20,000 persons).
For recent censuses:
Identify cells that represent very small numbers of persons in the population. Code them to a residual category or combine them.
Top- or bottom-code continuous variables that have a long tail that could identify small subpopulations.
Suppress specific categories of variables as requested by the National Statistical Office.

18. Translation Matrix for Marital Status
Harmonize Codes:
Translation Matrix for Marital Status
China
1982
Colombia
1973
Kenya
1989
Mexico
1970
U.S.A.
1990

19. Variable Programming
Some variable manipulations are too complex to be handled using the translation matrix tables. Typically these involve continuous variables or recoding logic that refers to multiple variables. This programming is written in C++.

20. Constructed “Pointer” Variables
(Simple household)
Spouse’s
Pernum
Relate
Age
Sex
Marst
Chborn 1
head 46
male
married
n/a 2
spouse 44
female 3
aunt 77
widow 7 4
child 15
single 5 13 6 11
Location 2 1
Mother’s
Father’s
Pernum
Relate
Age
Sex
Marst
Chborn 1
head 46
male
married
n/a 2
spouse 44
female 3
aunt 77
widow 7 4
child 15
single 5 13 6 11
Location
Location 2 1 2 1 2 1
(Colombia 1985)

21. Constructed “Pointer” Variables
(Complex household)
Spouse’s
Mother’s
Father’s
Pernum
Relationship
Age
Sex
Marst
Chborn 1
head 53
female
separated 6 2
child 28
male
single
n/a 3 22 4 21 5 25
married
child-in-law 7
grandchild 8 9
non-relative 32 10 11
Location
Location
Location 1 6 5 5 6 5 6 9 9
(Colombia 1985)

22. Original Data Dictionary – Kenya 1989

23. Original Data Dictionary – Romania 1992

24. Original Data Dictionary – China 1982

25. Original Data Dictionary – Mexico 1990

26. Enumeration Form: Original File

27. Enumeration Instructions: Original File (Mexico 1990)

28. Sample Information – from Statistical Office
Sample information is difficult for the IPUMS project to collect. Often only limited information can be gleaned from available documentation. It is extremely helpful when countries collate the information themselves, as was done below by the Netherlands:

29. Translate Documents to English
Many countries provide their census documentation in English. For those that do not, the IPUMS project hires translators from around the world. Often these are persons currently or formerly associated with National Statistical Offices. Some common languages are translated by staff in Minnesota.

30. Editable Enumeration Form – In English

31. IPUMS Data Dictionary

32. XML-Tagged Enumeration Form

33. Document Unharmonized Variables
The enumeration form and instruction text provides most of the documentation for the unharmonized input variables.
Other documentation is written as needed to clarify the interpretation of the variable for users.
We also empirically determine the universe of persons or households with valid values for each variable.

34. Variable Description (Literacy)

35. Sample Design