1. United Nations Expert Group Meeting on International Standards for Civil Registration and Vital Statistics Systems, 27-30 June 2011, New York Evaluation and assessment in civil registration and vital statistics systems Discussion, United Nations Statistics Division

2. United Nations Expert Group Meeting on International Standards for Civil Registration and Vital Statistics Systems, 27-30 June 2011, New York Australia – Evaluating the quality of vital statistics: paper for discussion Overarching assessment/evaluation framework - WHO and University of Queensland Health Information Systems Knowledge Hub Several components covering inputs, processes and outputs An assessment tool for independent evaluation studies Legal framework/organizational structure/coverage/accuracy/quality check/data dissemination The tool itself covers a sub-component that asks whether quality check is being conducted regularly (E.1)

3. United Nations Expert Group Meeting on International Standards for Civil Registration and Vital Statistics Systems, 27-30 June 2011, New York Australia – Evaluating the quality of vital statistics: paper for discussion Comments on Chapter IV of the Principles and Recommendations for a Vital Statistics System, Rev. 2 developing an assessment framework Technical and dense, may provide a set of guiding questions (see above framework) Consider the following additional criteria when evaluating vital statistics Relevance Ensuring international comparability Ensuring confidentiality of individuals

4. United Nations Expert Group Meeting on International Standards for Civil Registration and Vital Statistics Systems, 27-30 June 2011, New York The vital statistics system in China Three main sources of population statistics Population and housing censuses (every 10 years) Household sample surveys: 1% population survey (inter-censal) 1‰ population change survey (every year) Administrative records: Population by permanent residents (Ministry of Public Security) Registered temporary residents (Ministry of Public Security) Marriages and divorces (Ministry of Civil Affairs) Health statistics (Ministry of Health)

5. United Nations Expert Group Meeting on International Standards for Civil Registration and Vital Statistics Systems, 27-30 June 2011, New York The vital statistics system in China Data checking: Resident population and migrants’ information from the household registration administered by the Ministry of Public Security; birth information collected by family planning departments, the health department as well as the community and village committees – used to check and compare information collected in the 2010 censuses.

6. United Nations Expert Group Meeting on International Standards for Civil Registration and Vital Statistics Systems, 27-30 June 2011, New York Reflections/some thoughts on the revision of the P&R, rev 2. The chapter should answer the following questions: Why is quality assurance and assessment important? Against what standards are we evaluating? What is quality assurance and assessment? How should it be done? – What methods? – How often? – Who should be doing it?

7. United Nations Expert Group Meeting on International Standards for Civil Registration and Vital Statistics Systems, 27-30 June 2011, New York Measuring against the following standards Principles and Recommendations for a Vital Statistics System, Rev. 2 (para. 547) Completeness Correctness/accuracy Availability Timeliness Additional ones to consider? Relevance International/national comparability data confidentiality

8. United Nations Expert Group Meeting on International Standards for Civil Registration and Vital Statistics Systems, 27-30 June 2011, New York Quality assurance – what is it/how is it done? Occurs at each operation stage within the civil registration and vital statistics systems, for example – Registration stage: making sure that all events are registered without duplication; information verified; all required information provided – Data collection – Data transfer: Monitoring the submitted statistics report: timeliness, completeness (from all registration areas), and monitoring any irregularities – Data compilation, editing stage – Data dissemination Making sure that all the checks are in place and are integrated part of the two systems (Australia paper)

9. United Nations Expert Group Meeting on International Standards for Civil Registration and Vital Statistics Systems, 27-30 June 2011, New York Quality assurance – how often? Who does it? How often? – Constant Who does it? – Civil registrars at all levels – Statisticians compiling, processing and disseminating vital statistics

10. United Nations Expert Group Meeting on International Standards for Civil Registration and Vital Statistics Systems, 27-30 June 2011, New York Example: United States Hospital Completes Birth Record Passes all NCHS edits? Submit Record to VR Jurisdiction Via Web Correct Errors and Resubmit for Validation System Automatically Performs NCHS Validations N LIVE BIRTH VR Database END 2.0 Local Registration Print and Send Paper Record to Local Registrar Y Manually Logs, Reviews and Files Record Hold for X Days Record Received From Local Registrar? N Mail or Deliver Record to VR Jurisdiction END Match Records, Manually Stamp, Log, Key Local File Number Generate Routine Submissions to NCHS Register Record Y Timeliness Metrics: A.Number of days from date of birth to when paper record is received from local registrar (A1 to A2). B.Number of days from when record is received from local registrar to when record is registered (B1 to B2). C.Number of days from when record is registered to when it is submitted to NCHS (C1 to C2). C2C2 C1C1 B2B2 B1B1 A2A2 A1A1

11. United Nations Expert Group Meeting on International Standards for Civil Registration and Vital Statistics Systems, 27-30 June 2011, New York Process flow: deaths and stillbirths – South Africa CollectionPasting Export to Editing Data Analysis STORES Sorting Data Editing Publication & Dissemination Pre-coding Data Capture & QA Coding Derivation 0f UCD Data Capture (ID variables)

12. United Nations Expert Group Meeting on International Standards for Civil Registration and Vital Statistics Systems, 27-30 June 2011, New York Quality assessment studies – what are they? Answers specific questions and with specific objectives – What is the coverage of birth registration? – Who are the un-registered? What are their characteristics? – Is there difference in the quality of cause of death data from two provinces? What causes the difference and how to improve? – Is my new cause of death coder competent? – Is the information collected on birth records for District A accurate? Some studies carried out regularly, some ad-hoc basis Best if carried out by an independent group

13. United Nations Expert Group Meeting on International Standards for Civil Registration and Vital Statistics Systems, 27-30 June 2011, New York Quality assessment studies – how? Direct methods – matching birth and death records: identify un-registered records; an assessment of both registers – Matching civil registration records with other administrative records (e.g. social security, school records): such check may be automated and ran constantly if a comprehensive population register is in place – Match birth/death records with records from census and surveys: identify un- registered recent births/deaths; also check accuracy of other information – Compare cause of death information with autopsy reports, hospital records. – Assess coder’s qualification: ask 2 coders to work on the same set of statistics report and compare – Select a sample of birth records and send a questionnaire to parents asking similar information as on the birth reporting form – Include a question in census asking whether recent births and deaths are registered

14. United Nations Expert Group Meeting on International Standards for Civil Registration and Vital Statistics Systems, 27-30 June 2011, New York Quality assessment studies – how? (cont.) Indirect methods: Monitoring births trends over a number of years; Monthly figure monitoring Intervals between date of occurrence and registration – timeliness Work with census data: – Population balance equation: P2 – P1 = B – D + net in-Migration – Comparing births/deaths in the last 12 months – Comparing rates for similar periods

15. United Nations Expert Group Meeting on International Standards for Civil Registration and Vital Statistics Systems, 27-30 June 2011, New York Quality assessment studies – how? (cont.) Overall evaluation studies: touching on all aspects of the two systems but not in-depth – WHO/UQ HIS-Hub framework – Eurostat – UNSD assessment studies for countries through regional workshops and questionnaires Often carried out on ad-hoc basis Could be self-assessment

16. United Nations Expert Group Meeting on International Standards for Civil Registration and Vital Statistics Systems, 27-30 June 2011, New York Points for discussion What kind of information we would like this chapter to cover? Are the evaluation methods (direct/indirect) up-to-date? Any new methods emerged in the past 10 years?