1. Multiple Indicator Cluster Surveys Survey Design Workshop
MICS Evaluations and
Lessons Learned from MICS4
MICS Survey Design Workshop

2. Part 1: MICS Evaluations

3. MICS Evaluations MICS1 Evaluation MICS2 – No evaluation
MICS3 Evaluation – John Snow Inc
Comparable quality with DHS and other survey programs
Fulfills important mission in global monitoring
Mismatch between where technical expertise is (HQ) and where technical decisions are taken (Country), communication problems
Short-cuts are being taken in training, listing, fieldwork
Limited human resources an impediment

4. MICS Evaluations MICS4 Evaluation, Cluster 1 and Cluster 2
Cluster 1 completed

5. MICS4 Evaluation - Findings
Significant expansion of the envelope of technical support resources:
Regional coordinators, support by experts, UNICEF MICS Consultants, more structured process of technical support and quality assurance
Organizational structure, communication channels, decision-making authorities remain unchanged – suboptimal for the objectives. E.g. CO not complying with guidelines, quality assurance processes (large samples, additional questions)
Not on the agenda of senior managers at HQ or RO levels

6. MICS4 Evaluation - Findings
Universal adherence to training guidelines (duration)
No evidence of interviews or spot-checks
Field check tables an important tool, inconsistent use
Large sample sizes, large survey teams greater than recommended, manageable levels
Shorter time for production of final reports

7. MICS4 Evaluation - Findings
Dramatic improvement in data quality
MICS4 and DHS have comparable quality on most indicators
Quality of some MICS data need improvement

8. MICS4 Evaluation - Recommendations
CO to be compelled to hire UMCs
Increase regional consultant pool
Fully integrate technical review and quality assurance processes into key documents
When MICS reports are lagging, additional implementing agency or consultant to finalize report – include in MoU
UNICEF should invest more into other data collection efforts, without hampering MICS or other household surveys, for lower administrative level data generation

9. MICS4 Evaluation - Recommendations
Additional data processing staff needed
Strengthen use of field check tables
Increase guidance to Ros to gauge risks in advance of MoUs, and for course-correction and withdrawal from global MICS program
Do’s and don’t’s for CO and RO managers
Tools to be developed to ensure consistency of the work of regional consultants
Documentation for sample design and implementation

10. MICS4 Evaluation - Recommendations
Spot checks and observations
Measurements for further improvement of anthropometric data quality
Better documentation of Global MICS Consultations
Regional coordinator turn-over – overlaps needed

11. Part 2: Data Quality

12. Looking at data quality – Why?
Confidence in survey results
Identify limitations in results
Inform dissemination and policy formulation
All surveys are subject to errors

13. Data quality
Two types of errors in surveys
Sampling errors
Non-sampling errors: All other types of errors, due to any stage of the survey process other than the sample design
All survey stages are interconnected and play roles in non-sampling errors

14. Data quality
Sampling errors can be envisaged before data collection, and measured after data collection
More difficult to control and/or identify non-sampling errors

15. Data quality
We have discussed several features/recommendations for quality assurance to minimize non-sampling errors
Failure to comply with principles behind these recommendations leads to problems in data quality

16. Data quality analyses Looking at
Departures from recommended procedures/protocols
Internal consistency
Completeness

17. Before we begin

18. Monitoring Priorities Standard Survey Instruments
Countries, UNICEF, Interagency Groups, Partners in Development
Monitoring Priorities
Goals And Targets
Indicators
Operationalization
Validation, Testing, Piloting, National Surveys
Standard Survey Instruments
Questionnaires,
Data Processing Tools, Sampling Considerations, Analysis Plans, Dissemination Strategies

19. Monitoring Priorities
Goals And Targets
Indicators
Major source of poor data quality
Operationalization
Non-validated, untested Survey Instruments
Standard Survey Instruments

20. Completion, Age, Completeness

21. Household Completion Rates
Completed / Selected

22. Household Response Rates

23. Women’s Response Rates

24. Under-5 Response Rates

25. Age Distributions

26. Age Distributions

27. Age Distributions

28. Age Distributions

29. Age Distributions

30. Age Distributions

31. Age Distributions

32. Age Distributions

33. Women – Complete Birth Date

34. Under-5s – Complete Birth Date

35. Observations Selection
MICS Protocols
Observations
Selection

36. Selection for Child Discipline

37. Refusals to Observe Place for Handwashing

38. Observing Documents

39. Out-transference Omission
Serious Business
Out-transference
Omission

40. Years Since Last Birth

41. Years Since Last Birth

42. Years Since Last Birth

43. Years Since Last Birth

44. Ratio of children age 2 to age 1

45. Ratio of Population Age 2-3 to Age 0-1

46. Ratio of Population Age 5 to 4

47. Out-transference from age 15

48. Out-transference from age 15 (non-MICS)

49. Out-transference from age 15

50. Out-transference from age 15

51. Out-transference from age 15

52. Out-transference from age 15

53. Out-transference from age 15

54. Out-Transference/Omission

55. Easy Target
Anthropometry

56. Anthropometry Digit Heaping

57. Digit Recorded for Weight

58. Digit Recorded for Height

59. Digit Recorded for Weight

60. Digit Recorded for Height

61. Digit Recorded for Height

62. Digit Recorded for Height

63. Digit Recorded for Height

73. MENA Surveys

74. MENA Surveys

75. Operations

77. Summary Comply with the principles behind standard protocols
Think of the implications of each action on other stage of implementation, and data quality
Always check for errors/issues
“Understand” your data
Be transparent and report on problems
..before others detect them