1. Microdata Management Toolkit Tools to facilitate archive and dissemination of surveys Pascal Heus pascal.heus@gmail.com International Household Survey Network http://www.surveynetwork.org Session C3 - Wednesday, May 24 Effective Strategies for Metadata Management

2. Background International Household Survey NetworkInternational Household Survey Network –Marrakech Action Plan for Statistics (MAPS) (Feb 2004) –Established in September 2004 –International organizations actively sponsoring household surveys + NSO –Coordination: ILO, DFID, Paris 21, UNICEF, UNSD, WHO, World Bank –http://www.surveynetwork.org http://www.surveynetwork.org Survey under-use: limited access, poor return on investment limited impact, difficulties in policy makingSurvey under-use: limited access, poor return on investment limited impact, difficulties in policy making Common obstacles: quality, technical capacity, legal/political issuesCommon obstacles: quality, technical capacity, legal/political issues Common problems: Accessibility, Timeliness, Coherence, Lack of metadata / documentation / data, poorly organized archivesCommon problems: Accessibility, Timeliness, Coherence, Lack of metadata / documentation / data, poorly organized archives Need for new tools and guidelines  Microdata Management ToolkitNeed for new tools and guidelines  Microdata Management Toolkit

3. Toolkit Requirements User friendly software for microdataUser friendly software for microdata Facilitate metadata exchange (DDI, Dublin Core)Facilitate metadata exchange (DDI, Dublin Core) Facilitate archiving (metadata and data, quality control)Facilitate archiving (metadata and data, quality control) Facilitate preservation/dissemination: network, CD / DVD, web sitesFacilitate preservation/dissemination: network, CD / DVD, web sites Works with common data formatsWorks with common data formats Multilingual supportMultilingual support Free or InexpensiveFree or Inexpensive Availability of technical support and trainingAvailability of technical support and training Supported by national, international and research communitiesSupported by national, international and research communities

4. Generate HTML based CD-ROM Import metadata and prepare CD-ROM Import data and compile metadata The Toolkit Process 1 2 3

5. Toolkit Components Archiving: Metadata Editor (World Bank / Nesstar Ltd.)Archiving: Metadata Editor (World Bank / Nesstar Ltd.) –To compile survey data, documentation and metadata in a standard format (Nesstar/DDI). Free data reader for users. –Built on Nesstar Publisher Dissemination: CD Builder (World Bank / Mark Diggory)Dissemination: CD Builder (World Bank / Mark Diggory) –To facilitate the publication of survey data, documentation and metadata on CD-ROM and on the web (transforms DDI into HTML based navigation) –Based on Eclipse Platform, open source Nesstar ExplorerNesstar Explorer –Free software –Access metadata & data and export to common statistical formats IHSN ToolsIHSN Tools –Reporting, diagnostics and other utilities

6. What is the Metadata Editor? DDI / Dublin Core specialized editorDDI / Dublin Core specialized editor Template drivenTemplate driven Enhanced version of the Nesstar PublisherEnhanced version of the Nesstar Publisher Import/Export common data filesImport/Export common data files Integrated interface, multilingual supportIntegrated interface, multilingual support Metadata and data in single fileMetadata and data in single file Export to DDI / DCExport to DDI / DC Licensing agreement for developing countries and IHSN membersLicensing agreement for developing countries and IHSN members

7. What is the Metadata Editor?

8. What is the CD Builder? Publish survey metadata, documents and data on a CD-Rom or web site)Publish survey metadata, documents and data on a CD-Rom or web site) Transforms DDI into an HTML based interfaceTransforms DDI into an HTML based interface User can customize the layout (branding) and content of the CD (single or multi- surveys)User can customize the layout (branding) and content of the CD (single or multi- surveys) Open source applicationOpen source application Build on the Eclipse FrameworkBuild on the Eclipse Framework Based on DDI / Dublin CoreBased on DDI / Dublin Core Integrates with Metadata EditorIntegrates with Metadata Editor

9. CD Builder Process Create new CD-ROM Project Add a survey to the project and select its type and branding 1 2 Selecting a survey consist in opening the DDI-XML or Nesstar file The survey “branding” determines the overall look and feel of the CD The survey “type” determines the default metadata content Selecting a survey consist in opening the DDI-XML or Nesstar file The survey “branding” determines the overall look and feel of the CD The survey “type” determines the default metadata content Click the “Save” button to generate the HTML interface 3 After a few minutes, your CD Project is ready for publishing! 4

10. Sample output

11. Nesstar Explorer Free softwareFree software PDF philosophyPDF philosophy Access to survey metadataAccess to survey metadata Access to data (no need for specialized software)Access to data (no need for specialized software) Export to common formatsExport to common formats Single file holds data and metadataSingle file holds data and metadata

12. IHSN Tools Free / Open SourceFree / Open Source Utilities to complement existing softwareUtilities to complement existing software Integrates in EditorIntegrates in Editor DiagnosticsDiagnostics ReportingReporting Metadata sharingMetadata sharing

13. Benefits and Users Improves and facilitates documentation preservation, cataloguing, metadata exchange, dissemination, quality, etc.Improves and facilitates documentation preservation, cataloguing, metadata exchange, dissemination, quality, etc. Users: survey producers, survey sponsors, data archivesUsers: survey producers, survey sponsors, data archives Benefits: data producers, national & international survey sponsors, survey data repositories, data analysts policy makers, DDI CommunityBenefits: data producers, national & international survey sponsors, survey data repositories, data analysts policy makers, DDI Community

14. Status & Availability Metadata Editor:Metadata Editor: –Available in English, Commercial release in Nesstar 4.0 (end 2006?) CD BuilderCD Builder –Available in English & French IHSN ToolsIHSN Tools –Diagnostics reports available, Survey Documentation in PDF OtherOther –I18n: working on French, Spanish, Bosnian, Russian, Portuguese,… –User’s Guide in English –Open source project components hosted on SourceForge at http://www.sourceforge.net/projects/toolkit

15. Roll-out Roll-out in in 2006/2007 (global)Roll-out in in 2006/2007 (global) Completed training / pilot in about 20 countries, mainly in the Africa regionCompleted training / pilot in about 20 countries, mainly in the Africa region Expected use by UNICEF for next round of Multiple Indicators Cluster Surveys (MICS, 55 countries)Expected use by UNICEF for next round of Multiple Indicators Cluster Surveys (MICS, 55 countries) Asia: Partnership with United Nations Economic and Social Commission for Asia and the Pacific (ESCAP)Asia: Partnership with United Nations Economic and Social Commission for Asia and the Pacific (ESCAP) Latin America: partnership with Inter-American Development Bank (IADB) and Pan-American health Organization (PAHO)Latin America: partnership with Inter-American Development Bank (IADB) and Pan-American health Organization (PAHO) Used by IHSN member agencies (WHO, ILO, etc.)Used by IHSN member agencies (WHO, ILO, etc.)

16. Surveys and censuses in low-income countries in Africa Health survey in last 5 years Coverage: 82 % of countries 81 % of population Poverty survey in last 5 years Coverage: 56 % of countries 62 % of population Available Not conducted /unavailable Non-IDA country Pop. census in last 10 years Coverage: 67 % of countries 53 % of population 99 % in Europe, 95% in Asia, only 53% in Africa

17. Accelerated Data Program Growing demand for data for MDG, PRSP, IDAGrowing demand for data for MDG, PRSP, IDA Must improve timing, sequencing and frequency of surveys and censusesMust improve timing, sequencing and frequency of surveys and censuses More surveys is not enough, must address reliability, comparability, relevance & useMore surveys is not enough, must address reliability, comparability, relevance & use Activities:Activities: –Inventory and dissemination of existing data –Analysis of existing data –Assess past and on-going survey program; review and improve survey methods –Design and implement improved medium-term survey program (within NSDS)

18. Accelerated Data Program ImplementationImplementation –A MAPS activity (with Advisory Board) –Six pilot countries in Sub-Saharan Africa (12 pilot countries total) –Development Grant Facility (DGF) ($2m per year) –Likely to be extended to all African countries –3 to 5 years (from 2006) –Cooperation with other donors –Coordination with IHSN Customized to countries needs/capacityCustomized to countries needs/capacity

19. Next? Continue roll-out programContinue roll-out program –Packaging and Users Manual –Regional and national trainings, training of trainers –Country level capacity building, Accelerated Data Program Translations and support for additional languagesTranslations and support for additional languages Maintenance, bug fixes and new featuresMaintenance, bug fixes and new features Open source light web based cataloguing toolOpen source light web based cataloguing tool Guidelines for archiving, preservation and dissemination in developing countriesGuidelines for archiving, preservation and dissemination in developing countries Platform harmonization (Eclipse, XForms, DDI 3.0, XML DB)Platform harmonization (Eclipse, XForms, DDI 3.0, XML DB)

20. Thank you! QUESTION / ANSWER