Presentation on theme: "Census Microdata: findings and futures Manchester 1 – 3 September 2008 The future of microdata Professor Denise Lievesley UN African Centre for Statistics."— Presentation transcript:
Census Microdata: findings and futures Manchester 1 – 3 September 2008 The future of microdata Professor Denise Lievesley UN African Centre for Statistics Addis Ababa, Ethiopia and President, International Statistical Institute
Graduating from being data producers to generators of information and knowledge attention to data collection at expense of generation of information and knowledge –collection costly and difficult –importance of quality of data mountains of data – insufficiently processed and analysed most people not adept at understanding data –even more important for statisticians to get involved in interpretation and use of information
Partnerships for data use with –subject experts –data analysts –researchers in government, universities, private sector and civil society who can contribute to development of data production systems in countries this requires making census and survey data accessible to these stakeholders
The importance of international guidelines on ensuring data access The UN Principles and Recommendations for the 2010 Population and Housing Census urge countries to create census databases as part of the process of census data management. –in order to expand the life and usability of data, and as a complement to the standard production of tables, NSOs are encouraged to store census data in various computerized database forms including macro and micro- databases.
These recommendations give one of the main advantages of micro databases aspermitting the retrieval of data at least in principle, at any level of detail. Additional advantages: –to broaden data use and reuse; –to foster diversity and deepen the quality of data analysis thereby extracting more information from the data; –to add value to data by bringing subject- matter knowledge to data analysis; –to improve data quality (Data analysts can and often do detect errors in data and when they provide feedback to statistical agencies, this can lead to improvements in future data collection.)
According to the International Household Survey Network established as one of the six action points of the Marrakech Action Plan for Statistics, national and international micro databases should be established to: –promote the acquisition, documentation, dissemination and preservation of microdata essential for the production of national statistics, for research and for instruction in the social sciences, –promote the effective use of existing survey and census data, –ensure the continued viability and usability of microdata now and in the future, and, –promote equitable access to these data within the framework of the national statistical legislation.
Obstacles Despite these international guidelines and exhortations for data access, obstacles do exist and we must be sensitive to them. They include: –legal obstacles –technical and financial obstacles including in- house capacity to handle the complex aspects of micro-data dissemination such as data anonymization –political obstacles –psychological obstacles: the tendency to control access perhaps because of concerns over its mis-interpretation or because data is power
Legal constraints in many countries the statistical legislation in use is out dated and does not recognise the dissemination of electronic data particularly micro-data some legislation, or the interpretation of it, actually prevents such dissemination on account of confidentiality new legislation needed –the African Centre for Statistics is working with countries to help them to prepare both legislation and professional frameworks relevant to todays era of electronic information.
Data are collected using valuable resources – both financial but also the time of survey and census respondents – and are increasingly seen as being a vital part of democratic systems, since people are empowered though information. Official data are a public good: part of the enabling environment for national and international development, which all stakeholders in society should have access to and benefit from.
Conclusion - I It is the responsibility of statisticians to ensure that the widest possible use is made of data; consistent of course with the legal constraints and ethical undertakings.
Preservation is essential Having collected data at some cost to the taxpayer, it behoves official statisticians to manage them well. Alongside dissemination, this entails data preservation. Due to poor data management, human error as well as technical change and inadequate use of technology, many data sets including critical census data are no longer readable. Thus all that remains of this important legacy are the, often quite superficial, reports that were produced at the time. To this extent an important part of our heritage is lost and we will be severely limited in our analysis of change.
Long term preservation of electronic material is not a straightforward task, especially in resource-poor and technology-weak developing country statistical offices. It can be hard to persuade financial authorities to spend money on the preservation of data for historians and statisticians of the future, when there are so many pressing problems today.
Partnerships for data preservation To this end, partnership - for both technical work and advocacy – across the data archiving, data librarian, statistical and research communities is to be encouraged. Welcome the formation of new organisations such as the African Association of Statistical Data Archivists Value of international networks and support systems
Metadata It is necessary not only to preserve data but also to create and preserve metadata and contextual information. This is essential to ensure that the interpretation of the data will be informed. The documentation should include –data collection instruments and forms –instruction manuals –definitions and concepts –descriptions of scope and coverage and other aspects of quality –codebooks –basic tables –records of validation and post-enumeration checks
Documentation The documentation should be clear and easy to understand and should help users to: –identify and find the data they are interested in –understand what the data are measuring and how they have been created –assess the quality of data and fitness for their purpose International standards for metadata creation such as the DDI should be implemented
Conclusion - II The leadership of the national statistical agencies should ensure that census and survey macro- and micro-data are well documented and archived.
Case made for data dissemination and preservation. Data alone have no intrinsic value. –Their value is extrinsic: derived from the fact that they can be disseminated to those who need them, are understood and are used for a variety of purposes in a timely fashion. –So one way to justify the high cost of data production is to disseminate and mine them for information –especially for evidence-based policy and comparative analysis Data dissemination in a variety of ways (through reports, compilations of data, electronic extracts etc) is, therefore, important not only to complete the data cycle but also to enhance data relevance and usability. Data grow in value the more they are used, unlike most commodities which are diminished with use.
Evidence-based policy What is evidence-based policy? Need for use evidence at all stages in the policy cycle Why is evidence-based policy important? Evidence but one input into policy process What is evidence? Evidence initiatives Challenges for statisticians Examples from the developing world
What is evidence-based policy ? Helping people to make well-informed decisions about policies, programmes and projects, by putting the best available evidence from research at the heart of policy development and implementation Enlightening through making explicit what is known through scientific evidence and importantly what is not known Better statistics, better decisions, better outcomes Vision of the African Centre for Statistics
In contrast to opinion based policy which relies heavily on –either the selective use of information –or the untested views of individuals or groups often inspired by ideological standpoints, prejudices or speculative conjecture.
Need an evidence base at all stages in the policy cycle in shaping agendas, in defining issues, in identifying options, in making choices of action, in delivering them and in monitoring their impact and outcomes.
Context in the UK Modernising Government White Paper commits to policy making that is –strategic –outcome-focussed –joined up (works across organisational boundaries) –inclusive (is fair and takes account of the interests of all – gender sensitive, promotes equity) –flexible –innovative (means taking risks – and so risks must be identified, monitored and managed) –robust (works and continues to work) Tackles causes not symptoms. Is acceptable to the public.
Why is evidence-based policy important? Sherlock Holmes it is a capital mistake to theorise before you have all the evidence. It biases the judgement Policy makers may be well-intentioned but misguided
Results of a meta-analysis Collation of the results of many studies contradict this advice Extract from publicity prepared for the UK Reduce the Risk Campaign (early 1990s) The risk of cot death is reduced if babies are not put on the tummy to sleep. Place your baby on the back to sleep. ….Healthy babies placed on their backs are not more likely to choke.
Iain Chalmers No doubt like millions of his other readers, I passed on and acted on this apparently rational and authoritative advice. We now know that the advice promulgated so successfully in Spock's book led to thousands, if not tens of thousands, of avoidable cot deaths. (Letter to BMJ)
Judging what works Outcomes often defined too narrowly eg performance management whereby targets are often set top down and may have unintended consequences. –Hitting the target but missing the point –The good, the bad and the ugly report of the Royal Statistical Society –Difficulty is that sometimes different things work in different circumstances and this is interpreted as nothing works. (Ref Robert Martinsons 1974 review of American offender rehabilitation programmes- He concluded that there was no single approach which worked consistently – different things worked in different circumstances but this was misread as nothing works.)
The policy making process Policy making is the process by which governments translate their political vision into programmes and actions to deliver desired changes in the real world. Evidence but one input into policy process Ideology Lobbies Values/ beliefs Tradition Self interest Judgement Campaign promises Expert views Experience Resource constraints Acceptability to public
There is nothing a government hates more than to be well- informed: for it makes the process of arriving at decisions much more complicated and difficult. John Maynard Keynes
What is evidence? Expert knowledge; published research; existing statistics; stakeholder consultations; previous policy evaluations; the Internet; outcomes from consultations; costings of policy output from economic and statistical modelling. UK Cabinet Office 1999 Scientific, rigorous, critically appraised, well documented Fit for purpose Beware policy-based evidence? Unbiased (note problems of publication bias) But data are never value-free
All scientific evidence is imperfect. The absence of excellent evidence does not make evidence-based decision making impossible: what is required is the best evidence available not the best evidence possible Muir Gray 1997 Evidence rarely provides neat and tidy prescriptions to decision makers as to what they should do. Often it generates more questions to be resolved Petrosino et al 2001
Challenge : is it essential that evidence is based on experiments? Adrian Smith Mad cows and ecstasy : choice or chance in evidence-based society Experiments used too little in social and economic research Rich data from different sources Science is often misrepresented as the body of knowledge acquired by performing replicated controlled experiments – it is much broader – the acquisition of reliable knowledge about the world Jared Diamond Collapse
Challenges for analysts increase understanding of the policy process, where and how evidence can feed into it (eg SPATS) improve interpretation and communication of data particularly in relation to uncertainty speak the language of policy makers forge strong relationships of official statisticians with policy analysts, increase number of policy analysts combine humility and confidence improve training for analysts –(International Research Forum on statistical reasoning, thinking and literacy )
Particular challenges with respect to timing demand for quick fixes means that policy makers often do not have time for in-depth research –Demands by Ministers are short term, quick fix solutions rather than major carefully considered strategic initiatives ( Bullock et al) collection of quality data takes time – hence the existence of public use samples can accelerate research to be effective the research strategy needs to look beyond the timescales of one government many issues are chronic – beware unwarranted impatience (Boruch)
Evidence compilations – example initiatives Micro-data based on samples from Censuses Data archives Cochrane collaboration Campbell collaboration National Library of Health Communities of practice
European social survey CROP - the Comparative Research Programme on Poverty whose major aim is to produce sound and reliable knowledge, which can serve as a basis for poverty reduction RENCORE - encourage and enhance comparative empirical research of individual, national and institutional level data from the states of western, central and eastern Europe Cleveland conference on education research African Programme on Rethinking Development Economics
Evidence base urgently needed in developing countries Problems are severe and urgent Need ownership by countries… and to empower them Develop policies which are relevant to their needs… and ensure effective implementation Need to counteract corruption Paucity of data is a major obstacle Lack of recognition /acceptance of much data
Joint Marrakech Memorandum Affirm a commitment to fostering a global partnership on managing for development results. Awareness is growing that getting better development results requires management systems and capacities that put results at the center of planning, implementation and evaluation. We need to align cooperation programs with desired country results, define the expected contribution of our support to country outcomes and rely on and strengthen countries monitoring and evaluation systems to track progress and assess outcomes … better distil the lessons of countries experiences and disseminate knowledge about what gets results in different country contexts.
C.Scott for Paris 21 Use of good statistics having a positive effect on policy Uganda –Poor public service delivery caused by governments failure to ensure that budgeted funds reached front line agencies Brazil and Mexico –Tackled child poverty and education by a programme to give child benefits to mothers according to the attendance of their children in school
C.Scott for Paris 21 Absence of data or failure to use available information has negative effect on policy making Malawi –Data from Save the Children on child malnutrition disregarded by government because of conflict with crop data Botswana –Deficiencies in data on HIV/AIDS (from sentinel surveillance systems) legitimised the rejection of the message on the scale of the problem
Current challenge in policy development in poor countries Development literature of the 1990s dominated by the view that growth is central to any strategy aimed at poverty reduction. Since modified: not growth per se but structure of growth that matters. Further recognised that income inequality matters when making progress on poverty reduction. Example Nigeria –Aigbokhan asks why the rate of poverty is so high in Nigeria despite the strong growth performance and concludes it is because of a lack of explicit concern with inequality in public policy. He analyses data on Nigeria and on other countries for comparative purposes to argue that if growth occurs in the sectors that require skills that the poor of Nigeria do not possess although the economy grows it has little impact on poverty reduction. –Essential that access is provided to researchers which allows them to examine the distribution of resources within countries
The gross national product does not allow for the health of our children, the quality of their education, or the joy of their play; it does not include the beauty of our poetry or the strength of our marriages, the intelligence of our public debate or the integrity of our public officials. It measures neither our wit nor our courage, neither our wisdom nor our learning, neither our compassion nor our devotion to our country, it measures everything in short except that which makes life worthwhile. Robert Kennedy
Comparative research – building of cross-national resources Knowledge of the self is gained through knowledge of others Tension between the value and importance of cross-national data and its fragility To what extent are differences artefacts of the measurement ?
Benefits of cross-national research Contacts, enabling researchers to capitalise on their experience and knowledge of different intellectual traditions and to compare and evaluate a variety of conceptual approaches. Comparisons can lead to fresh, exciting insights and a deeper understanding of issues that are of central concern in different countries - the identification of gaps in knowledge and sharpen the focus of analysis of the subject under study by suggesting new perspectives. Cross-national projects give researchers a means of confronting findings in an attempt to identify and illuminate similarities and differences, not only in the observed characteristics of particular institutions, systems or practices, but also in the search for possible explanations in terms of national likeness and unlikeness. Cross-national comparativists are forced to attempt to adopt a different cultural perspective, to learn to understand the thought processes of another culture and to see it from the native's viewpoint, while also reconsidering their own country from the perspective of a skilled, external observer.
Cross-national data is a special case of comparative research Spatially comparative data –within countries –between countries Collecting data over time (the past is a foreign country ) Many parallel issues in spatially and time related comparative data such as achieving a balance between specificity versus generalisability, and coping with differences/changes.
Purposes of cross-national data To learn from one another (contrast and similarity) For purposes of national accountability (the indicator movement) To build a greater global understanding through comparison and through multiple instances of the same phenomena (Jared Diamond) To aggregate across national boundaries for a regional or global picture – for advocacy – for resource mobilisation To accelerate progress through sharing resources To make research more credible/ defensible recognising that research which displeases is attacked rather than accepted To distance the research from the political process (tension – policy relevance v. autonomy)
Challenges to comparability Language Culture Social systems and structure Administrative systems Ideology and politics Economics and resources History Context – events Different methodologies and types of methodological expertise (often deeply ingrained) Different research climate
Over-interpretation Jowell Good science should not turn a blind eye to known imperfections – nor should these be concealed from users –Concerns about lower standards being applied for cross-national studies than for national studies Goldstein (Surprising naivety) cross- national educational studies –Concerns about validity of cross-national measures, inappropriate conclusions about causation, over-simplification of complex relationships –Overall Goldstein argues for celebrating diversity and encouraging transparency
The UN model Attempting to achieve comparable data across the world Assumptions about universality of aims The survey hammer is owned by richer countries Under-resourced at UN and at country level Countries with resources play a bigger role (city groups) Fragmentation within the UN decentralised system (regionally and by substantive area) Concentrates on facts and often assumes that comparison is unproblematic Exhorted to Think globally, act locally
Problems with the UN model One size fits all –Difficulty of accommodating national differences –Cutting edge and trailing tail countries needs are very different –Context is critical but is often ignored The nation is too often the unit of analysis – variation within countries ignored Need to encourage country ownership and recognition of their data Accountability of countries v. their autonomy
Combining administrative with survey and census data Adds value to data Provides context Enables better understanding Improves communication – tells a better story Excites and interests users Engages the media
Summary Case made for data preservation and dissemination Data resources are critical for evidence based policy making and for cross-national, comparative research Welcome this meeting where we will discuss the value of this work, the challenges which face us and ways in which we can redouble our efforts to build these resources