Unit 3 Siobhan Carey Department for International Development Making cross-national comparisons using micro data

Content Why it’s important to get it right Preparation Assessing the data Organising your analysis Presentation of results

Does it matter if it’s wrong? Don’t be famous for the wrong reasons –Importance of reproducibility –Be able to defend your analysis if challenged Two examples of why it’s important

International Adult Literacy Survey A survey run by very reputable organisations on literacy skills of adults How do you explain the differences? Various theories

….cont’d Methodological review found –differences in sampling methods –differences in fieldwork practices –deviation from the survey design A second review –translation –scoring of tests –…………

Programme for International Student Achievement (PISA) How do you explain the differences? –didn’t lead to any investigation –results accepted but lead to huge debate Was IALS worse that other surveys? Why did PISA not lead to the same questions?

Preparation Know the data Read the documentation before you do the analysis Have clearly defined research questions Which countries are you going to include –Small number and expand? –Large number and reduce? –Combine countries?

What do you need to consider Was the survey intended to support cross- national research from the outset? Is the survey design the same across all countries? In what areas do the countries differ in design or execution and are these important? e.g. – all school types included?

Understanding the data Comparability of overall design What questions were asked – who was asked Sample design and coverage Mode Survey response Data collection methods Translation Data processing and imputation Missing values Topic measurement – any special features (anthropometry, biochemical…)

Understanding the variables Which variables are derived Which are imputed –How have they been imputed –What other modifications have been made e.g. hours spent watching TV – a categorical variable modified into a continuous variable

Preparation Creation or transformation of variables –Recording, derived variables, dummy variables, reference group Reducing dataset to improve speed Document your analysis - always Develop good habits – in-flight v programme based - Naming conventions

Weights Design weights Population weights Compound weights Replicate weights Which you use will depend on what comparisons you are making

Analysis Rarely unique Use existing analyses to learn Check and double check – –Reproduce, check bases, check population, check text to tables, check tables to source…. Check significance

Analysis - output Does it pass the common sense test? Are the bases right? Are the right weights on? Is it plausible? What can you triangulate against? Is your analysis simply a reflection of distribution? What could be behind it? – e.g. poverty reduction and population growth

Analysis Horse race? –Gets media attention but … Correlates are more interesting –e.g. relationship between sexual behaviour and HIV prevalence Analysis needs to be set in context Take into account system factors – e.g. school characteristics, hierarchical features, standardisation

Presenting your analysis Borrow from others –OECD, World Bank, UN, Lancet … Try to keep it simple – –what’s the story –and why is it interesting

Rewards? Help understand social condition Help unpick complex relationships Contribute to evidence base so as to improve policy and outcomes Interesting – only if you’re curious!

Activity Using ESS – on a topic of interest –formulate a research question - which countries are of interest? –investigate some variables that are relevant –look at difference when weights applied –are there differences in distribution between countries? Why might that be? –what is the unit of analysis and which weights should you use? –Develop an analysis plan

Suggestions Crime and fear of crime Religiosity Attitudes to organs of the state Values – attitudes to cheating on tax –attitudes to sexual behaviour