1. Government Social Research Unit Quality and purpose: issues and approaches Teresa Williams Head, Government Social Research Unit Research Methods Festival 2008 2 July 2008

2. www.gsr.gov.uk Why does quality matter? Quality Credibility Opinion ‘Evidence-based opinion’ ‘Stakeholder opinion’ ‘Person on the street’s view’ ‘Good qualitative research’ ‘Good quantitative research’ ‘Good RCT’ ‘Good systematic review’ ‘Trusted source’ view Slow or inconclusive research Adapted from J. A. Muir Gray Evidence-based healthcare, 1997, Bell and Bain Ltd, Glasgow.

3. www.gsr.gov.uk Who does it matter to? Research community – badge of professionalism, common language, sense of identity Research funders – value for money, continued investment Research users – confidence in the results, belief that they are relevant Research respondents – ethical considerations, cooperation

4. www.gsr.gov.uk Dimensions of quality…. Rigour – scientific method Respect – ethical frameworks But also Relevance –Range of stakeholders –Different stages in policy/implementation process –Tomorrow’s questions as well as today’s

5. www.gsr.gov.uk Frameworks for quality Principles setting out broad expectations Specific standards (can be independently assessed) on what ‘best’ means in different circumstances Clear processes to ensure the best design is identified and executed

6. www.gsr.gov.uk GSR Code: principles for quality and relevance in govt What is it? –Addendum to civil service code –7 principles under 2 key themes (People and Products –Specific guidance and resources What’s it for? –Cementing our identity –Communicating our standards –Assessing how well we’re doing

7. www.gsr.gov.uk

9. STANDARDS - Research Reviews

10. www.gsr.gov.uk STANDARDS: Survey Methods StandardDescription Level 5Representative random sample of sufficient size to provide robust estimates for main measures of interest including most or all sub-groups. Good response rate (average or above for similar studies), well developed and tested data collection instruments and competent execution. Level 4Representative random sample of sufficient size to provide robust estimates for whole sample and some sub-groups or interaction effects. Minor weaknesses on other aspects such as question wording. Response rate around average for similar studies. Level 3Random sample size where response rate is too low to provide robust estimates for the whole sample for the main questions (less than 50%). Some weightings to improve representativeness. Weaknesses on question wording or other aspects. Level 2Quota (non-random) sample, in which quotas are set for numbers of individuals in each of the sub-classes. Representative on some key attributes (such as sex, age, or ethnicity) but cannot be fully representative. Or random sample obtaining a very low response rate (30% or less) or failure to take account of differential response rates (e.g. by weighting data). Level 1Not random sample. No effort to ensure representativeness, or, where complete, poor reporting of sampling strategy so unable to determine likely representativeness. Alternatively, no sampling frame used (i.e. the list of individuals in the population from which the sample was selected), or serious bias in the sampling frame or sampling procedure. Critical weaknesses on other aspects, e.g. the true level of response are unknown.

11. www.gsr.gov.uk Processes Main tools –Publication: Inputs (research objectives, design protocols etc) as well as outputs (reports) to be in public domain for general scrutiny –peer review: ‘independent’ and ‘expert’ scrutiny to help inform decisions Not a one-off event. Good QA should be applied throughout project life cycle –Project design –Procurement –Project Delivery –Publication –Post project review

12. www.gsr.gov.uk Some key issues Building in and weighting non-methods aspects of quality? –Appropriateness of research questions –Ethical considerations –Research relevance/impact –Time/cost considerations Can we work towards minimum standards? Who should (and indeed can) judge quality –What/who counts as ‘independent’ –Methodological vs subject area expertise –Role of practitioners/end users as well as research experts Quality assurance processes as tools for assisting decision-making rather than a bureaucratic process

13. www.gsr.gov.uk Format for session Part 1: Methodological quality Jane Lewis (qualitative methods) Angela Dale (quantitative methods) Part 2: Quality from the User perspective Sandra Nutley (how knowledge is used) Juliet Mountford (capacity building in the Third Sector) Discussion – led by Ceridwen Roberts