European Conference on Quality in Survey Statistics

Len Cook, New ZealandLegislation and quality in official statistics Cardiff, April 2006 Legislation, Codes of Practice and the Quality of Official Statistics Len Cook New Zealand

The role for society of the statistician "The statistician keeps his fingers on the pulse of humanity, and gives the necessary warning when things are not as they should be." Quetelet

Official statistics Research Theory Policy Framework Anecdote Myth Stereotype Case studies Experience ? The place of official statistics in timely and decisive action

How much is confidence in official statistics in a state of change? Increased expectations Changing World Greater use Privacy Quality demands Technology Accessibility Advances in methods and practice

Prospects and trade-offs for the Global statistical system Delivery Performance Sense of Place of Official Statistics Trade-offs & risks in reputation & quality Competitive & influential Being able to be decisive and timely in actions Prospects Trade-offs

Summary of key elements of statistical legislation Unlikely to be in legislation Strengthened by legislation Usually in statistical legislation Quality standardsRoles of Head of National Statistical Institute, and responsible Minister Authority to obtain individual information RelevanceAccessibilityProtection of individual responses from disclosure ResponsivenessManaging respondent load Objectivity of methods Government/ community balance Co-ordination across government (survey approval) Impartiality of results, and selection of measures EfficiencyDefinition of official statistics Capacity to link records

Responsiveness of statistics and the nature of the professional/political boundary x comprehensive statistical legislation x international standards x other legislation x benchmarking, reviews x service agreement (key users) x structured user engagement/interaction x performance targets x quality framework RESPONSIVENESS PROFESSIONAL POLITICAL x manage obsolescence (resources) x political oversight x community partnerships (ethnicity) x political consensus x codes of practice

The diminishing capacity to make quality assessments x comprehensive statistical legislation x international standards x other legislation x benchmarking, reviews x service agreement (key users) x structured user engagement/interaction x performance targets x quality framework PROFESSIONAL POLITICAL x manage obsolescence (resources) x political oversight x community partnerships (ethnicity) x political consensus x codes of practice QUALITY MEASURES QUALITY INFORMATION

The gap between capacity to measure quality and expectations x comprehensive statistical legislation x international standards x other legislation x benchmarking, reviews x service agreement (key users) x structured user engagement/interaction x performance targets x quality framework PROFESSIONAL POLITICAL x manage obsolescence (resources) x political oversight x community partnerships (ethnicity) x political consensus x codes of practice QUALITY MEASURES Knowledge gaps USER EXPECTATIONS OF QUALITY QUALITY INFORMATION

The growing gap between expectations and quality measures PROFESSIONAL POLITICAL QUALITY MEASURES QUALITY INFORMATION Knowledge gaps USER EXPECTATIONS OF QUALITY

Statistics released without being undermined Objective selection of measures Access to administrative records Increased record linkage Strong survey response rates Continually robust survey frames Widen unit record access for research Expand small area statistics Reduce risk of accidental disclosure Legislation and quality x international standards x other legislation x benchmarking, reviews x service agreement (key users) x structured user engagement/interaction x performance targets x quality framework PROFESSIONAL POLITICAL x manage obsolescence (resources) x political oversight x community partnerships (ethnicity) x political consensus x codes of practice x comprehensive statistical legislation Authority to act Disclosure protection Objectivity of methods Impartiality of results {

Parliament is the place to recognise when ministerial authority has been transferred Ministers are elected to Parliament to make decisions for their peoples. If the authority to act in one sphere of responsibility, official statistics, is to be constrained, it should be recognised by Parliament, in the legislation of Parliament. Without this recognition, deciding what accountability will remain with Ministers may lack consensus, clarity, or consistency. Often statistical legislation will give a clear delineation in the responsibilities of the country’s chief statistician and the respective minister in the government responsible for it. This explicit relationship can be taken as that which applies to all Ministers and to all official statisticians. Comprehensive statistical legislation reduces expectation that each statistical decision or statistical event can be managed to have a good political outcome.

The place of legislation in the protection of confidentiality Obligation International security Protecting archives Data-matching and increased register use Challenging perceptions Quality of protection Increased capacity to access records from –Citizens (high response rates) –Business (tax records and market sensitive information) –Government (administrative records) Capacity to protect (court and police requests) Limit power of situational ethics Limit spread of less relevant oversight

The place of legislation in ensuring the objectivity of methods Selection of methodology by statistician Highly political interventions should be by legislation or judicial process Fitness for use in unknown situations outside design criteria International coherence Open peer review of practice Access to unit record data Immediacy of linkage to politically sensitive outcomes places a greater constraint on the reflective, consultative processes of the statistician Need high standing of role of National Statistician High rate of obsolescence of methods and conceptual reorientation External drivers grown tax liability, benefit eligibility, public funding, EU comparability, international quota Growth in international frameworks, Demand for cohesiveness across independently managed statistical sources Emphasis on residuals and balances from large gross flows Performance targets selected by Ministers

The place of legislation in impartiality of results and practices publish information about quality without approval independent publication of compendia on any topic have sufficient finance for assessing quality judgement on early access and immediacy of policy statements choice of comparisons obligation of consistency respondent relevant description and concepts Indigenous recognition, language use No intervention in senior appointments Limit damage caused by early access Limit juxtaposition of statistical release and policy reaction Independence of selection of comparisons

The place of legislation in impartiality of results and practices (contd) Release of information about the quality of results can affect confidence in policies based on them Timing of release and quality standards brings tensions that can result in belief in political intervention Need to publish methods and available evidence used in judgements even if confidential for other reasons Reliance on quality not being a sensitive concern

Can legislation advance quality, given that it is a matter of culture and method? Quality focus in statistical offices sometimes drives major strategies - statistical modernisation (UK) But mostly where staff have a quality focus they make sure that: Innovation often results from quality failure Value added by users does not damage the contributing products Avoidable limitations to quality are fixed

Innovation often results from quality failure Unanticipated revisions Delay in applying benchmark series revisions to rebased series User enquiries cannot be answered Users find errors Users get inconsistent answers on different occasions Derived measures use sources inconsistently Users find inconsistencies across series that require revisions Users fail to receive statistics as ordered Instability and inconsistency from results of related series from different sources Known policy applications delayed for statistical faults

Value added by users may reduce trust in NSI Users place statistical results into standard databases Users add metadata to series Users add to coherence with related series Users add explanations of series volatility Users need to check unusual results

Avoidable limitations to quality get fixed Inadequate range of quality measures Expected revisions delayed Insufficient length of series UK or only some of England, Wales and NI Inconsistent results across related series from different sources Inexplicable delays in getting results

What is essential alongside statistical legislation? Strong formal user involvement Rich publishing culture High degree of analysis, including complex derived statistics such as productivity, poverty and fiscal incidence Access to unit record datasets Large number of processes for providing transparency Regular publication of measures of cohesiveness/ confrontation across statistical sources Strong peer review processes Regular cross country benchmarking activity High degree of linkages across the collection and integrating frameworks produced by the system (Nat accounts, demography) Parliamentary ownership of system Strong methodology teams Engagement in cross country developments of statistical frameworks and standards International exchange of staff from both professional and operational areas of statistics

Codes of Practice and quality frameworks We may be overstating our capacity to measure quality We may not have sufficient resources to systematically assess quality through a comprehensive quality framework Hard to measure statistics that are very important may never be assessed We could give more attention to assessing the measurability of key phenomena, where measures greatly influence public life We may be undervaluing our judgements about quality in difficult uses of statistics by giving emphasis to tangible quality measures

Conclusion Legislation plays a key role in quality, but cannot substitute for user engagement, transparency and judgements on measurability. Legislation affirms the fundamental obligations on the statistician, and the authority to obtain information that that is central to the job The coming decades will see great change in our capacity to retain confidence in the quality of our work. We will give more confidence through the certainty of our processes The accessibility of statistical sources will be the most significant source of innovation, once technological transformations have been achieved

European Conference on Quality in Survey Statistics