1. Effective National Statistical System Assessments: Heading Towards Open Frameworks
Results of NSS Assessment Tools Study
Mary Strode – Independent Consultant
March 2017 – 49th UN Statistical Commission

2. Questions on NSS Assessments: raised by Study
What are we measuring?
Why are we measuring it?
What are we going to do with the results?
How can we develop a more harmonised approach to assessing National Statistical Systems in the light of new developments in data?

3. Early tools for NSS assessment
SCI/Statistical Performance Indicator
Data Quality Assessment Framework
DQAF
Indicators of Statistical Capacity
World Bank – Time Series – 10 years – 146 countries
IMF Data Quality Tool
PARIS21 Benchmarks Levels of DQAF

4. There are many more tools to chose from… 15 + and rising
TASC
US Bureau
STEP
PARIS21
ASDI
UNECA
Member Peer Reviews
OECD /EU
IdCA
FAO
SNAPSHOT EU
GLOBAL ASSESSMENT
ESS/EFTA
ODIN
Open Data Watch
Data4SDGs Toolbox
Partnership for SDGs

5. What are they measuring?
They measure a variety of issues… some several in one tool
Data quality
Outputs, indicators, openness, timeliness and periodicity
Adherence to standards and principles
Prerequisites of quality (laws, resources, structure etc.)
Compliance with Codes of Practice
Monitor Progress of statistical improvement
Advocacy – for statistical support
Statistical strategies
Project design
SDG coverage
Do they measure statistical capacity?
Why do we want to measure it? Impact of TA, design appropriate TA, ability to support Development Goals, trust in the resulting statistics

6. Four Types of Tool Identified
1. Tools used national authorities to inform statistical national planning and strategic processes
2. Tools for partners who want to ‘invest’ in statistics to inform project design and to discuss with governments ( includes data quality)
3. Tools to assess statistical performance internationally – Global public goods
4. Assess compliance with codes of practice (e.g. European Statistics Code Of Practice)

7. Tools classified by type
P21 Peer Review
NSDS tools
IdCA
Snapshot
TASC
SCI/SPI
STEP
ODIN
ASDI
OECD Peer Review
Global Assessment 1.
National Planning 2.
Project Design 3.
Global Monitoring 4.
Compliance

8. What do we mean by capacity? Statistics in the Development Context
CAPACITY DEFINITIONS
“ability of people, organizations and society as a whole to manage their affairs successfully”. OECD.
An essential ingredient in the UNDP capacity development approach is transformation… it must bring about transformation that is generated and sustained over time from within. Transformation of this kind goes beyond performing tasks;
If something does not lead to change that is generated, guided and sustained by those whom it is meant to benefit, then it cannot be said to have enhanced capacity, even if it has served a valid development purpose.
STATISTICAL CAPACITY
Statistical Capacity is a nation's ability to collect, analyze, and disseminate high-quality data about its population and economy. WBI
PARIS21 Capacity 4.0… to manage the Data Revolution
Surprisingly little guidance is offered on this topic. UN Inspectors evaluating capacity in 2016 provided a model based on the standard UN Generic Capacity Model

9. UNDP – GENERIC CAPACITY MODEL Adapted by ISWE for Statistics (2016)
National regulatory environment
Home user demand
Freedom of press
Rule of law
Strength of economy
Statistical Laws
Codes of Practice
Funding
Resources
Management
Business processes
Dissemination
Quality processes
Training
Attitude
Skills

10. Tools use different capacity dimensions
STEP (GAMSO Structure)
SNAPSHOT (ESS CoP)
TASC (DQAF & SCI)
1. Vision
Strategy for statistics (in national development)
1. Institutional Capacity 
2. Planning and Managing
2. Governance & Leadership of NSS
2. Organisation of the NSS
(Legal, institutional and strategic foundations)
3. Mapping 
4. Questionnaire Content and Testing
3. Manage Strategic Collaboration
4. Capability Management
3. Resources
5. Sampling
5. Corporate Support – Finances, Human Resources, IT, Consumer Management, Buildings, Suppliers etc
4. Quality determinants
6. Field Operations 
7. Data Processing
6. GBSPM (dissemination)
5. Relations with users
8. Data Analysis and Evaluation 
7. Output indicators
Key Indicators
9. Data Dissemination
Revised version

11. Do we have a Theory of Change that works behind our tools?
They are all GOOD - but all DIFFERENT
Statistical development is related to development – economic, policy, social, regulatory – strength of institutions.
SUPPORTING USE
Are statistics unique in development - having most of the demand external to the country?
We know surprising little about use of statistics in developing countries – why would governments invest in statistics if they don’t use the products?
ISWE (UN 2016) spoke of the disconnect between use and production of statistics. In countries with weak demand – we focus on needs of users (dismally small) rather than supporting it.

12. What do we need to know in order to build capacity to meet future needs?
AUGMENTATION OF CAPACITY
Capacity is strongly enhanced by donor funding and activity – for TA, field costs etc. – outputs valuable - sustainable capacity?
How much more augmentation is viable for SDGs, Big Data etc.? Could this overload fragile systems?
HOW WILL COUNTRIES MANAGE NEW DATA SOURCES
Should NSOs have stronger curation and quality role? – less data collection?
Transferring institutional processes from developed countries may be misplaced. How transferable are these practices? Isomorphic mimicry
How is this to be managed in the developing world

13. How will results of assessments contribute to the changed context for statistics?
To what end do we need to develop this capacity? What will be its purpose?
More Data? Is this overloading NSS?
More Capacity? Can this be done sustainably?
Whose capacities need to be developed?
NSS – Production? Curation? Quality? Interpretation?
Users – how to use this data
Other producers – to augment NSS capacity
What kinds of capacities need to be developed to achieve the broader development objectives?
Technical
Productive
Demand
Regulatory/Governance (e.g. laws for managing Dig Data)
Are our tools giving us information to answer these questions?
How can we move forward in a harmonised manner?
UNDP

14. Moving Forward We have different frameworks in use by the tools.
Different questions – often several hundred per tool – each with different formats and with implicit solutions to capacity problems.
Burden and transaction costs on countries significant.
Results are often not shared – inefficient & costly
How does the measurement support an agreed ‘theory of change’ to support the capacity changes we need.
Can this be agreed, globally or regionally?

15. Do we need a task team?