The challenges of obtaining reliable data World Bank Group Investment Climate Advisory Services Florentin BLANC Vilnius, 22 March 2012 Measuring reform success: business surveys, "client satisfaction" and administrative burden

The need for measurement – knowing where you stand, and where you are going Reforms are undertaken – but do the regulated/inspected businesses feel them? Do they see inspectors as a positive factor, or not? Do they complain about burden, lack of transparency etc. – or less? Essential to gather positive results in dealing with the regulated “clients” – while also maintaining or improving outcomes for the public – so the two sides should get measured Crucial for the Government as a whole – which has to show improvements to the citizens – and for the inspecting agencies, which have to report to the Government, and need success in improving effectiveness and efficiency in a difficult economic / fiscal context THE WORLD BANK World Bank Group Multilateral Investment Guarantee Agency

Range of instruments – all with specific upsides and downsides Business surveys: ask all, or a sample of, inspected businesses how their experience with inspectorate(s) was Focus-group discussions: gather several businesses with a certain profile and ask about experiences with range of structured questions Both of these methods can be implemented directly by inspectorate or (preferred) through a specialized company or researchers Open consultations with stakeholders (e.g. business associations) THE WORLD BANK World Bank Group Multilateral Investment Guarantee Agency

Focus of our measurement efforts – not just “what” but “how well”… Relatively easy to find out from businesses whether something took place or not (“Were you inspected?” – “Did you receive advice?”) More difficult to get reliable answers on questions of quality – which are inherently more subjective (and thus where answers can vary…) Specific focus in our case is to try and find out “satisfaction” with “quality of consultations” If possible, we should try and measure the results rather than subjectively expressed satisfaction THE WORLD BANK World Bank Group Multilateral Investment Guarantee Agency

Business consultations – what are we trying to achieve Immediate aim of consultations is to help business comply: knowing and understanding what they have to do means less violations Longer-term goal is to facilitate business creation, growth and diversification by providing more certainty for investment (if one operates within parameters explained in consultation, there is no risk of the business being closed down) Using indicators such as “compliance levels” and “business growth” to measure success of consultations is nonetheless problematic because they can be impacted by many other influences… THE WORLD BANK World Bank Group Multilateral Investment Guarantee Agency

“Client-satisfaction surveys” – an instrument that can be misleading Frequent approach: measure reform success by asking businesses whether they see improvements Several risks with such surveys:  Insincere answers – whenever the business fears about confidentiality (e.g. survey given just after inspection…)  Subjective answers – correlated with external events or “latest impressions” and not reflecting actual “average situation” with inspections May over-estimate “success” – or hide it… Such “subjective” questions are better asked by independent researchers, in focus-groups or in-depth interviews – where the reasons for the answers can be asked and discussed THE WORLD BANK World Bank Group Multilateral Investment Guarantee Agency

An example of qualitative, independent investigation of business perceptions From the Business End of the Telescope Commissioned by UK LBRO Focus-groups, in-depth interviews (with individual businesses, business associations etc.) to understand how businesses perceive inspections, inspectors Gives real insights on negatives and positives, what businesses resent and what they appreciate, expect etc. Trust in independence of interviewers – professional competence to investigate THE WORLD BANK World Bank Group Multilateral Investment Guarantee Agency

Designing surveys – asking the “right” questions First crucial step: ensure trust of interviewees in independence of survey, confidentiality Ask questions that will give more reliable answers:  Objective quantitative indicators: how often, how long, etc. – allowing to track inspections patterns  Qualitative but not “like” or “feel” questions:  Did you get advice that was clear?  Was the inspector proficient in technical issues? Develop questions that are clear, precise, yield meaningful answers – questionnaire testing is needed THE WORLD BANK World Bank Group Multilateral Investment Guarantee Agency

Examples of “quantitative surveys” World Bank Group has for 10+ years been supporting surveys of inspections in countries where it assists with reform work Provide clear picture on how many businesses inspected, how often, how long etc. – aggregated across country – by sector – by agency etc. Of course margin of error – and some answers may be wrong (misunderstandings, bad memory etc.) – but mostly this gets “averaged out” by large numbers of respondents Very useful information because (among other issues):  Consolidated data was simply missing – Government, even inspectorates, often did not know situation  Discrepancy between practices and “official numbers” THE WORLD BANK World Bank Group Multilateral Investment Guarantee Agency

The cost challenge – and the different technical solutions Business surveys have a cost – particularly if you don’t use the “questionnaire filled and collected at the end of the inspection” approach… Phone, mail, internet surveys have different costs – but also different levels of reliability Probably makes sense to combine “qualitative” instruments conducted by independent researchers, and “quantitative” ones – resources can be pooled by inspectorates, no need to always have a separate study for each Key is to properly conceive what we want to know: what are the important questions, and the target groups (all businesses, some specific sectors, sizes etc.?) THE WORLD BANK World Bank Group Multilateral Investment Guarantee Agency

Trust and credibility Businesses (and citizens – as some surveys could target inspected subjects that are not entrepreneurial ones) need to have full confidence that survey results will not harm them – so not only individual confidentiality, but even “group” confidentiality / results should not be traceable “Survey fatigue” will soon kick in if there are too many studies, and if they are not seen as having consequences Essential for Government, inspecting agencies to show that surveys / studies results are being acted upon and used to design / drive further changes Such link to policy actions can also be advertised before a new survey / study starts – to increase response rates / quality THE WORLD BANK World Bank Group Multilateral Investment Guarantee Agency

Some recommendations (1) – combine approaches Business surveys can be used for quantitative data (“did you receive advice?”, “were your questions answered?”) – particularly as long as most inspectorates do not have an information system to consolidate data on their activities Using (trusted) independent researchers to conduct qualitative research through focus-groups etc. is likely to provide more insights into satisfaction and remaining problems Direct discussions with business representatives are also essential to increase trust between the two sides Finally, trying to measure and track outcomes would be a very important step… THE WORLD BANK World Bank Group Multilateral Investment Guarantee Agency

Some recommendations (2) – measure outcomes Far more difficult because of “attribution” issues and of measurement quality issues (for compliance – since it is measured through inspections => conflict of interest…) Areas of focus should be determined depending on baseline – if businesses particularly complain in some areas, these are the ones where to check if changes have taken place More business creations or expansions, more jobs, in a sector that had problems, can be indicators that consultations (and other changes) are working Also very important: through surveys (possibly internet) or independent research, try and identify why some were discouraged from creating or expanding business activity – this may show the remaining gaps THE WORLD BANK World Bank Group Multilateral Investment Guarantee Agency

14 Thanks for your attention THE WORLD BANK World Bank Group Multilateral Investment Guarantee Agency