Presentation on theme: "Assessments and the WB Assessment Manual Linn Hammergren November 10, 2010."— Presentation transcript:
Assessments and the WB Assessment Manual Linn Hammergren November 10, 2010
Purpose Discuss why assessments are important (and what happens when you don’t do one) Discuss assessment inputs Provide ideas on methodologies and alternative strategies for doing them Outline a strategy for conducting an assessment
Why assessments are important Because obviously you can’t resolve a problem until you identify it and its causes Because the obvious answers to the two (implicit) questions are often wrong Because an assessment allows you to understand the broader context for an operation and the likely impediments to its success Because an assessment provides a format for discussing strategies with the client Because an assessment gives you baseline data and thus a means of monitoring progress
Why assessments are not done or are incomplete They cost money They take time Bank thinks identification mission is sufficient The client thinks they are unnecessary They reveal inconvenient details However, taking into account how long and how much it takes to “prepare a PAD suitable for Board approval” the investments in a good assessment are marginal. However inconvenient the truth of the matter, it is better to find it early on than to have it emerge post-approval
Examples of findings an assessment might discover Belief (Mexico) that primary problem with money claim cases is delay to judgment –Finding: only 20% of cases reach judgment –Finding: longer delays for enforcement Belief (Mexico and many LAC countries) that judges are “overloaded with work” –Finding: annual filings often quite low and even when not, most cases go “nowhere” and so require little work. –Finding: much of caseload is unattended backlog Belief (Colombia, Honduras) that new CrPC improves case processing: –Finding: most cases reaching verdicts are in flagranti –Finding: major crimes less attended. Belief (Ghana and other African countries) that fast-track and commercial courts are more efficient, but studies show low caseloads and delays
Inputs to assessments All prior diagnostics and studies – or at least the most important ones. Many assessments ignore prior work (especially if not by the WB), which is a waste of resources Discussions with the principal client (often the high court judges) as to their perception of the “problem” to be resolved Discussions with other stakeholders to get their views, which are often different Review of any existing statistical material, legal framework, history of prior and on-going programs Fieldwork of various kinds (more on this later) In short, a good part of the assessment can occur before the fieldwork and helps focus the latter. This saves time and resources.
A participatory approach? This is controversial although Bank seems to endorse The client is usually the host government or the judiciary, and both will have a self-interested cut on what is wrong and how it should be fixed. Participation is important for client buy-in, but in some cases may prove overly restrictive – e.g. when client attends and thus censors all meetings If the client is open to new approaches and inconvenient truths, it can be very helpful, but if this is not the case, it can limit findings. Still, to the extent possible: –Clients should be involved in assessment design –They should be invited to provide any material they consider essential (and what they consider essential is also revealing) –They should be involved in at least part of fieldwork –They should be informed about progress and problems encountered –They should be included in analysis and discussion of conclusions
Why not do a completely participatory assessment? Even in the best of cases this is not likely to produce the best results because: –Local appreciations are local –Participants lack the big picture (as regards their “problems” and the likely solutions) –Participatory planning encourages Christmas Tree projects – an ornament for everyone. External experts are important because they have broader experience, and if good can suggest a variety of different interpretations and solutions. The wisdom of crowds is rarely technically based
A problem-oriented approach using institutional, political economic analysis Can be as participatory as possible – techniques are not incompatible To maximize utility for project design – if there are no problems, an operation is unnecessary Problems can be defined by combination of CMU input, literature review, primary client views, broad-based interviews and team’s observations Initial problem definition tested and refined in later stages Seek explanations (and remedies) through institutional and political economic analysis
What is a problem? Defined here as shortcoming in desired outputs and downstream impacts, but there is often confusion on this point When judges, police are asked about problems, often mention salaries, equipment, overwork Not the problems of interest here – best treated as potential causes Challenge – to get “system operators” to redefine “their” problems in terms of output/impacts. Can ask about consequences of “overwork” for clients, for example
Institutional analysis looks at: Relevant state and non-state organizations and actors Formal laws, structure, mandates, and procedures; human and other resources; territorial distribution and other factors affecting use and outputs Real practices and informal rules Incentive structures – will have to be inferred How the above interact to shape outputs and create problems or performance gaps
Political Economy Bank has shied away from this (or mislabeled institutional analysis to this end) Political economy analysis (PEA) looks at power and access to resources, and at intra- and extra-sector interactions to affect both –Institutional analysis focuses on how organizational arrangements influence behavior –Political economy explains why – who benefits? PEA may be omitted from public report, but not from assessment. –If politicians need compliant judges or are in cahoots with criminals, this will affect the program –Example – IADB citizen security projects in Caribbean neglect ties between politicians and “dons” (gang lords). Bank need not engage in politics but should recognize its effects
Assessment stages and content – the Manual Preparation -- scope, resources, timing, and team S 1: Preliminary background S 2: Preliminary problem identification S 3: Field collection of information – institutional analysis S 4: Summary overview and analysis S 5: Comprehensive analysis of partial findings, prioritization of problems/causes, identification of areas for interventions S 6: Recommendations for Program
Preparation: Initial Considerations Initial definition of assessment scope by TTL, CMU, client –Often targeted at institution or problem –Team may need to negotiate some modification, especially as regards incorporating more context (Court’s problem is not always the Court) Resources and timing --- usually decided (too?) early –Both often underestimated, but there is never enough –2-4 weeks in field (with more time for writing) absolute minimum, but superficiality can be combated with experienced team (knows country, knows methods), access to documents, or with very limited focus, small country, etc. –Even with large team, more time, lack of focus/organization can produce ambiguous results Assessment team: skills needed; identification of local/foreign experts; decisions as to inclusion of Bank and/or client members Norms, ground rules, roles: set by team and TTL Design and selection of methodologies
Stage 1: Preliminary background Get team up to speed on country and overall characteristics, history, broader development problems Less necessary if use local experts or internationals with past experience However, will be needed as part of assessment document, and also to ensure common team understanding of where they are starting May be useful to have team meetings to discuss, especially if prior exposure to country varies greatly Example: team for Honduras labor assessment (1 local, 1 international with country experience, 3 substantive experts with no country exposure or language skills) – would have been helpful to start here
Stage 2: Preliminary problem identification Start with working hypotheses based on: –What people say (from interviews, written documents, international assessments, opinion polls) –Team’s initial observations – this is where international experience counts as a point of reference Over course of evaluation, problem definition may change – as may early guesses as to underlying causes Even if findings contradict conventional beliefs, latter should be mentioned in study. What people think is important and will affect what can be recommended Also useful to tie problems to groups mentioning them – not everyone agrees on what is wrong or why (which is why reviews should never be limited to judges and government authorities).
Stage 3: Data collection Some can be done prior to fieldwork– using written or internet documents (including laws, budgets, statistics) and out-of-country interviews Rest will inevitably be a mix of interviews, observation, surveys, statistical reviews, and whatever else the team can muster Aim – to do more than repeat what informants tell you; test their versions with other data sources and also explore alternative hypotheses
Methodology (Fieldwork and other Data collection) Assessments use multiple methodologies; several described in manual Combine hard with soft data sources Soft data (from interviews, observation) provide insights; hard data (statistics, surveys) used to check and to generate hypotheses directly No data source is perfect, and selection depends on what is available, time and financing, what is being sought Also depends on team skills – do they know statistics/surveys? Are they experienced with specific topics? Assessment may suggest further studies Most studies will only focus on formal systems – this can be a problem where most citizens uses informal ones. Even if someone on the team has experience, legal-pluralism does not lend itself to a fast assessment. (i.e. Countries with 100 or more local traditions?) But, legal pluralism may not be amenable to international assistance either.
S. 3 Softer data sources Document review –WB often overlooks prior studies, even its own – team should be provided with this material –Laws may not reveal reality, but they are important insofar as they impede change Interviews --- will continue to be a major source of information –Important to triangulate – multiple points of view –Structured (set questions) can be quantified, but richer information comes from unstructured. May thus combine two. –Try to transcend client’s definition of interview list – need to reach further. Observation – a widely ignored source of information –However requires skill and most probably wider experience –A good court administrator or IT expert doesn’t need to know the language to see problems –However, a local and an international specialist may achieve much more
S. 3: Harder sources Surveys – might be considered “soft” as they are quantified opinions, but in any event, seem to carry more weight Aggregate statistics -- often not available and in any case, usually limited Sampling – costly but revealing. –However, samplers should be carefully monitored to ensure 1) their sample is good and 2) their data are also good –Can always treat as “illustrative” if quality is in doubt. –In large, heterogeneous countries, what holds in one region may not in another
S 3, Harder data, some further observations Common lack of good statistics is a finding in itself – and may lead to recommendations on improvements Most systems only permit estimations of workloads, congestion, and possibly resolution rates; for delays, case profiles, differential outcomes need something more. Surveys, sampling, and related methods usually take more time and money than allowed in assessment. Can take “illustrative” short cuts –Quick review of random sample of filings from courts “of interest” –DB style survey of users on delays, outcomes – but their perceptions often don’t coincide with numbers (case of Mexico) –If courts agree, can use staff for some data collection, but supervision required and best to keep simple –See Dilbert (next slide) on iffy data
S. 3: Less conventional data sources and analysis Evaluating quality of judicial decisions – many have tried, but no consensus on how to do it. Also all methods are labor-intensive, time-consuming and ultimately, highly subjective Focus and study groups – also quasi-soft, but unlike surveys allow more detail and discussion; useful for explaining statistics and sampling results. Incentive analysis – more an art than a science and also the result of other methods (focus groups, statistics, surveys). Some interesting, if hypothetical conclusions: –Mexico: lawyers are less interested in curbing delays than clients – Generally – delay does not bother judges until someone (their bosses) tracks it –Generally – counting judgments encourages “cherry picking” and manipulation of numbers
S 4: Summary Overview and analysis Conducted on the basis of initial information, before fieldwork is completed Allows fieldwork to focus on further testing of hypotheses -- may call for surveys, focus groups, or sampling Except in longer-term studies may not be possible, but will at least generate areas “for further study.” If done, should orient collection of harder data – for example, extent of and causes of delays; hypotheses as to who uses courts and with what results
S. 5: Comprehensive analysis Initial list: –Performance and impact problems –Interrelated causes of each –Initial ideas on solutions Prioritization: –Of most significant or most urgent problems (can include perceptions as well) –Of causes, by susceptibility to change and broader impact –Potential points of program entry with short and long-term results, difficulty of addressing, sources of support and opposition to change.
S. 6: Recommendations Assessment TOR may not require, but is the obvious last step This and prior analysis should involve local stakeholders, as if they are not in agreement, nothing will happen May be useful to hold workshops or seminars to discuss tentative findings, conclusions, and recommendations Results and report content may consequently be negotiated Quick fixes (tactical measures) and longer term reforms (overall strategy)
Assessment Do’s and Don’ts There are no recipes, but have a plan, tentative organization, and role assignments even if they will change along the way Go beyond “descriptive studies” –an overview of the parts and how they fit together is useful, but assessments must do more than this and when they don’t, counterparts complain. Save the details for the Annexes? Likewise, assessment should not be a legal review – often a problem of local experts with little research experience Do try for a multi-disciplinary team – to spread the perspectives Mix internationals and locals – they each have their use Beware, however, of expert bias – trainers inevitably recommend training (even if not a priority) and IT experts will want systems; generalists with some substantive knowledge may be preferable, saving the dedicated experts for the identification mission.
Assessment Do’s and Don’ts (cont.) Triangulate sources and mix methodologies to test “what everyone knows” Avoid sending multiple team members to interview one person (simultaneously or sequentially) – a waste of time and can intimidate informants. “Round robin interviews” may work better with groups. Take informant claims seriously, but not as the unvarnished truth Protect informants and verify/respect their wish to be identified or not. Gather all relevant information, vetting for inclusion later – don’t censor along the way. (Consider multiple versions of report?) Encourage team meetings on findings and intermediate conclusions – this can reveal gaps, contradictions, and synergies. Be prepared to leave things out of the report (not only for sensitivity but for limited relevance). Data collection is a process, but the report does not need to repeat all the steps. Recognize areas for more research later, noting why they are important and how they might be explored.
Assessment is an iterative process The first cut is only that Assessments should include provisions and recommendations for further studies However, assessments should be used – and in many cases they remain on the shelves Can also be useful in motivating clients, other local participants to find out more.