Service Evaluation Gilbert O Ouma Institute of Climate Change and Adaptation University of Nairobi, Kenya

Outline Introduction Steps in conducting a survey Establishing goals Selecting sample population Methods of data collection Questionnaire design – the considerations Question structure Guidelines for formulating questions Setting metrics

Introduction Knowing what the client wants is the key factor to success in any type of business News media, government agencies and political candidates, etc. need to know what the public thinks We as meteorologists need to know what our users want The best way to find this information is to conduct surveys

Steps in conducting a survey Establish the goals of the project - What you want to learn Determine your sample - Who you will interview Choose interviewing methodology - How you will interview Create your questionnaire - What you will ask Pre-test the questionnaire, if practical - Test the questions Conduct interviews and enter data - Ask the questions Analyze the data - Produce the reports

Establishing Goals Before you even write the first question, it is important that you have a very clear idea about what you want your questionnaire to achieve Write down your research goals, and think about what information you need to elicit from respondents to give you evidence that those goals are met If your program has specific goals and objectives, the needed baseline/monitoring/final evaluation data will be quite obvious If the goals and objectives are vaguely defined or undefined, you will find it difficult to know what kind of data to gather

These goals represent general areas Examples of goals The potential market for a new product or service Ratings of current products or services User satisfaction levels User opinions These goals represent general areas The more specific you can make your goals, the easier it will be to get usable answers

Selecting Your Sample There are two main components in determining who you will interview The first is deciding what kind of people to interview - often called the target population If you conduct an employee attitude survey or an association membership survey, the population is obvious If you are trying to determine the likely success of a product, the target population may be less obvious Correctly determining the target population is critical If you do not interview the right kinds of people, you will not successfully meet your goals

The next thing to decide is how many people you need to interview A small, representative sample can reflect the group from which it is drawn The larger the sample, the more precisely it reflects the target group However, the rate of improvement in the precision decreases as your sample size increases For example, to increase a sample from 250 to 1,000 only doubles the precision You must make a decision about your sample size based on factors such as time available budget necessary degree of precision

Avoid bias in your samples Totally excluding all bias is almost impossible however, if you recognize bias exists you can intuitively discount some of the answers Source of bias depend on the nature of the survey For example a survey for a product aimed at retirees will not be as biased by daytime interviews as will a general public opinion survey A survey about Internet products can safely ignore people who do not use the Internet

Sample Probable bias Reason Your customers Favorable They would not be your customers if they were unhappy, but it is important to know what keeps them happy Your ex- customers Unfavorable If they were happy they would not have left you, but it is important to know why they left “Phone-in” Extreme views Only people with strong views on a subject (for or against) are likely to call in, and they may do so several times to load the vote Daytime Non- working A majority of people who are at home for interviews during the day do not work. Their opinion may not reflect the working population Internet Atypical people Limited to only to people with internet access. Usually not representative of general population

Methods of data collection There are many ways of getting information The most common methods are: literature searches talking with people focus groups personal interviews telephone surveys mail surveys email surveys internet surveys

Literature search Involves reviewing all readily available material These materials can include: internal company information relevant publications Newspapers magazines on-line data bases any other published materials It is a very inexpensive method of gathering information But does not often yield timely information

Talking with people A good way to get information during the initial stages of a program/project Can be used to gather information that is not publicly available, or that is too new to be found in the literature Basically these involve meetings with different users of your products Although often valuable, the information has questionable validity because it is highly subjective and might not be representative of the population

Focus group Used as a preliminary tool to explore peoples ideas and attitudes to discover customer concerns A group of 6 to 20 people meet in a conference-room-like setting with a trained moderator The moderator leads the group's discussion and keeps the focus on the areas you want to explore Their disadvantage is that the sample is small and may not be representative of the population in general

Personal interviews Good for getting in-depth and comprehensive information Involves one person interviewing another person for personal or detailed information Typically, an interviewer will ask questions from a written questionnaire and record the answers verbatim Sometimes, the questionnaire is simply a list of topics Personal interviews (because of their expense) are generally used only when subjects are not likely to respond to other survey methods

Telephone surveys The fastest method of gathering information from a relatively large sample (100-400 respondents) The interviewer follows a prepared script that is essentially the same as a written questionnaire However, unlike a mail survey, the telephone survey allows the opportunity for some opinion probing

Mail surveys A cost effective method of gathering information Ideal for large sample sizes or when the sample comes from a wide geographic area Because there is no interviewer, there is no possibility of interviewer bias The main disadvantage is the inability to probe respondents for more detailed information

Email and internet surveys Relatively new and little is known about the effect of sampling bias in internet surveys While it is clearly the most cost effective and fastest method of distributing a survey, the demographic profile of the internet user does not represent the general population, although this is changing

The choice of survey method will depend on the following Speed Email and internet are the fastest methods, followed by telephone interviewing Cost Personal interviews are the most expensive, followed by telephone and mail Internet usage Have significant advantages but you may not be able to generalize the results to the population as a whole Literacy levels Illiterate, less-educated people rarely respond to mail surveys Sensitive questions Personal interviews may not elicit the correct answers

Questionnaire design – general considerations Well-defined goals are the best way to assure a good questionnaire design When the goals of a study can be expressed in a few clear and concise sentences, the design of the questionnaire becomes considerably easier The questionnaire is developed to directly address the goals of the study

KISS - keep it short and simple If you present a 20-page questionnaire most potential respondents will give up in horror before even starting Response rate is the single most important indicator of how much confidence you can place in the results A low response rate can be devastating to a study

Start with an introduction or welcome message When practical, state who you are and why you want the information in the survey A good introduction or welcome message will encourage people to complete your questionnaire Include clear and concise instructions on how to complete the questionnaire These must be very easy to understand, so use short sentences and basic vocabulary

Begin with a few non-threatening and interesting items If the first items are too threatening or "boring", there is little chance that the person will complete the questionnaire People generally look at the first few questions before deciding whether or not to complete the questionnaire Make them want to continue by putting interesting questions first

Question Structure Broadly speaking, survey questions can be classified into three structures: Closed open-ended contingency

Closed or multiple choice questions These ask the respondent to choose, among a possible set of answers, the response that most closely represents his/her viewpoint May offer simple alternatives such as ‘Yes’ or ‘No’ They may also require that the respondent chooses among several answer categories, or that he/she uses a frequency scale, an importance scale, or an agreement scale

Have you observed delayed onset of rains in these seasons? Yes No Do you receive weather information? Does this information reach you in good time?

Advantages of closed questions are: the respondent is restricted to a finite (and therefore more manageable) set of responses they are easy and quick to answer they have response categories that are easy to code they permit the inclusion of more variables in a research study because the format enables the respondent to answer more questions in the same time required to answer fewer open-ended questions.

The disadvantages of closed questions are: They can introduce bias By forcing the respondent to choose between given alternatives By offering alternatives that otherwise would not have come to mind Where there is a tendency for the respondent to tick systematically either the first or last category Where respondents select what may be considered as the most socially desirable response alternative Where respondents answer all items in a list in the same way They do not allow for creativity or for the respondent to develop ideas They do not permit the respondent to qualify the chosen response or express a more complex or subtle meaning They require skill to write because response categories need to be appropriate, and mutually exclusive

Some closed questions may have only two mutually exclusive responses are provided Gender Female Male Should not be overused in a survey because it elicits much less information than multiple choice formats Do you read a daily newspaper? - can elicit a yes or no answer without giving a lot of information May be reworded

How many times per week do you read a daily newspaper? Seven times a week Five to six times a week Three to four times a week One to two times per week Less than once per week Never This would provide more specific and useful information

Open-ended or free response questions These are not followed by any choices The respondent must answer by supplying a response usually by entering a number a word a short text Answers are recorded in full, either by the interviewer or, in the case of a self-administered survey, the respondent records his or her own entire response

What additional weather information you would like to receive to improve your farming? __________________________________________________________________________ In which years (in the last two decades) have you had floods _____________________________________

The advantages of open-ended questions are: They allow respondents to express their ideas spontaneously in their own language They are less likely to suggest or guide the answer than closed questions because they are free from the format effects associated with closed questions They can add new information when there is very little existing information available about a topic

The disadvantages of open-ended questions are: They may be difficult to answer and even more difficult to analyze They require the development of a system of coded categories with which to classify the responses They require effort and time on behalf of the respondent They require the respondent to have some degree of writing ability Respondent handwriting can be illegible

When did you leave school? There is always the possibility with open-ended questions that responses may come in very different forms may lead to answers that cannot be systematically coded for analysis When did you leave school? Seven years ago When I got my first job When my brother started going to high school When my parents moved into this house Appropriate probing by the interviewer Clear instructions for self-administered

Contingency questions This is a special case of a closed-ended question because it applies only to a subgroup of respondents The relevance of the question for a subgroup is determined by asking a filter question The filter question directs the subgroup to answer a relevant set of specialized questions and instructs other respondents to skip to a later section of the questionnaire

In the last ten years have you bought or sold land Yes No > GO TO Q.21 The advantage of contingency questions is that detailed data may be obtained from a specific subgroup of the population Some questions may apply only to, for example, females and not to males At the base of good contingency questions are clear and specific instructions to respondents

Guidelines for writing questions Keep the vocabulary simple Use simple words Avoid acronyms, abbreviations, jargon, technical terms, abstract or general words, words with ambiguous meanings If a technical term is used, its meaning must be explained General or abstract words – examples must be provided to clarify meaning

Keep the question short Generally, it is recommended to hold questions to 25 words or less If a longer sentence is used then it should be broken up so that there will be several shorter sentences Avoid double-barrelled questions These are single questions that ask for two things and therefore require two answers Do you have your own table or your own room to do your homework? Do you think it is good idea for children to study geography and history in primary school? In such instances, respondents do not know what to do if they want to say ‘Yes’ to one part of the question but ‘No’ to the other

Avoid hypothetical questions Evidence has shown that hypothetical questions such as “Would you use this resource in your class if it were available?” are not good for the prediction of behaviour People are generally poor predictors of their own behaviour because of changing circumstances and because so many situational variables intervene You are able to collect more valid data if you ask the respondents about their past behaviour and present circumstances, attitudes, and opinions

Do not overtax the respondent’s memory It is risky to ask the respondent to recall past behaviour over a long retrospective period Especially when recurrent events or behaviours are concerned How many severe El-Nino events have occurred since 1961? If such a question must be asked, a shorter recall period might be more appropriate

Avoid double negatives Double negatives, either in the question or an answer category (or both), create difficulties for the respondent Customer satisfaction surveys by meteorological agencies should not be allowed, agree or disagree This is problematic to answer for respondents who are in favour of the surveys, that is those who do not agree that the surveys should not be allowed

Beware of ‘leading’ questions A leading question is a question phrased in such a way that it seems to the respondent that a particular answer is expected Do you favor inclusion of indigenous knowledge in deriving weather forecasts? Could be framed as You wouldn’t say that you were in favor of inclusion of indigenous knowledge in deriving weather forecasts, would you?” Or in a more subtle form: Would you say that you are not in favor of inclusion of indigenous knowledge in deriving weather forecasts?

Setting Metrics One of the important elements of performance management is the use of metrics A metric is a numerical measure used as baseline measurement for performance Metrics are used to define quantitative measurements for the quality of your service - they are basically a definition of the performance goals that you want to achieve They depend on the objectives and goals of your program/projects

Overall Objective The answer to Why does your agency exist? What do you want to achieve? Should have a clearly defined list of objectives Defines where you are going If you don't know where you are going then any road will take you there Also known as desirable outcomes

The objectives must be Doable Understandable Manageable Beneficial

Target Metrics are defined at the start of a project to: indicate the minimum quality which you want to achieve define how these factors will actually be measured When used properly these metrics can provide a useful tool When used irresponsibly they can be a time-wasting distraction You need to understand what you are measuring, how you are measuring it and why

Key Performance Indicator (KPI) Key Performance Indicators (KPIs) are metrics that help you understand how you are doing against your objectives For example, if the objective is Awareness about the products, the KPI could be average number of requests for specific products per month

Goals Specific strategies you'll leverage to accomplish your business objectives Objectives can be quite strategic and high level – support food security Goals – very specific Improve forecasts (can be broken down further) Create awareness, etc.

Baseline information You can’t solve a problem if you don’t know what exactly your problem is Information about your problem before you start fixing it or ‘baseline information’ is key Baseline data is basic information gathered before a program/project begins It is used later to provide a comparison for assessing program/project impact It establishes a basis for comparing what was happening before the program with what happens after

Monitoring Every program/project has a goal to achieve Activities implemented should lead to achieving this goal Implementation of some of the activities may not effectively lead to the goal Due to some unforeseen circumstances Constraints Need to ascertain that the activities produce desired results Constant collection of data on certain issues

Evaluation At the end of the program/project you need to find out if the objectives have been achieved Seen through the impacts Require further data collection targeted at the expected outcomes of your interventions Gives the picture of what is happening after the program/project intervention

Community-based climate services WMO GFCS Reaching the users New approach Transdisciplinary Participatory

Thank you