1. Service Evaluation Gilbert O Ouma
Institute of Climate Change and Adaptation
University of Nairobi, Kenya

2. Outline Introduction Steps in conducting a survey
Establishing goals
Selecting sample population
Methods of data collection
Questionnaire design – the considerations
Question structure
Guidelines for formulating questions
Setting metrics

3. Introduction
Knowing what the client wants is the key factor to success in any type of business
News media, government agencies and political candidates, etc. need to know what the public thinks
We as meteorologists need to know what our users want
The best way to find this information is to conduct surveys

4. Steps in conducting a survey
Establish the goals of the project - What you want to learn
Determine your sample - Who you will interview
Choose interviewing methodology - How you will interview
Create your questionnaire - What you will ask
Pre-test the questionnaire, if practical - Test the questions
Conduct interviews and enter data - Ask the questions
Analyze the data - Produce the reports

6. Establishing Goals
Before you even write the first question, it is important that you have a very clear idea about what you want your questionnaire to achieve
Write down your research goals, and think about what information you need to elicit from respondents to give you evidence that those goals are met
If your program has specific goals and objectives, the needed baseline/monitoring/final evaluation data will be quite obvious
If the goals and objectives are vaguely defined or undefined, you will find it difficult to know what kind of data to gather

7. These goals represent general areas
Examples of goals
The potential market for a new product or service
Ratings of current products or services
User satisfaction levels
User opinions
These goals represent general areas
The more specific you can make your goals, the easier it will be to get usable answers

8. Selecting Your Sample
There are two main components in determining who you will interview
The first is deciding what kind of people to interview - often called the target population
If you conduct an employee attitude survey or an association membership survey, the population is obvious
If you are trying to determine the likely success of a product, the target population may be less obvious
Correctly determining the target population is critical
If you do not interview the right kinds of people, you will not successfully meet your goals

9. The next thing to decide is how many people you need to interview
A small, representative sample can reflect the group from which it is drawn
The larger the sample, the more precisely it reflects the target group
However, the rate of improvement in the precision decreases as your sample size increases
For example, to increase a sample from 250 to 1,000 only doubles the precision
You must make a decision about your sample size based on factors such as
time available
budget
necessary degree of precision

10. Avoid bias in your samples
Totally excluding all bias is almost impossible
however, if you recognize bias exists you can intuitively discount some of the answers
Source of bias depend on the nature of the survey
For example
a survey for a product aimed at retirees will not be as biased by daytime interviews as will a general public opinion survey
A survey about Internet products can safely ignore people who do not use the Internet

11. Sample
Probable bias
Reason
Your customers
Favorable
They would not be your customers if they were unhappy, but it is important to know what keeps them happy
Your ex- customers
Unfavorable
If they were happy they would not have left you, but it is important to know why they left
“Phone-in”
Extreme views
Only people with strong views on a subject (for or against) are likely to call in, and they may do so several times to load the vote
Daytime
Non- working
A majority of people who are at home for interviews during the day do not work. Their opinion may not reflect the working population
Internet
Atypical people
Limited to only to people with internet access. Usually not representative of general population

12. Methods of data collection
There are many ways of getting information
The most common methods are:
literature searches
talking with people
focus groups
personal interviews
telephone surveys
mail surveys
surveys
internet surveys

13. Literature search Involves reviewing all readily available material
These materials can include:
internal company information
relevant publications
Newspapers
magazines
on-line data bases
any other published materials
It is a very inexpensive method of gathering information
But does not often yield timely information

14. Talking with people
A good way to get information during the initial stages of a program/project
Can be used to gather information that is not publicly available, or that is too new to be found in the literature
Basically these involve meetings with different users of your products
Although often valuable, the information has questionable validity because it is highly subjective and might not be representative of the population

15. Focus group Used as a preliminary tool
to explore peoples ideas and attitudes
to discover customer concerns
A group of 6 to 20 people meet in a conference-room-like setting with a trained moderator
The moderator leads the group's discussion and keeps the focus on the areas you want to explore
Their disadvantage is that the sample is small and may not be representative of the population in general

16. Personal interviews
Good for getting in-depth and comprehensive information
Involves one person interviewing another person for personal or detailed information
Typically, an interviewer will ask questions from a written questionnaire and record the answers verbatim
Sometimes, the questionnaire is simply a list of topics
Personal interviews (because of their expense) are generally used only when subjects are not likely to respond to other survey methods

17. Telephone surveys
The fastest method of gathering information from a relatively large sample ( respondents)
The interviewer follows a prepared script that is essentially the same as a written questionnaire
However, unlike a mail survey, the telephone survey allows the opportunity for some opinion probing

18. Mail surveys A cost effective method of gathering information
Ideal for
large sample sizes
or when the sample comes from a wide geographic area
Because there is no interviewer, there is no possibility of interviewer bias
The main disadvantage is the inability to probe respondents for more detailed information

19. Email and internet surveys
Relatively new and little is known about the effect of sampling bias in internet surveys
While it is clearly the most cost effective and fastest method of distributing a survey, the demographic profile of the internet user does not represent the general population, although this is changing

20. The choice of survey method will depend on the following
Speed
and internet are the fastest methods, followed by telephone interviewing
Cost
Personal interviews are the most expensive, followed by telephone and mail
Internet usage
Have significant advantages but you may not be able to generalize the results to the population as a whole
Literacy levels
Illiterate, less-educated people rarely respond to mail surveys
Sensitive questions
Personal interviews may not elicit the correct answers

21. Questionnaire design – general considerations
Well-defined goals are the best way to assure a good questionnaire design
When the goals of a study can be expressed in a few clear and concise sentences, the design of the questionnaire becomes considerably easier
The questionnaire is developed to directly address the goals of the study

22. KISS - keep it short and simple
If you present a 20-page questionnaire most potential respondents will give up in horror before even starting
Response rate is the single most important indicator of how much confidence you can place in the results
A low response rate can be devastating to a study

23. Start with an introduction or welcome message
When practical, state who you are and why you want the information in the survey
A good introduction or welcome message will encourage people to complete your questionnaire
Include clear and concise instructions on how to complete the questionnaire
These must be very easy to understand, so use short sentences and basic vocabulary

24. Begin with a few non-threatening and interesting items
If the first items are too threatening or "boring", there is little chance that the person will complete the questionnaire
People generally look at the first few questions before deciding whether or not to complete the questionnaire
Make them want to continue by putting interesting questions first

25. Question Structure
Broadly speaking, survey questions can be classified into three structures:
Closed
open-ended
contingency

26. Closed or multiple choice questions
These ask the respondent to choose, among a possible set of answers, the response that most closely represents his/her viewpoint
May offer simple alternatives such as ‘Yes’ or ‘No’
They may also require that the respondent chooses among several answer categories, or that he/she uses a frequency scale, an importance scale, or an agreement scale

27. Have you observed delayed onset of rains in these seasons?
Yes No
Do you receive weather information?
Does this information reach you in good time?

28. Advantages of closed questions are:
the respondent is restricted to a finite (and therefore more manageable) set of responses
they are easy and quick to answer
they have response categories that are easy to code
they permit the inclusion of more variables in a research study because the format enables the respondent to answer more questions in the same time required to answer fewer open-ended questions.

29. The disadvantages of closed questions are:
They can introduce bias
By forcing the respondent to choose between given alternatives
By offering alternatives that otherwise would not have come to mind
Where there is a tendency for the respondent to tick systematically either the first or last category
Where respondents select what may be considered as the most socially desirable response alternative
Where respondents answer all items in a list in the same way
They do not allow for creativity or for the respondent to develop ideas
They do not permit the respondent to qualify the chosen response or express a more complex or subtle meaning
They require skill to write because response categories need to be appropriate, and mutually exclusive

30. Some closed questions may have only two mutually exclusive responses are provided
Gender
Female
Male
Should not be overused in a survey because it elicits much less information than multiple choice formats
Do you read a daily newspaper? - can elicit a yes or no answer without giving a lot of information
May be reworded

31. How many times per week do you read a daily newspaper?
Seven times a week
Five to six times a week
Three to four times a week
One to two times per week
Less than once per week
Never
This would provide more specific and useful information

32. Open-ended or free response questions
These are not followed by any choices
The respondent must answer by supplying a response usually by entering
a number
a word
a short text
Answers are recorded in full, either by the interviewer or, in the case of a self-administered survey, the respondent records his or her own entire response

33. What additional weather information you would like to receive to improve your farming? __________________________________________________________________________
In which years (in the last two decades) have you had floods
_____________________________________

34. The advantages of open-ended questions are:
They allow respondents to express their ideas spontaneously in their own language
They are less likely to suggest or guide the answer than closed questions because they are free from the format effects associated with closed questions
They can add new information when there is very little existing information available about a topic

35. The disadvantages of open-ended questions are:
They may be difficult to answer and even more difficult to analyze
They require the development of a system of coded categories with which to classify the responses
They require effort and time on behalf of the respondent
They require the respondent to have some degree of writing ability
Respondent handwriting can be illegible

36. When did you leave school?
There is always the possibility with open-ended questions that responses may come in very different forms
may lead to answers that cannot be systematically coded for analysis
When did you leave school?
Seven years ago
When I got my first job
When my brother started going to high school
When my parents moved into this house
Appropriate probing by the interviewer
Clear instructions for self-administered

37. Contingency questions
This is a special case of a closed-ended question
because it applies only to a subgroup of respondents
The relevance of the question for a subgroup is determined by asking a filter question
The filter question directs the subgroup to answer a relevant set of specialized questions and instructs other respondents to skip to a later section of the questionnaire

38. In the last ten years have you bought or sold land
Yes
No > GO TO Q.21
The advantage of contingency questions is that detailed data may be obtained from a specific subgroup of the population
Some questions may apply only to, for example, females and not to males
At the base of good contingency questions are clear and specific instructions to respondents

39. Guidelines for writing questions
Keep the vocabulary simple
Use simple words
Avoid acronyms, abbreviations, jargon, technical terms, abstract or general words, words with ambiguous meanings
If a technical term is used, its meaning must be explained
General or abstract words – examples must be provided to clarify meaning

40. Keep the question short
Generally, it is recommended to hold questions to 25 words or less
If a longer sentence is used then it should be broken up so that there will be several shorter sentences
Avoid double-barrelled questions
These are single questions that ask for two things and therefore require two answers
Do you have your own table or your own room to do your homework?
Do you think it is good idea for children to study geography and history in primary school?
In such instances, respondents do not know what to do if they want to say ‘Yes’ to one part of the question but ‘No’ to the other

41. Avoid hypothetical questions
Evidence has shown that hypothetical questions such as “Would you use this resource in your class if it were available?” are not good for the prediction of behaviour
People are generally poor predictors of their own behaviour because of changing circumstances and because so many situational variables intervene
You are able to collect more valid data if you ask the respondents about their past behaviour and present circumstances, attitudes, and opinions

42. Do not overtax the respondent’s memory
It is risky to ask the respondent to recall past behaviour over a long retrospective period
Especially when recurrent events or behaviours are concerned
How many severe El-Nino events have occurred since 1961?
If such a question must be asked, a shorter recall period might be more appropriate

43. Avoid double negatives
Double negatives, either in the question or an answer category (or both), create difficulties for the respondent
Customer satisfaction surveys by meteorological agencies should not be allowed, agree or disagree
This is problematic to answer for respondents who are in favour of the surveys, that is those who do not agree that the surveys should not be allowed

44. Beware of ‘leading’ questions
A leading question is a question phrased in such a way that it seems to the respondent that a particular answer is expected
Do you favor inclusion of indigenous knowledge in deriving weather forecasts?
Could be framed as
You wouldn’t say that you were in favor of inclusion of indigenous knowledge in deriving weather forecasts, would you?”
Or in a more subtle form:
Would you say that you are not in favor of inclusion of indigenous knowledge in deriving weather forecasts?

45. Setting Metrics
One of the important elements of performance management is the use of metrics
A metric is a numerical measure used as baseline measurement for performance
Metrics are used to define quantitative measurements for the quality of your service - they are basically a definition of the performance goals that you want to achieve
They depend on the objectives and goals of your program/projects

46. Overall Objective The answer to
Why does your agency exist?
What do you want to achieve?
Should have a clearly defined list of objectives
Defines where you are going
If you don't know where you are going then any road will take you there
Also known as desirable outcomes

47. The objectives must be
Doable
Understandable
Manageable
Beneficial

48. Target Metrics are defined at the start of a project to:
indicate the minimum quality which you want to achieve
define how these factors will actually be measured
When used properly these metrics can provide a useful tool
When used irresponsibly they can be a time-wasting distraction
You need to understand what you are measuring, how you are measuring it and why

49. Key Performance Indicator (KPI)
Key Performance Indicators (KPIs) are metrics that help you understand how you are doing against your objectives
For example, if the objective is Awareness about the products, the KPI could be average number of requests for specific products per month

50. Goals
Specific strategies you'll leverage to accomplish your business objectives
Objectives can be quite strategic and high level – support food security
Goals – very specific
Improve forecasts (can be broken down further)
Create awareness, etc.

51. Baseline information
You can’t solve a problem if you don’t know what exactly your problem is
Information about your problem before you start fixing it or ‘baseline information’ is key
Baseline data is basic information gathered before a program/project begins
It is used later to provide a comparison for assessing program/project impact
It establishes a basis for comparing what was happening before the program with what happens after

52. Monitoring Every program/project has a goal to achieve
Activities implemented should lead to achieving this goal
Implementation of some of the activities may not effectively lead to the goal
Due to some unforeseen circumstances
Constraints
Need to ascertain that the activities produce desired results
Constant collection of data on certain issues

53. Evaluation
At the end of the program/project you need to find out if the objectives have been achieved
Seen through the impacts
Require further data collection targeted at the expected outcomes of your interventions
Gives the picture of what is happening after the program/project intervention

54. Community-based climate services
WMO GFCS
Reaching the users
New approach
Transdisciplinary
Participatory

55. Thank you