1. Training course on developing and using questionnaires for agricultural surveys
Field Testing
Field Testing
Marco Ballin
Istanbul, July 2011

2. Field Testing
Field testing methods are those methods used to asses the questionnaire in real conditions
They are usually implemented by a larger number of units than the pre-field tests.
These method can be implemented either in ad hoc test, in a pilot survey or during in ongoing surveys
Istanbul, July 2011 2

3. Interviewer debriefing Respondent debriefing
Field Testing
The following are some of the techniques available to the questionnaire designer to test the questionnaire:
Behavior coding
Interviewer debriefing
Respondent debriefing
Intense interviewers or follow-up interviews
Istanbul, July 2011 3

4. It can be both for CATI and face to face surveys
Field Testing
Behavior coding
It consists in classifying the interviewer/respondent interaction in order to evaluate the quality of the questionnaire.
It can be both for CATI and face to face surveys
With this method the behavior and reaction during the interviews have to be classified in a predetermined coding system
The analysis of the frequencies of reactions to each question can highlight the occurrence of problems due to administration, wording, questionnaire flow, etc
To implement this method you need to train a “coder”; that is a person that analyze the interview and classify the behavior using given scheme
Istanbul, July 2011 4

5. Interviewer question–reading codes
Field Testing
Examples of coding scheme
Interviewer question–reading codes
Indicator
description
Exact
Interviewer reads the question exactly as printed
Slight change
Interviewer reads the question changing a minor word that does not alter question meaning
Major change
Interviewer changes the question such that the question meaning is altered. Interviewer does not complete reading the question.
Istanbul, July 2011 5

6. Respondent behaviour codes
Field Testing
Examples of coding scheme
Respondent behaviour codes
Indicator
description
Interruption with answer
Respondent interrupts initial question-reading with answer.
Clarification
Respondent asks for repetition or clarification of question, or makes statement indicating uncertainty about question meaning
Adequate answer
Respondent gives answer that meets question objective.
Qualified answer
Respondent gives answer that meets question objective, but is qualified to indicate uncertainty about accuracy.
Inadequate answer*
Respondent gives answer that does not meet question
Don’t know
Respondent gives a “don’t know” or equivalent answer.
Refusal to answer
Respondent refuses to answer the question.
Istanbul, July 2011 6

7. To carry out a statistical analysis of the problem met;
Field Testing
This method allows:
To reveal a sizeble number of question problems (the number depends on the coding list categories)
To carry out a statistical analysis of the problem met;
To be implemented in real conditions
Main difficulties and limits are:
It is quite expensive and time consuming
It describe the problems but doesn’t help in finding out the underlying reasons
Istanbul, July 2011 7

8. Interviewer debriefing
Field Testing
Interviewer debriefing
It is a testing method applied to interviewers in order to have their feedbacks on the performance of the questionnaire
It is useful to capture the weakness from the point of view of interviewers point of view
Interviewers can report their feeling about respondets difficulties too
Istanbul, July 2011 8

9. Interviewer debriefing
Field Testing
Interviewer debriefing
The most common debriefing method is the group discussio after the field work
The discussion can be driven by some outlines prepared to cover the most important issues
This method can be easily applied in case of CATI surveys where interviewers are grouped in the same place
In case of face to face interviews, interviewers could be spread all over the country. In this case the debriefing could be carry out using a questionnaire
Istanbul, July 2011 9

10. Respondents debriefing
Field Testing
Respondents debriefing
It is analogue to the Interviewers debriefing method, but it is applied to respondents
Usually this method allow to find out probles caused by:
Wording
Definitions
Length
Clarity
Availability of data sources (important for business surveys)
Istanbul, July 2011 10

11. Respondents debriefing
Field Testing
Respondents debriefing
Most of times this method consists of a self-administered interview
It is necessary that the questionnaire designer are fully aware of the potential problems to develop a set of debriefing questions
Istanbul, July 2011 11

12. Intense Interviews or follow-up interviews
Field Testing
Intense Interviews or follow-up interviews
It is a method that shoul hel in providing information about the hidden processes involved in answering the questions:
This method can complement the behavior code method because it help in understanding the reasons of the problems that it has identified
The method consists in a second semi –structured interview.
The objective is to help the respondent to remind :
how he/she interpret and understood the questions
How he/she arrived to the answer
The importance he/she gave to the question …
Istanbul, July 2011 12

13. Field Testing Discussion
How can we apply some of the hints belonging to these methods in case of lack of resources ?
Istanbul, July 2011 13