1. Designing Questionnaires for Children and Young People
Alice Bell
Families and Children Group
National Centre for Social Research (NatCen)
Royal College of Nursing Conference
‘Consulting with Children and Young People’
Friday 9th June 2006
- Work in F&C Group, at NatCen nearly 5 years
- Quant and qual research
- Responsibility for annual workshop, which covers quant and qual.
- Focus today on quant research - questionnaires. Less common & less written about this.

2. Session outline Quantitative and qualitative research
When to use questionnaires with children and young people
Designing questions for children and young people - the Question and Answer Model
Question testing techniques
Interactive quiz!
- Will define key differences btwn quant and qual (apologies if already familiar).
- This will help explain why a range of things to consider when deciding whether or not to use a questionnaire in your study
- Will then move on to talk in some detail about designing questions - present to you the Question Answer model, wh is foundation for good questionnaire design (for adults and CYPs)
- Say a bit about question-testing techniques - NatCen pioneered many of these, recently set up Question Testing Unit with exclusive focus.
- Finally, get you involved in a quiz, to see whether you’ve learned to recognise a good question for CYPs.

3. Two types of research Quantitative Qualitative
Choosing which methodology(ies) to use is a key initial decision when planning any study.

4. Two types of research Quantitative Qualitative Fixed (questionnaires)
Flexible (e.g. topic guides)
Quant = questionnaires, more familiar. Can be administered in different ways - will come back to.
Qual = covers in-depth interviews; focus groups; observational methods, etc.
Tools used in qual research vary.
- Typically use a TG (sometimes called “interview schedule”)
- Guide to key topics & sub-topics to be covered
- Ours usually 6-10 pages long (for hr interview)
- May or may not be worded as questions, but not fixed list of Qs
- Used flexibly - may only cover part or in different order, may cover topics not on TG (interview/group led by respondent)
- We use similar instrument for focus groups

5. Two types of research Quantitative Qualitative Fixed (questionnaires)
Incidence & prevalence
Qualitative
Flexible (e.g. topic guides)
Reasons & explanation
Quant research QUANTIFIES. Tells you how many, how often etc.
Qual research tackles ‘why’ questions
- understanding and mapping complexity
- focus on rich description through eyes of particular people with different ‘worlds’
- searches for emergent concepts and underlying factors

6. Two types of research Quantitative Qualitative Fixed (questionnaires)
Incidence & prevalence
Representative sample
Qualitative
Flexible (e.g. topic guides)
Reasons & explanation
Purposive sample
Reason that 2 methodologies serve these different functions.
Quant research allows inferences re prevalence in wider population (whole population or some sub-group). Sample ideally drawn randomly. Larger samples required in order to give statistically significant findings (which depend on a combination of the size of the difference and the total sample size).
Qual samples much smaller. Don’t seek to represent but rather to reflect key groups within study population. We’d still argue you can infer to wider population, albeit without information on prevalence - but debatable.

7. Two types of research Quantitative Qualitative Fixed (questionnaires)
Incidence & prevalence
Representative sample
Data ordered & numerical
Qualitative
Flexible (e.g. topic guides)
Reasons & explanation
Purposive sample
Data disorderly & rich
2 methods result in different types of data:
Quant pre-ordered by fixed instrument, reducible to numerical for analysis (even if originally asked as open Qs, can’t do anything with these in raw form with such large samples).
Qual sample small, but generates large amount of rich & disorderly data. Various methods of reducing for analysis - coding, classifying etc.

8. Choosing an appropriate methodology (1)
Traditional leaning towards using qualitative techniques with children and young people
Quantitative research widely considered feasible from age 7…
…but age only a rough proxy for various aspects of ‘development’
- Qual techniques still only option for young children (under 7), and v young children don’t tend to participate in research, except for various kinds of testing (physical, cognitive - e.g. ‘Child of the New Century’).
- Development: cognitive, linguistic, memory, emotional.
- Cf later slides on the Question-Answer model - will come back to issue of cognitive capacities required for responding to questionnaires.

9. Choosing an appropriate methodology (2)
Be clear about your research questions
Identify your population
Consider time and budget
Quant and qual in combination?
- Crucial to be clear about research Qs. Have already mentioned prevalence vs explanation. Specific Qs might also have implications re age and type of participant.
- SO next step is to identify and refine the parameters of your sample population. How feasible to conduct quant research, given age and cognitive capacity? Might not want a fixed instrument if you think different CYPs will have very variable grasps of the topic.
- Of course, consider time and budget. Qual research usually quicker and cheaper because smaller samples, although some forms of survey research (e.g. paper questionnaires) can be too, and lower costs in terms of researcher time etc., can have several people interviewing using same questionnaire at same time.
Quant and qual in combination:
If budget, can be ideal.
- Qual before, alongside or after survey - different purposes, e.g. to inform design of questionnaire, add case illustrations & depth of description, explain statistical findings.
- Can also be challenging, as involve large teams!

10. Range of quantitative options
Face-to-face interviewing (paper or computer-assisted)
Telephone interviewing (paper or computer-assisted)
Self-completion questionnaires (paper or computer-based - including online)
Nowadays, face-to-face surveys tend to use CAI. Describe CAPI interview - ints follow exact wording (though may need more flexibility than with adults, as CYPs tend to ask more Qs), computer filters them through based on sample info & answers, may use showcards/other stimuli.
Telephone sometimes used with CYPs from about age 11, but several disadvantages compared with face-to-face - lack of visual aids, lack of privacy (so can’t ask sensitive Qs), importance of being able to read non-verbal signals.
Self-completion can be posted - cheaper - but young children will need help/supervision. Often conducted in schools/group settings which can enable you to get lots of data in short space of time, but issues here - access, who administers, ethical justification (taking away from lessons/breaks), consent (what to do with CYPs who don’t consent), etc.
How to make paper questionnaires appealing - smiley faces, pictures, clear layout, colours, etc.

11. Question-Answer Model
Comprehend
question
Respond
Retrieve
information
Question-Answer Model
To understand how awareness of a child’s cognitive development can be used in questionnaire design need to understand the question- answer model which explains how respondents answer survey questions.
Four process;
Comprehension of the question (understand the terms used)
Retrieve the information from memory
Make a judgement about information needed to answer the question
Respond to the question
This not necessarily a linear process - can be interactions between all these processes (give e.g.)
Will now consider each of these processes in turn and look at what you need to be aware of when designing questions for children. Note many of these also apply when designing questions for adults!
Judge
(Tourangeau 1984)

12. Comprehension Vocabulary Question length Ambiguity
Double-barrelled questions
Depersonalised/indirect questions
Complex constructions
Negative questions
Suggestively phrased questions
Comprehension
- is meaning of Q understood in way intended?
- does respondent understand what they’re being asked to do?
- can respondent keep all the info contained in Q in mind while formulating answer?
- generally harder to achieve the younger the age of the respondent, but principles apply to designing questions for all ages.
Vocabulary and syntax - simple words (if complex, consider use of examples - although can be disadvs). Appropriate language - not patronising, but not using slang etc.
Question length - reliability of answers decrease with increased question length.
Ambiguity - ie terms that can be interpreted in different ways in question and response categories - e.g. using “fairly” instead of “quite”. Children can be v literal - de Leeuw’s e.g. of “school trip” versus “class trip”.
Double-barrelled - ie contains more than one question e.g. “How would you rate your health and wellbeing?” “Do you like school and if so why?”
Depersonalised - children can’t adopt collective perspecitve e.g. “people your age”.
Complex constructions
- Semi-colons, brackets etc. in self-completion questionnaires.
-Too many sub-clauses in read-aloud questions e.g. “Looking at this card, which shows a list of possible reasons why young people sometimes take or consider taking drugs, please tell me which reasons apply to you?”
Negative Qs - especially if negative response options eg: “I never do my homework” with agree disagree scale. Just harder to comprehend.
Suggestive questions - eg “X is good - agree-disagree”. Younger children can be prone to prone to yay-saying. Consider potential suggestive effects of including examples.

13. Retrieval Level of recall Complexity of retrieval process
Reference periods
Level of recall
- Easier for rare, distinct or very meaningful/salient events - can also work for very familiar and routine events.
- May be partial recall, which can lead respondent to add to memory details which may be false or inaccurate, reality & fantasy more blurred for children so might not be able to distinguish.
- NB diary is alternative to some types of retrospective questioning, assuming certain level of literacy (but can be burdensome).
Complexity
- I.e. Qs that require retrieval of several items of info, and/or some processing of info in order to reach response. E.g: “On weekdays about how much time per day do you spend doing homework?” Need to think about what weekdays are/how much done on average per day/whether this varies from week to week/what counts as homework etc.
Children need more time to process questions and retrieve info - beware of them feeling rushed to give an answer, make explicit there’s no hurry
Reference periods - where recall likely to be good enough to ask Qs about the past, ref periods can help focus the mind. Use specific rather than vague quantifiers (e.g. “last 7 days” rather than “last week”). Can’t expect CYPs to make calculations over long reference periods.

14. Judgement Filtering responses - social desirability Context effects
‘No right or wrong answers’
Sensitive questions and topics
Judgement - how respondents formulate their answer based the info they have retrieved from memory and how this fits what asked in the questions.
Filtering:
- Children prone to social desirability (but so are adults!).
- Perceptions re confidentiality. CYPs may suspect you’ll tell adults. School settings can affect this too - even if believe answers are confidential, peers will quiz each other afterwards, and might feel more comfortable lying to you than to peers.
Context effects
- As with adults, responses can be influenced by ordering of Qs - e.g. asking what drugs they do after asking question on punishments for drug-taking will elicit different response to following balanced Q about rightness/wrongness of taking drugs.
No rights or wrongs:
- Can be afraid to say something wrong, or may assume adult interviewer knows everything already
- Emphasise right to refuse, ‘no right or wrong answers’. Children used to being asked by adults for a ‘correct’ answer.
- Design self-comps so they don’t look like tests, and take extra care to distinguish if administered in schools (e.g. in written and verbal introductions).
Sensitivity of the question/topic:
- Impacts on type of methodology you choose:
- qual/quant - we do surveys on a number of sensitive issues
- self-comp vs interviewer-administered (NB can have self-comp element in interviews, audio CASI etc)) - - Issues of privacy and confidentiality
- Be aware that issues that are non-sensitive for some may be sensitive for others

15. Response Number of response categories Ordering of response categories
Scales and labels
Explicit vs implicit ‘Don’t Know’s
Showcards
Responding -understand response options and select the response which best fits their answer.
Number of response categories - influenced by child’s reading skills and memory capacity three or four response categories, four or five, 16+ five to seven.
Ordering - experiments have shown CYPs likely to choose first on list (esp younger) - might consider putting most likely answer further down or asking one Q per option.
Scales and labels
-usually best to avoid abstract scales, e.g. 1-10, although pictorially represented versions can be good (smiley to sad faces).
-labelling scales improves reliability of responses as clear what each point means.
Don’t knows - tricky tension - don’t want CYP to feel pressured to give answer when genuinely don’t have information, but younger children will use explicit DKs as an opt-out.
Showcards - can help respondent remember response options. Consider using visual representations.
- Bear in mind can’t use where trying to elicit spontaneous responses (e.g. awareness) - these kinds of Qs more difficult for younger respondents anyway, and should certainly be left until respondent ‘warmed up’ (NB don’t tend to get good quality or plentiful spontaneous responses to self-comp; requires effort and affected by writing/typing skills).

16. Question testing Sometimes called ‘cognitive testing’
Used to revise and refine questionnaire
In-depth interviews; retrospective probing and ‘think-aloud’ techniques
Other options: expert appraisal, focus groups, observation
- Tests all 4 parts of Q-A model.
- NatCen build in as standard to development of new questionnaires, or parts of questionnaires.
Retro probing
- Asking how came to answers afterwards, perhaps after blocks of Qs. Most likely to use with CYPs as quite focused, can ask straightforward Qs, but beware of lack of recall.
Think aloud - describe
- Can come naturally to children and give useful insights - e.g, if they read question aloud, struggling with a word can indicate a comprehension problem.
- However, can be quite demanding on respondent, and requires level of confidence (children may be too shy).
Expert appraisal
- A cheaper & quicker alternative is to get researchers, experts in the area and others to comment on drafts of qre. Very valuable (can consider resps in all circumstances) but lacks practical ‘testing’ element. Ideal to do both.
- ‘Expert appraisal’ can mean something quite formal, and coding schemes specifying where Qs fail to fulfill certain criteria have been developed to assist this (as mentioned by de Leeuw).
Focus groups
Can be useful for getting feedback on data collection strategy, mode, structure of questionnaire etc., but not so useful for assessing individual Qs
Observation (less common in questionnaire testing)
Useful for assessing non-verbal signals (more important for children than adults, as don’t express as much in words), but can be ambiguous - e.g. looking away from interviewer could mean boredom, distraction or deep concentration.

17. Question testing - example (1)
Do you have a personal computer (PC) at
home?
(Tick one box)
Yes No
This is an example of a question that was cognitively tested for use in a paper self completion questionnaire with children aged 11 to 15.
At first a seemingly innocent question but the testing revealed problems.
- ‘personal’ was interpreted as their own computer and so didn’t include a computer for family use .
- Also didn’t understand the term PC in brackets ie didn’t add any further meaning to the term personal computer.

18. Question testing - example (1) revised
Do you or your family have a computer at
home?
(Tick one box)
Yes No

19. Question testing - example (2)
How often, if ever, are you punished at
school e.g. detention, sent to see the
head teacher, sent out of the classroom?
Never
Rarely
Sometimes
Often
Several problems:
No reference period - children tended to think about different periods from last week to last school year when really the question is trying to get a measure of usual behaviour
Use of vague quantifiers - meant difficulty in responding. Children tended to count the number of times they had been punished but then couldn’t decide if that was rarely, sometimes or often?
Nowhere to code “hardly ever”
List of examples confusing rather than helpful - may not use these forms of punishment in their school.
YPs tended to focus on the examples rather than punishment more generally.
Younger children didn’t understand “e.g.”

20. Question testing - example (2) revised
In the last four weeks, how many times
have you been punished at school?
Never
Once or twice
Three or more times
Added reference period
Removed examples
Change to more specific quantifiers for the response options.
Still some problems - ideally would do some re-testing.