Presentation on theme: "Designing Questionnaires for Children and Young People"— Presentation transcript:
1Designing Questionnaires for Children and Young People Alice BellFamilies and Children GroupNational Centre for Social Research (NatCen)Royal College of Nursing Conference‘Consulting with Children and Young People’Friday 9th June 2006- Work in F&C Group, at NatCen nearly 5 years- Quant and qual research- Responsibility for annual workshop, which covers quant and qual.- Focus today on quant research - questionnaires. Less common & less written about this.
2Session outline Quantitative and qualitative research When to use questionnaires with children and young peopleDesigning questions for children and young people - the Question and Answer ModelQuestion testing techniquesInteractive quiz!- Will define key differences btwn quant and qual (apologies if already familiar).- This will help explain why a range of things to consider when deciding whether or not to use a questionnaire in your study- Will then move on to talk in some detail about designing questions - present to you the Question Answer model, wh is foundation for good questionnaire design (for adults and CYPs)- Say a bit about question-testing techniques - NatCen pioneered many of these, recently set up Question Testing Unit with exclusive focus.- Finally, get you involved in a quiz, to see whether you’ve learned to recognise a good question for CYPs.
3Two types of research Quantitative Qualitative Choosing which methodology(ies) to use is a key initial decision when planning any study.
4Two types of research Quantitative Qualitative Fixed (questionnaires) Flexible (e.g. topic guides)Quant = questionnaires, more familiar. Can be administered in different ways - will come back to.Qual = covers in-depth interviews; focus groups; observational methods, etc.Tools used in qual research vary.- Typically use a TG (sometimes called “interview schedule”)- Guide to key topics & sub-topics to be covered- Ours usually 6-10 pages long (for hr interview)- May or may not be worded as questions, but not fixed list of Qs- Used flexibly - may only cover part or in different order, may cover topics not on TG (interview/group led by respondent)- We use similar instrument for focus groups
5Two types of research Quantitative Qualitative Fixed (questionnaires) Incidence & prevalenceQualitativeFlexible (e.g. topic guides)Reasons & explanationQuant research QUANTIFIES. Tells you how many, how often etc.Qual research tackles ‘why’ questions- understanding and mapping complexity- focus on rich description through eyes of particular people with different ‘worlds’- searches for emergent concepts and underlying factors
6Two types of research Quantitative Qualitative Fixed (questionnaires) Incidence & prevalenceRepresentative sampleQualitativeFlexible (e.g. topic guides)Reasons & explanationPurposive sampleReason that 2 methodologies serve these different functions.Quant research allows inferences re prevalence in wider population (whole population or some sub-group). Sample ideally drawn randomly. Larger samples required in order to give statistically significant findings (which depend on a combination of the size of the difference and the total sample size).Qual samples much smaller. Don’t seek to represent but rather to reflect key groups within study population. We’d still argue you can infer to wider population, albeit without information on prevalence - but debatable.
7Two types of research Quantitative Qualitative Fixed (questionnaires) Incidence & prevalenceRepresentative sampleData ordered & numericalQualitativeFlexible (e.g. topic guides)Reasons & explanationPurposive sampleData disorderly & rich2 methods result in different types of data:Quant pre-ordered by fixed instrument, reducible to numerical for analysis (even if originally asked as open Qs, can’t do anything with these in raw form with such large samples).Qual sample small, but generates large amount of rich & disorderly data. Various methods of reducing for analysis - coding, classifying etc.
8Choosing an appropriate methodology (1) Traditional leaning towards using qualitative techniques with children and young peopleQuantitative research widely considered feasible from age 7……but age only a rough proxy for various aspects of ‘development’- Qual techniques still only option for young children (under 7), and v young children don’t tend to participate in research, except for various kinds of testing (physical, cognitive - e.g. ‘Child of the New Century’).- Development: cognitive, linguistic, memory, emotional.- Cf later slides on the Question-Answer model - will come back to issue of cognitive capacities required for responding to questionnaires.
9Choosing an appropriate methodology (2) Be clear about your research questionsIdentify your populationConsider time and budgetQuant and qual in combination?- Crucial to be clear about research Qs. Have already mentioned prevalence vs explanation. Specific Qs might also have implications re age and type of participant.- SO next step is to identify and refine the parameters of your sample population. How feasible to conduct quant research, given age and cognitive capacity? Might not want a fixed instrument if you think different CYPs will have very variable grasps of the topic.- Of course, consider time and budget. Qual research usually quicker and cheaper because smaller samples, although some forms of survey research (e.g. paper questionnaires) can be too, and lower costs in terms of researcher time etc., can have several people interviewing using same questionnaire at same time.Quant and qual in combination:If budget, can be ideal.- Qual before, alongside or after survey - different purposes, e.g. to inform design of questionnaire, add case illustrations & depth of description, explain statistical findings.- Can also be challenging, as involve large teams!
10Range of quantitative options Face-to-face interviewing (paper or computer-assisted)Telephone interviewing (paper or computer-assisted)Self-completion questionnaires (paper or computer-based - including online)Nowadays, face-to-face surveys tend to use CAI. Describe CAPI interview - ints follow exact wording (though may need more flexibility than with adults, as CYPs tend to ask more Qs), computer filters them through based on sample info & answers, may use showcards/other stimuli.Telephone sometimes used with CYPs from about age 11, but several disadvantages compared with face-to-face - lack of visual aids, lack of privacy (so can’t ask sensitive Qs), importance of being able to read non-verbal signals.Self-completion can be posted - cheaper - but young children will need help/supervision. Often conducted in schools/group settings which can enable you to get lots of data in short space of time, but issues here - access, who administers, ethical justification (taking away from lessons/breaks), consent (what to do with CYPs who don’t consent), etc.How to make paper questionnaires appealing - smiley faces, pictures, clear layout, colours, etc.
11Question-Answer Model ComprehendquestionRespondRetrieveinformationQuestion-Answer ModelTo understand how awareness of a child’s cognitive development can be used in questionnaire design need to understand the question- answer model which explains how respondents answer survey questions.Four process;Comprehension of the question (understand the terms used)Retrieve the information from memoryMake a judgement about information needed to answer the questionRespond to the questionThis not necessarily a linear process - can be interactions between all these processes (give e.g.)Will now consider each of these processes in turn and look at what you need to be aware of when designing questions for children. Note many of these also apply when designing questions for adults!Judge(Tourangeau 1984)
12Comprehension Vocabulary Question length Ambiguity Double-barrelled questionsDepersonalised/indirect questionsComplex constructionsNegative questionsSuggestively phrased questionsComprehension- is meaning of Q understood in way intended?- does respondent understand what they’re being asked to do?- can respondent keep all the info contained in Q in mind while formulating answer?- generally harder to achieve the younger the age of the respondent, but principles apply to designing questions for all ages.Vocabulary and syntax - simple words (if complex, consider use of examples - although can be disadvs). Appropriate language - not patronising, but not using slang etc.Question length - reliability of answers decrease with increased question length.Ambiguity - ie terms that can be interpreted in different ways in question and response categories - e.g. using “fairly” instead of “quite”. Children can be v literal - de Leeuw’s e.g. of “school trip” versus “class trip”.Double-barrelled - ie contains more than one question e.g. “How would you rate your health and wellbeing?” “Do you like school and if so why?”Depersonalised - children can’t adopt collective perspecitve e.g. “people your age”.Complex constructions- Semi-colons, brackets etc. in self-completion questionnaires.-Too many sub-clauses in read-aloud questions e.g. “Looking at this card, which shows a list of possible reasons why young people sometimes take or consider taking drugs, please tell me which reasons apply to you?”Negative Qs - especially if negative response options eg: “I never do my homework” with agree disagree scale. Just harder to comprehend.Suggestive questions - eg “X is good - agree-disagree”. Younger children can be prone to prone to yay-saying. Consider potential suggestive effects of including examples.
13Retrieval Level of recall Complexity of retrieval process Reference periodsLevel of recall- Easier for rare, distinct or very meaningful/salient events - can also work for very familiar and routine events.- May be partial recall, which can lead respondent to add to memory details which may be false or inaccurate, reality & fantasy more blurred for children so might not be able to distinguish.- NB diary is alternative to some types of retrospective questioning, assuming certain level of literacy (but can be burdensome).Complexity- I.e. Qs that require retrieval of several items of info, and/or some processing of info in order to reach response. E.g: “On weekdays about how much time per day do you spend doing homework?” Need to think about what weekdays are/how much done on average per day/whether this varies from week to week/what counts as homework etc.Children need more time to process questions and retrieve info - beware of them feeling rushed to give an answer, make explicit there’s no hurryReference periods - where recall likely to be good enough to ask Qs about the past, ref periods can help focus the mind. Use specific rather than vague quantifiers (e.g. “last 7 days” rather than “last week”). Can’t expect CYPs to make calculations over long reference periods.
14Judgement Filtering responses - social desirability Context effects ‘No right or wrong answers’Sensitive questions and topicsJudgement - how respondents formulate their answer based the info they have retrieved from memory and how this fits what asked in the questions.Filtering:- Children prone to social desirability (but so are adults!).- Perceptions re confidentiality. CYPs may suspect you’ll tell adults. School settings can affect this too - even if believe answers are confidential, peers will quiz each other afterwards, and might feel more comfortable lying to you than to peers.Context effects- As with adults, responses can be influenced by ordering of Qs - e.g. asking what drugs they do after asking question on punishments for drug-taking will elicit different response to following balanced Q about rightness/wrongness of taking drugs.No rights or wrongs:- Can be afraid to say something wrong, or may assume adult interviewer knows everything already- Emphasise right to refuse, ‘no right or wrong answers’. Children used to being asked by adults for a ‘correct’ answer.- Design self-comps so they don’t look like tests, and take extra care to distinguish if administered in schools (e.g. in written and verbal introductions).Sensitivity of the question/topic:- Impacts on type of methodology you choose:- qual/quant - we do surveys on a number of sensitive issues- self-comp vs interviewer-administered (NB can have self-comp element in interviews, audio CASI etc)) - - Issues of privacy and confidentiality- Be aware that issues that are non-sensitive for some may be sensitive for others
15Response Number of response categories Ordering of response categories Scales and labelsExplicit vs implicit ‘Don’t Know’sShowcardsResponding -understand response options and select the response which best fits their answer.Number of response categories - influenced by child’s reading skills and memory capacity three or four response categories, four or five, 16+ five to seven.Ordering - experiments have shown CYPs likely to choose first on list (esp younger) - might consider putting most likely answer further down or asking one Q per option.Scales and labels-usually best to avoid abstract scales, e.g. 1-10, although pictorially represented versions can be good (smiley to sad faces).-labelling scales improves reliability of responses as clear what each point means.Don’t knows - tricky tension - don’t want CYP to feel pressured to give answer when genuinely don’t have information, but younger children will use explicit DKs as an opt-out.Showcards - can help respondent remember response options. Consider using visual representations.- Bear in mind can’t use where trying to elicit spontaneous responses (e.g. awareness) - these kinds of Qs more difficult for younger respondents anyway, and should certainly be left until respondent ‘warmed up’ (NB don’t tend to get good quality or plentiful spontaneous responses to self-comp; requires effort and affected by writing/typing skills).
16Question testing Sometimes called ‘cognitive testing’ Used to revise and refine questionnaireIn-depth interviews; retrospective probing and ‘think-aloud’ techniquesOther options: expert appraisal, focus groups, observation- Tests all 4 parts of Q-A model.- NatCen build in as standard to development of new questionnaires, or parts of questionnaires.Retro probing- Asking how came to answers afterwards, perhaps after blocks of Qs. Most likely to use with CYPs as quite focused, can ask straightforward Qs, but beware of lack of recall.Think aloud - describe- Can come naturally to children and give useful insights - e.g, if they read question aloud, struggling with a word can indicate a comprehension problem.- However, can be quite demanding on respondent, and requires level of confidence (children may be too shy).Expert appraisal- A cheaper & quicker alternative is to get researchers, experts in the area and others to comment on drafts of qre. Very valuable (can consider resps in all circumstances) but lacks practical ‘testing’ element. Ideal to do both.- ‘Expert appraisal’ can mean something quite formal, and coding schemes specifying where Qs fail to fulfill certain criteria have been developed to assist this (as mentioned by de Leeuw).Focus groupsCan be useful for getting feedback on data collection strategy, mode, structure of questionnaire etc., but not so useful for assessing individual QsObservation (less common in questionnaire testing)Useful for assessing non-verbal signals (more important for children than adults, as don’t express as much in words), but can be ambiguous - e.g. looking away from interviewer could mean boredom, distraction or deep concentration.
17Question testing - example (1) Do you have a personal computer (PC) athome?(Tick one box)YesNoThis is an example of a question that was cognitively tested for use in a paper self completion questionnaire with children aged 11 to 15.At first a seemingly innocent question but the testing revealed problems.- ‘personal’ was interpreted as their own computer and so didn’t include a computer for family use .- Also didn’t understand the term PC in brackets ie didn’t add any further meaning to the term personal computer.
18Question testing - example (1) revised Do you or your family have a computer athome?(Tick one box)YesNo
19Question testing - example (2) How often, if ever, are you punished atschool e.g. detention, sent to see thehead teacher, sent out of the classroom?NeverRarelySometimesOftenSeveral problems:No reference period - children tended to think about different periods from last week to last school year when really the question is trying to get a measure of usual behaviourUse of vague quantifiers - meant difficulty in responding. Children tended to count the number of times they had been punished but then couldn’t decide if that was rarely, sometimes or often?Nowhere to code “hardly ever”List of examples confusing rather than helpful - may not use these forms of punishment in their school.YPs tended to focus on the examples rather than punishment more generally.Younger children didn’t understand “e.g.”
20Question testing - example (2) revised In the last four weeks, how many timeshave you been punished at school?NeverOnce or twiceThree or more timesAdded reference periodRemoved examplesChange to more specific quantifiers for the response options.Still some problems - ideally would do some re-testing.