Presentation on theme: "MA Quantitative Methods Measuring Instruments Peter Ratcliffe."— Presentation transcript:
MA Quantitative Methods Measuring Instruments Peter Ratcliffe
Concepts and Indicators Key issue: translation of underlying concepts (which are not directly observable/measurable) into indicators (which are) Then assess external (or construct) validity - extent to which the indicator really does adequately ‘measure’ the underlying concept.
Examples Empirical measure of ‘housing classes’ Decomposition of ‘housing needs’ into ‘current’, ‘special’ and ‘future’ (Bradford 1996). [Key concept -‘housing need’: multi- dimensional involving current needs (quality, space, social/physical environment..), special needs (‘cultural’, health/disability....), and future needs (spatial and social mobility, fertility/fecundity, household transformation, net in-migration )]
Reliability and (internal) validity Reliability - replicability of the measurement process, i.e. measuring the ‘same thing’ each time. May relate to question wording, interviewer bias and/or recording error Validity is concerned with whether a particular ‘indicator/variable’ measures what you want it to measure. [In studies repeated at regular time intervals – also question of comparability]
Examples (a) Income Gross or net? Basic or ‘total’? (In cases of variable income) last week or last month? Does it include ‘benefits’? Does it include ‘unearned’ income? etc. (b) Rooms Definition?: Four walls, a ceiling and a floor?! Is R.03/04 (ground floor) one room or two? In the domestic sphere the number of ‘rooms’ important given a concern with overcrowding levels, e.g. is a kitchen a ‘room’, etc.. (c) Ethnic Group [Extremely complex! – next week] So - want BOTH reliability and validity (and, indeed, comparability)
The Structured Interview DEFINITION: Questions are listed on an interview schedule and asked in the same order in the case of each respondent. [Most extreme version - all questions are pre-coded.] Different theoretical perspectives: Positivist Phenomenological Feminist.
Positivist ‘The case for formal interviewing is simple. Only if all respondents are asked exactly the same questions in the same order can one be sure that all the answers relate to the same thing and are strictly comparable. Then, and then only, is one justified in combining the results into statistical aggregates.’ (Moser and Kalton, 1971: 296)
Positivist (cont.) All theoretical approaches stress importance of interaction between interviewer and respondent. Positivists - ‘interaction’ can be modelled statistically, in the following manner: Response = f[ Individual True Value(ITV), Interviewer Effect, Interaction Error]
Positivist (cont.) Solution? Standardise interview process by defining the respective roles of interviewer and respondent: ‘(i)f complete uniformity could be achieved and interviewers acted like machines, answers could be regarded as independent of the way the questions were asked’ (ibid.: 275-6) But could they? This is the classic defence of the structured interview as social scientific method par excellence.
Positivist (cont.) Reliability vital, hence concern devoted to achieving uniformity. Response Errors: how do they occur? Are errors systematic (rather than ‘random’)? How can they be countered? Random Errors, e.g. coding (box-ticking) error - unlikely to be systematic Interviewer Bias - systematic errors Interviewee Bias, may be systematic over the whole sample or simply across a sub-sample (say men/women, young/elderly).
Positivist (cont.) Sources of interviewer error: Interviewer’s opinions Interviewer’s expectations. Not quite so obvious. Three sorts: attitude structure, role expectations, and probability expectations.
Phenomenological One view is that ‘responses obtained are produced in part by dimensions of individual differences relevant to all social encounters.’ (Derek Phillips) Phillips uses a symbolic interactionist (Goffmanesque) approach to analyse the process underlying the survey interview……..
Phenomenological (cont.) Aaron Cicourel is especially critical of conventional positivist accounts of interviewing, in particular those which place great emphasis on fixed-choice schedules or questionnaires. He argues that the latter may be adequate for ‘factual’ information, but for data on social processes they might force a respondent to provide precise responses to events and issues about which s/he may be ignorant or vague.
Phenomenological (cont.) Three core arguments: Documents constitute ‘grids for the distortion of social process’. He asks: How much do we need to know about ‘language and cultural meanings’ and ‘structures of social action’ before a successful questionnaire can be written? What is the role of theory in coding and scaling fixed-choice responses? Much ‘spurious precision’ at the analysis phase of a study.
Feminist Ann Oakley shares the view of writers such as Phillips that role asymmetry is a pivotal issue, BUT goes further in suggesting that this is an explicit replication of male power and patriarchy. Rejects concerns about ‘over-rapport’ Rejects the view of the ideal interview schedule/process as a ‘scientific measuring instrument’ Argues that interviews should be occasions for sharing information, not for the extraction of data from the respondent by the interviewer.
Feminist (cont.) A number of concerns: (1)The implications of her approach for sociological research as a whole? (2) An overly rigid, deterministic division of male/female ‘characteristics’/social roles? (3) An underlying ethnocentrism [who are the ‘we’ when she talks about ‘women’?]
Questionnaire Design Two principal types of document: The Interview Schedule. Interviewer’s interests are paramount. Layout must be well designed and clear; particularly where sections of the schedule apply to only sub- groups of the overall sample. (Mistakes can be both fatal to the research and acutely embarrassing for the interviewer!) Computerisation has helped here. Mail questionnaire. Has to be relatively brief, clear in design and simple to complete.
Questionnaire Design Criteria (1) Should flow like a ‘normal conversation’ Questioning should develop in a logical way, and any change of direction should be flagged up clearly, with some explanation. ‘Classification’ questions should generally come at the end. (2) Certain things should be avoided: vague or leading questions rambling or overlong questionnaires ‘impractical’ questions (e.g. requiring long memory) (3) People need to be able to answer the questions (don’t like admitting ignorance), and willing to do so. (NB. willingness doesn’t imply response validity)
Questionnaire Design Criteria (4) Can broadly distinguish between ‘factual’ and ‘non- factual’ (or opinion) questions ‘Factual’ questions: issue of I.T.V. once again ‘Non-factual’ questions. There are a number of problem areas: May be no clear ‘knowing’ of the answer by the respondent Possibility of no one ‘correct’ answer May wish to measure intensity of opinion (how?) These questions may be particularly sensitive to wording, emphasis, sequence, etc. In general, no possibility of consistency checks (these may be available for ‘factual’ questions). Moser & Kalton suggest checks against ‘measurable’ behaviour (?)
Question wording/Question Type Need to be sufficiently specific (to avoid multi-dimensional response) Use simple language (but not too simple!) Avoid ambiguity, and vague words (fairly, generally, often, why.....) Avoid leading questions Avoid presuming questions (often require a ‘filter’) But, may wish to presume knowledge/behaviour if people are reluctant to discuss certain sensitive matters (?)
Question wording/Question Type Avoid hypothetical questions Need to give special thought to researching embarrassing/difficult issues. Can use indirect questioning, sentence completion, try to increase anonymity, use ‘check lists’ (or ‘randomised response technique’) Take care when studying periodical behaviour Special measures may be required if questions require memory (aided recall techniques, diaries, bounded recall, list of ‘available’ answers....)
Open-ended v. Pre-coded Questions Open-ended questions. Problem areas: loquacity, compression into codes difficulties in achieving verbatim record Pre-coded questions importance of piloting categories need to be mutually exclusive and exhaustive produce ‘forced choice’ answers (cf. Cicourel)
Pre-coded Questions: types A Likert scale: e.g. a range of answers from ‘Strongly agree’ through to ‘Strongly disagree’. A semantic differential scale: A range of positions between two extremes of a continuum (e.g. ‘Caring’ through to ‘Uncaring’). A checklist: e.g. a list of leisure activities. A ranking of items: e.g. placing the most important attributes of a potential partner in order. A choice between statements: e.g. a choice of responses to the knowledge that one’s best friend’s partner is being unfaithful.
(Mail) Questionnaire All questionnaires: Important to give thought to issues of questionnaire length and question order. Even more so for the mail (or self-completion) questionnaire. Latter is best considered in balance sheet format (advantages /disadvantages compared with the interview-based approach)
Advantages Cost Can cover widely dispersed populations Quick (but problem of late returns) Obviously, avoids interviewer errors/biases but.....(?) Useful when intra-household consultation (or considered response) is required Useful when asking embarrassing questions (more truthful responses?) Problem of ‘non-contacts’ eliminated
Limitations Questions need to be sufficiently simple and straightforward Answers need generally to be accepted as final (therefore, lack of flexibility) Inappropriate when spontaneity is required Answers to different questions cannot be seen as independent Cannot be sure the correct person answers (e.g. survey on ‘pig keeping’!) No chance to supplement data with observational material
Principal Limitation Some of these can be overcome by combining the method with interviewing, but this undermines two of the major advantages of the method (speed and low cost). Main limitation is its typically low response rate. Can be as low as 10 per cent. So: What factors affect response rates? How can we improve them?
Response rate issues Must give careful thought to design, content and length of questionnaire Can seek ‘prestigious/respectable’ sponsorship Give special thought to covering letter (including method of salutation, and assurance of anonymity and confidentiality) Include s.a.e. (?) Offer payment for completion (?), and (Most importantly) plan follow-ups carefully
The mail questionnaire and sociological research In sociological research: probably most useful EITHER for the collection of ‘factual’ data OR as part of a multi-phase study to locate ‘rare populations’ (OR BOTH). EXAMPLE: Heidi Mirza, Young, Female and Black
Coding and Data Analysis Coding (even if done pre-fieldwork) is a ‘theoretical’ task. Positivistic ‘solution’ is to go for the apparently ‘objective’ solution. Rather than suggest that the interviewer is the best person to ‘get into the respondent’s head’, Moser and Kalton argue that well trained coders are in the best position to make a dispassionate assessment of what respondents actually meant: ‘.....any summarising or coding can be carried out uniformly in the office, uninfluenced by the circumstances of the interview or the reaction of the respondent to the interviewer’ (p. 342). Do you agree? Coding is also a ‘theoretical’ task in that it needs to reflect the initial hypotheses/research problem.