Presentation on theme: "Good at listening or good at listening tests?"— Presentation transcript:
1 Good at listening or good at listening tests? Dr John FieldCRELLA, University of Bedfordshire, UKFaculty of Education, Cambridge UniversityANUPI Huatulco
2 this talk as part of their Professional Development programme With thanks to Oxford University Press for supportingthis talk as part of their Professional Development programme
3 Building bridges: research into practice Action researchTeacher investigates own practiceAcademic research into ELT / SLAMethodologyLearner performance / developmentApplying research from other domainsGeneral education – Discourse analysis – pragmaticsSociology – Psycholinguistics – Social psychologyPhonetics and phonology – vocabulary studiesProfessional Development
4 Building bridges: research into practice Action researchTeacher investigates own practiceAcademic research into ELT / SLAMethodologyLearner performance / developmentApplying research from other domainsGeneral education – Discourse analysis – pragmaticsSociology – Psycholinguistics –Social psychologyPhonetics and phonology – vocabulary studiesProfessional Development
5 A cognitive approach to testing listening Good at listening or …?A cognitive approach to testing listeningWhat learners tell us about listening testsWhat is listening?Recent developments in the testing of L2 listeningProfessional Development
7 Listening: where teaching and testing merge Listening is a process taking place in the mind of the listener.The only way we can find out if understanding has occurredis indirectly, by asking questions.This applies to checking understanding in the classroom as well as to designing tests.So what follows has implications for teachers as well as testers.Professional Development
8 The need for a cognitive approach There is a new interest among testers in what goes on in the mind of the test taker.We need to know whether high-stakes test actually test what they claim to test. Can a listening test, for example, accurately predict the ability of a test taker to study at an English medium university? Work in the health service? Survive as an immigrant?At a local level, we need to use tests to diagnose learner problems so that the tests can feed into learning. This is especially true of listening.
9 Cognitive validity asks… Does a test elicit from test takers the kind of process that they would use in a real-world context? In the case of listening, are we testing the kinds of process that listeners would actually use ?Or do the recordings and formats that we use lead test takers to behave differently from the way they would in real life?
10 Two possible research approaches A. Ask learners to report on the processes they adopted when taking a test (e.g. by explaining how they got their answers)B. Use a model of listening that is supported by evidence from psychology. Match the processes produced by a test against the model.
12 Listening and reporting A listening passage is paused after about every third (every 70 secs out of 3.5 mins.).Learners report their answers for this sectionLearner explain why they gave the answer.After a short piece of listening, learners’ recall is fresh and accurate.Answers serve as triggers to memory.Professional Development
13 1What learners tell us 1You will hear a talk by a man called Tom about the advantages and disadvantages of golf courses.Question 13:Tom suggests that golf courses could also be used as ………..(Source Cambridge CAE: past test)Professional Development
14 What learners tell us 1Question: Tom suggests that golf courses could also be used as NATURE RESERVES’(30) S: number 13 is I’m not sure but um + he said ‘crack’R: you heard the word ‘crack’?S: crack …but I don’t know the meaning of ‘crack’R: er you know it seemed to be an important wordS: yes I think soR: ok + how did you spell ‘crack’ if if you don’t know theS: c-r-a-c-kR: right so you guessed the spelling did you?S: I guess yesMost importantly, courses should be designed to attract rather than drive away wildlife.Professional Development
15 Conclusion: learner behaviour Learners sometimes simply listen out for prominent words – even if they do not understand them.This is partly a reflection of their level. At level B1 and below, listeners are very dependent upon picking up salient words rather than chunks or whole utterances.But this tendency is increased by the use of gap filling tasks, which focus attention at word level.Professional Development
16 What learners tell us 2R: is there anything that you heard that helped you?S: I have the problem about that because I am concentrate on the two of the questions so …I didn’t realiseR: soS: his his + he’s already go to the 9R: right ok so you were still listening out for number 8S: yeah and number 7R: so that’s why you missed number 9S: yes actually it’s a lot of problem about how to know where he’s talkingProfessional Development
17 DrawbacksIt is common practice to pre-set questions so as to ensure focused listening. Most conventional methods anticipate in writing information that is going to occur in the recording.Learners form predictions which may shape their listeningIt is a convention that questions follow the same order as the recordingLearners often listen out for an answer to Q1; miss it and as a result miss the answer to Q2 as well
18 What a conventional set of test items provides for free The items in (e.g.) a gap-filling task potentially provide a candidate with:A summary of what the recording coversA set of gapped sentences that follow the sequence of the recordingKey words with which to locate informationSequences which may echo the wording of the recording or the order of words
21 How close to real–life input? and the recordingDecodingWord searchParsingProfessional Development for IATEFL 2013
22 The recording (vs real-world input) Scripted (even semi-authentic) recordings bear little resemblance to natural everyday EnglishActors mark commas and full stopsThere are no hesitations and false startsUtterances are quite long and regular in rhythmVoices do not overlapTest setters sometimes put in distractors – making the recording much more informationally dense than a natural piece of speech would be.
23 Extending listening practice It is critical to maximise the exposure of learners to spoken L2 sources – ideally authentic onesThanks to the internet, there is now a wide range of potential L2 listening materials which can be downloaded and disseminated in MP3 form for:Classroom practiceListening homeworkSelf-study at homeStudy centresListening for pleasure
24 How close to a real–life task? and the taskDecodingWord searchParsingProfessional Development
25 Evaluating a task: multiple choice You hear an explorer talking about a journey he’s making. How will he travel once he is across the river?A. by motor vehicleB. on horsebackC. on foot(FCE Handbook, 2008: 60)
26 Recording 1 (FCE Sample Test 1:1) The engine’s full of water at the moment, it’s very doubtful if any of the trucks can get across the river in this weather. The alternative is to carry all the stuff across using the old footbridge, which is perfectly possible …and then use horses rather than trucks for the rest of the trip all the way instead of just the last 10 or 15 kilometres as was our original intention. We can always pick up the vehicles again on the way down…
27 Recording 1 (FCE Sample Test 1:1) The engine’s full of water at the moment, it’s very doubtful if any of the trucks can get across the river in this weather. The alternative is to carry all the stuff across using the old footbridge, which is perfectly possible …and then use horses rather than trucks for the rest of the trip all the way instead of just the last 10 or 15 kilometres as was our original intention. We can always pick up the vehicles again on the way down…
28 Conclusion Conventional formats require the listener to: Map from written information to spokenEliminate negative possibilities as well as identify positive ones (esp with MCQ and T/F)Read and write as well as listen (esp gap filling)Engage in complex logistical tasks which take us well beyond listening (esp. multiple matching)
29 Some solutionsProvide items after a first playing of the recording and before a second. This ensures more natural listening, without preconceptions or advance information other than general context.Keep items short. Loading difficulty on to items (especially MCQ ones) just biases the test in favour of reading rather than listening.Items should not echo words in the recordingSome tasks (e.g. multiple matching) allow items to ignore the order of the recording and to focus on global meaning rather than local detail.
30 The listening / reading conflict It is important to ensure that a task really practises listening and is not too dependent upon reading.Visual tasks avoid the problem – but are limited in the complexity of what they can showMultiple-choice questions make very heavy reading demands. The learner needs to discriminate finely between options and to hold the options n the mind.Gap filling makes multiple demands – reading, writing and listening.Solution: Keep questions short or rely on oral questions.
31 Reducing the reading load (Oxford Test of English B level) Prices and time 0 Tickets cost A £6.50 B £8.50 C £ The trip is on A Friday B Saturday C Sunday 2 The coach leaves school at A 8.30 B 9.30 C At the film studio 3 We’ll see actors performing in A a comedy B a drama C a thriller 4 We’ll have the chance to A use the cameras B talk to actors C try on clothes 5 Visitors can’t take into the studios A bags B phones C sandwiches
32 Does a given test represent all levels of listening? Speech signalWordsDecodingMeaningWord searchParsingMeaning constructionDiscourse construction32Professional Development
33 Higher levels of listening Meaning construction (= context).Take a bare fact and relate it to what was said beforeInterpret the attitude or intentions of the speakerBring in world knowledge or knowledge of the speakerDiscourse constructionLink pieces of information togetherIdentify main and minor pointsReport the line of argument or overall main pointProfessional Development for IATEFL 2013
34 Targets An item in a test can target any of these levels: Decoding: She caught the (a) (b) (c) (d) 5.50 train.Lexical search:She went to London by …….Factual information:Where did she go and how?Meaning construction:Was she keen on going by train?Discourse construction.What was the main point made by the speaker?
35 Targeting levels of listening In theory, a good test should target all levels of listening in order to provide a complete picture of the test taker’s command of all the relevant processes.In practice, higher levels of listening may be too demanding in the early stages of L2 study. Novice listeners focus quite heavily on word-level decoding, which does not leave them enough spare attention to give to wider meaning. It is therefore appropriate to focus mainly on factual information.
36 Question focusTest formats often focus narrowly on one level of listening.Gap-filling items tend to focus on word recognitionItems in other task types tend to focus on facts.At higher levels of L2 knowledge, questions rarely tap into more complex levels of listening:What are the speaker’s intentions or attitude?What is the overall main point or points?What is the line of argument that links the facts?What can we infer that the speaker did not say?What will the speaker go on to say?
38 The inflexibility of high stakes tests Large scale high-stakes tests have major constraints which prevent them from testing listening in a way that fully represents the skill.Reliability and ease of markingHighly controlled test methods, using traditional formats that the candidate knowsLittle attention possible to individual variation or alternative answers
39 Solutions for local testers Ask questions at discourse level:What is the main point of the recording? / Give three main points.What is the connection between Point A and Point B?Complete a skeleton summary of the text with main points and sub-pointsAsk learners to compare two recordings for similarities and differencesAsk learners to summarise a recording orally or in the form of notes (in L1 or L2)
40 Recent developments in the testing of L2 listening
41 Middle and low-stakes tests Test providers have recently begin to offer middle- and low-stakes tests. These tests can be much more flexible because they are not bound by some of the limitations of high-stakes tests.They can be used asEntry testsProgress testsEnd of course testsProfessional selection testsDiagnostic tests (useful in listening)
42 OTE Level B listening specifications 15 short monologues5 MCQ visualSpecific information2Longer monologueNote completion5 MCQ items3 optionSpecificinformation3Longer dialogue(speaker opinion)5 MM itemsStated opinionimplied meaning45 short monologues / dialoguesattitude/feeling/opiniongistfunction/reason/purposespeaker relationshiptopictype/genre
43 Oxford Test of English - multiple matching: opinion You will hear two students talking about a local history project they are doing at college. Match the opinion to the person who says it. Choose the correct answer (the woman, the man, or both) for each statement (1–5). There is one example (0).0 It would be a good idea to check the research topics with the teacher.1 The project will be improved by researching what local people did in the past.2 Preparing questions for the interviews will be useful.3 It is advisable to interview some extra people.4 The best people to interview are family members.5 People who are interviewed should get a copy of the project.
44 Advantages of this kind of task Items do not need to follow the order of the recordingItems can test opinion – main point – fact – inference etcItems minimise the reading load.
45 Meaning level questions (OTE B Level) 1. A man and a woman are talking about a concert they have just seen. What did the woman think of the concert?A The music was too loud. B The tickets were too expensive. C The stage was too far away.2. A woman and her son are in a newsagent’s at the airport. Which magazine do they choose?A a football magazine B a computer magazine C a politics magazine3. Two friends are talking on the phone. What is the man’s job?A a receptionist B a chef C a waiter4 A man is leaving a voic message for his friend. Why?A to complain B to apologize C to suggest something5. Two friends are talking about a new computer game. What is the boy’s opinion of the game?A. Not good value. B Not challenging. Not original.
46 Computer deliveryMany test boards offer computer delivered versions of their tests. Unfortunately, they do not always take advantage of the new possibilities that computer delivery offers. In the case of listening, these include:Improved audio perception through individual headphonesThe possibility of delivering the questions after the first listening. This means that the first listening is much more natural and not unduly influenced by seeking answers.Thanks to an adaptive algorithm, the level of each question presented to the test taker can be based on their success in answering the previous question, so most questions are pitched at the approximate ability of the test taker.
47 ReferencesField, J. (2008) Listening in the Language Classroom. Cambridge: CUPField, J. (2009). The cognitive validity of the lecture listening section of the IELTS listening paper. IELTS Research Reports 9, CambridgeField, J. (2013) Cognitive validity. In Geranpayeh, A. & Taylor, L. (eds.) Examining Listening. Cambridge: Cambridge University PressOxford Test of English: https://elt.oup.com/feature/global/ote