Good at listening or good at listening tests? Dr John Field CRELLA, University of Bedfordshire, UK Faculty of Education, Cambridge University ANUPI 2013 Huatulco

this talk as part of their Professional Development programme With thanks to Oxford University Press for supporting this talk as part of their Professional Development programme

Building bridges: research into practice Action research Teacher investigates own practice Academic research into ELT / SLA Methodology Learner performance / development Applying research from other domains General education – Discourse analysis – pragmatics Sociology – Psycholinguistics – Social psychology Phonetics and phonology – vocabulary studies Professional Development

Building bridges: research into practice Action research Teacher investigates own practice Academic research into ELT / SLA Methodology Learner performance / development Applying research from other domains General education – Discourse analysis – pragmatics Sociology – Psycholinguistics –Social psychology Phonetics and phonology – vocabulary studies Professional Development

A cognitive approach to testing listening Good at listening or …? A cognitive approach to testing listening What learners tell us about listening tests What is listening? Recent developments in the testing of L2 listening Professional Development

A cognitive approach

Listening: where teaching and testing merge Listening is a process taking place in the mind of the listener. The only way we can find out if understanding has occurred is indirectly, by asking questions. This applies to checking understanding in the classroom as well as to designing tests. So what follows has implications for teachers as well as testers. Professional Development

The need for a cognitive approach There is a new interest among testers in what goes on in the mind of the test taker. We need to know whether high-stakes test actually test what they claim to test. Can a listening test, for example, accurately predict the ability of a test taker to study at an English medium university? Work in the health service? Survive as an immigrant? At a local level, we need to use tests to diagnose learner problems so that the tests can feed into learning. This is especially true of listening.

Cognitive validity asks… Does a test elicit from test takers the kind of process that they would use in a real-world context? In the case of listening, are we testing the kinds of process that listeners would actually use ? Or do the recordings and formats that we use lead test takers to behave differently from the way they would in real life?

Two possible research approaches A. Ask learners to report on the processes they adopted when taking a test (e.g. by explaining how they got their answers) B. Use a model of listening that is supported by evidence from psychology. Match the processes produced by a test against the model.

Listening to listeners

Listening and reporting A listening passage is paused after about every third (every 70 secs out of 3.5 mins.). Learners report their answers for this section Learner explain why they gave the answer. After a short piece of listening, learners’ recall is fresh and accurate. Answers serve as triggers to memory. Professional Development

1 What learners tell us 1 You will hear a talk by a man called Tom about the advantages and disadvantages of golf courses. Question 13:Tom suggests that golf courses could also be used as ……….. (Source Cambridge CAE: past test) Professional Development

What learners tell us 1 Question: Tom suggests that golf courses could also be used as NATURE RESERVES’ (30) S: number 13 is I’m not sure but um + he said ‘crack’ R: you heard the word ‘crack’? S: crack …but I don’t know the meaning of ‘crack’ R: er you know it seemed to be an important word S: yes I think so R: ok + how did you spell ‘crack’ if if you don’t know the S: c-r-a-c-k R: right so you guessed the spelling did you? S: I guess yes Most importantly, courses should be designed to attract rather than drive away wildlife. Professional Development

Conclusion: learner behaviour Learners sometimes simply listen out for prominent words – even if they do not understand them. This is partly a reflection of their level. At level B1 and below, listeners are very dependent upon picking up salient words rather than chunks or whole utterances. But this tendency is increased by the use of gap filling tasks, which focus attention at word level. Professional Development

What learners tell us 2 R: is there anything that you heard that helped you? S: I have the problem about that because I am concentrate on the two of the questions so …I didn’t realise R: so S: his his + he’s already go to the 9 R: right ok so you were still listening out for number 8 S: yeah and number 7 R: so that’s why you missed number 9 S: yes actually it’s a lot of problem about how to know where he’s talking Professional Development

Drawbacks It is common practice to pre-set questions so as to ensure focused listening. Most conventional methods anticipate in writing information that is going to occur in the recording. Learners form predictions which may shape their listening It is a convention that questions follow the same order as the recording Learners often listen out for an answer to Q1; miss it and as a result miss the answer to Q2 as well

What a conventional set of test items provides for free The items in (e.g.) a gap-filling task potentially provide a candidate with: A summary of what the recording covers A set of gapped sentences that follow the sequence of the recording Key words with which to locate information Sequences which may echo the wording of the recording or the order of words

What is listening?

Expert listening (Field 2008) Speech signal Words Decoding Meaning Word search Parsing Meaning construction Discourse construction 20 Professional Development

How close to real–life input? and the recording Decoding Word search Parsing Professional Development for IATEFL 2013

The recording (vs real-world input) Scripted (even semi-authentic) recordings bear little resemblance to natural everyday English Actors mark commas and full stops There are no hesitations and false starts Utterances are quite long and regular in rhythm Voices do not overlap Test setters sometimes put in distractors – making the recording much more informationally dense than a natural piece of speech would be.

Extending listening practice It is critical to maximise the exposure of learners to spoken L2 sources – ideally authentic ones Thanks to the internet, there is now a wide range of potential L2 listening materials which can be downloaded and disseminated in MP3 form for: Classroom practice Listening homework Self-study at home Study centres Listening for pleasure

How close to a real–life task? and the task Decoding Word search Parsing Professional Development

Evaluating a task: multiple choice You hear an explorer talking about a journey he’s making. How will he travel once he is across the river? A. by motor vehicle B. on horseback C. on foot (FCE Handbook, 2008: 60)

Recording 1 (FCE Sample Test 1:1) The engine’s full of water at the moment, it’s very doubtful if any of the trucks can get across the river in this weather. The alternative is to carry all the stuff across using the old footbridge, which is perfectly possible …and then use horses rather than trucks for the rest of the trip all the way instead of just the last 10 or 15 kilometres as was our original intention. We can always pick up the vehicles again on the way down…

Recording 1 (FCE Sample Test 1:1) The engine’s full of water at the moment, it’s very doubtful if any of the trucks can get across the river in this weather. The alternative is to carry all the stuff across using the old footbridge, which is perfectly possible …and then use horses rather than trucks for the rest of the trip all the way instead of just the last 10 or 15 kilometres as was our original intention. We can always pick up the vehicles again on the way down…

Conclusion Conventional formats require the listener to: Map from written information to spoken Eliminate negative possibilities as well as identify positive ones (esp with MCQ and T/F) Read and write as well as listen (esp gap filling) Engage in complex logistical tasks which take us well beyond listening (esp. multiple matching)

Some solutions Provide items after a first playing of the recording and before a second. This ensures more natural listening, without preconceptions or advance information other than general context. Keep items short. Loading difficulty on to items (especially MCQ ones) just biases the test in favour of reading rather than listening. Items should not echo words in the recording Some tasks (e.g. multiple matching) allow items to ignore the order of the recording and to focus on global meaning rather than local detail.

The listening / reading conflict It is important to ensure that a task really practises listening and is not too dependent upon reading. Visual tasks avoid the problem – but are limited in the complexity of what they can show Multiple-choice questions make very heavy reading demands. The learner needs to discriminate finely between options and to hold the options n the mind. Gap filling makes multiple demands – reading, writing and listening. Solution: Keep questions short or rely on oral questions.

Reducing the reading load (Oxford Test of English B level) Prices and time 0 Tickets cost A £6.50 B £8.50 C £10.50 1 The trip is on A Friday B Saturday C Sunday 2 The coach leaves school at A 8.30 B 9.30 C 11.30 At the film studio 3 We’ll see actors performing in A a comedy B a drama C a thriller 4 We’ll have the chance to A use the cameras B talk to actors C try on clothes 5 Visitors can’t take into the studios A bags B phones C sandwiches

Does a given test represent all levels of listening? Speech signal Words Decoding Meaning Word search Parsing Meaning construction Discourse construction 32 Professional Development

Higher levels of listening Meaning construction (= context). Take a bare fact and relate it to what was said before Interpret the attitude or intentions of the speaker Bring in world knowledge or knowledge of the speaker Discourse construction Link pieces of information together Identify main and minor points Report the line of argument or overall main point Professional Development for IATEFL 2013

Targets An item in a test can target any of these levels: Decoding: She caught the (a) 9.15 (b) 9.50 (c) 5.15 (d) 5.50 train. Lexical search: She went to London by ……. Factual information: Where did she go and how? Meaning construction: Was she keen on going by train? Discourse construction. What was the main point made by the speaker?

Targeting levels of listening In theory, a good test should target all levels of listening in order to provide a complete picture of the test taker’s command of all the relevant processes. In practice, higher levels of listening may be too demanding in the early stages of L2 study. Novice listeners focus quite heavily on word-level decoding, which does not leave them enough spare attention to give to wider meaning. It is therefore appropriate to focus mainly on factual information.

Question focus Test formats often focus narrowly on one level of listening. Gap-filling items tend to focus on word recognition Items in other task types tend to focus on facts. At higher levels of L2 knowledge, questions rarely tap into more complex levels of listening: What are the speaker’s intentions or attitude? What is the overall main point or points? What is the line of argument that links the facts? What can we infer that the speaker did not say? What will the speaker go on to say?

Spread of targets

The inflexibility of high stakes tests Large scale high-stakes tests have major constraints which prevent them from testing listening in a way that fully represents the skill. Reliability and ease of marking Highly controlled test methods, using traditional formats that the candidate knows Little attention possible to individual variation or alternative answers

Solutions for local testers Ask questions at discourse level: What is the main point of the recording? / Give three main points. What is the connection between Point A and Point B? Complete a skeleton summary of the text with main points and sub-points Ask learners to compare two recordings for similarities and differences Ask learners to summarise a recording orally or in the form of notes (in L1 or L2)

Recent developments in the testing of L2 listening

Middle and low-stakes tests Test providers have recently begin to offer middle- and low-stakes tests. These tests can be much more flexible because they are not bound by some of the limitations of high-stakes tests. They can be used as Entry tests Progress tests End of course tests Professional selection tests Diagnostic tests (useful in listening)

OTE Level B listening specifications 1 5 short monologues 5 MCQ visual Specific information 2 Longer monologue Note completion 5 MCQ items 3 option Specific information 3 Longer dialogue (speaker opinion) 5 MM items Stated opinion implied meaning 4 5 short monologues / dialogues attitude/feeling/opinion gist function/reason/purpose speaker relationship topic type/genre

Oxford Test of English - multiple matching: opinion You will hear two students talking about a local history project they are doing at college. Match the opinion to the person who says it. Choose the correct answer (the woman, the man, or both) for each statement (1–5). There is one example (0). 0 It would be a good idea to check the research topics with the teacher. 1 The project will be improved by researching what local people did in the past. 2 Preparing questions for the interviews will be useful. 3 It is advisable to interview some extra people. 4 The best people to interview are family members. 5 People who are interviewed should get a copy of the project.

Advantages of this kind of task Items do not need to follow the order of the recording Items can test opinion – main point – fact – inference etc Items minimise the reading load.

Meaning level questions (OTE B Level) 1. A man and a woman are talking about a concert they have just seen. What did the woman think of the concert? A The music was too loud. B The tickets were too expensive. C The stage was too far away. 2. A woman and her son are in a newsagent’s at the airport. Which magazine do they choose? A a football magazine B a computer magazine C a politics magazine 3. Two friends are talking on the phone. What is the man’s job? A a receptionist B a chef C a waiter 4 A man is leaving a voicemail message for his friend. Why? A to complain B to apologize C to suggest something 5. Two friends are talking about a new computer game. What is the boy’s opinion of the game? A. Not good value. B Not challenging. Not original.

Computer delivery Many test boards offer computer delivered versions of their tests. Unfortunately, they do not always take advantage of the new possibilities that computer delivery offers. In the case of listening, these include: Improved audio perception through individual headphones The possibility of delivering the questions after the first listening. This means that the first listening is much more natural and not unduly influenced by seeking answers. Thanks to an adaptive algorithm, the level of each question presented to the test taker can be based on their success in answering the previous question, so most questions are pitched at the approximate ability of the test taker.

References Field, J. (2008) Listening in the Language Classroom. Cambridge: CUP Field, J. (2009). The cognitive validity of the lecture listening section of the IELTS listening paper. IELTS Research Reports 9, Cambridge Field, J. (2013) Cognitive validity. In Geranpayeh, A. & Taylor, L. (eds.) Examining Listening. Cambridge: Cambridge University Press Oxford Test of English: https://elt.oup.com/feature/global/ote

Thanks for listening! jcf1000@dircon.co.uk