1. Good at listening or good at listening tests?
Dr John Field
CRELLA, University of Bedfordshire, UK
Faculty of Education, Cambridge University
ANUPI Huatulco

2. this talk as part of their Professional Development programme
With thanks to Oxford University Press for supporting
this talk as part of their Professional Development programme

3. Building bridges: research into practice
Action research
Teacher investigates own practice
Academic research into ELT / SLA
Methodology
Learner performance / development
Applying research from other domains
General education – Discourse analysis – pragmatics
Sociology – Psycholinguistics – Social psychology
Phonetics and phonology – vocabulary studies
Professional Development

4. Building bridges: research into practice
Action research
Teacher investigates own practice
Academic research into ELT / SLA
Methodology
Learner performance / development
Applying research from other domains
General education – Discourse analysis – pragmatics
Sociology – Psycholinguistics –Social psychology
Phonetics and phonology – vocabulary studies
Professional Development

5. A cognitive approach to testing listening
Good at listening or …?
A cognitive approach to testing listening
What learners tell us about listening tests
What is listening?
Recent developments in the testing of L2 listening
Professional Development

6. A cognitive approach

7. Listening: where teaching and testing merge
Listening is a process taking place in the mind of the listener.
The only way we can find out if understanding has occurred
is indirectly, by asking questions.
This applies to checking understanding in the classroom as well as to designing tests.
So what follows has implications for teachers as well as testers.
Professional Development

8. The need for a cognitive approach
There is a new interest among testers in what goes on in the mind of the test taker.
We need to know whether high-stakes test actually test what they claim to test. Can a listening test, for example, accurately predict the ability of a test taker to study at an English medium university? Work in the health service? Survive as an immigrant?
At a local level, we need to use tests to diagnose learner problems so that the tests can feed into learning. This is especially true of listening.

9. Cognitive validity asks…
Does a test elicit from test takers the kind of process that they would use in a real-world context? In the case of listening, are we testing the kinds of process that listeners would actually use ?
Or do the recordings and formats that we use lead test takers to behave differently from the way they would in real life?

10. Two possible research approaches
A. Ask learners to report on the processes they adopted when taking a test (e.g. by explaining how they got their answers)
B. Use a model of listening that is supported by evidence from psychology. Match the processes produced by a test against the model.

11. Listening to listeners

12. Listening and reporting
A listening passage is paused after about every third (every 70 secs out of 3.5 mins.).
Learners report their answers for this section
Learner explain why they gave the answer.
After a short piece of listening, learners’ recall is fresh and accurate.
Answers serve as triggers to memory.
Professional Development

13. 1
What learners tell us 1
You will hear a talk by a man called Tom about the advantages and disadvantages of golf courses.
Question 13:Tom suggests that golf courses could also be used as ………..
(Source Cambridge CAE: past test)
Professional Development

14. What learners tell us 1
Question: Tom suggests that golf courses could also be used as NATURE RESERVES’
(30) S: number 13 is I’m not sure but um + he said ‘crack’
R: you heard the word ‘crack’?
S: crack …but I don’t know the meaning of ‘crack’
R: er you know it seemed to be an important word
S: yes I think so
R: ok + how did you spell ‘crack’ if if you don’t know the
S: c-r-a-c-k
R: right so you guessed the spelling did you?
S: I guess yes
Most importantly, courses should be designed to attract rather than drive away wildlife.
Professional Development

15. Conclusion: learner behaviour
Learners sometimes simply listen out for prominent words – even if they do not understand them.
This is partly a reflection of their level. At level B1 and below, listeners are very dependent upon picking up salient words rather than chunks or whole utterances.
But this tendency is increased by the use of gap filling tasks, which focus attention at word level.
Professional Development

16. What learners tell us 2
R: is there anything that you heard that helped you?
S: I have the problem about that because I am concentrate on the two of the questions so …I didn’t realise
R: so
S: his his + he’s already go to the 9
R: right ok so you were still listening out for number 8
S: yeah and number 7
R: so that’s why you missed number 9
S: yes actually it’s a lot of problem about how to know where he’s talking
Professional Development

17. Drawbacks
It is common practice to pre-set questions so as to ensure focused listening. Most conventional methods anticipate in writing information that is going to occur in the recording.
Learners form predictions which may shape their listening
It is a convention that questions follow the same order as the recording
Learners often listen out for an answer to Q1; miss it and as a result miss the answer to Q2 as well

18. What a conventional set of test items provides for free
The items in (e.g.) a gap-filling task potentially provide a candidate with:
A summary of what the recording covers
A set of gapped sentences that follow the sequence of the recording
Key words with which to locate information
Sequences which may echo the wording of the recording or the order of words

19. What is listening?

20. Expert listening (Field 2008)
Speech signal
Words
Decoding
Meaning
Word search
Parsing
Meaning construction
Discourse construction 20
Professional Development

21. How close to real–life input?
and the recording
Decoding
Word search
Parsing
Professional Development for IATEFL 2013

22. The recording (vs real-world input)
Scripted (even semi-authentic) recordings bear little resemblance to natural everyday English
Actors mark commas and full stops
There are no hesitations and false starts
Utterances are quite long and regular in rhythm
Voices do not overlap
Test setters sometimes put in distractors – making the recording much more informationally dense than a natural piece of speech would be.

23. Extending listening practice
It is critical to maximise the exposure of learners to spoken L2 sources – ideally authentic ones
Thanks to the internet, there is now a wide range of potential L2 listening materials which can be downloaded and disseminated in MP3 form for:
Classroom practice
Listening homework
Self-study at home
Study centres
Listening for pleasure

24. How close to a real–life task?
and the task
Decoding
Word search
Parsing
Professional Development

25. Evaluating a task: multiple choice
You hear an explorer talking about a journey he’s making. How will he travel once he is across the river?
A. by motor vehicle
B. on horseback
C. on foot
(FCE Handbook, 2008: 60)

26. Recording 1 (FCE Sample Test 1:1)
The engine’s full of water at the moment, it’s very doubtful if any of the trucks can get across the river in this weather. The alternative is to carry all the stuff across using the old footbridge, which is perfectly possible …and then use horses rather than trucks for the rest of the trip all the way instead of just the last 10 or 15 kilometres as was our original intention. We can always pick up the vehicles again on the way down…

27. Recording 1 (FCE Sample Test 1:1)
The engine’s full of water at the moment, it’s very doubtful if any of the trucks can get across the river in this weather. The alternative is to carry all the stuff across using the old footbridge, which is perfectly possible …and then use horses rather than trucks for the rest of the trip all the way instead of just the last 10 or 15 kilometres as was our original intention. We can always pick up the vehicles again on the way down…

28. Conclusion Conventional formats require the listener to:
Map from written information to spoken
Eliminate negative possibilities as well as identify positive ones (esp with MCQ and T/F)
Read and write as well as listen (esp gap filling)
Engage in complex logistical tasks which take us well beyond listening (esp. multiple matching)

29. Some solutions
Provide items after a first playing of the recording and before a second. This ensures more natural listening, without preconceptions or advance information other than general context.
Keep items short. Loading difficulty on to items (especially MCQ ones) just biases the test in favour of reading rather than listening.
Items should not echo words in the recording
Some tasks (e.g. multiple matching) allow items to ignore the order of the recording and to focus on global meaning rather than local detail.

30. The listening / reading conflict
It is important to ensure that a task really practises listening and is not too dependent upon reading.
Visual tasks avoid the problem – but are limited in the complexity of what they can show
Multiple-choice questions make very heavy reading demands. The learner needs to discriminate finely between options and to hold the options n the mind.
Gap filling makes multiple demands – reading, writing and listening.
Solution: Keep questions short or rely on oral questions.

31. Reducing the reading load (Oxford Test of English B level)
Prices and time 0 Tickets cost A £6.50 B £8.50 C £ The trip is on A Friday B Saturday C Sunday 2 The coach leaves school at A 8.30 B 9.30 C At the film studio 3 We’ll see actors performing in A a comedy B a drama C a thriller 4 We’ll have the chance to A use the cameras B talk to actors C try on clothes 5 Visitors can’t take into the studios A bags B phones C sandwiches

32. Does a given test represent all levels of listening?
Speech signal
Words
Decoding
Meaning
Word search
Parsing
Meaning construction
Discourse construction 32
Professional Development

33. Higher levels of listening
Meaning construction (= context).
Take a bare fact and relate it to what was said before
Interpret the attitude or intentions of the speaker
Bring in world knowledge or knowledge of the speaker
Discourse construction
Link pieces of information together
Identify main and minor points
Report the line of argument or overall main point
Professional Development for IATEFL 2013

34. Targets An item in a test can target any of these levels: Decoding:
She caught the (a) (b) (c) (d) 5.50 train.
Lexical search:
She went to London by …….
Factual information:
Where did she go and how?
Meaning construction:
Was she keen on going by train?
Discourse construction.
What was the main point made by the speaker?

35. Targeting levels of listening
In theory, a good test should target all levels of listening in order to provide a complete picture of the test taker’s command of all the relevant processes.
In practice, higher levels of listening may be too demanding in the early stages of L2 study. Novice listeners focus quite heavily on word-level decoding, which does not leave them enough spare attention to give to wider meaning. It is therefore appropriate to focus mainly on factual information.

36. Question focus
Test formats often focus narrowly on one level of listening.
Gap-filling items tend to focus on word recognition
Items in other task types tend to focus on facts.
At higher levels of L2 knowledge, questions rarely tap into more complex levels of listening:
What are the speaker’s intentions or attitude?
What is the overall main point or points?
What is the line of argument that links the facts?
What can we infer that the speaker did not say?
What will the speaker go on to say?

37. Spread of targets

38. The inflexibility of high stakes tests
Large scale high-stakes tests have major constraints which prevent them from testing listening in a way that fully represents the skill.
Reliability and ease of marking
Highly controlled test methods, using traditional formats that the candidate knows
Little attention possible to individual variation or alternative answers

39. Solutions for local testers
Ask questions at discourse level:
What is the main point of the recording? / Give three main points.
What is the connection between Point A and Point B?
Complete a skeleton summary of the text with main points and sub-points
Ask learners to compare two recordings for similarities and differences
Ask learners to summarise a recording orally or in the form of notes (in L1 or L2)

40. Recent developments in the testing of L2 listening

41. Middle and low-stakes tests
Test providers have recently begin to offer middle- and low-stakes tests. These tests can be much more flexible because they are not bound by some of the limitations of high-stakes tests.
They can be used as
Entry tests
Progress tests
End of course tests
Professional selection tests
Diagnostic tests (useful in listening)

42. OTE Level B listening specifications1
5 short monologues
5 MCQ visual
Specific information 2
Longer monologue
Note completion
5 MCQ items
3 option
Specific
information 3
Longer dialogue
(speaker opinion)
5 MM items
Stated opinion
implied meaning 4
5 short monologues / dialogues
attitude/feeling/opinion
gist
function/reason/purpose
speaker relationship
topic
type/genre

43. Oxford Test of English - multiple matching: opinion
You will hear two students talking about a local history project they are doing at college. Match the opinion to the person who says it. Choose the correct answer (the woman, the man, or both) for each statement (1–5). There is one example (0).
0 It would be a good idea to check the research topics with the teacher.
1 The project will be improved by researching what local people did in the past.
2 Preparing questions for the interviews will be useful.
3 It is advisable to interview some extra people.
4 The best people to interview are family members.
5 People who are interviewed should get a copy of the project.

44. Advantages of this kind of task
Items do not need to follow the order of the recording
Items can test opinion – main point – fact – inference etc
Items minimise the reading load.

45. Meaning level questions (OTE B Level)
1. A man and a woman are talking about a concert they have just seen. What did the woman think of the concert?
A The music was too loud. B The tickets were too expensive. C The stage was too far away.
2. A woman and her son are in a newsagent’s at the airport. Which magazine do they choose?
A a football magazine B a computer magazine C a politics magazine
3. Two friends are talking on the phone. What is the man’s job?
A a receptionist B a chef C a waiter
4 A man is leaving a voic message for his friend. Why?
A to complain B to apologize C to suggest something
5. Two friends are talking about a new computer game. What is the boy’s opinion of the game?
A. Not good value. B Not challenging. Not original.

46. Computer delivery
Many test boards offer computer delivered versions of their tests. Unfortunately, they do not always take advantage of the new possibilities that computer delivery offers. In the case of listening, these include:
Improved audio perception through individual headphones
The possibility of delivering the questions after the first listening. This means that the first listening is much more natural and not unduly influenced by seeking answers.
Thanks to an adaptive algorithm, the level of each question presented to the test taker can be based on their success in answering the previous question, so most questions are pitched at the approximate ability of the test taker.

47. References
Field, J. (2008) Listening in the Language Classroom. Cambridge: CUP
Field, J. (2009). The cognitive validity of the lecture listening section of the IELTS listening paper. IELTS Research Reports 9, Cambridge
Field, J. (2013) Cognitive validity. In Geranpayeh, A. & Taylor, L. (eds.) Examining Listening. Cambridge: Cambridge University Press
Oxford Test of English:

48. Thanks for listening!