1. Using Verbal Reports for Data Collection and Analysis
Maja Gjurovikj
STANAG 6001 testing team
Ministry of Defence
Republic of Macedonia

2. Why trialling? Collect data to be analyzed:
Do items test what they were meant to test?
Do they distinguish between masters or non- masters?
Are the ambiguous?
Inform about test administration procedures:
time
instructions
answer keys
Bank items to generate new tests:
determine characteristics of items
keep/revise/reject

3. Test item characteristics
CODE
DATA FROM
AS NO.
GROUP A B C D ? FV DI
IMMIGRATION
Pretesting Feb , Perugia 32 T 4 13
26%
-0,36 M 7 10 12 2 %
36%
24%
13% 0%
CODE
DATA FROM
AS NO.
GROUP A B C D ? FV DI
CAR ALARMS 2
PretestingOct. 2011, BiH 40 T 3 17 4 5 2
36%
0,30 M 8 10 6 7 %
14%
16%
21%
12%

4. Psychometric approaches vs. mentalistic approaches
empirical validity
theoretical validity
concurrent validity
predictive validity
QUANTITATIVE APPROACHES
EXAMINE PSYCHOMETRIC PROPERTIES OF THE TEST
face validity
content validity
construct validity
QUALITATIVE APPROACHES PROVIDE MORE INFORMATION ON TEST CONTENT, TEST TASK PROPERTIES AND TEST- TAKING PROCESSES

5. What is verbal report analysis?
a special label used to describe the verbal data gathered
a technique that draws inferences about the cognitive processes that produced the verbalization
comprised of ‘protocols’ - utterances made by an individual
invaluable at the trialling and pretesting stage
supplement other quantitative data

6. Variations on verbal report procedure
Form of Report TALK ALOUD THINK ALOUD
Temporal Concurrent Retrospective Concurrent Retrospective
Variations
Procedural Mediated Non-med. Mediated Non-med. Mediated Non-med. Mediated Non-med.

7. Stages in collecting and analyzing verbal reports
DATA COLLECTION STAGES
Task identification
Task analysis
Selecting an appropriate procedure
Selecting subjects
Training subjects
Collecting verbal reports
Collecting supplementary data
Transcribing verbal reports
DATA ANALYSIS STAGES
Developing an encoding scheme
Segmenting protocols
Encoding protocols
Calculating encoder reliability
Analyzing data
Source: Green, A. (1998). Verbal Protocol Analysis in Language Testing Research: A Handbook. Cambridge: Cambridge University Press and University of Cambridge Local Examinations Syndicate

8. Examples of verbal report methodology
Cohen (1994)
five test-takers
a two-part test
concurrent reports
observe test-takers during the test-taking process
give test-takers a questionnaire
conduct in the test-takers’ L1
Buck (1994)
six test-takers
54-item test based on a single listening text
13 short sections, one at a time
attend a post-test interview
take the test items again

9. Advantages vs. disadvantages
reveals test validity
provides direct information
has wide application
commitment in time
disruption of behaviour
individual differences

10. Some qualitative data analysis softwares
Coding Analysis Toolkit (CAT; 
Weft QDA (
Text Analysis Markup System (TAMS; 
Transana (
QDA Miner Lite ( data-analysis-software)

11. Points to note reveals the dynamics of the test taking processes
uncovers students’ unobservable behavior
resourceful feedback concerning the mental processes or test-taking strategies
complements the analysis generated from quantitative investigation
a developing and maturing method

12. References: (https://www.youtube.com/watch?v=BwpPIiBK0cA)
Green, A. (1998). Verbal Protocol Analysis in Language Testing Research: A Handbook. Cambridge: Cambridge University Press and University of Cambridge Local Examinations Syndicate.
Grotjahn, R. (1986). Test validation and cognitive psychology: Some methodological considerations. Language Testing 3: 159–185.
Zheng, Y. (2009) Protocol Analysis in the Validation of Language Tests:
Potential of the Method, State of the Evidence. International Journal of Pedagogies and Learning 5: 124–137
Council of Europe (2009). Relating Language examinations to the Common European Framework of Reference for Languages: learning, teaching, assessment Cambridge: Cambridge University Press. (