1. Roadmap Towards a Validity Argument
Peggy Garza
Associate BILC Secretary

2. Views of Validity Traditional Current Adapted from Chapelle (1999)
Validity was considered a characteristic of a test: the extent to which a test measures what it is supposed to measure.
Validity is considered an argument concerning test interpretation and use: the extent to which test interpretations and uses can be justified.
Reliability was seen as distinct from and a necessary condition for validity.
Reliability can be seen as on type of validity evidence.
Validity was often established through correlations of a test with other tests.
Validity is argued on the basis of a number of types of evidence, including the consequences of testing.
Adapted from Chapelle (1999)

3. Validity Argument
Glossary definition: Interpretive argument that presents evidence to make a case justifying the score-based inferences and the intended uses of the test.
Validity
Argument

5. Testing Purpose Develop a Validity Argument for a particular test use
NATO Interoperability Capability Targets
Job Descriptions
Training or exercise requirements
From STANAG 6001:
Participating nations adopt the appended table of language proficiency levels for the purpose of:
Communicating language requirements for international staff appointments.
Recording and reporting, in international correspondence, measures of language proficiency.
Comparing national standards through a standardized table while preserving each nation’s right to maintain its own internal proficiency standards.

6. How are SLPs Officially Used in NATO?
E1101 outlines English Language Capability Targets
For NATO Command Structure staff / participation:
As described by job description, OR
Officers SLP
NCOs SLP
For NATO operations, exercises or training
Officers in command positions and principal staff officers SLP
All other officers SLP
NCOs OR-5 and above likely to have contact with personnel from other nations SLP
Enlisted personnel planned to operate tactical comms, or on NATO comms networks or are members of tactical air control elements SLP

7. Job Description Requirement for English: SLP 2222
What does this really mean?
SLP
Not Qualified 
SLP
Qualified 
Test validation is the process of making a case for the interpretation and uses of test scores.

8. Audience for Validity Argument
Determine the audience for your validity argument
“An argument is made and judged by an audience” Chapelle (2011)
National stakeholders
BILC: IAW sound practices in STANAG 6001 testing

9. Tasks, criteria, instruction
Score need
Test design & development
Administration/ performance
Rating/
evaluation
Score use
Purpose of assessment
Tasks, criteria, instruction
Performance Criteria
Scores
Validity evidence is created and collected throughout the test development cycle.
Starting at the planning phase, tasks, criteria and instructions must correspond to STANAG 6001 descriptors, CTA statements, and established test development guidelines.
Diagram adapted from Luoma 2004

10. Getting started : collecting & organizing evidence to support a validity argument for National STANAG 6001 Tests
Table of Contents:
Test Information for Stakeholders
Testing Personnel Qualifications and Training Records
Outline or Summary of Training Session Content
Test Specifications
Test Moderation Checklists and Documents
Test Validation Documents and Statistical Data
Test Administration Procedures
Test Security Handbook
Records of Norming Sessions for Raters
Statistical Data on Rater Reliability
Records and Summaries of a Priori Research
Records and Summaries of a Posteriori Research
_________________

11. Roadmap Activity Tasks Identify your audience(s)
List evidence or documentation in as many Building Blocks as possible
Share/exchange ideas and examples of evidence

12. Bibliography
Chapelle, C. A. (2012). Validity argument for language assessment: The framework is simple. Language Testing, 29(1), 19–27. Chapelle, C. A. (1999). Validity in language assessment. Annual Review of Applied Linguistics, 19(1), 254–272. Luoma, S. (2004). Assessing Speaking. Cambridge: Cambridge University Press.