World-Class Instructional Design and Assessment Alignment of Standards as a Source of Validity for ACCESS for ELLs® Margo Gottlieb, Ph.D. Lead Developer

Validity is the superstructure that defines testing.

Evidence of validity must be systemic throughout the testing cycle… Evidence of validity must be systemic throughout the testing cycle….and alignment between the components of the system is one source of validity.

Evidence of Alignment in Test Development 1. Second language acquisition theory and research P D R E O V F E E L S O S P I M A E L N T V A L I D T O N 2. WIDA’s English Language Proficiency Standards 3. Test Specifications 4. ACCESS for ELLs®

Theoretical Background: A Model of English Language Proficiency Listening Speaking Contexts of Interaction Support Language Complexity Level of Cognitive Engagement Reading Writing (Gottlieb, 2003)

The English language proficiency standards are aligned to state academic content standards. Social and Academic Language Concepts Knowledge and Skills

The English Language Proficiency standards are defined by a hierarchical structure. Formative and Summative Frameworks f English Language Proficiency Standards Language Domains Grade Level Clusters Levels of Language Proficiency Model Performance Indicators

WIDA’s English Language Proficiency Standards Standard 1: English language learners communicate for social and instructional purposes within the school setting.

Standard 2: English language learners communicate information, ideas, and concepts necessary for academic success in the area of language arts. Standard 3: English language learners communicate information, ideas, and concepts necessary for academic success in the area of mathematics. Standard 4: English language learners communicate information, ideas, and concepts necessary for academic success in the area of science. Standard 5: English language learners communicate information, ideas, and concepts necessary for academic success in the area of social studies.

WIDA’s English Language Proficiency Standards: The Interaction between Language and Content The language of LISTENING SPEAKING READING WRITING LANGUAGE ARTS MATH SCIENCE SOCIAL STUDIES

English Language Proficiency Standard: The Matrix English Language Proficiency Standard: Grade Level Cluster: Level 1 Level 2 Level 3 Level 4 Level 5 Listening Speaking Reading Writing Levels of English Language Proficiency A Model Performance Indicator A strand of Model Performance Indicators

The Second Language Acquisition Continuum Level 1 Entering Level 2 Beginning Level 3 Developing Level 4 Expanding Level 5 Bridging Level 6 Reaching Vertical alignment occurs between contiguous levels of English language proficiency to form a strand of Model Performance Indicators. The criteria used to determine the proficiency level definitions are couched in terms of the language used in schools to impart content area information. Issues of linguistic complexity and semantic & pragmatic knowledge are brought to bear in formulating the definitions. At the two lower proficiency levels, it is assumed that ELLs would need extralinguistic support via graphic and visual aids in order to carry out language functions. This requirement also motivates the use of graphics for test items at these levels.

Performance Definitions Movement along the continuum is based on an English language learner’s increasing Vocabulary Usage- The specificity of words and phrases Linguistic Complexity- The amount and quality of speech or writing Language Control- Comprehensibility based on the amount and types of errors.

A Strand of Model Performance Indicators The language of science Language domain: Speaking or writing Levels of English Language Proficiency Level 1 Level 2 Level 3 Level 4 Level 5 The criteria used to determine the proficiency level definitions are couched in terms of the language used in schools to impart content area information. Issues of linguistic complexity and semantic & pragmatic knowledge are brought to bear in formulating the definitions. At the two lower proficiency levels, it is assumed that ELLs would need extralinguistic support via graphic and visual aids in order to carry out language functions. This requirement also motivates the use of graphics for test items at these levels.

Expectations for English Language Learners Level 1 Level 2 Level 3 Level 4 Level 5 Relate information about weather from photographs, charts, or graphs (such as, ‘It’s hot.’) The criteria used to determine the proficiency level definitions are couched in terms of the language used in schools to impart content area information. Issues of linguistic complexity and semantic & pragmatic knowledge are brought to bear in formulating the definitions. At the two lower proficiency levels, it is assumed that ELLs would need extralinguistic support via graphic and visual aids in order to carry out language functions. This requirement also motivates the use of graphics for test items at these levels.

Level 1 Level 2 Level 3 Level 4 Level 5 Describe weather conditions based on information from photographs, charts, or graphs (such as, ‘I see dark clouds in the sky.’) The criteria used to determine the proficiency level definitions are couched in terms of the language used in schools to impart content area information. Issues of linguistic complexity and semantic & pragmatic knowledge are brought to bear in formulating the definitions. At the two lower proficiency levels, it is assumed that ELLs would need extralinguistic support via graphic and visual aids in order to carry out language functions. This requirement also motivates the use of graphics for test items at these levels.

Level 1 Level 2 Level 3 Level 4 Level 5 Compare and contrast weather information from charts, tables, or visually supported text (such as, ‘It is colder today than yesterday.’) The criteria used to determine the proficiency level definitions are couched in terms of the language used in schools to impart content area information. Issues of linguistic complexity and semantic & pragmatic knowledge are brought to bear in formulating the definitions. At the two lower proficiency levels, it is assumed that ELLs would need extralinguistic support via graphic and visual aids in order to carry out language functions. This requirement also motivates the use of graphics for test items at these levels.

Level 1 Level 2 Level 3 Level 4 Level 5 Match predictions about weather from graphs, charts, or visually supported text (such as, ‘I think that tomorrow it will’ ….) The criteria used to determine the proficiency level definitions are couched in terms of the language used in schools to impart content area information. Issues of linguistic complexity and semantic & pragmatic knowledge are brought to bear in formulating the definitions. At the two lower proficiency levels, it is assumed that ELLs would need extralinguistic support via graphic and visual aids in order to carry out language functions. This requirement also motivates the use of graphics for test items at these levels.

Level 1 Level 2 Level 3 Level 4 Level 5 Make inferences about weather based on modified grade level material (such as, ‘It seems as if it might…’) The criteria used to determine the proficiency level definitions are couched in terms of the language used in schools to impart content area information. Issues of linguistic complexity and semantic & pragmatic knowledge are brought to bear in formulating the definitions. At the two lower proficiency levels, it is assumed that ELLs would need extralinguistic support via graphic and visual aids in order to carry out language functions. This requirement also motivates the use of graphics for test items at these levels.

Assessing Comprehension and Communication in English State to State for English Language Learners ACCESS for ELLs®

ACCESS for ELLs® is built on: A theoretical model of English language proficiency Social and academic language proficiency across standards Five levels of language proficiency Four language domains Developmentally appropriateness for each grade level cluster Second language acquisition research and best practice

The purposes of ACCESS for ELLs® … to provide one data source for: Identifying initial placement of English language learners (ELLs) on the L2 acquisition continuum (W-APT) Monitoring annual progress of ELLs toward meeting English language proficiency standards Marking attainment of ELLs of full English language proficiency Complying with the requirements of No Child Left Behind

Samples of alignment from a string of Model Performance Indicators, representative of a standard, to ACCESS for ELLs® items.

Sample Items from a Theme (from the Pilot test) English Language Proficiency Standard: The language of science Language Domain: Listening Grade Level Cluster: 1-2 Remember that all sample items presented here are themes and items that appeared on the pilot test. These items do not appear on the operational test. The sample items give a good sense of what themes and items are like and how they are aligned to the standards and performance indicators, but for a variety of reasons, they did not survive to become operational. Therefore, criticisms of the content, theme, possible bias, or other observations about these specific items are not particularly useful. However, overall comments and suggestions are always welcome.

Model Performance Indicator (PI) for Language Proficiency Level 1 PI: “identify objects according to chemical or physical properties from pictures and oral statements” SCRIPT: “A seed is small. Find the small seed.” These slides illustrate the direct relationship between the performance indicators (PIs) from the large-scale assessment framework of the standards and the items. This slide and the following two show how items of contiguous proficiency levels are located within a single theme.

Model Performance Indicator for Language Proficiency Level 2 PI: “match objects with their chemical or physical properties from pictures and oral statements” SCRIPT: “One day the seed will grow into something large, round, and heavy. Find what the seed grows into.”

Model Performance Indicator for Language Proficiency Level 3 PI: “identify and group objects according to chemical or physical properties from oral statements” SCRIPT: “Seeds grow into plants. Find something else that grows.”

Validation of the construct of English language proficiency, from theory to standards to specifications to test items…

….is a set of interlocking, interdependent gears within an aligned system. Academic content standards English language proficiency standards ACCESS for ELLs®