The Lower Order Expectations of High-Stakes Tests: A Four-State Analysis of Social Studies Standards and Test Alignment

Research team – Scott DeWitt, University of Cincinnati, Blue Ash College – Nancy Patterson, Bowling Green State University – Whitney Blankenship, Rhode Island College – Brooke Blevins, Baylor University – Lorrei DiCamillo, Canisius College – David Gerwin, Queens College - CUNY – Jill M. Gradwell, Buffalo State College – SUNY – John Gunn, Queens College – CUNY – Lamont Maddox, University of North Alabama – Cinthia Salinas, University of Texas – Austin – John Saye, Auburn University – Jeremy Stoddard, The College of William and Mary – Caroline C. Sullivan, Georgia State University

Issues with Conventional Testing Assessment to both promote and measure student achievement (NCLB, 2004). Multiple choice questions can range along a continuum of cognitive skills from lower to higher order, but there are no set rules for producing test items targeted to particular cognitive skill levels (Haladyna, Downing, & Rodriquez, 2002).

Alignment as Essential Assessment does not accurately reflect achievement when standards and tests do not match (developing understanding v. rote learning) (Black & Wiliam, 1998).

The Research Gap While high-stakes accountability is premised on the fact that tests measure the extent to which students have met objectives set by state content standards, there are surprisingly few empirical studies that explore the correspondence between test scores and the competencies laid out in state standards (Reich, 2009, p. 326).

The Study To what extent do high-stakes tests hold students accountable for demonstrating the cognitive skills required in the corresponding state academic standards in social studies? – What is the cognitive level of the standards in each state? – What is the cognitive level of the test items in each state? – Across the four states, how well are the tests aligned with the standards?

Authentic Intellectual Work (Newmann, King, & Carmichael, 2007) CRITERIAInstructionAssignmentsStudent Work Construction of Knowledge Disciplined Inquiry Value Beyond School Higher Order Thinking Deep Knowledge Substantive Conversation Connections to the World Beyond the Classroom Construction of Knowledge Elaborated Written Communication Connections to Students Lives Analysis Disciplinary Concepts Elaborated Written Communication

Data Analysis Framework Revised Blooms Taxonomy (Anderson, et al., 2001) – Blooms Taxonomy is widely used by standards writers and test developers to differentiate item expectations and intended cognitive difficulty. The six Taxonomy categories are grouped into Lower Order (Remember and Understand) and Higher Order (Apply, Analyze, Evaluate, Create).

Comparison by State of Standards Ratings NYOHTXVA # Higher Order 14/3224/4264/13430/173 % Higher Order Findings

Norming Guidelines 1.Questions with a document that do not require application of skills beyond comprehension are Lower Order. 2.Chronology questions are Lower Order since the primary expectation is for students to remember events in sequence. 3.Questions whose answers provide only one response related in any way to the prompt are Lower Order. 4.Assume when reading the question that students do not have the content knowledge necessary to answer by remembering. Any question that provides students information that creates a possibility of figuring out the answer, e.g., through analysis, is Higher Order.

Findings Comparison of Multiple Choice Question Ratings NYOHTXVA Global History & Geog US History & Govt. Social Studies World History I World History II US & VA History # Higher Order/ Total 6/505/503/326/553/601/605/60 % Higher Order 12%10%9%11%5%2%8%

Comparison by State of Percentage of Higher- Order Standards to Higher-Order MC Questions NYOHTXVA % Higher- Order Standards % Higher- Order MC Questions 119 5

Conclusions Better tools are needed to authentically assess secondary social studies students learning. The assessments do not allow students to demonstrate a level of cognitive ability matching the expectations of the standards or necessary to become productive citizens.

Conclusions Testing policies vary by: – Defining critical thinking as a distinct testing criterion (OH, TX) – Excluding particular standards from the testing process (VA, NY)*

Comparison by State of Percentage of Higher-Order Standards to Higher-Order MC Questions NYOHTXVA % Higher-Order Standards % Higher-Order MC Questions Testing policies vary by: Defining critical thinking as a distinct testing criterion (OH, TX) Excluding particular standards from the testing process (VA, NY)

Conclusions These high-stakes tests institutionalize lower order expectations for students. Students receive a message that the knowledge that matters in social studies is lower-order remembering and explaining. Narrowing content and focusing on teacher- centered strategies are rational reactions by teachers to their contexts.

Questions & Comments

Norming In recent years, information about foods and food preparation around the world has had a wide audience on U. S. television and on internet websites. People who are eager to try new recipes have created a demand for ingredients not usually produced in the United States. To meet this demand, U.S. supermarkets have been stocking goods formerly unknown to U.S. buyers. What has been the cultural effect of the trend described above? A. a decrease in religious restrictions on particular foods B. a reduction in the numbers of people eating out in restaurants C. the inclusion of additional international food choices in the U.S. diet D. people in foreign countries eating more foods produced in the United States (Ohio Department of Education, 2009a, p. 9)

Norming The federal lands shown above were set aside as a response to concerns about: A.border conflicts B.community development C.environmental conservation D.property values (Texas Education Agency, 2009b, p. 31)

Norming Which heading below best completes the partial outline below? ___________________________________ – Magna Carta – House of Burgesses – Town meetings – John Locke A.Ideas of Social Darwinism B.Basis of British Mercantilism C.Contributions to American Literature D.Influences on United States Constitutional Government (The University of the State of New York, 2010c, p. 2)

Norming In the 1940s, President Franklin D. Roosevelt made winning World War II a priority over extending the New Deal. In the 1950s, President Harry Trumans focus shifted from the Fair Deal to the Korean War. In the 1960s, President Lyndon B. Johnsons attention to the Great Society gave way to preoccupation with the Vietnam War. These presidential actions best support the conclusion that A. Presidents prefer their role as commander in chief to that of chief legislator B. Domestic programs are often undermined by the outbreak of war C. Presidents Roosevelt, Truman, and Johnson were not committed to their domestic initiatives D. Large domestic reform programs tend to lead nations toward involvement in foreign wars (The University of the State of New York, 2010c, p. 7)

Norming Identify four factors (social, political, economic and/or environmental) that contributed to the migration of large numbers of people from Europe to the United States in the late 19th century (Ohio Department of Education, 2009a, p. 5). The Scoring Guide provides the following as the exemplar response: U. S. industrialization offered opportunities for jobs. Better wages were available in the United States. Land was abundant in the United States. Ethnic minorities came to the United States to escape persecution (Ohio Department of Education, 2009b, p. 15).

Norming A prince, therefore, ought always to take counsel, but only when he wishes and not when others wish… also, on learning that any one, on any consideration, has not told him the truth, he should let his anger be felt. – Machiavelli, The Prince Based on this quote, Machiavellis political writings supported the idea of A. absolute power B. limited monarchy C. popular sovereignty D. parliamentary government (Virginia Department of Education, 2007b, p. 34)

All of the following contributed to the trend shown in the graph EXCEPT – a. increasing industrialization B. poor worker-management relations C. discontent with working conditions D. a decline in the population (Virginia Department of Education, 2007a, p. 19)