Presentation is loading. Please wait.

Presentation is loading. Please wait.

Understanding Standardized Testing: A Deeper Look

Similar presentations


Presentation on theme: "Understanding Standardized Testing: A Deeper Look"— Presentation transcript:

1 Understanding Standardized Testing: A Deeper Look

2 Percentage of graduates to the population at the typical age of graduation (1995, 2000, 2005)

3 In response to slipping student and school performance, there is a growing emphasis on establishing greater accountability. Testing plays a more important role than ever in this effort, by setting higher standards and accurately measuring performance. The SAT is the premier college admissions test and a critical component in the American educational system. Like the problems, the solutions are complex. One key to the solution to slipping school and student performance is a growing emphasis on accountability Testing plays a more important role than ever in this effort, by setting higher standards and accurately measuring performance

4 History and Background
10b_1585_SATforRAS History and Background 4/19/2017 Developed more than 80 years ago as a tool to help democratize higher education access for all students, the SAT ® has grown to become the world’s most widely used college entrance exam. The SAT continues to evolve and improve to meet the needs of a dynamic education landscape. However, the SAT has always served the primary mission of the College Board: to connect students to college opportunity and success. As a not-for-profit membership organization, the College Board and its programs (including the SAT) are backed by a diverse group of institutions with broad educational experience. The College Board’s membership includes more than 5,700 institutions: secondary schools, colleges and universities located in more than 50 countries throughout the world. How has the SAT evolved? The SAT was evolved from the original College Board college entrance exams. Founded in 1900, the College Board was organized to facilitate the successful transition of secondary school students to higher education. Prior to those tests, each American university had a separate entrance test. Why work with the College Board? The College Board is responsible for having developed and guided an evolving, yet highly valid, admission test that provides a level playing field for students of all backgrounds. Your institutions have been, and will continue to be, concerned with reaching more and different types of students -- the SAT has been one way that the College Board has responded to this need.

5 What is the SAT®? A Measure of College Readiness
10b_1585_SATforRAS 4/19/2017 What is the SAT®? A Measure of College Readiness The SAT evaluates the cognitive tools necessary to succeed in college and beyond, including the ability to: Think critically Solve problems Communicate effectively A Predictor of College Outcomes Is a valid predictor of meaningful college outcomes on a student’s path to a college degree, including: Freshman Year GPA 2nd, 3rd and 4th year GPAs College Retention The goal of the SAT is simple: to measure what students have learned in school, and how well they apply that knowledge. It’s based on the subjects and skills that students study in high school classrooms every day. It’s developed by teachers and educators. Periodic curriculum surveys validate that the test measures what’s learned in high school and what’s expected in college. The SAT’s Mathematics section is less limiting than other multiple-choice exams: “grid-ins” allow for more than one correct answer. The writing section also allows students to express their learning in another format.

6 What’s On The Test… Critical Reading 200 - 800 Mathematics Writing
The SAT® assesses the core academic skills necessary for college success and how students apply those skills. 44 multiple-choice items, 10 student- produced response items Number & Operations, Algebra, Geometry, Data Analysis 25 minute essay, take sides on an accessible topic 49 multiple-choice grammar/usage questions Essay sub-score 2-12 Short (paragraph) and long (up to 800 word) passages 48 questions, make inferences, draw conclusions 19 sentence completion questions Critical Reading Mathematics Writing Developing rigorous, evidence-based and internationally benchmarked college and career readiness standards in English language arts and mathematics is the goal of the Common Core State Standards initiative. For more than 80 years, the SAT has been a proven, valid and respected standardized measure of the English language arts and mathematics skills and knowledge that students need to succeed in college. The College Board is finishing a study that examines the relationship between the Common Core State Standards and the SAT®. This slide outlines the skills and knowledge topics that are tested on the SAT. This study will show that each and every one of the SAT skills and knowledge topics is represented in the Common Core State Standards. This means that states and districts adopting the Common Core State Standards can be confident that their students will be acquiring the skills and knowledge necessary to succeed on the SAT.

7 Every SAT® Knowledge and Skills Topic is represented in the Common Core State Standards
The knowledge and skills covered on the SAT are directly linked to what students are learning in the classroom SAT Critical Reading Knowledge and Skills Topics SAT Mathematics Knowledge and Skills Topics SAT Writing Knowledge and Skills Topics Determining the Meaning of Words Author’s Craft Reasoning and Inference Organization and Ideas Understanding Literary Elements Number & Operations Algebra & Functions Geometry & Measurement Data, Statistics & Probability Problem Solving Representation Connections Communication Manage Word Choice and Grammatical Relationships Between Words Manage Grammatical Structures Used to Modify or Compare Manage Phrases and Clauses in a Sentence Recognize Correctly Formed Sentences Manage Order and Relationships of Sentences and Paragraphs Developing rigorous, evidence-based and internationally benchmarked college and career readiness standards in English language arts and mathematics is the goal of the Common Core State Standards initiative. For more than 80 years, the SAT has been a proven, valid and respected standardized measure of the English language arts and mathematics skills and knowledge that students need to succeed in college. The College Board is finishing a study that examines the relationship between the Common Core State Standards and the SAT®. This slide outlines the skills and knowledge topics that are tested on the SAT. This study will show that each and every one of the SAT skills and knowledge topics is represented in the Common Core State Standards. This means that states and districts adopting the Common Core State Standards can be confident that their students will be acquiring the skills and knowledge necessary to succeed on the SAT. Source: Vasavada, N., Carman, E., Hart, B. Luisier, D.; Common Core State Standards Alignment: readiStep™, PSAT/NMSQT® and SAT; College Board, 2011

8 Students enrolled in a core curriculum perform better on the SAT®
Students who take a core curriculum in high school perform better on the SAT and are better prepared for college than students who do not. SAT® Mean Scores by Curriculum +48 points +47 points Score Core curriculum is defined by at least four years of English, and at least three years of mathematics, three years of natural science, and three years of social science and history. We cannot overstate the importance of rigorous coursework in preparing for success in college. College success takes hard work. It’s a simple point – the best way to prepare for success in college and success on the SAT is to work hard and take rigorous courses. This chart shows how much higher students who took a core curriculum scored on each section of the SAT points in reading; +47 points in math; and +48 points in writing. Source: College-Bound Seniors 2011 Cohort Data

9 Students enrolled in rigorous courses perform stronger on the SAT ®
Not surprisingly, we see that students who took the most rigorous English courses scored higher on the SAT. Students who took AP or honors English scored +59 point higher on reading; +46 points on math; and +58 points on writing. Students enrolled in rigorous courses perform stronger on the SAT ® Students enrolled in AP ® or Honors English outperform the general SAT populations in all sections of the SAT. +59 points +46 points +58 points Score Source: College-Bound Seniors 2011 Cohort Data SAT Mean Scores by AP or Honors English Participation Not surprisingly, we see that students who took the most rigorous English courses scored higher on the SAT. Students who took AP or honors English scored +59 point higher on reading; +46 points on math; and +58 points on writing.

10 Students enrolled in rigorous courses perform stronger on the SAT ®
Students enrolled in AP ® or Honors Mathematics outperform the general SAT populations in all sections of the SAT. +64 points +76 points Score SAT Mean Scores by AP or Honors Math Participation We see here that students who took the most rigorous math courses scored higher on the SAT. Students who took AP or honors math scored +64 point higher on reading; +76 points on math; and +64 points on writing. Source: College-Bound Seniors 2011 Cohort Data

11 High school grades are increasing over time
The need for a consistent national measure is more important than ever as high school grades have been increasing over time. Percentage of Students by Self-Reported High School GPA 1991 1996 2001 2006 2011 Grade Average for All Subjects* 3.10 3.20 3.28 3.33 3.34 *Based on four-point system, where A=4.0 Note: 1990 GPAs reflect both SAT Subject Test™ takers and SAT® takers. GPAs for reflect SAT® takers only.

12 West Ed Alignment | Summary of Findings
Competency SAT Alignment ACT Alignment UC – Academic Literacy Critical Reading: 90% Strong, 10% Implicit (B8) Writing: 100% Strong (B8) Essay: 3 standards aligned Reading: 85% Strong, 13% Implicit, 2% None English: 100% Strong UC – Mathematics Mathematics: 85% Strong, 13% Partial, 2% Implicit Mathematics: 82% Strong, 12% Partial CA – English Language Arts (Grades 9-10) Critical Reading: 66% Strong, 34% None (B8) Essay: 5 standards aligned Reading: 25% Strong, 8% Partial, 67% None CA – English Language Arts (Grades 11-12) Critical Reading: 37% Strong, 63% None (B1) Writing: 98% Strong, 2% None (B1) Reading: 40% Strong, 60% None English: 97% Strong, 3% None CA – Mathematics Mathematics: 63% Strong, 35% Partial, 2% None SAT II 3YBC: 82% Strong, 12% Partial, 6% None Mathematics: 77% Strong, 22% Partial, 1% None CA – Science SAT II Physics: 57% Strong, 23% Partial, 20% None SAT II Chemistry: 58% Strong, 32% Partial, 10% None SAT II Biology: 39% Strong, 17% Partial, 44% None Science: 0% Strong, 3% Partial, 97% None *All information taken from WestEd SAT/ACT Alignment Study Let’s compare how well the SAT and ACT align with curriculum… In several rigorous studies, the SAT has been shown to align as well (or better) than the ACT with state learning standards California (WestEd) Maine (Norm Webb) Michigan (Norm Webb) HS Teachers and college faculty, working with the College Board, have developed Standards for College Success, with which the SAT is firmly aligned

13 SAT® Participation

14 More students are planning for college as evidenced by the increase in SAT ® participation
The SAT reaches more students than ever before while the number of graduating high school seniors in the U.S. has decreased. SAT Participation Relative to U.S. High School Graduates SAT Participation by Cohort US SAT Takers US High School Graduates There is good news that the universe of students participating in the college planning process is expanding – 1,647,123 students in the Class of 2011 took the SAT. The largest number of SAT testers in history. Over the past year, the number of SAT testers has increased while the number of students graduating high school has decreased. Starting in 2011 all SAT cohort populations included members of that class who took the SAT for the first time in May or June of that year. All previous cohorts have been recalculated to included first-time May and June senior SAT takers. Source: 2011 College-Bound Seniors Total Group Report; WICHE (2008)

15 The SAT® Is reaching more underserved students
The SAT continues to reach more underserved students – with significant increases over the past five years. The SAT® Is reaching more underserved students The SAT is reaching more low-income and first-generation students who are traditionally underserved in the college-going process. 2011 SAT Takers  Beneficiaries of the SAT Fee-Waiver Program 2011 SAT Takers  Prospective First-Generation College Goers 45% of all SAT-takers are first generation. 21% of all test takers used a fee waiver. The SAT continues to reach more underserved students – with significant increases over the past five years. Taking a college entrance exam is a critical step on the road to higher education, but many traditionally underserved students face additional financial, familial and geographic barriers that can prevent them from testing. Through the College Board’s SAT Fee-Waiver Program, the SAT is extending its reach to help more of these college-aspiring students take the SAT than ever before. Since 1970, the College Board has provided SAT fee waivers to low-income students for whom exam fees would present an unnecessary challenge in the college-going process. With the assistance of high school counselors throughout the country, the College Board’s SAT Fee-Waiver Program is making it possible for more low-income students than ever before to get on the road to college. More than 350,000 students in the class of 2011 took the SAT for free via the SAT Fee-Waiver Program, which represents more than 21 percent of all test-takers. The number of college-bound seniors who benefitted from SAT fee waivers increased nearly 77 percent since 2007 (from 198,729 students in the class of 2007 to 351,068 students in the class of 2011). During the academic year, the College Board provided more than $37 million in SAT fees and services to students in grades 9– 12. 77% increase in SAT Fee-Waiver usage since 2007 Represents more than $37 million in fees and services Source: 2011 College-Bound Seniors

16 The SAT ® is the most diverse U.S. college entrance exam
43% of SAT takers report an ethnic or racial background other than Caucasian 11% Asian 13% Black 15% Hispanic/Latino 4% Other 53% White No Response 1% American Indian 2011 College-Bound Seniors by Race/Ethnicity Indian or Alaska Native 9,244 Asian, Asian American, or Pacific Islander 183,853 Black or African American 215,816 Mexican or Mexican American 99,166 Puerto Rican 26,520 Other Hispanic, Latino, or Latin American 127,017 White 865,660 Other 58,699 No Response 61,148 The Class of 2011 is the most diverse group of SAT testers in history. Note: Due to rounding, percentages do not add up to 100. Source: 2011 College-Bound Seniors

17 Minority SAT ® takers are increasing at a faster rate than U. S
Minority SAT ® takers are increasing at a faster rate than U.S. public high school grads The growth rate of SAT takers is highest for racial and ethnic minority sub-groups Growth rate of SAT takers 5% over 2007, Growth rate of U.S. public high school graduates 1% over 2007. Source: 2011 College-Bound Seniors Total Group Report; WICHE (2008)

18 SAT® reflects the diversity of the nation’s classrooms
SAT participation closely reflects the distribution of minority students graduating from U.S. public schools. US Public School SAT Takers Graduates SAT participants reflect the diversity present in our nation’s classrooms. Public US SAT Test Takers only. 84% of SAT Takers report attending public school Source: 2011 College-Bound Seniors Total Group Report; WICHE (2008)

19

20 SAT ® Validity

21 Why is Predictive Validity important?
Predictive Validity refers to the ability of a factor (e.g. test scores) to predict future performance. For colleges and universities, it is important to understand how well factors used for college admissions or placement predict desired academic outcomes Factors (predictors) - typically include high school grades, SAT ® scores or class rank Academic Outcomes – GPA, retention, course grades The measurement of how well predictors do is usually expressed as a correlation (from +1.0 to -1.0) Necessity for colleges to do this – way to prove that your process Is working accurately and throroughly Yourself or college board can conduct these studies.

22 National SAT ® Validity Study
Data supplied by four-year institutions from around the U.S. and matched to College Board data. 110 institutions 216,081 students Broad institutional characteristics Size Public/Private Geographic distribution Selectivity Cross-institutional, longitudinal validity and higher education research informing ways to ensure that students are ready for and successful in college. It is the responsibility of the test developer to provide information and research that shows that the assessment predicts the performance it is intended to predict. You may ask what does the SAT do? What does it measure? The answer is, simply: the SAT predicts college outcomes. If you want to know whether a student will do well in college, you can look at his or her SAT score and get a good idea. To say the exam has predictive validity is to say it works for its intended purpose. CB has partnered w schools to form a HED database. For the 2007 sample, a total of 110 institutions provided data, covering 216,081 students. Mattern et al. (2010). * College Board Research and Development Database, 2007 Cohort

23 The SAT ® is a strong predictor of first year college performance
10b_1585_SATforRAS 4/19/2017 The SAT ® is a strong predictor of first year college performance The SAT continues to predict just as well as high school grades. When used together, grades and SAT scores are the best predictors of college performance Correlation* of SAT® and High School GPA to First-Year College GPA Predictor Correlation SAT Mathematics .49 SAT Critical Reading .50 SAT Writing .53 Combined SAT (CR + M + W) .56 High School GPA SAT Total + High School GPA .64 CB has partnered w schools to form a HED database. For the 2007 sample, a total of 110 institutions provided data, covering 216,081 students. Mattern et al. (2010). In this study, high school grades are a pretty good predictor of first-year college GPA (.56). The SAT is equally as good (.56). However, when used together, grades and SAT scores are the best predictors of FYGPA (.64). Why do we do that? To show how well high school grades and SAT scores actually differentiate among all students Isn’t that using statistics to prove your point? Extreme example: College that accepts only students with 4.0 GPA and 2400 SAT Correlation between FYGPA and HSGPA = 0.0 Correlation between FYGPA and SAT = 0.0 Each section is a valid and strong predictor of college performance The combined SAT predicts as well as high school GPA The SAT in combination with HSGPA is the best predictor of college performance * Correlations corrected for restriction of range Source: Patterson, B.; Mattern, K.; Kobrin, J.; Validity of the SAT for Predicting FYGPA: 2007 SAT Validity Sample; College Board, 2011

24 What does a correlation of .56 mean?
10b_1585_SATforRAS 4/19/2017 What does a correlation of .56 mean? The SAT® provides a meaningful prediction of how students will perform in their first year of college. SAT Scores: Mathematics + Critical Reading + Writing Source: Patterson, B.; Mattern, K.; Kobrin, J.; Validity of the SAT for Predicting FYGPA: 2007 SAT Validity Sample; College Board, 2011

25 What does a .56 correlation mean?

26 Other Correlation Examples for Context
Variable 1 Variable 2 Correlation Sugar Consumption Child’s Behavior .00 Aspirin Heart Attack Death .02 Lead Exposure Child IQ .12 Parental Divorce Child Well-Being .09 Ibuprofen Pain Reduction .14 Alcohol Aggressive Behavior .23 Viagra Sexual Functioning .38 Gender Height .67 SAT (CR, M, W) with FYGPA ≈ .56 From: Meyer, G., et. al. (2001). Psychological testing and psychological assessment: A review of evidence and issues. American Psychologist, 56,

27 SAT ® Scores Still Matter, Even After Controlling for High School GPA
Mean FYGPA by SAT Score Band, Controlling for HSGPA SAT 2.79 3.09 3.34 3.56 2.22 2.52 2.51 2.74 2.94 3.15 1.98 2.26 2.12 2.27 2.47 2.54 1.81 FYGPA 1.0 1.5 2.0 2.5 3.0 3.5 4.0 HSGPA C or Lower B A Freshman Year GPA

28 The SAT® continues to predict college outcomes through students’ 2nd and 3rd years
The relationship between SAT scores and cumulative GPA remains strong and consistent as students progress through their college career. Percent of Students Earning a 2nd Yr and 3rd Yr Cum GPA of a B or Higher by SAT Score Band Source: Patterson, B.; Mattern, K.; Kobrin, J.; Validity of the SAT for Predicting Second-Year Grades: 2006 SAT Validity Sample; College Board, 2011; Patterson, B.; Mattern, K.; Kobrin, J.; Validity of the SAT for Predicting Third-Year Grades: 2006 SAT Validity Sample; College Board, 2011;

29 The SAT® continues to predict college persistence
College Retention Rates by SAT Score Band

30 College Retention As SAT ® scores increase, so does the likelihood of a student returning for the second and third year of college. Source: Patterson, B.; Mattern, K.; The Relationship between SAT Scores and Retention to the Second Year: 2007 SAT Validity Sample; College Board, 2011; Patterson, B.; Mattern, K.; The Relationship between SAT Scores and Retention to the Third Year: 2006 SAT Validity Sample; College Board, 2011

31 The SAT ® provides incremental validity over HSGPA in predicting college retention
Once again we see that the SAT in combination with High School grades provides a greater understanding of how students might perform Incremental Validity of SAT Scores over HSGPA for Predicting Second Year Retention The SAT provides substantial benefit over and above high school grades. SAT and HSGPA are both pretty good measures when considered alone, but in when you look at them together you have the best predictive indicator of future college success. SAT Scores add meaningful information over and above HSGPA. The best predictor of college outcomes is the SAT and HS record taken together. Source: Patterson, B.; Mattern, K.; The Relationship between SAT Scores and Retention to the Second Year: 2007 SAT Validity Sample; College Board, 2011

32 The SAT ® helps predict four-year graduation rates
Rates by score band with continuous enrollment at original institution The SAT provides substantial benefit over and above high school grades. SAT and HSGPA are both pretty good measures when considered alone, but in when you look at them together you have the best predictive indicator of future college success. SAT Scores add meaningful information over and above HSGPA. The best predictor of college outcomes is the SAT and HS record taken together.

33 Four-Year Graduation Rates by SAT Scores, holding HSGPA constant
The SAT ® provides incremental validity over HSGPA in predicting four-year graduation rates Four-Year Graduation Rates by SAT Scores, holding HSGPA constant The SAT provides substantial benefit over and above high school grades. SAT and HSGPA are both pretty good measures when considered alone, but in when you look at them together you have the best predictive indicator of future college success. SAT Scores add meaningful information over and above HSGPA. The best predictor of college outcomes is the SAT and HS record taken together.

34 External validity studies confirm the value and validity of the SAT ®
University of Minnesota “SAT retains virtually all of its predictive power when SES is controlled” After controlling for SES, SAT scores are still related to college grades (correlation of .44) University of Georgia SAT-Writing scores are the most predictive at predicting academic achievement For every 100-point increase in SAT-Writing there is a .07-point increase in Freshman GPA and a .18-point increase in Freshman English University of California In 2004 study, HSGPA was slightly more predictive than SAT V+M In 2006 study, SAT CR+M+W was slightly more predictive than HSGPA U of Minnesota - (Sackett, Kuncel, Arneson, Waters, and Cooper, 2009) Analyzed validity data from 41 colleges, controlling for SES Findings There is a relationship between SAT scores and SES (correlation of .42) There is a stronger relationship between SAT scores and college grades (correlation of .47) After controlling for SES, SAT scores are still related to college grades, and almost as strongly (correlation of .44) “SAT retains virtually all of its predictive power when SES is controlled” U of Georgia SAT-Writing scores more effective than Critical Reading or Math scores at predicting academic achievement For every 100-point increase in SAT-Math .36-point increase in Pre-calculus .52-point increase in Calculus I For every 100-point increase in SAT-Writing .07-point increase in Freshman GPA .18-point increase in Freshman English Students enroll in .44 more hours per academic year Students earn .54 more hours Students withdraw from .20 fewer hours University of California Zwick, R., Brown, T., & Sklar, J. C. (2004, July). SAT Critical Reading and Math slightly more predictive in 2006 than in 2004 SAT Writing slightly more predictive than the other SAT sections In 2004 study, High School GPA was slightly more predictive than SAT V+M In 2006 study, SAT CR+M+W was slightly more predictive than High School GPA Sources: University of Minnesota - (Sackett, Kuncel, Arneson, Waters, and Cooper, 2009); University of Georgia (Cornwell, Mustard, and Parys, 2008); University of California (Agronow, 2007)

35 SAT ® Fairness

36 SAT ® score performance does vary by sub-group
10b_1585_SATforRAS 4/19/2017 SAT ® score performance does vary by sub-group The existence of score gaps amongst different groups does not necessarily indicate that the assessment is unfair or biased. Source1: U.S. Census Bureau, 2006 American Community Survey (ACS) Public Use Microdata Sample (PUMS) File. Source: 2011 College-Bound Seniors

37 The achievement gap exists among many measures of academic achievement and attainment
Unfortunately, inequities in American society and education exist as evidenced by multiple academic measures Academic Achievement Academic Attainment Average CR+M+W Average GPA American Indian Avg SAT 1437 HSGPA N/A HS graduates 79.9% College graduates 39% Unfortunately, sub-group score differences reflect inequities in American society and education The SAT sub-group differences reflect these educational and societal inequities Since the test measures what students have learned in school and how well they apply that knowledge, a test such as the SAT is going to reflect these differences.… The SAT can’t undo a bad education or sub-par schools. The College Board is deeply troubled by these inequities and has instituted a range of programs to strengthen K-12 education across the country. We are committed to excellence, equity and access. Setting high standards for achievement is not the problem; the problem is addressing all the factors that impede achievement along the way. % - Percent of population age with a high school credential, 2006 % National six-year graduation rates of bachelor's degree-seeking students, 2007 Sources: 2011 College-Bound Seniors; U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, High School Transcript Study (HSTS), various years, 1990–2009. National Center for Higher Education Management Systems, 2009; U.S. Census Bureau, 2006 American Community Survey (ACS)

38 Achievement gaps persist beyond undergraduate studies
African Americans Asian Americans Hispanics Caucasians SAT Verbal 434 498 456 526 SAT Math 426 460 528 ACT Composite 17.1 21.8 18.9 21.7 GRE Verbal 391 487 438 495 GRE Quant 416 598 482 540 GMAT 533 492 531 LSAT 142.7 152.7 145.2 153.5

39 The SAT ® is rigorously developed and researched to ensure fairness
10b_1585_SATforRAS 4/19/2017 The SAT ® is rigorously developed and researched to ensure fairness Regular Curriculum Surveys Ensure the content on the SAT aligns with what is taught in high school and what colleges expect entering freshman to know and understand Test Development Committees Comprised of high school and college faculty review all test items and test forms Pre-Test (variable section) Field tests new items to collect statistics about item performance, ensure fairness for students of all backgrounds and that students of equal ability perform the same Rigorous Psychometric Analysis Evaluates the performance of each item to validate that items are performing as expected The SAT® is the most rigorously researched and designed standardized test in the world. Every question is field-tested across the entire testing population All 50 U.S. states Over 170 countries every question goes through a sensitivity review to avoid concerns with: Gender Ethnicity/Race Disabilities Controversial topics like war, violence and politics Built Fair / Treats Fair The SAT has consistently proven valid across all subgroups, ethnicities, genders and socioeconomic groups. The SAT is a fair test because it is also the most rigorously researched and designed standardized test in the world. Every single question is assessed and screened for potential bias. New test questions are researched to ensure that students of equal ability perform the same on a test question, regardless of gender, race and ethnicity. Each question goes through at least four content reviews and a sensitivity review to ensure the content is fair to all students. Field tests are then conducted and analyzed to confirm that, in live testing situations, sub-group differences do not occur. Every question goes through many internal and external reviews, including a sensitivity review to avoid concerns with: Gender Ethnicity/Race Disabilities Controversial topics like war, violence and politics Every question is field-tested in 50 states and eliminated if students from different groups perform inconsistently. The Unscored Section In addition to the nine scored sections of the SAT, there is one 25-minute section that ensures that the SAT continues to be a fair and valid test. This unscored section is used to “try out” new questions for future editions of the test. It also ensures that scores on new editions of the SAT are comparable to scores on earlier editions of the test. Including an unscored section on all exams is part of good standardized test design. Exams without a mandatory unscored section do not allow for as comprehensive pre-testing of questions. You will find the same technique used on the GRE and LSAT, for example. This helps make the SAT one of the most rigorously tested exams in the US and ensures its fairness for students of all backgrounds, which is a primary objective of the College Board. Building and Scoring All questions tried out in the 25-minute Variable Section of the SAT Statistics collected from actual test-takers: Difficulty (percent correct), r-biserial (relationship of question to rest of test) and DIF (fairness measure) Pre-tested questions are assembled into final form “Equating” questions, from a prior SAT, are in some versions of the Variable Section when a new form is administered Statisticians use the new test-takers’ responses to those equating questions to determine the raw-score to scale-score conversion of the new form, based on the difficulty of the new form Differential Item Functioning Compares subgroup performance on pre-test questions, matched on total score Male vs. Female White vs. Black, Hispanic and Asian Looks for questions where students with the same total score answer differently If the differences are large enough, the question is taken out of the pool

40 Eliminating Questions that Behave Differently for Groups of Students
Gardens in one form or another has been planted, tended, and harvested since antiquity by people from a wide variety of cultures. No error. Easier/more difficult for which gender or racial/ethnic group(s), matched on total score? African Americans – more difficult Asian Americans – more difficult

41 The SAT ® holds as a valid predictor of college performance by all sub-groups
When we evaluate the predictive validity of the SAT by racial/ethnic sub- groups we see that SAT continues to be a strong and valid predictor of college performance, in most cases even more so than high school GPA Correlation* of SAT® and High School GPA to First-Year College GPA by Race/Ethnicity RACE/ETHNICITY SAT (CR+M+W) HSGPA SAT+HSGPA American Indian .54 .49 .63 Asian American .48 .47 .56 African American .44 Hispanic .50 .46 .57 White .53 Total .64 If the SAT were not succeeding as a strong predictor of college performance by different sub-groups we would expect to see low to medium correlation coefficients for those sub-groups. That is simply not the case, in fact we see that the SAT is a STRONGER predictor of first-year college performance than high school GPA for all racial and ethnic sub-groups except white students. It is important to underscore, however, that the best predictor of college success is the combination of SAT and HSGPA When we talk about validity, we do so in terms of Correlation Coefficients: A statistic that expresses the degree and direction of the relationship between 2 variables. Correlation coefficients can be positive or negative. Perfect correlation (ie: formula) = 1 Small Correlation = 0.1 Medium = 0.3 Large = 0.5 or higher Positive: One variable increased and the other variable increases. Ie: Positive correlation between SAT scores and FGPA because FGPA increases as SAT scores increase Negative: One variable increases, and the other variable decreases. Ie: Number of days absent from school increases, the high school GPA decreases. Writing is the single most predictive of the 3 SAT sections. Zwick - Another reason that admissions tests can be valuable is that using only high school grades, without test scores, to predict freshman grades tends to produce predictions that are systematically off target for some ethnic groups, a problem that can occur despite the sizeable correlation between high school and college grades. The SAT in combination with HSGPA remains the best predictor of college performance across all sub-groups

42 Extensive external research confirms that the SAT ® is not biased
There is a substantial body of literature indicating that individual item bias has been largely mitigated in today’s admission test due to extensive external research and development of question items on both the SAT and ACT®. NACAC Testing Commission Report September 2008

43 Interpreting and Comparing Scores

44 Interpreting SAT Scores
Each section score is reported on a 200- to 800- point scale, each section is an independently valid predictor of college success. The Writing sub-score is a combination of a multiple-choice score from (70%) and an essay score from 2 to 12 (30%) Each essay is independently graded by two qualified readers The SAT is designed so that a student who answers about half of the questions correctly will receive an average score of approximately 500. Scores on any standardized assessment are approximations rather than precise measures of skill. The standard error of measure (SEM) of the SAT usually falls in a range of 30 points for reading and mathematics and 40 points for writing above or below a student’s true skill level. When comparing scores there must be a difference of points before more skill can be assumed in one area than another.

45 Comparing SAT ® and ACT ® scores
The ACT and SAT are scored on different scales The concordance table provides a way for individuals and institutions to compare a student’s performance on one exam with their likely performance on a second exam Comparing percentiles is not accurate because the pools of students taking the two tests are different It’s important to help people understand that scores on the ACT and SAT are not equivalent. This is a widely misunderstood fact: First off, in most cases, the pool of students taking the two tests is different The SAT is taken by a greater number of high achievers It’s not fair to compare the two scores Concordance tables are necessary to compare “apples to oranges” when pool of SAT and ACT overlap Two separate concordance tables have been developed:  Table 1 provides a concordance between the ACT Composite Score and the sum of SAT Critical Reading and Mathematics scores for 300,437 students.  Table 2 provides a concordance between the ACT Combined English/Writing Score and the SAT Writing Score for 190,148 students who completed the ACT Plus Writing. Both tables are based on scores from students who took both tests between September 2004 (for the ACT) or March 2005 (for the SAT) and June Students in the sample represent the first high school graduating cohort since the introduction of the SAT with Writing and the optional Writing section on the ACT. The sample includes students who completed both tests and were matched across ACT and SAT files. Note: The concordance table is based on scores from over 300k students who took both versions of the tests with writing from the Class of 2006 45

46 The population of students taking the SAT® perform better than the population taking the ACT ® across all sub-groups To compare the performance of students or populations of students you should concord an ACT score to the SAT scale 2011 National Composite ACT Scores Concorded 2011 ACT Scores to SAT CR+M Scale 2011 National Combined SAT CR+M Total 21.1 994 1011 American Indian 18.6 894 972 Asian 23.6 1094 1112 Black 17 830 855 Hispanic 18.7 898 914 White 22.4 1046 1063 Sources: 2011 College Bound Seniors; ACT: Condition of College and Career Readiness, 2011

47 Resources – For Students

48 SAT Practice Tools At a Glance for Students
Most SAT Practice Tools are FREE! Practicing for the SAT and SAT Subject Tests Planning for Test Day SAT Question of the Day – FREE SAT Question of the Day Mobile App – FREE SAT Practice Questions – FREE SAT Subject Test Practice Questions – FREE SAT Practice Test – FREE Mathematics Review – FREE Effective Writing Review – FREE SAT® Skills Insight™ – FREE The SAT® Practice Booklet – FREE Official SAT Study Guide™: 2nd Ed. Official SAT Subject Test Study Guide™: 2nd Ed. The Official SAT Online Course™ My SAT Study Plan™ – FREE Answers Imagined – FREE SAT Test Taking Approaches – FREE SAT Essay Strategies – FREE SAT Subject Test Taking Approaches – FREE SAT Subject Test Recommended Skills and Prerequisites – FREE SAT Subject Test Web Resources – FREE Expanded Practice Questions Answer Explanations Recommended Preparation Important Test Day information on: How to Do Your Best What to Bring Standby Testing If You’re Absent SAT Test Center Closing Make-up Testing Test Security and Fairness Talking Points: The College Board also offers a number of practice tools for students. Most of them are free and easily accessible on our website. They are effective These are the only official practice tools available from the test maker Demo: - Show the SAT practice site homepage Walk through features on the home page Show how students can access the practice page from the SAT homepage Helping Students Get Ready Models, Lesson Plans, and Strategies for: Argumentative writing skills School-based SAT Practice The Official SAT Teacher’s Guide™ ESL/ELL students Professional Development Workshops: SAT Skills Insight Animating Student Writing Holistic Scoring Workshop School-Based SAT Practice Writing Preparation for Educators of ESL/ELL Students 5

49 SAT Skills Insight SAT Skills Insight identifies the academic skills that typical students should focus on to improve their scores, depending on their target score range Free, online resource, paired with My SAT Online Score Report to make SAT In Focus Sample SAT questions and answers help students better understand the skill descriptions given Here’s another screen shot of the Skills Map interface. Again, this is NOT final. But let’s take a look at this in a little greater detail… This shows the SAT SKILLS MAP – CRITICAL READING While not pictured here, there is a similar page for the Math section. Let me tell you a little bit about it. It shows Level 500–590 Skill Descriptions A typical student in this score interval can do the following: Number and Operations Solve problems using ideas from basic set theory and basic number theory Recognize and apply ratio, proportion, or percent in solving problems Use properties of real number operations, ordering, and the zero-product property Solve problems involving counting techniques Algebra and Functions Formulate and solve problems involving proportions Solve multistep problems involving linear and quadratic relationships Use and interpret graphs, including graphs of step functions Solve problems involving algebraic inequalities Solve problems involving exponential growth and decay Evaluate an operation in three variables represented by unfamiliar symbols Data, Statistics, and Probability Interpret and solve problems involving data displays (e.g., circle graphs) By reviewing this, a student (and their teacher) can know ahead of time what a typical student needs to master to achieve a score in this range.

50 SAT® Offers the Most Generous Fee Waiver Program of Any College Entrance Exam
More than 360,000 low-income students in the graduating class of 2011 benefitted from SAT Fee Waivers (1 in 5) More than $35 million in services made available at no cost to low-income high school students last academic year Eligible students can receive: Two SAT fee waivers and two SAT Subject Tests™ fee waivers (up to six Subject Tests) Four Free Additional Flexible Score Reports Free Question and Answer Service or Student Answer Service Four Request for Waiver of College Application Fee forms Discount on The Official SAT Online Course™

51 Resources – For Institutions

52 The Importance of the SAT® for Institutions
There are many factors that colleges and universities use to assess students today, but the SAT remains a key component, along with high school grades and a rigorous curriculum. The SAT: Provides a national yardstick to compare students across the country High schools vary widely in courses, teachers and grading practices Grade inflation is more common than ever Is a strong predictor of college success and retention As good as high school grades at predicting student performance in the first, second and even third years of college Gives students a better sense, based on their scores, of how ready they are academically for the rigors of college For over 80 years, the SAT has been a powerful tool helping colleges achieve their recruitment and enrollment goals. The test continues to evolve and help you meet these needs. The SAT: Is the most widely used college entrance exam, taken by over 2 million students in over 270 countries. It can substantially increase your institution’s reach.  Provides a standard measure of students’ readiness for college-level work, presented in an objective way to support your admission activities. Is rigorously developed to be fair for all students and a strong predictor of freshman year GPA, thus helping to increase the richness of the prospective students admitted to your institution

53 Admitted Class Evaluation Service™
10b_1585_SATforRAS Admitted Class Evaluation Service™ 4/19/2017 To assist colleges and universities in conducting research about their students, the College Board offers the Admitted Class Evaluation Service (ACES™), a free online service that predicts how admitted students will likely perform at your institution and how successful they can be in specific classes. ACES admission validity studies identify which measures best predict a student’s future performance and recommends the best combination of predictors for your institution. ACES offers two types of validity studies—admission and placement. These studies identify the optimum combination of measures to predict a student's future performance at your institution. ACES studies evaluate the differences for predicting the success of specific student groups and document the probability of error. Each ACES report features: In-depth analysis of findings General background information to help you examine the study in greater detail Interpretive text highlighting key findings Colorful presentation and graphics ACES admission validity studies typically use high school grade point average (HSGPA) or high school rank, along with SAT scores, to establish the best combination of variables to predict student performance at your institution. You choose additional variables based on what you believe to be important contributors for predicting the academic success of your students. Examples of additional predictors are: SAT Subject Tests™ scores Years of study in a particular subject area Scores on tests developed by your institution ACES placement validity studies predict how students will perform in future college classes by comparing student performance in selected courses with student scores from College Board tests such as: SAT SAT Subject Tests ACCUPLACER® tests AP® Exams CLEP® exams ACES gives you continuous validation It is critical to verify that current admission and placement measures are valid measures for future classes. To be certain your admission and placement standards remain valid, you should conduct an ACES study at least every three years. ACES placement validity studies predict how students will perform on different academic measures by comparing student performance in selected courses with student scores on various College Board assessments*. * SAT ®, SAT Subject Tests™, ACCUPLACER® tests, AP ® Exams and CLEP ® exams


Download ppt "Understanding Standardized Testing: A Deeper Look"

Similar presentations


Ads by Google