Presentation on theme: "East Bay Career Pathways Regional Convening #2 December 4, 2014"— Presentation transcript:
1 East Bay Career Pathways Regional Convening #2 December 4, 2014 Let Icarus Fly: Unleashing student achievement through multiple measures assessment John J. Hetts, Ph.D. Senior Director of Data Science Educational Results PartnershipEast Bay Career Pathways Regional Convening #2December 4, 2014John Hetts, Senior Data ScientistKen Sorey, VP ERPShort Bios go here.
2 OverviewStandardized tests systematically underestimate student capacityStudents of colorFirst generation college studentsLower SESWomenMultiple measures (esp. GPA) fairer and far more accurate predictor of college performance/graduationIncrease grad rates, decrease time to completionVery low cost, exceptionally high ROITo studentsTo collegesTo state(optional intro) 0) Standardized assessments systematically underestimate student’s capacity to do college-level work, particularly students of color, first-generation college students, students with lower SES, and women, and do so so both at four year colleges and community colleges.1) Using multiple measures, particularly high school GPA, in assessment, admissions, and merit-based aid, provides a fairer, far more accurate assessment of students’ ability both to perform well in college and to graduate.2) As a result, using multiple measures will increase graduation rates, decrease time to graduation, and meaningfully help close equity gaps and can do so at next to no cost.3) In fact, using multiple measures will save public education systems and taxpayers tens of millions of dollars annually in California alone in unnecessary costs of reteaching successfully mastered skills and millions of dollars annually in unnecessary assessments.(for longer elevators) 4) Even more importantly, using multiple measures will save California students billions of dollars in lost salary/other opportunity costs that they currently lose to the time necessary to retake the same courses they successfully completed in high school and will meaningfully improve a wide range of their long-term life outcomes as well as their productive contributions to the economy and their communities.
3 Daedalus and IcarusDaedalus crafted the labyrinth of inescapable complexity for King MinosTo escape from Minos, Daedalus built wings of feather and wax for his son Icarus and himselfDon’t fly too high, lest sun melt the wax and you plummet to your doomDangers of innovation/invention, hubris,Importance of knowing your limits, listening to your wiser eldersBut most of us forget the rest of that story…
4 Student transition to college Community colleges rely nearly entirely on standardized assessmentMost CC students placed below college-levelSignificant barrier (Bailey, Jeong, and Cho, 2010)First interaction is to tell students they don’t belongImply that most students are not ready for college and are likely to failConvinces many, including our studentsSubstantial majority of community colleges rely heavily if not nearly entirely on standardized assessmentMost community college students placed into coursework below college-levelSignificant barrier to completion (Bailey, Jeong, and Cho, 2010)First interaction with most students is to tell them they’re not ready for collegeImplies that most students are not ready for college and are likely to failConvinces manyIncluding our studentsVery first interaction with student is to tell them they’re not ready for college“You’re not good enough”You don’t belongYou’re not really college materialAgain, and again.Internalized by faculty, staff, administration, counselors, publicEventually, tragically, many of the studentsPlacement into remediation is a significant barrier to completionTwo enormous systemic changes:More, longer, slower remediationReduction of access (offerings at bottom of sequences first to get restricted)
5 Conventional Wisdom Explaining Assessment Results It is a problem with today’s studentsStudents are simply, vastly unprepared for collegeKids these days ….It is a problem with public educationPublic education is failing to prepare studentsTeachers these days…Not my students, not my teachers.. Teachers in general, students in general, sometimes other problematic adjectives added like urban.
6 What If the Conventional Wisdom is Wrong? Substantial, long-term increase in IQ: bit.ly/FlynnEffectIQ18-24 with HS degree: 90% - highest ever: bit.ly/HS18-24National Assessment of Educational Progress: at all-time highs in virtually every demographic category:bit.ly/NAEPInfoResearch increasingly questions effectiveness of standardized assessment for placementLittle relation to college course outcomes. (e.g., Belfield & Crosta, 2012; Edgescombe, 2011; Scott-Clayton, 2012; Scott-Clayton & Rodriguez, 2012): bit.ly/CCRCAssessNAGB, 2012: Incredible variability in cutscores; 2-year colleges often use HIGHER cutscores than 4-year bit.ly/NAGB2012Hiss & Franks, 2014: Underestimates capability of students of color, women, first gen college students, low SES: bit.ly/DefiningPromisethe National Assessment of Educational Progress from the NCES National Center for Educational StatisticsNational Assessment Governing BoardAustin Peay – 50% remediation, poor completion – eliminate remediation, less prepared students enroll in college-level with 2 hours/week in learning lab required. Pre-53% pass developmental math, 30% college level w/in 2 years. Post – 67% complete college-level. English – 54% to 76%:
7 Big questionsWhat if the problem is not with our students, but with how we have assessed their capabilities?ORWhat if one of the barriers to our students’ successful transition to college is one that we fully control?
8 LBCC ResearchFive longitudinal cohorts tracking more than 7,000 HS grads who attend LBCC directly after high schoolbuilt with help of Cal-PASSExamined predictive utility of wide range of high school achievement datamost notably 11th grade California Standards Test (CST) scores and high school gradesFor predicting:How students are assessed and placed into developmental skills sequencesHow students perform in those classes(and for understanding alignment between them)Highly simplified presentation of unstandardized ordinal and logistic regressionLots more too it and RP has posted a lot of it
9 * p <.05 **, p <.01, *** p<.001, x = p< 1 x 10-10 Alignment in English(unstandardized ordinal and logistic regression coefficients)High school, course you took, A-G classes, more. What’s more important than 11th or 12th grade English – Math CST score – almost as strong as overall GPA.Most important predictor of performance – what course you took – after that – it’s your overall GPA.* p <.05 **, p <.01, *** p<.001, x = p< 1 x 10-10
10 * p <.05 **, p <.01, *** p<.001, x = p< 1 x 10-10 Alignment in MathMath much more complicated than this – the last course you took/which CST matters a lot in performance as does the difficulty level of the which course you take.* p <.05 **, p <.01, *** p<.001, x = p< 1 x 10-10
11 Key TakeawaysAssessment should predict how students will perform at our collegesInstead:Standardized tests best predict standardized testsClassroom performance best predicts classroom performanceMore information tells us more about students than less informationReplicated statewide by STEPS project, conceptually replicated by CCRC workSignificant opportunities exist to improve placement, student achievement, and students’ college experience.
12 Multiple Measures Placement: Transfer-level Placement Rates F2012 Attending a community college>75% students of color, >50% HispanicLargely eligible for the BOGG fee waiver and Pell grantsTwo major communities of English as a Second Language studentsMostly first person in family to go to collegeMany first or second generation immigrants
13 F2012 Promise Pathways vs. Fall 2011 2-year rates of achievement Compares first year rates of achievement to most recent 6-year cohort. Pathways is ahead, in many cases quite significantly.
14 Success rates in transfer-level courses Fall 2012 Neither of these differences approach significance, p >.30
15 Equity impact: F2011 Baseline Equity Gaps for 2-year rates of achievement Our equity gaps are typical of equity gaps seen at most California Community Colleges…
16 Equity impact: F2012 Pathways 2-year rates of achievement Our equity gaps are typical of equity gaps seen at most California Community Colleges…
17 How might this change how we understand college readiness? Notes: unduplicated by student but counting bottom category first each time. Majority of the LBCC placement is the alternative placements.
18 What was gained through evidence based approach to transition to college Dramatic increases in students attaining early educational milestones, & shorter times to do soNew discussion of research and instructional pedagogy, kick-starting experimentation and innovationStrong challenges to conventional wisdom and perceptions of students by administration, staff, faculty, and students themselvesSaved thousands of units of unnecessary remediationReminder of the forgotten second instruction of DaedalusWe keep on using these tests. I do not think they mean what we think they mean…Just as important not to fly too low.Concrete achievable steps that any college can take to dramatically improve all of our students’ futures.Can your college do this research?Can this research help your students too?Can your college afford to implement something like this?Making the primary changes to assessment and placement are not only virtually cost-free – it can save your college money and direct its resources more effectively and improve outcomesLow cost, outcome accelerating, potential long-term savings – can your college afford not to try?We’ve built a model, made it available through the work of the RP Group, and have already been providing support to over 20 California Community College
19 Contact InformationResearch questions/data requestsJohn Hetts, Educational Results Partnershipext. 208cellGeneral questions about Promise Pathways or Long Beach College PromiseAlicia Kruizenga, Director of School Relations and International Education:(562)
20 Additional Resources Background research Achieving the Dream/Jobs for the Future summary of alternative assessmentbit.ly/AlternativeAssessmentCCRC research on Assessment, Placement, and Progression in Developmental Educationbit.ly/CCRCAssess and bit.ly/CCRCDevEdRP Group’s Student Transcript-Enhanced Placement (STEPS) Projectbit.ly/RPSTEPSStep by Step process for replication: bit.ly/RPSTEPS2More information about our researchbit.ly/PathwaysResearchSimilar CCC research and implementation:Peralta CCD: bit.ly/LaneySTEPS2, bit.ly/PeraltaSTEPSGrossmont-Cuyamaca: bit.ly/GrossmontState-level redesigns of assessment and placementNorth Carolina – 2.6 GPA & 4 years of math transfer-levelFlorida – developmental educational optionalVirginia – re-design of assessment in, modularization of mathConnecticut – dev. education cut to 1 semester, coreq onlyHawaii CCs- pilots using HSGPA in assessmentColorado, Montana, Mississippi* - move to corequisite modelsNational level attention on assessment and placement2013: US Governmental Accountability Office report on Developmental Education in Community Colleges2014: Center for the Analysis of Postsecondary Readiness established by CCRC, MDRC