1. LEARNING FROM EXPERIMENTATION IN DEVELOPMENTAL MATH IN COMMUNITY COLLEGES IN CALIFORNIA: RESULTS FROM A LONG-TERM RESEARCH PARTNERSHIP The Steinhardt Institute for Higher Education Policy, New York University, October 19, 2015 Tatiana Melguizo Associate Professor, University of Southern California melguizo@usc.edu This research was funded by a grant from the U.S. Department of Education’s Institute of Education Sciences (IES).

2. Overview  Educational & professional background  Key results from a federally funded long-term research collaboration with a large urban community college district in California  Other research interests  Q & A

3. Inequalities in college access and outcomes by socioeconomic status in Colombia Only 25% of students from the lowest income strata enter college compared to 60% of students from the highest income strata (Melguizo, Sanchez & Velasco, 2015) Los Andes University Bogota-Colombia

4. Increasing inequalities in educational outcomes in the U.S.

5. Banning of Affirmative Action Policies in the U.S. in the mid 1990s I started my Ph.D. at the time that Affirmative Action policies were banned in California That major policy shift motivated me to study the impact of attending more selective colleges and universities for students of color Findings suggested that students of color both in the 1980s and in the 1990s had higher graduation rates at the most selective colleges and universities (Melguizo, 2007) Findings cited as part of Amicus Brief to the Supreme Court by AERA

6. Creating a Research Partnership to Study Assessment and Placement in Developmental Math  The purpose of the partnership is to inform developmental education research, policies, and practices  Site: The Los Angeles Community College District  Research Focus: To examine the impacts and implementation of test-based, alternative placement policies, and delivery methods for developmental math  Research approaches: Experimental and quasi- experimental, qualitative, & descriptive

7. Strength of partnership built on commitment & unique expertise of partners  A joint commitment to improve achievement in dev. ed. through policy and practical change  An firm belief that each partner is an equal; brings unique contextual, technical expertise  LACCD: Knowledge about local context, access to administrative records, faculty, and administrators  USC: A strong record of community college research  AIR: Technical expertise in conducting experimental and quasi-experimental research

8. Problem Statement Every year about 80 percent of community college students in California are placed into preparatory mathematics. This percentage is higher than the national average Community college students have widely varying initial skills levels Colleges have to offer classes to meet these levels and have to keep heterogeneity in the classrooms manageable Placing students incorrectly can reduce the likelihood that students succeed

9. Why LACCD? Los Angeles Community College District - a natural laboratory  Diverse student population that varies by college.  Nine colleges with 130,000 plus students.  “Common data system.”  Large number of observations.  Presumption of representativeness—likely to capture the wide variation across community colleges in the United States.

10. Literature on Impacts of Remedial Education Proponents argue that remedial education provides the preparation necessary for students to succeed in college (Boylan, Bliss, & Bonham, 1994; 1997; Lazarik, 1997) Critics contend that the benefits that students obtain are not clear (Calcagno & Long, 2008; Martorell & McFarland, 2011; Scott-Clayton & Rodriguez, 2015).

11. Developmental Math Sequence ArithmeticPre-Alg.Elem. Alg.Int. Alg. Transfer- Level Developmental Math

12. Remediation needs of LACCD students

14. Five Key Findings Finding 1: Establishing an effective A&P system is complex. More support and training is needed for faculty and administrators charged with this task. (Melguizo, Kosiewicz, Prather & Bos, 2014). Finding 2: The largest barrier for developmental math students is attempting their initial course (Fong, Melguizo, & Prather, 2015). Finding 3: Community college faculty and administrators have the opportunity to improve placement and success in developmental math by engaging in a systematic process of calibration of the cut scores of assessment and placement tests (Melguizo, Bos, Ngo, Mills & Prather, 2015). Finding 4: The diagnostic test places students more accurately than the computer- adaptive test (Ngo & Melguizo, 2015). Finding 5: The inclusion of multiple measures in the placement process can increase access to higher-level math without decreasing students’ chances of success (Ngo & Kwon, 2014; Fong & Melguizo, 2015).

15. F2: Over 30 percent of students are NOT attempting the assigned courses

16. F3: Systematic Process of Calibration of the Cut Scores  Math faculty set the cut points between the different levels based on who applies and how their course offerings are distributed  If the cut points are too high, too many students languish in remedial courses  If the cut points are too low, too many students fail higher-level courses and present a challenge to the instructors  Getting the cut points just right is important

17. F3: Different Pathways to Success (Arithmetic vs. Pre Algebra) Test Placed in Pre- Algebra Placed in Arithmetic Enroll in Pre- Algebra Enroll in Arithmetic No enrollment Success Failure Success Failure Next course

18. F3: Ideal Regression Discontinuity Situation  Regression discontinuity analysis is the strongest non-experimental method to estimate causal effects  It depends on a continuous forcing variable and an exogenously established cut point  Those two conditions are present in this situation

19. Outcome Placement Test Score Regression Line Arithmetic Pre-Algebra Impact

20. F3: Negative Impact of Starting in AR Time Dimensions is Key

21. F4: Placement Accuracy: Diagnostic versus Computer-AdaptiveTests Do diagnostics improve placement accuracy? Methods I. Logistic Regressions  Predicting EA outcomes with skill-specific math information II. Placement Accuracy  Sum of accurate placements III. Regression Discontinuity  Traditional (with single-score)  Binding-score (with multiple criteria)

22. Placement Accuracy Table 3. Percent of accurate placements by level of developmental math. AR vs PAPA vs EAEA vs IA ACCUPLACER Colleges College A73.275.372.9 College B-75.548.4 College C19.822.624 College D77.847.271.4 MDTP Colleges College G (w/subscores)51.676.356.1 College G (w/o subscores)49.968.244.9 College H90.065.554.1 Note: Placement accuracy calculated using method described by Scott-Clayton (2012). Percentage shown is the sum of the proportion of students predicted to pass higher-level course and placed there plus the proportion of students not predicted to pass the higher-level course and placed in the lower level course.

23. F4: Diagnostic tests are Placing Students more Accurately than Computer-Adaptive Tests  Students placed using results from computer- adaptive tests were more negatively impacted by the placement decision than prior cohorts placed by MDTP.  Students were less likely to enroll and persist onto the next math course after the placement test switch.  Consistent with other studies, we found that the diagnostic test can provide information on student proficiency on a range of subtopics such as fractions, exponents, and reasoning which can improve math placement decisions and/or tailor instruction in math courses.

24. F5: The Inclusion of Multiple Measures can Increase Access w/out Decreasing Student Success

25. F5: Findings  Only 6% of the students benefitted from multiple measures at the LACCD  Major benefits for African American and Latino students who could enroll in higher-level math courses  We found no evidence that “boosted” students were less likely to complete the course

26. Conclusions  Engaging in research partnerships with large districts are a way of increasing research capacity of the district while gaining a much more nuanced understanding of the context for the researcher  Research findings not only contribute to the knowledge but are also policy relevant and actionable  Researcher-practitioner partnerships can be conducive to high-quality and high-impact research

27. Other Current Projects  Mixed-methods evaluation of the Susan Thompson Learning Communities. Buffett Foundation  Using high school transcript information to refine A&P in math. LAUSD-LACCD-USCNSF: EAGER  Accurately estimating Student Learning Outcomes (SLOs) in higher education  Brazil (Melguizo & Wainer, under review)  Colombia (Melguizo, Zamarro, Sanchez & Velasco, 2015)  Proposed an RCT evaluation to test student self- placement in Dev Ed Math

28. THANK YOU! Questions Tatiana Melguizo melguizo@usc.edu http://www.uscrossier.org/pullias/research/projects/s c-community-college/