1. Implementation and Evaluation of Project REAL Rural Early Adolescent Learning Allen Murray ACRES March 14, 2009

2. Key Personnel for Project Real Principal Investigators Thomas Farmer, Ph.D. & Jill Hamm, Ph.D. Investigators Kimberly Dadisman, Ph.D. & Linda Mason, Ph.D. Intervention Staff Allen Murray, Intervention Director Kirsten Feil Abby Hoffman Project Support Staff Mallory Vinson, Project Manager Jenny Westrick Kelli O’Brien Courtney Mann

4. What is NRCRES? The National Research Center on Rural Education Support (NRCRES) was established in 2004 with funding from the Institute for Educational Sciences of the U. S. Department of Education. The center is based at the University of North Carolina at Chapel Hill. NRCRES conducts a focused program of control-trial research that addresses significant problems in rural education. More than 40% of all American schools are in rural areas and 30% of all students attend rural schools. The research and development work of NRCRES will seek solutions that will improve the quality of rural education.

5. What is REAL In short, Project REAL is a Professional Development Program for teachers in rural schools (with a focus on teachers of early adolescents). It is designed to help them promote the academic, behavioral, and social adjustment of rural youth as they transition from childhood to adolescence. The program is designed to help support youth who are most likely to experience difficulty in school, but the strategies and techniques have been carefully selected to promote the engagement of all students including typical learners and high achieving students. A primary emphasis of this program is to help teachers establish classroom and instructional contexts responsive to a broad and diverse range of learning needs.

7. Common Challenges: –Poverty and changing employment base; –Financial impact of large geographic distances; –Shrinking population of communities and schools; –Changing economic foundation of communities. –Teacher recruitment, retention, and licensure issues; Common Strengths: –Faculty are a part of the community; –Schools are focal point for community; –Home ownership rates above the national average –Generational affiliation with local school and/or school system; –Social indicators (church, civic clubs) show strong community ties, –Higher rates of participation in extra-curricular activities Overview of Project REAL Schools

8. Pilot Sites: –2 Appalachian states, 4 middle school transition schools Full Implementation Sites: –Northern Plains: 4 k-12 schools (completed) –Appalachia: 4 middle school transition schools (completed) –Midwest: 2 k-8, 2 middle school transition schools (completed) –Southwest: 4 middle school transition schools (completed) –Deep South: 4 k-12 schools (completed) –Southeast: 2 k-8, 2 middle school transition schools (5/09) –Far West, 4 middle school transition schools (5/09) –Pacific Northwest, 4 k-8 schools (5/09) Total Number of Districts/Schools: 28 districts/56 schools Total Number of Teachers: 392 Total Number of Students: approximately 4000 Training and Research Sites

9. Rationale for Project Design The wide variances in grade level Standard Course of Study made a content-specific focus and a universal design mutually exclusive. So the focus is on addressing issues that impact academics for all populations and across curricula. Small school size means the grade-level team is often very small (and may consist of only one teacher). Small teams can cause teachers to work in isolation, with little opportunity to see others model new teaching techniques, curricula, or management strategies. The program combines a topic-focused training model with a teacher- focused consultation model.

11. Training vs. Consultation Training The agenda is owned by the trainer. Trainer has information to give others. Trainer’s job is to get that information out. A good model when many folks need the same content. Consultation The agenda is owned by the teacher. The teacher needs help in finding a solution. The consultant’s job is problem-solving. A good model when individuals or schools have unique needs.

12. Training is available to local schools; Consultation may be the greater need In many ways it is easier to provide training than consultation. Training is more time efficient, can be done in larger scale, can be made universal, and can be individually pursued (self- instructional modules, etc.). Consultation, by its very nature, is a person to person process; it is not adaptable to self- instructional models (by reading, videos, on-line, etc.). It is time-intensive, requires knowledge of both the teacher and community, and needs a face-to-face component.

13. Training Consultation Our training is not designed as stand alone, Instead, it is designed to set the stage for consultation, Consultation is the “real goal”, especially for special educators or teachers of students with disabilities, Consultation is student specific, with hope that it will generalize. Sets the stage for

15. Project REAL Delivery Project Real combines in-person training with a technology enhanced (distance delivery) consultation to both regular education teachers and specialists who have “at-risk” students in their class Focus is on how the transition to early adolescence impacts academic performance, both in class-wide issues and for “at-risk” students (as defined by teacher). Prior to the beginning of intervention, relationships are formed during site visits. These relationships, and the in- person summer institutes, keep the video-conferences from feeling “artificial”. Site visits are made during the intervention year to observe classrooms and follow-up on training topics.

16. Project REAL Intervention Universal Components Site Visits, Needs Assessments, and Summer Institutes Training modules on topics including: –Early Adolescent Development –Motivation and Academic Engagement –Instruction for low-achieving students –School and classroom social dynamics –Information processing –Literacy support and Writing Targeted Components Video-conference consultation with Project REAL staff A forum for intervention specialists to facilitate discussions of: –Training modules –Strategies for at-risk students –Strategies for class-side issues

17. Sample Training Unit Each module typically consists of some or all of these components: –Narrated PowerPoint Presentation, –Brief (non-academic) article, –Further readings on the topic for those teachers interested in a more in-depth look at the topic. –A video illustrating a teaching technique or topic in adolescent development. –An activity that teachers will complete with each other or in class.

19. Research Design for Project REAL Intervention schools matched with control schools –1/2 with middle school transition configuration –1/2 alternative configuration (e.g., k-8, k-12) –Schools are matched closely on demographic data Baseline student data is collected in the spring of 5 th grade and fall of 6 th grade. Student outcome data on school adjustment and academic achievement collected in spring of 6 th grade. A follow-up year of data is collected. (7 th grade, fall & spring)

20. Plan of Analysis Hierarchical Linear Regression Models Control variables : –Baseline, fall scores as controls –Ethnicity (Native American, White), Gender Intervention Main Effect Intervention by Ethnic Group, Intervention by Gender effects Tests for improvement to fit of model for block of variables; Test for significance of beta weight within significant block of variables

21. Perceptions of Social-Academic Context –Peer norms for effort and achievement (Hamm, 2001) –Emotional risk of participation (Hamm & Faircloth, 2005) –Bullying Context (Song, 2008) Affective Relationships with Schooling –Sense of Belonging (Hagborg, 1994; 1998) –School Valuation (Voelkl, 1996) Achievement –End-of-year state-level standardized test (school records) –Grades (school records) Three Key Outcome Domains

23. Findings for Intervention Effects: Northern Plains Site Site Description Recruited from all 4 th, 5 th, & 6 th grade classrooms of four public k-12 schools in a state in the Northern Plains –72% agreed to participate, N=165 student participants –45% Native American, 55% White –92 boys, 73 girls Schools were eligible for U.S. Department of Education’s Rural and Low-Income School Program (RLISP) –locale code 7 or 8 and at least 20% of students are from families living below the federal poverty level –In our sample, 64.6% of students were eligible for subsidized meals through the National School Lunch Act

24. After controlling for the previous year scores on each outcome, and for student demographic characteristics, students in Project REAL intervention schools, compared to their peers in matched control schools reported... –More supportive peer norms for effort and achievement; –More peer protection from bullying; –A greater sense of school belonging; –Greater valuing of school; –Higher year-end grades. Results

25. Intervention effects were particularly pronounced for Native American (versus White) students for: –State-level standardized test scores –Sense of belonging –Perceived peer norms for effort and achievement

33. Hierarchical Regression Analysis of Indicators of Students’ Academic Achievement Note: Standardized Betas reported: +p <.06, *p <.05, **p <.01, and ***p <.001 (one-tailed). Standardized State Test End of Course Grades 123123 Native American -0.13** -0.27-0.15*-0.16*-0.29** Female 0.03 0.010.27***0.26***0.36** Prior year score (spring) 0.71*** 0.32*** 0.30*** Intervention 0.01-0.090.27***0.25** Intervention * Native American 0.23*0.18 Intervention * Female 0.03-0.15 R2R2 0.58 0.590.240.310.33 F for change in R 2 72.2***0.12.2*15.9***15.2***1.5

34. Hierarchical Regression Analysis of Indicators of Students’ Affective Relationship with Schooling School Valuing/ Involvement School Belonging 123123 Native American0.17*0.14*0.010.080.06-0.18 Female0.110.100.160.07 0.17 Current year score (fall)0.38***0.36***0.35***0.55***0.53***0.50*** Prior year score (spring)0.20* 0.100.070.08 Intervention0.15*0.130.12*0.07 Intervention * Native American0.190.33** Intervention * Female -0.09-0.13 R2R2 0.380.400.410.400.420.45 F for change in R 2 18.2***4.4*0.920.6***2.9* Note: Standardized Betas reported: *p <.05, **p <.01, and ***p <.001 (one-tailed).

35. Hierarchical Regression Analysis of Indicators of Students’ Perceptions of the Social Academic Context of Classrooms and School Peer Norms Emotional Risk Bullying Protection Variable123123123 Native American 0.19**.17**-0.10-0.13*-0.110.130.11*0.07-0.09 Female 0.09 0.01-0.03-0.02-0.22*0.060.050.01 Current year score (fall) 0.47***0.46***0.44***0.42***0.39***0.38***0.50***0.51***0.47*** Prior year score (spring) 0.22**0.18**0.21**0.16**0.15*0.130.23*0.160.18** Intervention 0.13*-0.05-0.12-0.130.21***0.11 Intervention * Native American 0.36**-0.33*0.21 Intervention * Female 0.110.29*0.07 R2R2 0.430.450.480.270.290.330.500.540.55 F for change in R 2 22.6***3.2*3.6*11.4***2.23.6*30.1***10.0***1.5 Note: Standardized Betas reported: *p <.05, **p <.01, and ***p <.001 (one-tailed).

36. Equivalence of Intervention and Comparison Groups on Selected Child and School Attributes Total ComparisonInterventionp-value Female child, %44.240.046.7.410 Native American child, %43.040.044.8.555 White child, %55.260.052.4.347 Other race/ethnicity, %1.802.9.189 Child in 4 th grade at baseline, %30.925.034.3.217 Child in 5 th grade at baseline, %38.138.338.1.976 Child in 5 th grade at baseline, %30.936.727.6.229 School involvement at baseline, mean4.24.14.3.161 End of course grade at baseline, mean652.1650.9652.7.711 School size, mean11590140.085 Eligible for Free lunch, %53.851.955.6.896 Minority students, %38.541.935.1.897 Pupil/teacher ratio10.68.712.6.058 Note. There were 165 participants (105 intervention and 60 control students) and 4 schools (2 intervention and 2 control schools).