ED PACE Educational Development for Planning and Conducting Evaluations West Virginia Department of Education

Project Design

Goals Provide empirical data on student achievement in virtual foreign language courses using a quasi- experimental design with experimental elements Develop a framework for scientifically based research that can be used at the local, state, and national level to measure the impact of technology enhanced programs on student achievement

Description ED PACE will generate and implement three research models and methods that can be replicated in other settings: a summative research model a formative research model a training model See revised project outputs chart FOR MORE INFO...

Summative Research Model Research Question –Does participation in program lead to student achievement gains? Methodology –Quasi-experimental Outcome Measures –Individual achievement data from WVEIS* FOR MORE INFO... *West Virginia Education Information System (WVEIS)

Sample Studied Treatment Group Randomized Treatment Control Group I Control Group II* Control Group III* FOR MORE INFO... *see revised sample and data collection chart

Outcome Measures A state standardized test –WESTEST ( ) –SAT9 ( ) A rubric-scored writing assessment –7th and 10th grade only UNIACT Interest Inventory –ACT Plan (8th) –ACT Explore (10th) Language Acquisition Assessment Spanish I

Analytic Methods Step 1) Analysis of Students Outcomes Comparison of Means and Mean Gains (randomized groups) Regression Analysis (non-randomized groups) -Control for students past achievement -Control for schools past achievement (controls for current year non-program education when appropriate)

Analytic Methods Step 2) Explore Controls to Generate Matching Criteria Goal: provide state with set of school and student-level variables that are related to outcomes. Process: -Analyze other student and school-level variables (OLS and HLM) -Determine extent to which they are subsumed within models current controls / account for additional variation

Formative Research Model Research Question: –How is the program implemented in different settings? Methodology –Documentation of full and sub-sample of treatment group over three year period Sample –Year 1: all sites with treatment (27 schools) –Year 2: new sites and 5 focus sites (multiple visits) –Year 3: new sites and 5 focus sites (3 repeat, 2 new)

Formative Research Goals: –Provide feedback for virtual both Spanish 1A/1B courses and Virtual School courses in general about what elements lead to successful implementation –Inform the summative analysis of achievement data Measures –Observation protocol –Interview protocol (principal/teacher/parent) –Surveys (student/teacher)

Training Model Goal: –Transfer models and methods developed during summative and formative research to state and local practitioners Methodology Mentor-Novice model to develop training materials –Year One: shadow researchers –Year Two: develop materials for training and pilot –Year Three: Implement training with local entity* FOR MORE INFO... * efforts will be made to extend training model to other states

Intervention

Spanish Project West Virginia Board of Education Policy 2510 required the offering of foreign language at the 7 th and 8 th grade levels by the fall of In response to requests from numerous school systems, the West Virginia Virtual School (WVVS) submitted and received a grant from the United States Department of Education to develop and deliver via Internet technology a middle school Spanish course.

Spanish Delivery The course is delivered in a two–year split model option of Spanish I. Spanish IA (7th grade) covers approximately one-half of the content standards for Spanish I. Students who successfully complete Spanish IA enroll in Spanish IB (8th grade) Upon completion of both years of study, students who have passed the courses will be awarded one (1) high school credit.

A Small Class at Paw Paw

A Large Class at Warm Springs

Classroom Visits

Time for El teléfono

Handbook – example of numerous hands-on activities

Los Cuadernos – 4 Modules

Spanish 1A/1B

Implementation

ED PACE Team The West Virginia Department of Education will draw on several resources both within and outside of the department: –Rockman et al –The EdVenture Group –WVEIS –Virtual School –SETDA FOR MORE INFO... See related links

Management Plan FOR MORE INFO... See organization plan with reporting schedule

Schedule: Year One Gather historical and current data on program FOR MORE INFO... Visit project calendar at: OctNovDecJanFebMarAprMayJunJul Aug Classroom Observations WVEIS Request Develop Instruments WVEIS request Analyze data Observe researchers Gather and Clean data Analyze data

Schedule: Year Two Track short and long-term gains and facilitate transfer to state FOR MORE INFO... Visit project calendar at: OctNovDecJanFebMarAprMayJunJul Aug Student Observations WVEIS Request Develop Instruments Analysis Analyze data Develop training modelCollect observation data Gather and Clean DataGather and Clean data

Schedule: Year Three Summative Analysis of program impact and evaluation of transfer to state FOR MORE INFO... Visit project calendar at: OctNovDecJanFebMarAprMayJunJul Aug WVEIS Request Evaluation of training model and transfer Analysis Analyze field data Mentor trainees in the field Receive training Train Virtual School Collect field Data Gather and Clean data

Schedule: Year Four Build capacity at the state level for future evaluation efforts FOR MORE INFO... Visit project calendar at: OctNovDecJanFebMarAprMayJunJul Aug WVEIS Request Analysis Disseminate model to other statesTrain additional WV agencies Collect field data on new programs

Related Links Project Calendar Rockman et al website Virtual School website WVEIS website