1. ED PACE Educational Development for Planning and Conducting Evaluations West Virginia Department of Education

2. Project Design

3. Goals Provide empirical data on student achievement in virtual foreign language courses using a quasi- experimental design with experimental elements Develop a framework for scientifically based research that can be used at the local, state, and national level to measure the impact of technology enhanced programs on student achievement

4. Description ED PACE will generate and implement three research models and methods that can be replicated in other settings: a summative research model a formative research model a training model See revised project outputs chart FOR MORE INFO...

5. Summative Research Model Research Question –Does participation in program lead to student achievement gains? Methodology –Quasi-experimental Outcome Measures –Individual achievement data from WVEIS* FOR MORE INFO... *West Virginia Education Information System (WVEIS)

6. Sample Studied Treatment Group Randomized Treatment Control Group I Control Group II* Control Group III* FOR MORE INFO... *see revised sample and data collection chart

7. Outcome Measures A state standardized test –WESTEST (2004-2006) –SAT9 (2001-2003) A rubric-scored writing assessment –7th and 10th grade only UNIACT Interest Inventory –ACT Plan (8th) –ACT Explore (10th) Language Acquisition Assessment Spanish I

8. Analytic Methods Step 1) Analysis of Students Outcomes Comparison of Means and Mean Gains (randomized groups) Regression Analysis (non-randomized groups) -Control for students past achievement -Control for schools past achievement (controls for current year non-program education when appropriate)

9. Analytic Methods Step 2) Explore Controls to Generate Matching Criteria Goal: provide state with set of school and student-level variables that are related to outcomes. Process: -Analyze other student and school-level variables (OLS and HLM) -Determine extent to which they are subsumed within models current controls / account for additional variation

10. Formative Research Model Research Question: –How is the program implemented in different settings? Methodology –Documentation of full and sub-sample of treatment group over three year period Sample –Year 1: all sites with treatment (27 schools) –Year 2: new sites and 5 focus sites (multiple visits) –Year 3: new sites and 5 focus sites (3 repeat, 2 new)

11. Formative Research Goals: –Provide feedback for virtual both Spanish 1A/1B courses and Virtual School courses in general about what elements lead to successful implementation –Inform the summative analysis of achievement data Measures –Observation protocol –Interview protocol (principal/teacher/parent) –Surveys (student/teacher)

12. Training Model Goal: –Transfer models and methods developed during summative and formative research to state and local practitioners Methodology Mentor-Novice model to develop training materials –Year One: shadow researchers –Year Two: develop materials for training and pilot –Year Three: Implement training with local entity* FOR MORE INFO... * efforts will be made to extend training model to other states

13. Intervention

14. Spanish Project West Virginia Board of Education Policy 2510 required the offering of foreign language at the 7 th and 8 th grade levels by the fall of 2002. In response to requests from numerous school systems, the West Virginia Virtual School (WVVS) submitted and received a grant from the United States Department of Education to develop and deliver via Internet technology a middle school Spanish course.

15. Spanish Delivery The course is delivered in a two–year split model option of Spanish I. Spanish IA (7th grade) covers approximately one-half of the content standards for Spanish I. Students who successfully complete Spanish IA enroll in Spanish IB (8th grade) Upon completion of both years of study, students who have passed the courses will be awarded one (1) high school credit.

16. A Small Class at Paw Paw

17. A Large Class at Warm Springs

18. Classroom Visits

19. Time for El teléfono

20. Handbook – example of numerous hands-on activities

21. Los Cuadernos – 4 Modules

22. Spanish 1A/1B 2003-2004

23. Implementation

24. ED PACE Team The West Virginia Department of Education will draw on several resources both within and outside of the department: –Rockman et al –The EdVenture Group –WVEIS –Virtual School –SETDA FOR MORE INFO... See related links

25. Management Plan FOR MORE INFO... See organization plan with reporting schedule

26. Schedule: Year One Gather historical and current data on program FOR MORE INFO... Visit project calendar at: http://wvde.state.wv.us/calendar http://wvde.state.wv.us/calendar OctNovDecJanFebMarAprMayJunJul Aug Classroom Observations WVEIS Request 2003-2004 Develop Instruments WVEIS request 2002-2003 Analyze 2003-2004 data Observe researchers Gather and Clean data Analyze data

27. Schedule: Year Two Track short and long-term gains and facilitate transfer to state FOR MORE INFO... Visit project calendar at: http://wvde.state.wv.us/calendar http://wvde.state.wv.us/calendar OctNovDecJanFebMarAprMayJunJul Aug Student Observations WVEIS Request 2004-2005 Develop Instruments Analysis 2003-2004 Analyze 2004-2005 data Develop training modelCollect observation data Gather and Clean DataGather and Clean data

28. Schedule: Year Three Summative Analysis of program impact and evaluation of transfer to state FOR MORE INFO... Visit project calendar at: http://wvde.state.wv.us/calendar http://wvde.state.wv.us/calendar OctNovDecJanFebMarAprMayJunJul Aug WVEIS Request 2005-2006 Evaluation of training model and transfer Analysis 2004-2005 Analyze 2005-2006 field data Mentor trainees in the field Receive training Train Virtual School Collect field Data Gather and Clean data

29. Schedule: Year Four Build capacity at the state level for future evaluation efforts FOR MORE INFO... Visit project calendar at: http://wvde.state.wv.us/calendar http://wvde.state.wv.us/calendar OctNovDecJanFebMarAprMayJunJul Aug WVEIS Request 2006-2007 Analysis 2005-2006 Disseminate model to other statesTrain additional WV agencies Collect field data on new programs

30. Related Links Project Calendar http://wvde.state.wv.us/calendar Rockman et al website http://rockman.com/ Virtual School website http://virtualschool.k12.wv.us/vschool/ WVEIS website http://wveis.k12.wv.us/