1. American Educational Research Association Annual Conference New York – March 24-28, 2008 Noelle Griffin, Ph.D. Evaluation of an Arts-Based Instructional Program for Primary Grades: Preliminary Findings and Lessons Learned from WebPlay

2. Section 1: WebPlay Evaluation Overview Section 2: Survey Design and Analysis Section 3: Pre-/Post- Comparison Results Overview

3. Section 1 WebPlay Evaluation Overview

4. WebPlay Program Internet-enhanced arts education project Integrates theater arts in school curriculum Performing arts, literacy, social studies, and technology

5. WebPlay Curriculum Two weekly lessons Use of internet: Connections with partner classrooms WebPlay curriculum involves activities in the following four areas: Performing Arts English Language Arts History/Social Studies Technology

6. Evaluation 3 year evaluation beginning 2005-06 3 rd and 5 th grade classes/large urban school district Quantitative methods/small scale Both summative and formative purposes

7. Evaluation Questions: WebPlay Program 1. Will WebPlay improve student performance? 2. What is the impact of WebPlay on student skill development? 3. Do effects persist across different student/site cohorts? 4. Do different student/site characteristics interact with treatment?

8. Sampling and Methodology Sample (2006-07 school year) 18 WebPlay schools 12 matched comparison schools (for survey) Student survey (pre-/post-) Standardized state test data (English Language Arts) – future analysis

9. Evaluation Roadblocks Selecting outcome measures The focus on standardized data Lack of resources Comparison participation

10. Section 2 Survey Design and Analysis

11. Key Constructs in WebPlay Curriculum 1. Theatrical knowledge 2. Internet knowledge and safety 3. Engagement 4. Efficacy and academic esteem 5. Collaboration and broadening horizons

12. Sample Items by Construct AreasItems 1. Theatrical knowledge The location or place (“setting”) of a play is where the play is being presented. 2. Internet knowledge and safety E-mails as well as the documents sent via e-mail (“attachments”) can have viruses. 3. School engagementI think the activities I do in school are boring. 4. Efficacy and academic esteem I would be able to do a good job if I had to write a story for school. 5. Collaboration and broadening horizons I can learn new things from students who live in other countries.

13. Factor Analysis Pre- surveys N = 590 (424 WebPlay, 166 comparison) Exploratory and confirmatory factor analysis techniques

14. Exploratory Factor Analysis Based on Likert-style items Three higher order factors determined Theatrical engagement/interest General academic confidence/engagement External connections

15. Confirmatory Factor Analysis Initial fits indices mixed (i.e., not all indices reached cut-offs) Initial model revised (allow for cross- loadings, correlated residuals) Consistent with survey theoretical design Fit indices improved

16. WebPlay Student Survey: Final CFA Model

17. Section 3 Pre/Post Comparison Results

18. Survey Pre-/Post- Analyses Comparing pre/post results for WebPlay and comparison students Based on only students with link-able pre/post data (i.e., smaller sample) Hierarchical linear modeling techniques used (due to nested nature of data) Two-level model (student and school)

19. Applicability of Hierarchical Linear Models (HLMs) to Evaluation Studies HLM allows analysis of outcome variable relationship to predictors at multiple data levels Evaluation studies often involve: Intact clusters Nested data Background characteristics of individuals that may vary appreciably across different clusters

20. Overall Results Two sets of outcomes analyzed Knowledge Attitude/Engagement Knowledge: No significant group differences (p =.41) Engagement/attitude: WebPlay students increased pre-/post- relative to comparison students (p =.02)

21. HM Results for Knowledge Outcome Fixed EffectsCoefficientSEp Value Intercept7.100.21<.0001 WebPlay-0.220.260.41 Pre-test0.200.060.00 Random Effects Variance Component SEp Value Adjusted means0.180.100.03 Student residual2.120.17<.0001

22. HM Results for Engagement/Attitude Outcome Fixed EffectsCoefficientSEp Value Intercept3.800.07<.0001 WebPlay0.190.080.02 Pre-test0.570.05<.0001 Random Effects Variance Component SEp Value Adjusted mean0.000.010.43 Student residual0.360.03<.0001

23. Sub-scale Analyses Descriptive/exploratory in nature No variation in impact between types of Knowledge items (theater, internet) Some apparent variation in impact on attitude/engagement sub-items Theatrical engagement/interest WebPlay vs. comparison group differences most pronounced

24. Attitude Outcome sub-area scores by treatment group

25. Next Steps Integration of state standardized test data Revision of Knowledge items (multiple choice) Collection and integration of teacher-level data

26. griffin@cse.ucla.edu