The Effect of Computers on Student Writing: A Meta-Analysis of Studies from 1992 to 2002 Amie Goldberg, Michael Russell, & Abigail Cook Technology and Assessment Study Collaborative, Boston College

Computers in the schools Increase in presence: –1983: one computer for every 125 students –1995: one for every nine students in 1995 –2001: one for every 4.2 students –(Glennan & Melmed, 1996; Market Data Retrieval, 2001) –Most common educational use of computers by students is for word processing (Becker, 1999; inTASC, 2003)

Our study: Builds on earlier research (Cochran-Smith, 1991; Bangert-Drowns, 1993) Covers the new generation of research spanning from 1992 through 2002 Combines quantitative and qualitative measures Focuses on a generation of research that differs in two ways from earlier research: –vast improvements in technology –increased student usage and comfort levels with the technology

Research Questions: Does word processing impact K-12 writing? If so, in what ways? (quantity and/or quality) Does the impact of word processing on student writing vary according to other factors, such as student-level characteristics (grade level, keyboarding skills, urban/suburban/rural school setting, etc.)

Meta-Analysis Gene Glass - the first to propose such methods and coined the term in “Meta analysis refers to the analysis of analyses … it … refer[s] to the statistical analysis of a large collection of results from individual studies for the purpose of integrating the findings. It connotes a rigorous alternative to the casual, narrative discussions of research studies which typify our attempts to make sense of the rapidly expanding research literature”.

Methodology Employed procedures detailed by Lipsey and Wilson (2001) as well as those set forth by Hedges and Olkin (1985). Five main phases: –identification of relevant studies –determination for inclusion –coding –effect size extraction and calculation, and –data analyses.

Determination for inclusion Criteria: Quantitative study conducted between Results reported in a way that would allow effect size calculation Research design that employed a measure of word-processing’s impact on writing over time OR a direct comparison between paper-and-pencil and computerized writing Quality and/or quantity of student writing and/or revision of student writing as its outcome measure(s). Not specifically focus on the effects of grammar- and spell- checkers or heavily multimedia enhanced software Not examine the differences in writing within the context of a test administration (i.e., mode of administration rather than mode of learning) focus on students in Grades K-12.

Articles collected (N=99)

Coding of outcome measures Quality holistic (n=10) vs. individual dimension scores (n=5) Quantity number of words (n=14) Revision diverse operational definitions- insertions, deletions, corrections, surface/format changes, content/meaning changes (n=6)

Extracting and calculating effect sizes Calculation: mean performance difference between computerized and paper-and pencil groups divided by the pooled standard deviation.2 thru.5 - small.5 thru.8 - medium.8 or higher - large Unit of analysis: “independent study finding” -- controls for Type I errors

Adjusting for bias and applying inverse variance weights Each effect size multiplied by the inverse of its sampling variance in order to give more weight to findings based on larger sample sizes Outlier analysis (+ or - 2 SD), publication bias (Forest plots, funnel plots, and fail-safe N analysis)

Significance and homogeneity analysis Set of independent effect sizes were aggregated and tested for homogeneity: is the group of effect sizes part of the same population, and thus, are not influenced by any other variable?

Summary of findings- Quantity

Summary of findings- Quantity(2) Regression analyses: following groups of variables were not significant factors affecting the quantity: student supports (keyboard training in study, technical assistance, teacher feedback, and peer editing) student characteristics (keyboard training prior to study, student achievement level, school setting and grade level) study characteristics (i.e., publication type, presence of control group, pre-post design, length of study) Regression analyses on subset of studies with interventions longer than six weeks revealed: effect sizes were larger for studies situated in middle and high school students as compared with elementary school students.

Summary of findings- Quality

Summary of findings- Quality(2) Regression analyses: following groups of variables were not significant factors affecting the quantity: student supports (keyboard training in study, technical assistance, teacher feedback, and peer editing, etc.) study characteristics (i.e., publication type, presence of control group, pre- post design, length of study, etc.) However in analyzing student characteristics, a significant, positive relationship was found between grade level and effect size. Regression analyses on subset of studies with interventions longer than six weeks revealed no significant relationships, indicating relationship between school level and quality occurred regardless of length of study.

Summary of findings- Revisions Meta-analytic techniques could not be applied Nonetheless, all six studies reported more changes to their writing between drafts with word processors as compared with paper-and-pencil.

Summary of findings Excluded studies Sixty-five articles were determined to be focused on the effects of computers on student writing Some not on variables of our interest Some did not report statistics necessary to calculate effect sizes Others employed qualitative methodologies and covered such topics as: writing as a social process writing as an iterative process computers and motivation keyboarding and computers generally positive effects on student writing

Conclusions Computers have a positive impact on the quantity and quality of student writing Similar to previous era of research: fairly large positive effect on quantity of writing Relationship between computers and quality of writing appears to have strengthened over time: On average students who develop their writing skills while using a computer produce written work that is.4 standard deviations higher in quality than those students who learn to write on paper. Qualitative study analysis is consistent with earlier research syntheses