Situated Evaluation Chip Bruce March 9, 1999

Constructivism “Our perceptions do not come simply from the objects around us, but from out past experience as functioning, purposive organisms “... always wrong in any particular instance [book example]” --E. C. Kelley, 1947, p. 34

Hermeneutics Not occasionally only, but always, the meaning of a text goes beyond its author. That is why understanding is not merely a reproductive, but always a productive attitude as well. – H. Gadamer, Truth & Method

Intertextuality No member of a verbal community can ever find words in the language that are neutral, exempt from the aspirations and evaluations of the other, uninhabited by the other’s voice. On the contrary, he receives the word by the other’s voice and it remains filled with that voice. He intervenes in his own context from another context, already penetrated by the other’s intentions. His own intention finds a word already lived in. – M. Bakhtin, Problems of Dostoevsky’s Poetics

My history with IT evaluation 1969 generative CAI Quill => situated evaluation Statistics Workshop Cheche Konnen ENFI 1997Reader Focused Writing Evaluation of IT course

Stereotype view Quantitative only Little attention to antecedent conditions or classroom transactions Assumes fixed, knowable entities Unable to address unanticipated effects No model for diversity of realizations

Quill evaluation Differentially significant growth in writing ability across expository, persuasive, narrative modes, but... Not relevant to most stakeholders Ignored changes in classroom practices, reconceptions of curricula, etc. Obscured process of change

Electronic Networks for Interaction New social dimensions in the classroom Immersion in a writing community Collaboration in writing Writing for authentic purposes Writing across the curriculum Writing process made visible

Data sources Classroom observations; videotapes Interviews with students, teachers,... Student writing Survey data Students. teachers, experts Curriculum guides Teachers’ writing about their classrooms Participant feedback

Changes mediated by the contexts of use Students’ (parents’) beliefs and values Teacher’s pedagogical approach Teacher’s view of the educational potential Classroom management issues Support Institutional realities

Realizations of ENFI Text sharing Drama Socratic tutoring Scenarios Small group discussions Brainstorming Collaborative writing Devil’s Advocate Distance networking Twenty questions Cross-age tutoring Discussion of reading Discussion of issues Open discussion

Alternate Realizations

Evaluation questions Summative: How well does it work? Formative: How can it be improved? Situated: What practices emerge as the innovation(s) are incorporated into different settings?=> How well do they work? How can they be improved?

Aspects of situated evaluation Analyze the innovation Analyze existing practices Observe changes over time as the innovation is incorporated into practice Examine the functional relevance of differences in realizations Reanalyze innovation => emergent properties

Abstraction vs. Generalization It is a mistake to equate “abstract” with “general”. Only the concrete permits a general understanding of systemic interconnectedness. –Y. Engestrøm, “Learning by Expanding”

Pasteur’s death-bed words “Bernard is right; the pathogen is nothing; the terrain is everything. – Oliver Sacks, Awakenings

Implications: Re-creation as central to change Need to understand diverse realizations Innovation begins with the teacher Technology as tool for the re-creation

Technology as Social Practice How do we relate classroom goals to IT characteristics? How does that analysis inform further development? How do participants interpret IT resources? How do they construct their activities? What elements persist across realizations? What specific materials, activities, and collaborative environments facilitate the most successful projects?

Orientation constructivism particulars, e.g., Donald Graves reader response theory

Types of evaluation Formative Summative Situated Innovation Effects Practices Developer User Both Improve Judge Describe Minimize var. Control Use User feedback Experiment Ethnographic Development End Either List of change Comparisons Ethnography

Evaluation “the systematic collection and interpretation of evidence, leading, as part of the process, to a judgment of value with a view to action” --Beeby, 1977

Stakeholder positions software developer curriculum developer teacher teacher educator parent researcher

Countenance of educational evaluation “too little effort to spell out antecedent conditions and classroom transactions... [and] to couple them with the various outcomes” --R. E. Stake, 1967, p. 524 [reprinted in Classics of Instructional Technology, Ely & Plomp, 1996]

Responsive evaluation “if it orients more directly to program activities than to program intents; responds to audience requirements for information; and if the different value perspectives present are referred to in reporting the success and failure of the program” --R. E. Stake, 1975, p. 14