Evaluation of Learning Systems Bertram C. Bruce Graduate School of Library and Information Science University of Illinois, Urbana-Champaign

Evaluation “the systematic collection and interpretation of evidence, leading, as part of the process, to a judgment of value with a view to action” (Beeby, 1977)

Basic questions for assessment What do you want to assess? What evidence can you gather? How, when, where? How will you interpret the results? To whom will you report? What will you do about the results?

Assessing the measurement Reliability: Is the measurement representative? Is it repeatable? Validity: Does it measure what matters?

Assessments should... Draw on multiple sources of evidence Occur on a regular basis Closely align with activity Measure work on authentic tasks Provide information for multiple stakeholders

Purpose and method Ongoing assessments: frequent analyses of performance; informal,brief, based on existing practice; formative information to guide action Focused assessments: periodic analyses of performance designed to measure progress in specific areas; formal, summative information to evaluate mastery of specific skills or concepts

Evaluation approaches interpretivist/ functionalist internal/external responsive/objective qualitative/quantitative longitudinal./snapshot formative/summative process/product goal-free/goal-directed descriptive/normative emergent/pre-set case study/systemic analysis unique uses/uniform impact open inquiry/audit review

Challenges (esp. for ICT)...complex characteristics...changes with scale...reshapes geography...limited trail of use...radically changeable...appropriated within social practices...access issues

Technology as social practice How do we relate learning to ICT use? How can that inform development? How do participants interpret ICT’s? How do they construct their activities? What aspects persist across realizations? What materials, activities, and collaborative environments facilitate success?

Organizational change Developmental Multidimensional Constructive Situated in existing practices Concerns-based adoption model Component checklist Scenario-based design/evaluation Situated evaluation

Concerns-based adoption model Stage of Concern 6. Refocusing 5. Collaboration 4. Consequence 3. Management 2. Personal 1. Informational 0. Awareness Expression of Concern How can I improve it? …do I relate it to others? …does it affect learners? …do I organize it? …does it affect me? What is it? ---

Component checklist Low-cost assessment Can be done by participants (cf. responsive) Can incorporate existing practices Assumes developmental process Identify components needing attention Applies at site and project levels

Quill components Planning Integration of reading and writing Publishing Meaningful communication Collaboration Revision Classroom management Use of PLANNER Integration with content areas Sharing writing Writing for different audiences Use of LIBRARY and MAILBAG Writing in different genres Working in pairs Teacher's comments Teaching revision Conferencing Frequency of revision Nature of revision Frequency of use Scheduling of QUILL Composing at the computer Students using QUILL Classroom structure

Component checklist example Active use Never Rare, few participants, forced Occasional Frequent Fully integrated into daily practice

Scenario-based design and evaluation Example: Border learning Describe a scenario of successful implementation Identify the key components Specify levels of adoption

Situated evaluation How well does it work? (summative) How can it be improved? (formative) What practices emerge as the innovation is incorporated into different settings? (situated) => How well do they work? How can they be improved?

Electronic Networks for Interaction New social dimensions in the classroom Immersion in a writing community Collaboration in writing Writing for authentic purposes Writing across the curriculum Writing process made visible

Realizations of ENFI Text sharing Drama Socratic tutoring Scenarios Small group discussions Brainstorming Collaborative writing Devil’s Advocate Distance networking Twenty questions Cross-age tutoring Discussion of reading Discussion of issues Open discussion

Alternate realizations

Aspects of situated evaluation Analyze the innovation Analyze existing practices Observe changes over time as the innovation is incorporated into practice Examine the functional relevance of differences in realizations Reanalyze innovation => emergent properties

Data sources Classroom observations; videotapes Interviews with students, teachers,... Student writing Survey data Students. teachers, experts Curriculum guides Teachers’ writing about their classrooms Participant feedback

Changes mediated by the contexts of use Students’ beliefs and values Parents’ beliefs and values Teacher’s pedagogical approach Teacher’s view of the educational potential Classroom management issues Support Institutional realities

Implications of situated evaluation Evaluation: Understand use in diverse contexts Teacher education: Innovation begins with the teacher Curriculum development: Useful tools for the re-creation process Re-creation: Vital part of the process of change

References Bruce, B. C., & Rubin, A. D. (1993). Electronic Quills: A situated evaluation of using computers for writing in classrooms. Hillsdale, NJ: Lawrence Erlbaum. (§§8) Electronic Quills: A situated evaluation of using computers for writing in classrooms Bruce, B. C., Peyton, J. K., & Batson, T. W. (Eds.). (1993). Network-based classrooms: Promises and realities. New York: Cambridge University Press. (§§2)Network-based classrooms: Promises and realities Loucks-Horsley, S., & Bybee, R. W. (1998 ). Implementing the National Science Education Standards. Science Teacher, 65(6),