1. Can Democratic Evaluation be Scientific? Ann Ooms Frances Lawrenz University of Minnesota College of Education and Human Development Department of Educational Psychology

2. Overview of Presentation What is Democratic Evaluation? History of Democratic Evaluation. What is Scientific Evaluation? Two examples of Democratic Scientific Evaluation Efforts. Conclusion.

3. Democratic Evaluation: defined “Government by the people” Exercised directly or through elected representatives Based on principles of social equality and respect for the individual

4. Democratic Evaluation: History MacDonald (1973, 1977): democratic evaluation Bryk (1983): stakeholder-based evaluation Fetterman (1994): empowerment oriented evaluation

5. Democratic Evaluation: History Floc’hlay and Plottu (1998): Model for the Operationalization of Democratic Evaluation Empowerment Evaluation Participatory Evaluation Multi-criteria Evaluation Counterpower exercised by those who do not agree

6. Democratic Evaluation: History House and Howe (1999): Model of Democratic Deliberative Evaluation Inclusion Dialogue Deliberation Impact: Better informed decision-making parties A thoughtful and deliberated population

7. Democratic Evaluation: History Patton (2002): think evaluatively suggests including a methodological dialogue to maintain methodological quality.

8. Democratic Evaluation: our definition If an evaluation is democratic: All interests are represented Procedures for controlling any imbalances of power All groups participate seriously and authentically in meaningful ways Groups participate in appropriate ways Reflective deliberation about findings and implications

9. Scientific Evaluation: defined No Child Left Behind Act of 2001 U.S. National Research Council’s Committee of Scientific Principles for Education Research

10. Scientific Evaluation: our definition Scientific Evaluations are: Evaluations that use experimental and comparison groups or some other sort of quasi- experimental design that can demonstrate causality

11. Democratic, Scientific Evaluation: Two Examples Collaboratives for Excellence in Teacher Preparation (CETP) Program Frances Lawrenz AlphaSmart Evaluation Ann Ooms

12. CETP Collaboratives for Excellence in Teacher Preparation evaluation Frances Lawrenz

13. CETP: Democratic? There were power imbalances Evaluation was required by the funder and was conducted for the funder There was no full participation in the CETP evaluation There were limited decisions about how results were to be used

14. CETP: Scientific? The pre post assessments of faculty instructional approaches Comparison of existing qualities of institutions, classrooms, teachers, etc. to pre determined standards A random selection of participants Comparison of students in matched classes

15. CETP: Scientific? None of these meet the “gold standard” of randomly assigned experimental and comparison groups but they do have some elements of experimental design and causality Also through the negotiation process, additional “non scientific” data were also included

16. AlphaSmart Evaluation Ann Ooms

17. AlphaSmart: Democratic? Not all stakeholders were represented. For example, there were no principals, parents or students in the inquiry group There were minor imbalances of power in the group: the teachers had more power

18. AlphaSmart: Democratic? The members of the inquiry group were involved seriously and authentically and they participated in ways which matched their skills There was reflective discussion about the findings and whether or not the AlphaSmarts should be used

19. AlphaSmart: Scientific? There was no comparison group Students were not pre tested: the study was a post test only However, there were pre and post interviews and regular meetings with the teachers which might provide some indication of causality

20. AlphaSmart: Scientific? The students’ retrospective opinions were gathered Additionally the evaluators and the teachers observed the students as they used the AlphaSmarts and were able to form their own opinions of the success or failure of the technology

21. Conclusion of two examples As in past work, producing democratic evaluations appears to be quite difficult The two examples did NOT accomplish a full democratic scientific evaluation

22. Limitations of Scientific Evaluation Scientific Evaluation narrows the meaning of evaluation to achievement of specific outcomes Scientific Evaluation deemphasizes the importance of understanding the process and meaning

23. Limitations of Scientific Evaluation Scientific Evaluation does not accommodate for methodological dialogs Scientific Evaluation precludes the participation of anyone who has different philosophical perspectives

24. Conclusion Scientific Evaluation seems only possible if everyone involved has the same philosophical perspective. Democratic, Scientific Evaluation: a wonderful but inaccessible dream?

25. Contact Information Ann Ooms: ooms0001@umn.eduooms0001@umn.edu Frances Lawrenz: lawrenz@umn.edulawrenz@umn.edu