1. QUALITATIVE PROGRAM EVALUATION RESEARCH: EMPLOYING THE COMPLEMENTARY ANALYSIS RESEARCH MATRIX APPLICATION (CARMA) LeAnn Grogan Putney, PhD Educational Psychology & Higher Education Professor and Department Chair University of Nevada, Las Vegas Joan Wink, PhD Professor Emerita California State University, Stanislaus joan@joanwink.com EQRC – Las Vegas, NV February 11, 2014 putneyl@unlv.nevada.edu

2. YOU DO KNOW THESE HEAPS OF DATA NEED MORE ANALYSIS, RIGHT? …I TOLD YOU TO USE CARMA TO GET OUT OF THE HEAPS AND OVER TO CONCEPTS!

3. Vygotsky’s Conceptual Development – AKA, THE HEAPS 3 Heaps Complexes Concepts Generalized understandings Group similar items Connections among items

4. Critical Action/Complementary Analysis Research Matrix Application (CARMA) 4 CARMA helps you move from  Heaps of Data to  Complexes of Relationships to  Concepts related to a theoretical perspective or producing a grounded theory Putney, L. G., Wink, J & Perkins, P. (2006). Teachers as Researchers: Using the Critical Action Research Matrix Application for Reflexive Classroom Inquiry. Florida Journal of Teacher Education, Vol IX, p. 23 - 35.

5. 5 NOTE-TAKING EXPECTATIONS Who are the intended participants? Who is supposed to provide what service? How are participants to be served? What will be produced in the activity? NOTE-TAKING EVIDENT IMPLEMENTATION Who are the actual participants? Who is actually providing what service? How are participants actually served? What is actually being produced? NOTE-MAKING / ALIGNMENT OR DEPARTURE?  Compare/contrast expected with evident using Action Researcher Interpretations  For each question do you see alignment or departure? o What might be the implications? NOTE-REMAKING / RECOMMENDATIONS/  Modify or maintain program activity?  In what ways?  For which participants?  With what intended outcomes?

6. Complementary Analysis Research Matrix Application [CARMA] 6

7. Complementary Analysis Research Matrix Application (CARMA) 7 Content of fieldnotes  NoteTaking  Descriptions of setting/people/activities  Mapping of physical space  Direct quotes or substance of what was said  Note Making  Observer comments in margins – interpretations & questions  Note Remaking  What did you learn – maintain or modify?  Reflective Cycle – transformative stage

8. Potential Uses for CARMA 8 Program evaluation Applied classroom research  Self study (action research) or  Team approach with classroom teacher Educational research  School wide behavioral practice  Single-sex based classes  Comparison of several schools – English Language Learners in high ranking schools for “best practices” Policy analysis Critical incident study

9. Scale of use 9 MicroMacro LessonClassroomSchoolDistrictCommunit y StateNationalContinentalGlobal Unit/Project/Lesson/Guest speaker (Teacher’s, Speaker’s, and Students’ expectations) Classroom project/Tolerance School-wide project/Action Research – 111 teachers District-wide project/Empowerment Community-wide/Hospice – EPY 716 2008/Critical Incident dissertation

10. Scale of use 10 State-wide project/Proposition 227 in California- dissertation National-level project/No child Left Behind policy analysis – dissertation Continental level/Quality Assurance in European Higher Education Area - policy analysis – dissertation Global level/recommendations for future research studies MicroMacro LessonClassroomSchoolDistrictCommunityStateNationalContinentalGlobal

11. PARTICIPANT DISCUSSION FOR YOUR OWN RESEARCH, WHAT WAS YOUR INTENDED VS ENACTED? DID YOU ENCOUNTER WHAT YOU EXPECTED? DO YOU NEED TO COLLECT MORE DATA OR ARE YOU SEEING REPEATING PATTERNS? 11 CARMA APPLIED TO YOU

12. Complementary Analysis Research Matrix Application (CARMA) 12 WHAT - Tool for critical evaluation, to help the researcher/evaluator to describe and interpret  what is being accomplished in the setting, and  in what ways the cultural space is being utilized by the participants, by  measuring program expectations, against evident implementation of practices and outcomes,  from the perspectives of all levels of participants (Putney, Wink & Perkins, 2006).

13. Complementary Analysis Research Matrix Application (CARMA) 13 WHY - Roots of CARMA:  Vygotskian view of life as a social construction, steeped in the context of our cultural experiences  to evaluate whether the context of a program is meeting the needs of the constituents, we would have to determine what the constituents and program administrators are socially constructing through the program implementation.

14. Complementary Analysis Research Matrix Application (CARMA) 14 WHY - Roots of CARMA:  Ethnographic perspective of learning from people about their cultural patterns of interaction (Agar, SBCDG)  people, who work in a common setting for individual yet related purposes, begin to co- construct norms and expectations for being together in their local time and space.

15. Complementary Analysis Research Matrix Application (CARMA) 15 HOW – Observations and Interviews of constituents Fieldnotes – written account of observations  Continuous notes, jottings, head notes, “ cooked up ”

16. TEACHER’S REFLECTIONS ON HER OWN PRACTICE 16 Example from action research Putney, L. G., Wink, J & Perkins, P. (2006). Teachers as Researchers: Using the Critical Action Research Matrix Application for Reflexive Classroom Inquiry. Florida Journal of Teacher Education, Vol IX, p. 23 - 35.

17. Step 1 – NoteTaking Notations on classroom expectations 17

18. Step 2 – Notetaking from observations what is evident 18

19. Step 3 – results and conclusions after contrastive analysis 19

20. Step 4 - Recommendations - decisions 20