1 © 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation See the PDE booklet, Collecting evaluation data: Direct observations Using observation to collect evaluation data

2 © 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation Checking in… What do you think? Answer YES or NO to each of the following 1.Observation involves “seeing” and “listening” 2.People may behave differently when they know they are being observed so it is better not to tell them 3.Structured observations provide more accurate and useful information 4.As long as you see it, it doesn’t matter if you record what you saw 5.You, as someone who “knows” the program and the participants, are best suited to conduct the observations 6.The same principles of sampling apply to observation as to other forms of data collection Check your answers at the end

3 © 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation Observation… Involves all 5 senses: sight, hearing, smell, touch, and taste

4 © 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation Purpose and benefits of observation as a data collection method It is unobtrusive You can see things in their natural context You can see things that may escape conscious awareness, things that are not seen by others You can discover things no else has ever really paid attention to, things that are taken for granted You can learn about things people may be unwilling to talk about It is inconspicuous – least potential for generating observer effects It is the least intrusive of all methods You can be totally creative – flexibility to yield insight into new realities or new ways of looking at old realities

5 © 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation Observation is useful when… You want direct information You are trying to understand an ongoing behavior, process, unfolding situation, or event There is physical evidence, products, or outcomes that can be readily seen Written or other data collection methods seem inappropriate

6 © 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation Observations Advantages –Most direct measure of behavior –Provides direct information –Easy to complete, saves time –Can be used in natural or experimental settings Disadvantages –May require training –Observer’s presence may create artificial situation –Potential for bias –Potential to overlook meaningful aspects –Potential for misinterpretation –Difficult to analyze

7 © 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation Is observation culturally appropriate? Things to consider: Discomfort, threat of being observed Issue of being an “outsider” Observer effect Possibilities for misinterpretations

8 © 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation Observation – Ethical issues Unobtrusiveness is its greatest strength; also potential for abuse in invasion of privacy You can venture into places and gather data almost anywhere so be ethical Remember our Human Subjects Protection guidelines –Consent form for participating in an observational study

9 © 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation Types of observation Structured Unstructured Observing what does not happen may be as important as observing what does happen. Looking forLooking at Sometimes we have something specific we want to observe – leadership skills; level of participation; etc. We use a structured, preset guide of what to observe or a checklist. Sometimes we want to see what is naturally occurring or exists without predetermined ideas. We use have an open-ended approach to observation and record all that we observe

10 © 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation Practice: Structured/unstructured observations Imagine you are sitting in a room where ten youth are sitting at computers learning about Web 2.0 applications. 1) If you want to assess to what extent students are interested and learning, what specifically would you look (listen) for? 2) If you aren’t sure what specifically indicates student interest or learning and you want to see what is going on during the demonstration, how would you proceed?

11 © 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation Steps in planning for observation Determine who/what will be observed. Determine aspects that will be observed (characteristics, attributes, behaviors, etc.). Determine where and when observations will be made. Develop the observation guide Pilot test the observation guide Train the observers and have them practice. Conduct the observations Analyze and interpret the collected information. Write up and use your findings.

12 © 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation Who/what can you observe People (individuals, groups, communities) –Characteristics –Interactions –Behaviors –Reactions Physical settings Environmental features Products/physical artifacts Use sampling strategies as you do for other methods of data collection

13 © 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation Example – Observing participation in an after school program Who you will observe: youth attending the program What you will observe: –Age, gender –Length of time student stays in the program –Involvement in activities: which activities Level of involvement –Interactions with other youth; with staff When you will observe: all hours the program is open for one week each month during 2007

14 © 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation ons Recording your observations It is not good enough to just observe, you need to record your observations. You might use: –Observation guide –Recording sheet –Checklist –Field note –Picture –Combination of the above

15 © 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation Sample Observation Guides Guide for structured observationsGuide for unstructured observations

16 © 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation Structured observation guide used for pre and post program evaluation

17 © 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation Who does the observations? You – program staff Participants - Youth Parents Teachers Volunteers Other stakeholders Colleagues

18 © 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation Training – preparation/orientation may be necessary –To learn what to look for –To learn how to record observations –To practice –To ensure that observations across sites are consistent: observers use the same methods, rate an observation in same way

19 © 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation How well do you observe? Take 15 seconds and look at the picture below. Move to the next slide or turn away from the computer screen and write down everything you observed. Then, come back to the photo and see what you missed (or thought was there and isn’t!).

20 © 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation Practice your observation skills everyday in everyway! And, add observation to your data collection toolbox.

21 © 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation Checking back in…Answers 1.Observation involves “seeing” and “listening” YES 2.People may behave differently when they know they are being observed so it is better not to tell them NO – they often DO behave differently but that is not a reason not to tell them they are being observed. 3.Structured observations provide more accurate and useful information NO – unstructured observation also can be very useful 4.As long as you see it, it doesn’t matter if you record what you saw NO – you must record your observations to have evidence of it existing 5.You, as someone who “knows” the program and the participants, are best suited to conduct the observations NO – various people might be well suited, though training may be necessary 6.The same principles of sampling apply to observation as to other forms of data collection YES