1. Incorporating an Evaluation Plan into Program Design: Using Qualitative Data Connie Baird Thomas, PhD Linda H. Southward, PhD Colleen McKee, MS Social Science Research Center Mississippi State University Positioned for Progress Conference April 30, 2014 Jackson, MS

2. Workshop Goals Understand the difference between qualitative and quantitative data Learn how to capture qualitative data in a rigorous objective way Explore best practices related to qualitative data collection

3. PROJECT ABC – Literacy Intervention Program Conceptualize—“If we do X, then Y will result” ACTIVITIES: (X)  Pilot program in 10 schools  Group tutoring  One-on-one tutoring  Curriculum  Take home materials  Parent participation  Trained facilitators OUTCOME (Y): Proficient reading by fourth grade

4. What is Qualitative Research? Data collection method in which the researcher: Seeks an in-depth, “rich” view of topic Explores an idea or topic to describe or give it meaning Asks participants broad, general questions Collects detailed views of participants or subject Analyzes and codes data for description or themes

5. Characteristics of Qualitative and Quantitative Research Qualitative Goal: describe, discover Design: flexible, emergent Sample: small, non-random Data collection: interviews, observation Data interpretation: subjective Quantitative Goal: predict, confirm Design: structured Sample: random, large representative Data collection: surveys, questionnaires Data interpretation: objective

6. Qualitative Research … Does – Explain “how” and “what” – Describe “what happened” – Explore processes over time – Explore meaning of event to those involved Does not – Prove/disprove hypothesis – Measure variables – Test theories – Show relationship between variables – Make generalizable statements

7. Methods of Qualitative Research Methods In-depth Interview Focus group Case Study Direct Observation Data “Words don’t fly individually- they fly in flocks.” Geoffrey Nunberg- University of CA, Berkley School of Information

8. Methodology In-depth Interview Advantages Open structure allows for individual to express detailed opinions and experiences Interviewer can easily follow-up and/or clarify statements Appropriate for sensitive or controversial topics or special populations Limitations Requires skilled interviewer Responses may be biased Not generalizable Time/labor intensive One-on-one conversation between interviewer and respondent regarding a specific topic Sometimes audio/video recorded

9. Methodology Focus Group Advantages Group setting often stimulates additional thoughts and ideas Gives more insight into why certain opinions are held Can better understand processes (changes that have occurred over time) Observe nonverbal responses Limitations Group/time management Peer pressure among respondents May stress resources Focused small group (usually 6- 10 participants) discussion guided by a leader or moderator Useful for populations with shared experiences

10. Methodology Case Study Advantages Provides rich, detailed information Lends to understanding of processes, interactions and context Uses different methods of data collection (interviews, observations, document analysis, etc.) Limitations Applicable only to site or event under study (limited generalizability) Requires extended time and effort In-depth study of a single unit- can be a person, organization, program, etc. Sometimes referred to as “telling a story” When to Use Case Study* Is the program is unique? Is it premature to establish impact measures? Is there unexplained variation in program impacts? Will understanding the program implementation in detail help design future programs? *E. Balbach. Using Case Studies to do Program Evaluation. (1999) California Department of Health.

11. Methodology Direct Observation Advantages Helps to understand the social, physical, political, and economic context in which participants live Can provide important information that was previously unknown Observe participants in their natural settings Complements other forms of data collection (triangulation) Limitations Time consuming/resource intensive Difficult to document everything- may have to rely on memory Objectivity is a challenge (observer bias) Observer effect Observing the participants in their natural environment and setting- either as becoming a part of the setting or being strictly an observer

12. Issues to Consider Purpose of research (re: evaluation) Subjective method (reliability and validity) Researcher bias Labor intensive Cost

13. Words fly in flocks…

14. Example: Assessing the Impact of the Mississippi Healthy Student Act: Year Four Report

15. Questions

16. Contact us at:  mskidscount@ssrc.msstate.edu  www.kidscount.ssrc.msstate.edu  Tel: 662-325-7127 Connie Baird Thomas, PhD. Tel: 601-407-2742 Email: connie.baird@ssrc.msstate.edu Linda H. Southward, PhD Tel: 662-325-0851 Email: linda.southward@ssrc.msstate.edu Colleen McKee, MS Tel: 662-325-0851 Email: colleen.mckee@ssrc.msstate.edu Thank You