Incorporating an Evaluation Plan into Program Design: Using Qualitative Data Connie Baird Thomas, PhD Linda H. Southward, PhD Colleen McKee, MS Social Science Research Center Mississippi State University Positioned for Progress Conference April 30, 2014 Jackson, MS

Workshop Goals Understand the difference between qualitative and quantitative data Learn how to capture qualitative data in a rigorous objective way Explore best practices related to qualitative data collection

PROJECT ABC – Literacy Intervention Program Conceptualize—“If we do X, then Y will result” ACTIVITIES: (X)  Pilot program in 10 schools  Group tutoring  One-on-one tutoring  Curriculum  Take home materials  Parent participation  Trained facilitators OUTCOME (Y): Proficient reading by fourth grade

What is Qualitative Research? Data collection method in which the researcher: Seeks an in-depth, “rich” view of topic Explores an idea or topic to describe or give it meaning Asks participants broad, general questions Collects detailed views of participants or subject Analyzes and codes data for description or themes

Characteristics of Qualitative and Quantitative Research Qualitative Goal: describe, discover Design: flexible, emergent Sample: small, non-random Data collection: interviews, observation Data interpretation: subjective Quantitative Goal: predict, confirm Design: structured Sample: random, large representative Data collection: surveys, questionnaires Data interpretation: objective

Qualitative Research … Does – Explain “how” and “what” – Describe “what happened” – Explore processes over time – Explore meaning of event to those involved Does not – Prove/disprove hypothesis – Measure variables – Test theories – Show relationship between variables – Make generalizable statements

Methods of Qualitative Research Methods In-depth Interview Focus group Case Study Direct Observation Data “Words don’t fly individually- they fly in flocks.” Geoffrey Nunberg- University of CA, Berkley School of Information

Methodology In-depth Interview Advantages Open structure allows for individual to express detailed opinions and experiences Interviewer can easily follow-up and/or clarify statements Appropriate for sensitive or controversial topics or special populations Limitations Requires skilled interviewer Responses may be biased Not generalizable Time/labor intensive One-on-one conversation between interviewer and respondent regarding a specific topic Sometimes audio/video recorded

Methodology Focus Group Advantages Group setting often stimulates additional thoughts and ideas Gives more insight into why certain opinions are held Can better understand processes (changes that have occurred over time) Observe nonverbal responses Limitations Group/time management Peer pressure among respondents May stress resources Focused small group (usually participants) discussion guided by a leader or moderator Useful for populations with shared experiences

Methodology Case Study Advantages Provides rich, detailed information Lends to understanding of processes, interactions and context Uses different methods of data collection (interviews, observations, document analysis, etc.) Limitations Applicable only to site or event under study (limited generalizability) Requires extended time and effort In-depth study of a single unit- can be a person, organization, program, etc. Sometimes referred to as “telling a story” When to Use Case Study* Is the program is unique? Is it premature to establish impact measures? Is there unexplained variation in program impacts? Will understanding the program implementation in detail help design future programs? *E. Balbach. Using Case Studies to do Program Evaluation. (1999) California Department of Health.

Methodology Direct Observation Advantages Helps to understand the social, physical, political, and economic context in which participants live Can provide important information that was previously unknown Observe participants in their natural settings Complements other forms of data collection (triangulation) Limitations Time consuming/resource intensive Difficult to document everything- may have to rely on memory Objectivity is a challenge (observer bias) Observer effect Observing the participants in their natural environment and setting- either as becoming a part of the setting or being strictly an observer

Issues to Consider Purpose of research (re: evaluation) Subjective method (reliability and validity) Researcher bias Labor intensive Cost

Words fly in flocks…

Example: Assessing the Impact of the Mississippi Healthy Student Act: Year Four Report

Questions

Contact us at:    Tel: Connie Baird Thomas, PhD. Tel: Linda H. Southward, PhD Tel: Colleen McKee, MS Tel: Thank You