S-005 Collecting data: What methods to use

Common methods Interviews – Face-to-face – Focus group – Telephone – Skype / video conference Questionnaires – “mailed” or distributed to individuals – Completed in a group setting (classroom) – On-line questionnaires Observations – Open-ended (unstructured) Using field notes – Structured Event-driven Time sampling – Unit of observation Teacher Student group Tests – Standardized – Norm-referenced – Criterion-referenced Standards-based

More ideas Interviews Questionnaires Observations Tests Interviews Questionnaires Observations Tests Secondary data sources – Existing information that is already collected

More ideas Interviews Questionnaires Observations Tests Secondary data sources Interviews Questionnaires Observations Tests Secondary data sources Content analysis – Documents, books, newspapers – TV, media – Other artifacts

More ideas Interviews Questionnaires Observations Tests Secondary data sources Content analysis Interviews Questionnaires Observations Tests Secondary data sources Content analysis Physiological recordings – Eye movements – Heart rate, Blood pressure – Stress levels – Brain activity MRI fMRI

More ideas Interviews Questionnaires Observations Tests Secondary data sources Content analysis Physiological measures Interviews Questionnaires Observations Tests Secondary data sources Content analysis Physiological measures Unobtrusive measures – Video – Audio – Wear-and-tear assessments

More ideas Interviews Questionnaires Observations Tests Secondary data sources Content analysis Physiological measures Unobtrusive measures Interviews Questionnaires Observations Tests Secondary data sources Content analysis Physiological measures Unobtrusive measures Others? – Diaries or logs – Other ideas?

Features to consider Participation rates Costs – Time / money Flexibility Clarification Order of items – Fixed order – no looking ahead – Flexible order – based on responses Non-verbal information Literacy levels Cultural differences Participation rates Costs – Time / money Flexibility Clarification Order of items – Fixed order – no looking ahead – Flexible order – based on responses Non-verbal information Literacy levels Cultural differences Social desirability – What is the “right” answer – What do they want me to say? Sensitive topics Response sets Timing Practice / familiarity – With the topics – With the terminology – With the format Setting – School, home, other – Formal or informal Format Technology Administering – Who will do this? Social desirability – What is the “right” answer – What do they want me to say? Sensitive topics Response sets Timing Practice / familiarity – With the topics – With the terminology – With the format Setting – School, home, other – Formal or informal Format Technology Administering – Who will do this? Other issues to add?

Examples to consider We are interested in students’ attitudes toward school. How engaged and involved are they? 1. Ask with a questionnaire Consistency, face validity, efficiency. But... Will they know this? Accuracy of self reports? Superficial? 2. Observe the students in classrooms (and other settings?) Consistency, face validity. But... Who will observe? Coverage? Where to observe? Consistency? 4. Track attendance, grades, test scores, participation in school activities Unobtrusive, indirect, extended over time, quantifiable But... Validity of this? 3. Interview students, teachers, parents Direct (face validity), comprehensive But... Response rates? Knowledge of issues? Interviewer bias? Feasibility?

A mix of strategies Mixed methods tend to work best – In developing best methods – In analyzing and interpreting Quantitative and qualitative approaches – Questionnaires and interviews and observations (and others) – Open-ended and closed-option options – Expert opinions and a variety of other voices – Multiple perspectives Be rigorous and careful at all stages – Consider alternative approaches – Consider alternative interpretations Mixed methods tend to work best – In developing best methods – In analyzing and interpreting Quantitative and qualitative approaches – Questionnaires and interviews and observations (and others) – Open-ended and closed-option options – Expert opinions and a variety of other voices – Multiple perspectives Be rigorous and careful at all stages – Consider alternative approaches – Consider alternative interpretations