Educational Action Research Todd Twyman Week 2

Gathering Quantitative Data Numbers! Attendance records, test scores, grades, specific counts of behavior, responses to selected response survey questions, etc. Attendance records, test scores, grades, specific counts of behavior, responses to selected response survey questions, etc. Consider the validity of the source of your numbers for the specific use you intend. Consider the validity of the source of your numbers for the specific use you intend. Has this use been empirically validated? Has this use been empirically validated? How do you know the numbers are ‘good’? How do you know the numbers are ‘good’?

Designing Your Dependent Measure 1. Identify the construct. 2. Identify the purpose of the measure. 3. Define the content to address the purpose and construct. 4. Decide on the structure of the test. 5. Formalize administration protocols. 6. Write down the scoring methods.

Structure of the Measure Consider the construct under investigation and the intended use of the observed scores. What level of knowledge are you hoping to measure? Format of the Measure Format of the Measure Selection/Production Response Selection/Production Response Written/Oral/Demonstration/etc. Written/Oral/Demonstration/etc.

Structure of the Measure Context of the Problem Context of the Problem Situated/Not Situated Situated/Not Situated Context of Administration Context of Administration Timed/Not Timed Timed/Not Timed Group/Individual Group/Individual Standardized/Not Standardized Standardized/Not Standardized

Scoring the Behavior Consider the construct under investigation and the intended use of the observed scores Type of Scoring System Type of Scoring System Objective/Subjective (analytic/holistic rubrics) Objective/Subjective (analytic/holistic rubrics) Dichotomous/Partial-Credit Dichotomous/Partial-Credit

Scoring the Behavior Type of Reporting Type of Reporting Letter grade/number or percentage correct/2 category/ checklist/narrative report Letter grade/number or percentage correct/2 category/ checklist/narrative report Basis for Interpreting Scores Basis for Interpreting Scores Norm-/Criterion-/Individual-reference Norm-/Criterion-/Individual-reference

Selected Response Items Make sure each answer choice is a viable option (syntactically possible). Make sure each answer choice is a viable option (syntactically possible). Make sure each answer choice is approximately the same length. Make sure each answer choice is approximately the same length. Randomly order the correct answer choice location. Randomly order the correct answer choice location. Avoid confusing question stems that might detract from people’s understanding. Example Avoid confusing question stems that might detract from people’s understanding. Example

Poor Example During the past month, have you felt depressed? 0 = No 1 = Yes, once in a while 2 = Yes, some of the time 3 = Yes, most of the time 4 = Yes, all of the time Harder to develop questions and response categories May force invalid responses Less depth and substance

Poor Example, cont. Do you think it should be possible for a pregnant woman to obtain a legal abortion? Circle your response. 1 = Yes 2 = No Respondents unable to explain, qualify, or clarify answer

Dos and Don’ts Be Concise Poor:How do you feel about building an ice arena in downtown Coquitlam where the railroad property has been sitting unused for a number of years? Better:An ice arena should be built on the railroad property in downtown Coquitlam. 1 = Strongly agree 2 = Agree 3 = Disagree 4 = Strongly disagree

Dos and Don’ts, cont. Use simple, clear language Poor:How often do you punish your toddler? Better:How often do you put your toddler into timeout? Check only one. ___ Once a day ___ Several times a day ___ Once a week ___ Several times a week

Dos and Don’ts, cont. Make questions concrete Poor:Did you enjoy the book? Better:Have you recommended the book to anyone else? Use mutually exclusive and exhaustive categories Poor:What is your marital status? ___ Married___ Single Better:What is your marital status? ___ Married___ Divorced___ Separated ___ Widowed___ Never Married

Dos and Don’ts, cont. Limit “skip”patterns Do you participate in sports? 1 = No (GO TO QUESTION 3) 2 = Yes (Check all sports that apply) ___ Football___ Volleyball ___ Basketball___ Soccer ___ Swimming___ Other (Specify_________)

Dos and Don’ts, cont. Use caution when asking personal questions Poor:How much do you earn each year? $______________ Better:In which category does your annual income last year best fit? ___ Below $10,000___ $10,001-$20,000 ___ $20,001-$30,000___ $30,001-$40,000 ___ $40,001-$50,000___ $50,001-$60,000 ___ $60,001-$70,000___ over $70,001

Formatting a Survey Focus on clarity of questions. Focus on clarity of questions. Consider carefully the # of answer choices. Consider carefully the # of answer choices. Use appropriate language for your target audience. Use appropriate language for your target audience. Have your survey reviewed for appropriateness prior to use. Have your survey reviewed for appropriateness prior to use.

Formatting a Survey Include a short explanation of what you are planning to do with the results/ why you are seeking their input. Make a plug for why their input is really important to you (how it will help improve something if possible). Include a short explanation of what you are planning to do with the results/ why you are seeking their input. Make a plug for why their input is really important to you (how it will help improve something if possible). Try to fit all the survey items on 1 piece of paper – double sided is OK. Try to fit all the survey items on 1 piece of paper – double sided is OK.

Formatting a Survey If possible, phrase the question stems as “I statements” rather than as questions. Example Include demographic information that will allow you to group respondents for later analysis – be sure to consider how you will analyze the data prior to administering the survey so you have this covered! Example

Your Turn! Write questions for a survey or a selected response test appropriate for your Action Research study. (10 minutes) Write questions for a survey or a selected response test appropriate for your Action Research study. (10 minutes) We’ll hold ‘content review’ focus groups as a means of providing you with feedback on your instrument. We’ll hold ‘content review’ focus groups as a means of providing you with feedback on your instrument.

Content Review Focus Groups Groups of 5 people. Groups of 5 people. Decide on appropriate questions for the group to consider. (2 minutes) Decide on appropriate questions for the group to consider. (2 minutes) Take turns facilitating [for discussion of your own measurement instrument] and acting as note-taker. (5 minutes) Take turns facilitating [for discussion of your own measurement instrument] and acting as note-taker. (5 minutes) Review each person’s survey or test and provide feedback on ways to improve. Review each person’s survey or test and provide feedback on ways to improve.