Quantitative SOTL Research Methods Krista Trinder, College of Medicine Brad Wuetherick, GMCTE October 28, 2010

Session outcomes O Identify components of good survey questions O Choose an appropriate research method O Willingly and enthusiastically participate in more workshops in this O Meet and work with other dedicated and scholarly teachers

Quantitative Tools O Statistics O Numbers O Measurement O Surveys/Questionnaires are the most common tool.

Survey Selection O Good to use pre-existing instruments that have been found to be reliable and valid. O May have to create your own if no suitable tools exist.

Creating a Survey O Decide what you want to measure O Generate a large number of items O Decide on question format O Have experts review your items O Consider including validation items O Give items to a development sample (some suggest 300) O Evaluate items O Adjust scale length

Examples of Survey Questions O This course was enjoyable and provided me with information that will benefit my future career. O A. Strongly Disagree O B. Disagree O C. Neutral O D. Agree O E. Strongly Agree

Examples of Survey Questions O I dislike courses that do not include small group work. O Better: I like courses that include small group work.

Good Survey Questions O Keep items short and concise. O Aim for a Grade 6 reading level O Avoid double negatives O Avoid double-barreled items (only ask one thing per question) O Avoid ambiguous pronoun references and phrases with unintended meanings

Quantitative Approaches O Experimental O Comparative and Associational O Descriptive

Experimental O Randomized Experimental Approach O Causality O Gold Standard O Randomly assign participants to experimental and control groups

Quasi-Experimental O Similar to an experimental approach, but without random assignment. O Poor designs: O One group posttest only O One group pretest-posttest O Posttest only with non-equivalent groups O Better design: O Pretest-posttest non-equivalent comparison group

Comparative and Associational Approaches O Examine effect of an attribute independent variable on the dependent variable. O No random assignment O No control over independent variables (IVs) O Comparative: ordered IVs with few levels (men, women). O Associational: continuous IVs with more levels (age, grades)

Descriptive Approach O Purpose is to summarize data O One variable; no comparisons or relationships investigated O Report percentages, means, frequencies etc.

Discussion of Approaches O What approach might you use?

Parametric vs. Non-Parametric O Usually use parametric analyses, which compare means (i.e., ANOVAs, t-tests). O Sometimes it is more appropriate to conduct non-parametric analyses, which compare distributions (i.e., Chi-square). O Very small sample O Different data collection methods

Reliability O Shows similar performance at different times. O Sources of reliability: O Test-retest O Internal Consistency (i.e., Cronbach’s alpha – should be at least.70) O Rater Consistency (i.e., inter-rater reliability)

Content Validity O Extent to which the content of the instrument measures what you want it to. O Define your concept. O See how it is represented in the literature O Develop test items assessing this concept. O Expert judges review items. O See handout

Criterion Validity O How well your instrument compares to an outside criterion that is known to measure the same concept. O E.g., How well does self-reported voting correlate with actual voting shown through records?

Construct Validity O The extent to which a measure correlates with other measures that should be related. O Expect correlations with measures that measure the same thing (convergent validity). O Expect low correlations with measures that should not be related (discriminant validity)

Reliability and Validity O A measure can be reliable without being valid, but cannot be valid without being reliable. O See cartoon

Mixed Methods O Do not have to use strictly quantitative or qualitative tools! O Research projects often benefit from including both quantitative and qualitative methods.

Choosing your method O See handouts O Work independently O Exchange thoughts and ideas: consult O More individual work O Leave at your own discretion Thank you! See you on November 25 th !