1. Depending on the Mean and the Midpoint, Random Responders Cause Type 1 Error Zdravko Marjanovic, Noor Shubear, Tsz Yin Fung, & Lisa Bajkov Thompson Rivers University The validity of questionnaire data is premised on the expectation that responders are conscientious responders (CR)—they answer items as honestly and accurately as they can. The problem is, for a variety of reasons such as disinterest and fatigue, some responders are random responders (RR)—they answer items carelessly or indiscriminately, without regard for what items are asking them to consider. If RR go undetected and are by default interpreted as CR, their data may lead testers to reach wrong conclusions and make erroneous descriptions, diagnoses, and predictions. Despite this, testers generally do not seek to identify and remove RR because they are (1) notoriously difficult to detect because the means to do so are limited and (2) only thought to affect statistics by increasing the likelihood of Type 2 error. The purpose of this study was to show that (1) a newly developed validity scale called the CRS (see below) reliably identifies RR in data and (2) random responding increases the likelihood of Type 1 erro r when a measure’s mean score is far from its response scales’ midpoint value. The farther away a measure’s mean is from its midpoint, the greater the likelihood that RR causes Type 1 error. Hypothesis 2 supported: t-test differences between responder groups varied depending on the distance between each measure’s mean and midpoint. Table 2 Mean-Midpoint Differences Predict t-Test Magnitudes between Responder Groups RRG CRG Measure D Dt-testd CRS -60.0076.9228.96***3.07 SES0.3731.718.61***0.90 SDT-P1.18-40.54-14.77***-1.51 GSES1.1739.7013.33***1.38 CESD-R-0.56-54.79-21.10***-2.32 __________________________________________________________ Note. D=distance between a scale mean and its midpoint, where -100=a mean farthest below the midpoint, 100=a mean farthest above the midpoint, and 0=a mean at the midpoint. ***=p<.001. All t-test dfs=372. d=Cohen’s d effect size. Hypothesis 2 further (mixed) support. Adding random data to conscientiously generated data sets increases correlation magnitudes between variables with great distances between their means and midpoints Table 3 % of CR Affects Correlation Sizes Depending on Mean-Midpoint Differences Measures % of Data of CR1234 1. SES 1001.00 501.00 2. SDT-P 100-.15*1.00 50-.31***1.00 3. GSES100.70*** -.071.00 50.51***-.37***1.00 4. CESD-R100-.60***.20**-.42***1.00 50-.53***.50***-.56***1.00 _________________________________________________________________ Note. % of Data of CR = either 100% conscientious data or 50% conscientious & 50% random. Introduction The CRS utilizes instructional item content. Each CRS item, randomly embedded in a questionnaire, instructs responders exactly how to respond to that particular item. CRS 1. To answer this question, please choose number three, neither agree nor disagree. 2. Choose the first option—strongly disagree—in answering this question. 3. To respond to this question, please choose number five, strongly agree. 4. Please answer this question by choosing number two, disagree. 5. In response to this question, please choose number four, agree. Compliant responses are scored as 1s and incompliant responses are scored as 0s. Items are summed to make a score that ranges from 0 (all incompliant responses) – 5 (all compliant responses). Because, on a 5-point response scale, it is so statistically unlikely that RR can answer several CRS items compliantly by chance alone, high CRS sum scores (3 – 5) must reflect conscientious responding whereas low CRS sum scores (0– 2) probably reflect random responding. Design and ResultsResults and Discussion Hypotheses 1.CRS scores can reliably discriminate between RR and CR responders 2. RR causes Type 1 error when measures’ means are distant from their response scales’ midpoints Method Participants and Procedure The Conscientious Responders Group (CRG) was made up of 187 undergraduate students who completed a 66-item paper-and-pencil questionnaire under standard testing instructions (i.e., “answer all items as honestly and accurately as possible”). Once these data were collected, an equally sized sample (187) of uniform random data was created using a random number generator. These data made up the Random Responders Group (RRG). Total sample size = 374. Measures All items were answered on a 5-point Likert scale Conscientious Responders Scale (CRS, Marjanovic et al., 2014) Self-Esteem Scale (SES, Rosenberg, 1965) Short Dark Triad—Psychoticism (SDT-P, Jones et al., 2014) General Self-Efficacy Scale (GSES, Chen et al., 2001) Center for Epidemiologic Studies Depression Scale (CESD-R, Eaton et al., 2004) Results Hypothesis 1 supported: logistic regression analysis showed that the CRS reliably discriminated between RRG and CRG participants with 93.05% accuracy. Table 1 % of Participants in Each Responder Group Correctly Classified using CRS Scores as the Predictor CRS CutoffRRGCRGAverage Classification RRG/CRG Accuracy 2/3 95.19% 90.91% 93.05% Conclusions 1. CRS was effective at discriminating between RR and CR 2. With 2 exceptions, RR caused Type 1 error (i.e., significant group differences) in measures with means distant from the midpoint