1. Psychometric Benefits of Removing Different Types of Bad Responders
Hung Loan Nguyen, Megan Wertheimer, Dr. Michael Hein, Dr. Mark Frame, and Dr. Judith Van Hein Middle Tennessee State University, Murfreesboro, Tennessee
Introduction
The results of surveys are often used to make inferences that lead to making big decisions in organizations and therefore it is important for survey results to be valid.
Research has shown that respondents may respond carelessly and that this can have a profound effect on data quality.
There has been increased attention in detecting and removing bad responders but less attention has been paid in preventing bad responding from occurring in the first place.
One type of bad responding may be due to survey fatigue. To this point, understanding the different types of bad responding and the conditions they most commonly occur will help practitioners better combat careless responding.
This study investigated the effects that survey length can have on psychometric properties of the HEXACO-60. There is evidence to indicate that careless responding, also known as insufficient effort responding (IER), increases as surveys go on (Baer, Ballenger, Berry & Wetter, 1997; Berry et al., 1992; Meade & Craig, 2012). One possible explanation of this is survey fatigue. This study looked at survey fatigue and its possible connection with IER by comparing the factor structure of the HEXACO-60 (Ashton & Lee, 2009) when placed at the very beginning of a survey and when placed at the end of a lengthy survey. Furthermore, types of IER were identified using latent class analysis in order to classify cases to types of IER based on their responses. Cases were then removed based on their classification and psychometric properties were evaluated to assess the differential effects of removing different types of IER would have on psychometric properties.
Measures
͏Long Survey. Job analysis survey from a southeastern state highway patrol unit. This survey contained approximately 1700 items and had the HEXACO-60 close to the end. MAJOR LIMITATION: Participants could exit and continue this online survey as they wished.
͏͏Short Survey. Adaptive performance survey conducted for the purpose of completing the thesis requirement for the graduate Industrial Organizational Psychology program at Middle Tennessee State University of two students. This survey contained approximately 215 items and had the HEXACO-60 at the very beginning.
Participants
Long Survey
732 patrol officers from a southeastern highway patrol unit. In addition, Survey 1 was conducted completely online via the survey platform Qualtrics.
Short Survey
1008 participants that were recruited via Amazon’s Mechanical Turk and the Middle Tennessee State University research pool. Was also conducted on Qualtrics.
Procedure
Data from two surveys (i.e. Long and Short) were analyzed using the R statistical environment.
Confirmatory factor analysis was conducted for the Hexaco-60 for the normal six factor model and a seven factor model for both surveys. The additional factor in the seven factor model is a distinct negatively worded/reversed scored factor that is a result of IER and negatively worded items (Woods, 2006).
Using various detection indices, (i.e., long string, within-person standard deviation, etc.) a latent class analysis (LCA) was conducted for the Long survey.
Removed the different classes found in the LCA and evaluated differential results on the factor structure and reliability (internal consistency) of the Long Survey.
Results
Fit indices indicated that there was a potential seven factor in the Long Survey but not in the Short Survey (Support for Hypothesis 1). Contact researchers for fit indices.
Almost all factor loadings were of a smaller magnitude in the Long Survey versus the Short Survey and some items did not significantly load onto their respective factors in the Long Survey (Support for Hypothesis 2). Contact researchers for factor loadings.
Two types of bad responders found were pattern and random responders. Removing the two types, as well as removing bad responders overall, typically lead to better model fit, refer to the table below, and reliability. (General support for Hypothesis 3). Contact researchers for Cronbach’s alpha table.
Removing cases using various detection indices resulted in different psychometric consequences.
References
Ashton, M. C., & Lee, K. (2009). The HEXACO-60: A short measure of the major dimensions of personality. Journal of Personality Assessment, 91(4),
Baer, R. A., Ballenger, J., Berry, D. T., & Wetter, M. W. (1997). Detection of random responding on the MMPI- A. Journal of Personality Assessment, 68(1),
Berry, D. T., Wetter, M. W., Baer, R. A., Larsen, L., Clark, C., & Monroe, K. (1992). MMPI-2 random responding indices: Validation using a self-report methodology. Psychological Assessment, 4(3), 340.
Meade, A. W., & Craig, S. B. (2012). Identifying careless responses in survey data. Psychological Methods, 17(3), 437.
Woods, C. M. (2006). Careless responding to reverse- worded items: Implications for confirmatory factor analysis. Journal of Psychopathology and Behavioral Assessment, 28(3),
Hypotheses
The HEXACO-60 placed at the end of the Long Survey will have a distinct negative worded item factor but will not when placed at the beginning of the Short Survey.
Items in the HEXACO-60 placed at the end of the Long Survey will be less likely to load on to their respective latent factors than items in the HEXACO-60 placed at the beginning of the Short Survey.
Removing the different types of careless responders identified by the latent class analysis will result in different, albeit largely positive, results on the psychometric properties of the HEXACO-60 in the Long Survey.
Contact Information
Hung Loan Nguyen: