The use of scores from self-report measures are very common in several areas of research. Since those instruments provide researchers to measure some psychological constructs such as personality, attitudes, beliefs, emotions of too many respondents in a short time, they are preferred widely for data collection process (Alarcon & Lee, 2022; Curran, 2015; Ulitzsch et al., 2022). However, some important problems may occur when responders do not give their best effort to select the response correctly reflecting themselves (which is very common especially with the unmotivated responders) (Rios & Soland, 2021; Schroeders et. al., 2022). Individuals may respond to items without reading them, by misinterpreting them or be unmotivated to think about (Huang et al., 2012; Ward & Meade, 2022). This type of responding behaviour have been stated as random (Beach, 1989), careless (Meade & Craig, 2012), insufficient effort (Huang et al., 2012), disengaged responding (Soland et al., 2019) in the literature. In the context of this study, the term ‘careless responding’ with CR abbreviation is preferred. Careless responding (CR) behaviour is a major concern based on the data taken from self-report scales in any type of research (Meade & Craig, 2012). Even the amount is small, it may affect the data quality and results of the study severely. Careless responses may introduce a measurement error, weaken the relationship between variables and inflate the Type II error. It may also introduce a new source of construct-irrelevant variance to the process and end up with an undesirable effect on psychometric properties of the scale (item difficulty, average scores, test reliability, factor structure etc.). Briefly, CR has the potential to weaken the test scores’ validity in different ways (Beck et. al., 2019; Rios & Soland, 2021).

Considering the factors stated above, CR have become an important and interesting research topic for researchers with a growing interest. One of the most important aspects on CR research is the way how we can detect and cope with them to ensure the quality of survey data. Identifying careless responders and removing them from the dataset is one of the suggested ways to increase data quality. In the literature, there are several data screening methods mainly classified in two groups; priori and post-hoc. Priori methods are the ones that are planned and incorporated into data collection process before the administration of survey. On the contrary, post-hoc methods get involved in the process after data collection. They are implemented on the collected dataset and typically based on a statistical calculation.

While there are several studies focusing on the effect of careless responding on datasets and comparison of the efficacy of CR identification methods, there is still no clear answer about the detection accuracy of CR identification methods (Goldammer et al., 2020). Besides, this study will focus on prior methods that have been studied and focused less on previous studies.

The present study will handle three different ways of prior methods (instructed response items, reverse items and self-report items) which will be explained in method part in detail. In the context of this study, these three ways of CR identification will be used, examinees will be removed from dataset according to those methods separately and their effects on psychometric properties of data will be investigated. This study addresses the following research questions;

-       How was the distribution of careless responders with respect to three different CR identification methods?

-       How did psychometric properties of the data (scale mean, reliability, correlation between factors, factor structure etc.) change when careless respondents were removed from the data with respect to different CR identification methods?

The purpose of that study is to examine self-report data for careless responding, to investigate the effect of CR on psychometric properties of dataset and to compare the performance of CR identification methods. Three different priori methods will be used for this purpose; instructed response, reverse and self-report items. Instructed response items are special items instructing respondents to select one specific category and the ones that choose another option than the instructed response, are assumed as careless. Reverse items are used as attention control items. Individuals are expected to select responses in opposite directions for reverse items. When they give same or too similar answers, it is assumed as an indicator of CR. Lastly, self-report items directly ask individuals about their effort (e.g. ‘I put forth my best effort in responding to this survey’; Meade & Craig, 2012). In the context of this study, a self-report scale will be used for data collection purpose. A manipulated version of this instrument will be formed by adding one instructed-response item (‘Please select ‘strongly agree’ for this item’), one manipulated reverse item and one self-report item (‘I did my best while responding to the scale’). It is planned that manipulated form will be applied to approximately 500 students. Only one instructed response item will be added to the original scale and individuals selecting the response other than the instructed one will be handled as CR. Additionally, one reverse item for one of the items on the original scale will be purposefully added and individuals choosing the same or similar responses for reverse items will be assumed as CR. Lastly, only one self-report item will be included at the end of the scale and responders will be evaluated according to their own answers in terms of CR. Percentage of careless responders will be calculated for each method separately and psychometric properties of data (scale mean, factor loadings, reliability, explained variance etc.) will be examined. After that, careless responders will be excluded from dataset according to three methods separately and the three separate remaining datasets will be examined again to see how psychometric properties (scale means, reliabilities, correlation between factors etc) were affected by that removal. Lastly, in order to see which CR identification method performed most efficiently and improved data quality, psychometric properties (reliability, factor structure etc.) of remaining datasets will be compared separately.

The finding of this study is important for the researchers and practitioners who are using self-report measures for data collection and making conclusions based on that data. Careless responses may cause ‘dirty data’ and may affect the results significantly. So, some investigations should be considered in order to make data cleaning. In addition, result will investigate the efficiency of using of different prior methods and some suggestions will be made on CR identification. I hope that this study will help to fill some gaps in careless responding identification and eliminating its’ effect in a better way.

Alarcon, G. M., & Lee, M. A. (2022). The relationship of insufficient effort responding and response styles: An online experiment. Frontiers in Psychology, 12. https://www.frontiersin.org/article/10.3389/fpsyg.2021.784375 Beck, M. F., Albano, A. D., & Smith, W. M. (2019). Person-fit as an index of inattentive responding: A comparison of methods using polytomous survey Data. Applied Psychological Measurement, 43(5), 374–387. https://doi.org/10.1177/0146621618798666 Curran, P. G. (2015). Methods for the detection of carelessly invalid responses in survey data. Journal of Experimental Social Psychology, 66(2016), 4–19. https://doi.org/10.1016/j.jesp.2015.07.006 Goldammer, P., Annen, H., Stöckli, P. L., & Jonas, K. (2020). Careless responding in questionnaire measures: Detection, impact, and remedies. The Leadership Quarterly, 31(4). https://doi.org/10.1016/j.leaqua.2020.101384 Huang, J. L., Curran, P. G., Keeney, J., Poposki, E. M., & DeShon, R. P. (2012). Detecting and deterring insufficient effort responding to surveys. Journal of Business and Psychology, 27(1), 99–114. https://doi.org/10.1007/s10869-011-9231-8 Meade, A. W., & Craig, S. B. (2012). Identifying careless responses in survey data. Psychological Methods, 17(3), 437–455. https://doi.org/10.1037/a0028085 Rios, J. A., & Soland, J. (2021). Parameter estimation accuracy of the effort-moderated item response theory model under multiple assumption violations. Educational and Psychological Measurement. Schroeders, U., Schmidt, C., & Gnambs, T. (2022). Detecting careless responding in survey data using stochastic gradient boosting. Educational and Psychological Measurement, 82(1), 29–56. https://doi.org/10.1177/00131644211004708 Ward, M. K., & Meade, A. W. (2022). Dealing with careless responding in survey data: prevention, identification, and recommended best practices. Annual Review of Psychology, 74(1). https://doi.org/10.1146/annurev-psych-040422-045007 Ulitzsch, E., Yildirim-Erbasli, S. N., Gorgun, G., & Bulut, O. (2022). An explanatory mixture IRT model for careless and insufficient effort responding in self-report measures.