In large-scale assessments, many psychometric challenges occur, which compromise the validity of measures used to draw conclusions on students’ abilities. One of these issues refers to the assumption of local independence of items when using item response theory models such as the Rasch model (e.g., Wainer, Bradlow, & Wang, 2007). In this regard, Marais and Andrich (2008) distinguished between trait and task dependence. The first assumes that item dependencies are only manifested in residual correlations, whereas the latter assumes that students’ responses on a specific item significantly affect their responses on subsequent items. Both assumptions are reasonable, especially in assessments which are comprised of tasks that contain a set of items. For instance, researchers have already found substantial effects of item bundles (i.e., tasks that are composed of a common stimulus and different items) on the estimation of students’ abilities or the dimensionality of assessments (Brandt, 2012). These testlet or bundle effects are often regarded as design issues in studies such as PISA and TIMSS.

Until now, there has been surprisingly little research on item dependencies within large-scale assessments of problem solving competence.  Scherer (2014) has conducted a study on dependencies for complex problem solving, which requires students to interact with a responsive system in order to solve the problem. He found that item dependencies were mainly due to the cognitive processes and problem settings within the assessment. Jonassen (2011) furthermore stressed that problem solving frameworks assume interactions between the different steps that should be performed to solve the problem. Within the PISA framework of cross-curricular problem solving, three types of problems are distinguished: system analysis and design, decision-making, and troubleshooting (OECD, 2004). These types have recently been proven to be essential when considering the dimensionality of analytical problem-solving competence (Leutner et al., 2012). More precisely, a three-dimensional Rasch model assuming the three types as correlated but separate factors outperformed the unidimensional approach. Consequently, there is evidence on the relatedness of items within the problem types. However, it has not yet been investigated whether these relations lead to substantial item dependencies, which might compromise traditional estimation approaches such as Rasch modeling. Since Rasch modeling forms the basis for establishing proficiency scales and evaluating relations among core constructs in large-scale assessment such as the PISA study, the effects of model violations are of particular importance.

Based on the research gaps mentioned above, the present study is concerned with the relevance of item dependencies in problem solving assessments. More precisely, the main objective is two-fold: (1) Quantify item dependencies and locate their main source(s); (2) Compare different approaches of modeling item dependencies for problem types and problem tasks (i.e., stimuli) with the default reference approach as given by the Rasch model. In psychometric research, a number of approaches have been presented to deal with local item dependencies (Ip, 2010; Yen, 1993). These range from establishing polytomous “super” items (e.g., Thissen, Steinberg, & Mooney, 1989), Rasch testlet models (e.g., Wang & Wilson, 2005), Rasch copula models (e.g., Braeken, Tuerlinckx, & De Boeck, 2007) to autoregressive models (e.g., Hoskens & De Boeck, 1997).

Braeken, J., Tuerlinckx, F., & De Boeck, P. (2007). Copula functions for residual dependency. Psychometrika, 72(3), 393-411. Brandt, S. (2012). Robustness of multidimensional analyses against local item dependence. Psychological Test and Assessment Modeling, 54(1), 36-53. Hoskens, M., & De Boeck, P. (1997). A parametric model for local dependence among test items. Psychological Methods, 2(3), 261-277. Ip, E. (2010). Empirically indistinguishable multidimensional IRT and locally dependent unidimensional item response models. British Journal of Mathematical and Statistical Psychology, 63, 395-416. Jonassen, D. (2011). Learning to Solve Problems: A Handbook for Designing Problem-Solving Learning Environments. New York/London: Routledge. Kiefer, T., Robitzsch, A., & Wu, M. (2013). TAM – Test Analysis Module [R package]. Leutner, D., Fleischer, J., Wirth, J., Greiff, S., & Funke, J. (2012). Analytische und dynamische Problemlösekompetenz im Lichte internationaler Schulleistungsvergleichsstudien [Analytical and complex problem-solving competence in large-scale assessments]. Psychologische Rundschau, 63(1), 34-42. Marais, I., & Andrich, D. (2008). Formalizing dimension and response violations of local item independence in the unidimensional Rasch model. Journal of Applied Measurement, 9, 200-215. Muthén, B., & Muthén, L. (2010). Mplus 6 [Computer software]. Los Angeles, LA: Muthén & Muthén. OECD (2004). Problem solving for tomorrow’s world. Paris: OECD. Robitzsch, A. (2013). SIRT – Supplementary functions for Item Response Theory [R package]. Scherer, R. (2014, in press). Psychometric challenges in modeling scientific problem-solving competency: An item response theory approach. In H. Bock et al. (Eds.), Studies in Classification, Data Analysis, and Knowledge Organization. New York, NY: Springer. Thissen, D., Steinberg, L., & Mooney, J. A. (1989). Trace lines for testlets: A use of multiple-categorical-response models. Journal of Educational Measurement, 26, 247-260. Wainer, H., Bradlow, E., & Wang, X. (2007). Testlet response theory and its applications. Cambridge, UK: Cambridge University Press. Wang, W.-C., & Wilson, M. (2005). The Rasch Testlet Model. Applied Psychological Measurement, 29(2), 126-149. Yen, W. (1984). Effects of local item dependence on the fit and equating performance of the three-parameter logistic model. Applied Psychological Measurement, 8, 125-145.