Language background plays a central role in how students understand, process, and respond to items in cognitive assessments. Extensive research indicates that even subtle changes in test item wording can significantly influence student performance (Abedi et al., 2000; Abedi & Lord, 2001). For instance, employing simpler vocabulary, shorter sentences, and active voice allows students to demonstrate their true abilities more accurately. By reducing the linguistic burden, educators ensure that assessments measure intended knowledge rather than a student's ability to decode complex syntax.

This issue is particularly critical in international and European assessment contexts. In many countries, students participate in large-scale assessments conducted in a language that is not their first. While these students may possess profound knowledge in subjects like mathematics or science, they often perform poorly because they struggle with the specific language used in test items. Evidence suggests these language barriers reduce both the validity and reliability of test scores, creating systemic disadvantages for English Language Learners (ELLs) and students from language-minority backgrounds (Abedi, 2002, 2004).

From a measurement perspective, this phenomenon is explained by construct-irrelevant variance (Messick, 1994). This occurs when test scores are influenced by factors that are not part of the specific construct intended for measurement. In this context, linguistic complexity acts as an unintended source of difficulty. Consequently, test scores may inadvertently reflect a student’s reading proficiency or language exposure rather than their actual mastery of the subject matter.

Previous research has pinpointed several linguistic features that negatively affect comprehension. Long, convoluted sentences, unfamiliar academic vocabulary, and passive voice increase cognitive load, slowing down reading and making misunderstandings more likely (Abedi, 2006). While math performance typically improves when wording is concise, most existing studies focus on traditional domains.

There is currently a significant gap regarding Creative Thinking Tasks (CTTs). This is a crucial oversight, as CTTs frequently utilize open-ended and language-rich prompts. Students must generate original ideas and justify their reasoning, requiring not only divergent thinking but also high-level language comprehension. Because of this, item wording may exert an even stronger influence on CTTs than on standardized multiple-choice tests. Students with high creative potential but limited language exposure may be severely disadvantaged, raising concerns regarding fairness and equity in international assessments.

Recent evidence from ePIRLS 2016 (Bulut et al., 2023) supports this, showing that item complexity significantly reduces the probability of success in online reading. This study aims to examine if similar effects occur in creative thinking tasks. By considering both item-level factors (complexity) and student-level factors (reading achievement, language background, gender, and socioeconomic status), the study contributes to the global dialogue on assessment validity.

1. Does the linguistic complexity of creative thinking task items significantly affect the probability of correct answer?
2. Do student characteristics (reading achievement, language background, gender, and SES) significantly affect the probability of correct answer?
3. Does student characteristics moderate the association between item complexity and the probability of correct answer?

The factors influencing students’ probability of correctly answering PISA 2022 CTT items were examined using a two-level Generalized Linear Mixed Model (GLMM) approach. The dataset comprised responses from 83,295 students across 15 countries, all of whom completed the cognitive assessment in English. For the purposes of analysis, Level 1 (item level) included the linguistic complexity of the items, while Level 2 (student level) included reading achievement, language most frequently spoken at home (test language [English] vs. another language), time spent on homework related to the test language (English) per day, socioeconomic status (ESCS), and gender. Item linguistic complexity was operationalized using ATOS readability scores for the 18 released CTT items (Renaissance Learning, 2012). Higher ATOS scores indicate greater linguistic complexity. The ATOS scores were computed based on the item texts publicly available at https://pisa2022-questions.oecd.org/. ATOS values ranged from 5.37 to 9.14 (M = 7.45, SD = 1.02). The least linguistically complex item was Space Comic Q02, whereas Carpooling Q01 exhibited the highest linguistic complexity. At the student level, analyses were repeated for each of the ten plausible values of reading achievement, and the results were pooled according to Rubin’s rules (Rubin, 1987). Students with missing responses on the language background, homework, ESCS, or gender variables were excluded from the analyses, resulting in a final analytic sample of 73,135 students. The probability of correctly answering CTT items was examined through the relative comparison of four hierarchical models: • Model 0: Null model • Model 1: Item-level predictors only • Model 2: Model 1 + student-level predictors • Model 3: Model 2 + moderation effects of student-level predictors Model comparisons were conducted using the Akaike Information Criterion (AIC), Bayesian Information Criterion (BIC), log-likelihood (LL), and R² indices (Nakagawa & Schielzeth, 2013). In addition, likelihood ratio tests (LRTs) were used to assess whether the inclusion of additional predictors significantly improved model fit. Lower AIC and BIC values, along with higher LL and R² values, indicate better model–data fit.

Based on the model comparison results, Model 3 was identified as the model providing the best fit to the data. Likelihood ratio tests (LRTs) indicated that each model differed significantly from its preceding model in the hierarchical sequence. For Model 3, the explained variance was R² = 27.8%, indicating that the predictors included in the model accounted for 27.8% of the variance in students’ probability of correctly answering CTT items. These findings suggest that the probability of a correct response is jointly determined by both item-level characteristics and student-level characteristics. Item linguistic complexity substantially reduced the probability of a correct response (OR = 0.517, p < .001), indicating that the linguistic structure of the item text constitutes a barrier to student performance and represents a source of CIV. However, this effect varied significantly as a function of student characteristics. The Complexity × Reading Achievement interaction was negative (OR = 0.942, p < .001), suggesting that as reading proficiency increases, the detrimental effect of linguistic complexity decreases. Similarly, the negative effect of complexity was more pronounced for students with a different home language background (OR = 0.869, p < .001). The Complexity × Homework interaction was also negative (OR = 0.958, p < .001), indicating that linguistically complex items are relatively more challenging for students who spend more time on homework in the test language. In contrast, the Complexity × Gender interaction was positive (OR = 1.158, p < .001), suggesting that the negative impact of linguistic complexity is comparatively weaker for male students. The Complexity × ESCS interaction was not statistically significant. In summary, reading achievement and gender emerge as mitigating factors that attenuate the negative effect of linguistic complexity, whereas language background intensifies this effect. The absence of a significant interaction between linguistic complexity and socioeconomic status suggests that linguistic barriers function as a direct cognitive constraint, largely independent of social class differences.

Abedi, J. (2002). Standardized achievement tests and English language learners: Psychometrics issues. Educational Assessment, 8(3), 231–257. https://doi.org/10.1207/S15326977EA0803_02 Abedi, J. (2004). The No Child Left Behind Act and English language learners: Assessment and accountability issues. Educational Researcher, 33(1), 4–14. https://doi.org/10.3102/0013189X033001004 Abedi, J. (2006). Language issues in item development. In S. M. Downing & T. M. Haladyna (Eds), Handbook of Test Development (pp. 377–398). Lawrence Erlbaum Associates. Abedi, J., & Lord, C. (2001). The language factor in mathematics tests. Applied Measurement in Education, 14(3), 219–234. https://doi.org/10.1207/S15324818AME1403_2 Abedi, J., Lord, C., Hofstetter, C., & Baker, E. (2000). Impact of accommodation strategies on English language learners’ test performance. Educational Measurement: Issues and Practice, 19(3), 16–26. https://doi.org/10.1111/j.1745-3992.2000.tb00034.x Bulut, H. C., Bulut, O., & Arikan, S. (2023). Evaluating group differences in online reading comprehension: The impact of item properties. International Journal of Testing, 23(1), 10–33. https://doi.org/10.1080/15305058.2022.2044821 Messick, S. (1994). Foundations of validity: Meaning and consequences in psychological assessment. ETS Research Report Series, 2. https://doi.org/10.1002/j.2333-8504.1993.tb01562.x Nakagawa, S., & Schielzeth, H. (2013). A general and simple method for obtaining R2 from generalized linear mixed‐effects models. Methods in Ecology and Evolution, 4(2), 133-142. https://doi.org/10.1111/j.2041-210x.2012.00261.x Renaissance Learning. (2012). Text complexity: Accurate estimates and educational recommendations. Wisconsin Rapids. https://renaissance.widen.net/view/pdf/ob12ylc2hr/R54882.pdf?t.download=true&u=zceria Rubin, D. B. (1987). Multiple imputation for nonresponse in surveys (1. ed.). Wiley. https://doi.org/10.1002/9780470316696