Background and Rationale

The integration of artificial intelligence (AI) into teacher education has attracted increasing attention from scholars and policymakers worldwide, intensifying calls to define and evaluate teachers’ competencies for an AI-rich educational landscape (Zhai & Nehm, 2023). Within teacher education, a growing body of work has sought to conceptualise teachers’ capabilities to use AI tools effectively, ethically, and pedagogically. Building on the Technological Pedagogical Content Knowledge (TPACK) framework, recent scholarship has proposed AI-TPACK by incorporating AI-related dimensions into teachers’ professional knowledge base (Celik, 2023; Ning et al., 2024). Correspondingly, researchers have developed and validated self-report AI-TPACK scales to capture pre- and in-service teachers’ perceived readiness to integrate AI into teaching (Ning et al., 2024), and these instruments are increasingly adopted in empirical studies as a basis for competency evaluation (Karatas & Atac, 2025; Xu et al., 2025).

However, reliance on self-report instruments raises enduring validity concerns, particularly when the target competency involves complex, design-based practice rather than general dispositions or confidence (Akyuz, 2018). This challenge becomes especially salient with generative AI (GenAI), which can support multimodal science teaching—an area where teachers must coordinate multimodal resources (e.g., text, image, audio, animation) to foster conceptual understanding (Kress, 2001; Nielsen & Yeo, 2022). These conditions make it crucial to examine whether self-reported AI-TPACK aligns with demonstrated competence in authentic, AI-supported multimodal design tasks.

Research context

To examine this measurement gap in an authentic teacher-education setting, this study is situated in a pre-service science teacher education course where participants completed an AI-supported multimodal design task. Specifically, pre-service teachers used generative AI to support the creation of web-based animations and interactive representations intended to teach abstract scientific concepts. The same cohort also completed a widely used AI-TPACK self-report questionnaire, enabling a direct comparison between perceived competence and demonstrated performance within a shared instructional context.

Empirically, the study draws on 116 GenAI-assisted multimodal assignments, which were evaluated using a TPACK-aligned performance rubric across the seven TPACK dimensions, and these scores were matched with participants’ questionnaire responses. To contextualise and interpret patterns of convergence and divergence, we further conducted six focus group interviews with 24 pre-service teachers, exploring how they perceived AI’s role in science teaching and how they evaluated their own capabilities when working with AI-supported design tasks.

Research Questions:

RQ1: To what extent do pre-service science teachers’ AI-TPACK self-reports align with performance-based TPACK scores derived from an AI-supported multimodal design task?

RQ2: What patterns of misalignment emerge across the seven TPACK dimensions?

RQ3: How can the observed misalignment be interpreted in terms of construct validity?

Theoretical Framework

This study integrates TPACK (Mishra & Koehler, 2006) with Kane’s (2013) argument-based validation to examine the meaning and limits of AI-TPACK self-report scores in an AI-supported multimodal design context. TPACK specifies the competence domain of interest—teachers’ integrated use of content, pedagogy, and technology knowledge (CK, PK, TK) and their intersections in designing multimodal science representations. Kane’s framework then examines the use of AI-TPACK questionnaires as a validity claim. First, we examine relations to other measures by comparing self-report scores with rubric-based performance scores from an AI-supported multimodal design task. Second, we use qualitative data as response-process evidence to explain why self-reports misalign with task performance and to refine defensible score interpretations.

International Implications

Globally, AI-TPACK self-report scales are increasingly used to monitor teacher readiness and evaluate AI-focused teacher education. Using data from China—a major, fast-evolving AI-in-education context—this study provides internationally relevant validity evidence. It cautions against relying solely on self-reports when benchmarking teacher competence and supports combining questionnaires with performance-based assessment for cross-country comparisons.

This study adopted a convergent mixed-methods design in which quantitative and qualitative data were collected within the same course context, analysed in parallel, and then integrated to generate meta-inferences about alignment and misalignment between measures (Greene et al., 1989; Creswell & Clark, 2018). Sampling Participants were 116 pre-service primary science teachers enrolled in a compulsory course at a normal university in China. The course required an assessed design task where students used generative AI to support the production of multimodal teaching materials. Ethical procedures included informed consent, anonymisation, etc. Data Collection 1.AI-TPACK self-report questionnaire. All participants completed a validated AI-TPACK survey (Ning et al., 2024), yielding item-level responses across seven theoretically defined dimensions. 2.Performance-based assessment (coursework). Each participant submitted a GenAI-assisted multimodal assignment (i.e., creating an HTML-based animation/interactive representation to explain the Earth's seasons). Artefacts were evaluated using a TPACK-aligned, 21-item performance rubric developed for this study to operationalise observable indicators across the seven TPACK dimensions. Two raters conducted initial independent scoring, followed by calibration rounds to refine decision rules before full-scale scoring. 3.Focus groups. Six semi-structured focus groups (n = 24) explored how participants interpreted AI-TPACK items, how they approached the design task, and perceived reasons for misalignment between self-perceptions and performance. Data Analysis Quantitative analysis (aligned to RQ1–RQ2). Confirmatory factor analysis (CFA) was conducted to examine the factorial validity of the AI-TPACK measurement model. A seven-factor model was specified, with each item loading on its corresponding latent construct. Estimation used maximum likelihood (ML), and fit was evaluated using χ², CFI, TLI, RMSEA, and SRMR. To examine effects of background variables on self-reported AI-TPACK and course performance, multivariate regression analyses were conducted in Mplus with gender, year of study, and programme track as predictors, reporting standardised coefficients. For subsequent alignment analyses, factor scores for latent variables were estimated and exported. Alignment analysis (RQ1–RQ2). To assess alignment, AI-TPACK factor scores were correlated with performance scores from the multimodal coursework assessment at both overall and dimension levels. Misalignment patterns were further profiled through score comparisons (e.g., distributional contrasts and categorical banding of high/medium/low scores) to identify systematic overestimation and dimension-specific gaps. Qualitative analysis and integration (RQ3). Focus group transcripts were analysed using deductive coding to generate response-process and context-anchoring explanations for divergence. Integration was conducted via joint displays, linking dominant quantitative misalignment profiles to qualitative accounts of how participants construed questionnaire items and evaluated their own competence in relation to the task.

Quantitative analyses indicate a systematic misalignment between pre-service teachers’ self-reported AI-TPACK and their demonstrated competence in an AI-supported multimodal design task. CFA supports the intended seven-factor structure of the AI-TPACK questionnaire, suggesting acceptable internal structure for the target sample. However, when AI-TPACK factor scores were compared with performance scores, alignment was largely absent. Across dimensions, self-report–performance correlations were non-significant, with one exception: a small positive association between self-reported PK and task-based TK (p < .05). Moreover, when scores were categorised into low/medium/high bands, participants’ self-ratings were consistently higher than their coursework performance, indicating a pattern of overestimation rather than random noise. Regression analyses further suggest that gender, year of study, and programme track offer limited explanatory power for this misalignment. Focus group evidence provides interpretive insight into why self-report AI-TPACK may not translate into enacted competence in this task context. First, participants highlighted that questionnaire items are context-general, prompting them to anchor responses in familiar teaching scenarios rather than the specific demands of AI-supported multimodal design. Second, many expressed doubts about the current maturity of AI tools for complex design and debugging, which shaped their strategies, persistence, and reliance on AI assistance. Third, participants commonly viewed AI as useful for compensating gaps in content knowledge (CK), yet reported low confidence in technological knowledge (TK), producing a “content-support” orientation that did not necessarily yield robust technical implementation in the final artefact. Taken together, the study offers validity-oriented implications for AI-TPACK measurement and teacher education. It cautions against using self-report AI-TPACK scales as stand-alone indicators of competence in AI-supported multimodal design and supports combining questionnaires with performance-based assessment to enable defensible score interpretations. Conceptually, the findings suggest that AI-TPACK self-reports may primarily capture perceived readiness and scenario-based confidence, whereas rubric-based assessment captures task-enacted design competence in a technology-intensive context.

Akyüz, D. (2018). Measuring technological pedagogical content knowledge (TPACK) through performance assessment. Computers & Education, 125, 212–225. https://doi.org/10.1016/j.compedu.2018.06.012 Celik, I. (2023). Towards intelligent-TPACK: An empirical study on teachers’ professional knowledge to ethically integrate artificial intelligence (AI-based) tools into education. Computers in Human Behavior, 136, 107468. https://doi.org/10.1016/j.chb.2022.107468 Creswell, J. W., & Plano Clark, V. L. (2018). Designing and conducting mixed methods research (3rd ed.). SAGE. Greene, J. C., Caracelli, V. J., & Graham, W. F. (1989). Toward a conceptual framework for mixed-method evaluation designs. Educational Evaluation and Policy Analysis, 11(3), 255–274. https://doi.org/10.3102/01623737011003255 Kane, M. T. (2013). Validating the interpretations and uses of test scores. Journal of Educational Measurement, 50(1), 1–73. https://doi.org/10.1111/jedm.12000 Karataş, F., & Ataç, B. A. (2025). When TPACK meets artificial intelligence: Analyzing TPACK and AI-TPACK components through structural equation modelling. Education and Information Technologies, 30, 8979–9004. https://doi.org/10.1007/s10639-024-13164-2 Kress, G., & van Leeuwen, T. (2001). Multimodal discourse: The modes and media of contemporary communication. Arnold. Mishra, P., & Koehler, M. J. (2006). Technological pedagogical content knowledge: A framework for teacher knowledge. Teachers College Record, 108(6), 1017–1054. https://doi.org/10.1111/j.1467-9620.2006.00684.x Nielsen, W., & Yeo, S. (2022). Multimodal representations in primary science: A systematic review. Research in Science Education, 52, 1343–1367. https://doi.org/10.1007/s11165-020-09974-6 Ning, Y., Zhang, C., Xu, B., Zhou, Y., & Wijaya, T. T. (2024). Teachers’ AI-TPACK: Exploring the relationship between knowledge elements. Sustainability, 16(3), 978. https://doi.org/10.3390/su16030978 Xu, G., Yu, A., Gao, A., Chen, H., & Li, M. (2025). Developing an AI-TPACK framework: Exploring the mediating role of AI attitudes in pre-service TCSL teachers’ self-efficacy and AI-TPACK. Education and Information Technologies, 30, 22471–22495. https://doi.org/10.1007/s10639-025-13630-5 Zhai, X., & Nehm, R. H. (2023). AI and formative assessment: The train has left the station. Journal of Research in Science Teaching, 60(7), 1227–1233. https://doi.org/10.1002/tea.21885