Session Information
99 ERC SES 04 I, Assessment and evaluation design and methodology
Paper Session
Contribution
Social media has become core infrastructure for accessing information, communicating, and participating in civic life. European policymakers have called for stronger media literacy to counter disinformation and support informed decision-making. Frameworks like DigComp 2.2 describe what competence looks like. But these frameworks cannot function as psychometrically validated measurement instruments—they describe competence but cannot locate individuals on a calibrated scale (Vuorikari et al., 2022). The lack of validated measurement creates difficulties. Evidence-informed curriculum development, assessment design, and cross-context proficiency comparison all require instruments that can establish where individuals stand.
Substantial work on social media literacy (SML) has accumulated, yet the field remains conceptually fragmented (Cho et al., 2024; Valle et al., 2025). Most instruments rely on self-report measures. These capture what users perceive they can do, rather than what they actually demonstrate (Tandoc et al., 2021). Some newer performance measures move beyond self-report using IRT-based approaches. This is progress. But these tools typically target specific populations and do not address standards-referenced interpretation across age groups (Drake et al., 2023). Making defensible proficiency judgements across educational systems is difficult.
We conducted a systematic scoping review and thematic synthesis of SML literature from 2015 to 2025. From this work, we propose a measurement framework applicable across ages and contexts. The framework operationalises SML through three components: Platform Awareness, Critical Evaluation, and Responsible Participation. These consolidate recurring domains in the literature. We describe proficiency using four levels. The four-level structure reflects theoretically distinct transitions in metacognitive regulation. L1 (Unaware) represents minimal recognition. L2 (Retrospective) involves post-hoc awareness—individuals identify relevant influences after engagement (Flavell, 1979). L3 (Real-time) marks a shift to in-process monitoring and deliberate adjustment during engagement (Schraw & Dennison, 1994). L4 (Habitual–Reflexive) describes internalised monitoring across contexts, including low-stakes use, where strategies transfer beyond single situations (Mezirow, 1997). Self-awareness runs through all levels as a difficulty factor rather than a separate component.
Three questions guide the study: What are the core components and indicators of SML for standards-referenced measurement? How do we operationalise and design a rubric-based framework for Rasch calibration? What validity evidence is needed to support the intended uses of the measures?
Our goal is translating competence language into a calibratable rubric that supports defensible proficiency judgements. The conference paper presents the three-component construct map, candidate indicators, draft performance level descriptors, and the validation workflow. We plan to validate the framework across cultural and educational contexts. Cross-cultural comparability will be addressed in later stages. The study contributes a validation logic for standards-referenced interpretation, supporting European efforts to assess SML comparably across populations and systems.
Method
Psychometrically validated instruments for standards-referenced SML measurement are absent (Drake et al., 2023; Valle et al., 2025). We propose a Rasch-based workflow to address this gap, translating competence descriptions into calibrated proficiency scales for defensible cross-context interpretation. 1. Construct Definition through Systematic Synthesis (Completed) We conducted a systematic scoping review following PRISMA-ScR conventions (Tricco et al., 2018) of SML literature from 2015 to 2025. Twenty-five studies proposed competency frameworks or measurement dimensions. Thematic synthesis identified three recurring components: Platform Awareness (algorithmic curation, business logic, platform architecture), Critical Evaluation (source credibility, evidence-based claims, bias/framing analysis), and Responsible Participation (pre-posting and in-interaction responsibility). Construct mapping (Wilson, 2023) operationalised these into eight indicators with observable behaviours across four proficiency levels: L1 Unaware, L2 Retrospective, L3 Real-time, L4 Habitual–Reflexive. Self-awareness is embedded as a cross-cutting difficulty factor rather than a separate dimension. This is reflected in Performance Level Descriptors (PLDs) that discriminate adjacent levels, grounded in Flavell's (1979) awareness emergence and Schraw and Dennison's (1994) regulatory monitoring. 2. Expert Validation of Rubric Structure (Designed; Initial Implementation by Conference) Measurement specialists and SML domain experts (n=8–12) will evaluate construct coverage, indicator clarity, and PLD discriminability through structured focus groups. Qualitative feedback will inform descriptor refinement. Content Validity Index (CVI) ratings will quantify expert agreement on indicator relevance and level appropriateness, providing evidence of interpretive coherence prior to empirical calibration. 3. Rasch Calibration Workflow (Designed; Implementation Post-Conference) Scenario-based assessment tasks anchored to specific PLDs will undergo polytomous Rasch calibration if model fit is adequate. A field trial with adequate sample size for stable parameter estimation is planned. The validation workflow addresses four psychometric requirements: First, model specification compares the Rating Scale Model (RSM) with the Partial Credit Model (PCM) to determine optimal parameterisation. Second, Wright maps verify the theoretically predicted difficulty ordering across proficiency levels. Third, fit diagnostics using Infit/Outfit mean square (MNSQ) statistics identify misfitting items requiring revision. Fourth, Differential Item Functioning (DIF) tests examine whether demographic subgroups respond differently to items, establishing a foundation for later cross-context validation (e.g., Australia, Taiwan). 4. Conference Contribution We will present the construct map (three components, eight indicators) and draft PLDs, alongside interim evidence from expert consultation if available. We will also outline the pre-specified Rasch validation workflow for subsequent calibration.
Expected Outcomes
Three related outcomes emerge from this study, addressing the research questions and supporting standards-referenced SML assessment. 1. Framework specification. The construct map defines SML through three components grounded in theory: Platform Awareness, Critical Evaluation, and Responsible Participation. Eight indicators span four proficiency levels. Performance Level Descriptors treat metacognitive self-awareness as a cross-cutting difficulty factor. Post-hoc recognition appears at L2. L3 requires real-time monitoring. L4 captures habitual reflexivity. Recurring competency domains from across the literature are consolidated into a coherent measurement structure. This addresses conceptual fragmentation identified in recent reviews (Cho et al., 2024; Valle et al., 2025). 2. Validation evidence. Expert review provides content validity evidence. Measurement specialists and SML domain experts evaluate construct coverage, indicator clarity, and level discriminability through structured feedback. CVI ratings quantify how much experts agree on relevance and appropriateness. This strengthens interpretive coherence and helps determine readiness for field testing. 3. Calibration workflow specification. We present a fully specified Rasch validation workflow. Rubric-based competence descriptions are translated into calibrated measures. The workflow includes model comparison (RSM vs PCM), hierarchy evaluation using Wright maps to check alignment with expected difficulty ordering, fit diagnostics using Infit/Outfit MNSQ, and DIF checks across demographic groups for measurement invariance. This provides a replicable pathway. Descriptive frameworks can be converted into psychometrically sound instruments. Standards-referenced calibration has lacked these operational requirements. Beyond these methodological contributions, the study addresses a critical gap in EU digital literacy policy. While frameworks like DigComp 2.2 (Vuorikari et al., 2022) describe competence, they cannot support defensible proficiency judgements or cross-national comparison. The validated SML scale enables evidence-based curriculum placement and intervention targeting across European educational systems. It offers policymakers a measurement model that moves from descriptive taxonomies to calibrated assessment.
References
Andrich, D. (2004). Controversy and the Rasch model: a characteristic of incompatible paradigms? Med Care, 42(1 Suppl), I7-16. https://doi.org/10.1097/01.mlr.0000103528.48582.7c
Cho, H., Cannon, J., Lopez, R., & Li, W. B. (2024). Social media literacy: A conceptual framework. New Media & Society, 26(2), 941-960, Article 14614448211068530. https://doi.org/10.1177/14614448211068530
Drake, A. P., Masur, P. K., Bazarova, N. N., Zou, W. T., & Whitlock, J. (2023). The youth social media literacy inventory: development and validation using item response theory in the US. Journal of Children and Media, 17(4), 467-487. https://doi.org/10.1080/17482798.2023.2230493
Flavell, J. H. (1979). Metacognition and Cognitive Monitoring: A New Area of Cognitive-Developmental Inquiry. American Psychologist, 34, 906-911. https://doi.org/https://doi.org/10.1037/0003-066X.34.10.906
Mezirow, J. (1997). Transformative Learning: Theory to Practice. New Directions for Adult and Continuing Education, 1997(74), 5-12. https://doi.org/https://doi.org/10.1002/ace.7401
Schraw, G., & Dennison, R. S. (1994). Assessing Metacognitive Awareness. Contemporary Educational Psychology, 19(4), 460-475. https://doi.org/https://doi.org/10.1006/ceps.1994.1033
Tandoc, E. C., Yee, A. Z. H., Ong, J., Lee, J. C. B., Xu, D., Han, Z., Matthew, C. C. H., Ng, J., Lim, C. M., Cheng, L. R. J., & Cayabyab, M. Y. (2021). Developing a Perceived Social Media Literacy Scale: Evidence from Singapore. International Journal of Communication, 15, 2484-2505.
Update Modus of this Database
The current conference programme can be browsed in the conference management system (conftool) and, closer to the conference, in the conference app.
This database will be updated with the conference data after ECER.
Search the ECER Programme
- Search for keywords and phrases in "Text Search"
- Restrict in which part of the abstracts to search in "Where to search"
- Search for authors and in the respective field.
- For planning your conference attendance, please use the conference app, which will be issued some weeks before the conference and the conference agenda provided in conftool.
- If you are a session chair, best look up your chairing duties in the conference system (Conftool) or the app.