Session Information
31 SES 07 B, Assessing Receptive Language Skills
Paper Session
Contribution
In order to participate actively in 21st century society, not only being literate in one’s first language (L1) but also becoming literate in at least one foreign language (FL), is amongst the key requirements. The most frequently learnt foreign language is English, which has become a lingua franca across the globe (EACEA, 2017; Stein-Smith, 2017). Hence, it is of particular importance that students do not only develop the oral skills (speaking and listening; necessary for successful social participation in an English speaking country) but also the print-related skills reading and writing. With regard to the latter, reading is of particular importance since reading provides access to information for everyday life, for the work place as well as for academic purposes (Darcovich et al., 1997; PISA, 2001).
However, reading is a complex process that requires the orchestration of a number of cognitive and meta-cognitive processes in order to extract meaning from text (Cain & Oakhill, 2012; Koda, 2005; Perfetti & Stafura, 2014). Thus, the FL-teaching context very much relies on transfer of L1 skills onto FL. Its success depends on (a) enough knowledge in L1 and (b) enough experience with reading in general (Cummins, 1979; Koda, 2005). This puts students from deprived backgrounds at a disadvantage. Therefore, instruction responding to individual learner needs is key for enabling these students to develop sufficient literacy skills in the FL successfully (Koda, 2005). In order to adequately tailor instruction to learners’ needs, teachers need to assess where the issues are. Reading comprehension tests can be of assistance. Such tests, however, are frequently very broad with regard to the feedback they provide about students’ abilities in the different areas assessed.
One such test developed for supporting teachers in identifying learner needs is the National educational standards test for English in Austria (BIST-UE). This test assesses students’ English as a foreign language (EFL) abilities after four years of instruction and provides feedback about students’ skills in EFL Reading, Listening and Writing. Of particular importance to this presentation is the EFL reading test of the BIST-UE, which reports very broadly by referring to performance level descriptors derived from the Common European Framework of Reference (CEFR) and the national education standards for EFL reading (Siller & Kulmhofer, 2018). These performance level descriptors summarize the expected EFL reading abilities as described in the respective CEFR levels (A2 and B1), in the national curriculum for foreign languages and the national education standards for EFL reading. Thus, a holistic picture of students’ reading abilities is drawn which is enough for system monitoring purposes but maybe too broad for the identification of adequate instructional goals on the classroom and individual learner level. Therefore, an approach with a more detailed focus on FL reading skills involved in the respective test is required in order to learn about the targeted linguistic, cognitive and strategic skills. One such approach would be the application of cognitive diagnosis assessments in combination with cognitive diagnostic models (CDMs; DiBello, Roussos, & Stout, 2007). Applying CDMs requires a multi-step process. This tiered process is the focus of the prevailing presentation aiming at addressing the following research questions:
1) Domain analysis: What are the FL reading skills required for the successful completion of items in the Austrian national EFL reading test?
2) Q-matrix construction: How do subject experts assign the respective FL reading skills to these reading items?
Learning about such reading skills could be helpful for teachers and teacher trainers to better understand the abilities involved in EFL reading and to better understand the reports provided so that responsiveness of instruction to individual learner needs increases.
Method
CDMs are discrete latent variable models that allow tracing back a test taker’s performance on the item level to the skills addressed in the respective item. Thus, CDM results can be used to identify different ability groups, which may allow for more targeted support and instruction (de la Torre & Minchen, 2014). Hence, this approach seems appropriate to gain an understanding for the more subtle elements of FL reading. Applying a CDM requires the identification of the relevant set of skills and their assignment to the items; a multi-step process. In a first step, a list of FL reading skills is developed. This step is often called domain analysis and refers to the first research question. Following the procedure as described in Kim (2015), the list of skills for this project was developed after thorough analysis of common models of FL reading such as Khalifa and Weir’s model of cognitive processing in reading (Khalifa & Weir, 2009; Brunfaut & McCray, 2015) and the communicative language ability model (Bachman & Palmer, 1996). The skills described in these models were analyzed towards fit with the language abilities as described in the national curriculum for foreign languages, the abilities described in the national educational standards and the targeted CEFR levels. In a second step, summarized in the second research question, three content experts familiarized themselves with this final list of skills in a training session. Next, they analyzed 200 reading items. The goal of the item analysis was to allocate the pre-defined reading skills to each item with particular focus on the major skills used when responding to each individual item. By assigning skills to items, experts constructed so-called Q-Matrices. Finally, the initial set of data consisting of three Q-Matrices was further analyzed. Items with three or more agreed skills were selected for a final Q-Matrix. The content experts then judged the items in the final Q-Matrix again with regard to whether (a) all the assigned skills were necessary to solve the item successfully or (b) the assigned skills could be compensated. Empirical data was used to evaluate the expert judgements. This step is of importance for the final step, the calculation of cognitive diagnostic models (CDMs) allowing to gain a more profound understanding for the strengths and weaknesses of different learner groups which is beyond the scope of this presentation.
Expected Outcomes
By identifying the FL reading skills of individual reading items teachers and test-developers knowledge regarding the expected abilities of 8th grade EFL learners in Austria is increased. This knowledge can then directly be fed back to teacher trainers and teacher education institutions in order to foster in-service and pre-service EFL teachers’ understanding for FL reading. Effective classroom interventions and teaching approaches can thus be identified and implemented. Further, such an analysis also provides important information for item development valuable to large scale testing institutions who can use the findings in order to refine the item development process or improve their reports. Regarding the latter, the identified reading skills may directly be linked to the performance level descriptors and thus lead to a better understanding for the expected learning outcomes of 8th grade students in Austria. This may lead to a more detailed picture about how the ability groups as described in the performance level descriptors differ with regard to the expected reading abilities.
References
Bachman, L. F. & Palmer, A. S. (1996). Language testing in practice. Oxford: Oxford University Press. Brunfaut, T. & Mc Cray, G. (2015). Looking into test-takers’ cognitive processes while completing reading tasks: a mixed-method eye-tracking and stimulated recall study. ARAGs research report online. https://www.britishcouncil.org/sites/default/files/brunfaut-and-mccray-report_final.pdf Cain, K. & Oakhill, J. (2012). Reading comprehension development from seven to fourteen years: Implications for assessment. In. J. P. Sabatini, E. R. Albro, & T. O. O’Reilly (Eds.). Measuring up: Advances in how we assess reading ability (pp. 59-76). Lanham, MD: Rowan & Littlefield Education. Cummins, J. (1979). Linguistic interdependence and the educational development of bilinguial children. Review of educational research, 49(2), 222-251. Darcovich, N., Jones, S., Kirsch, I. S., Levin, H. M., Murray, T. S., Rubenson, K., . . .Tuijnman, A. C. (1997). Literacy skills for the knowledge society: further results from the International Adult Literacy Survey. Paris, France: OECD. de la Torre, J., & Minchen, N. (2014). Cognitively diagnostic assessments and the cognitive diagnosis model framework. Psicología Educativa, 20, 89-97. DiBello, L., Roussos, L., & Stout, W. (2007). Review of cognitively diagnostic assessment and a summary of psychometric models. In C. R. Rao, & S. Sinharay (Eds.), Handbook of statistics (Volume 26, Psychometrics, pp. 979–1030). Amsterdam: Elsevier. EACEA (Ed.) (2017). Key data on teaching language at school in Europe – 2017 Edition. Eurydice Report. Luxembourg: Publications Office of the European Union. https://publications.europa.eu/s/kjTZ [retrieved January 29, 2019] Kim, A. (2015). Exploring ways to provide diagnostic feedback with an ESL placement text: cognitive diagnostic assessment of L2 reading ability. Language Testing, 32(2). 227-258. Khalifa, H., & Weir, C. J. (2009). Examining reading: Research and practice in assessing second language reading. Cambridge: Cambridge University Press. Koda, K. (2005). Insights into second language reading. A cross-linguistic approach. Cambridge: CUP. Perfetti, C. & Stafura, J. (2014). Word knowledge in a theory of reading comprehension. Scientific studies of reading. 18(1). 22-37. PISA. (2001). Knowledge and skills for life. First results from the OECD programme for international student assessment (PISA) 2000. Paris: OECD. Siller & Kulmhofer (2018). The development of the Austrian Educational Standards Test for English Reading at grade 8. In: Sigott, G. (Ed.). Language Testing in Austria: Taking Stock. (pp. 85-108). Berlin: Peter Lang. Stein-Smith, K. (2017). The multilingual advantage: foreign language as a social skill in a globalized world. International Journal of Humanities and Social Science, 7(3). 48-56
Search the ECER Programme
- Search for keywords and phrases in "Text Search"
- Restrict in which part of the abstracts to search in "Where to search"
- Search for authors and in the respective field.
- For planning your conference attendance you may want to use the conference app, which will be issued some weeks before the conference
- If you are a session chair, best look up your chairing duties in the conference system (Conftool) or the app.