During a period of 10 to 15 years we have seen an increasing number of studies measuring different aspects of competence related to Information Society Technology (IST) from a perspective of self-efficacy within Information and Communication Technology (ICT) Literacy (e.g. Fraillon, Ainley & Schulz, 2015). With such indirect methods there is an obvious validity problem, depending on the informants’ abilities to evaluate their own level of competence. With this paper we will try to describe and discuss the process of developing a digital test tool based on direct performance and the problems connected to data collection with such a tool. Furthermore, questions regarding construction of an instrument using Item Response Theory (IRT) for validation of direct measurement of ICT competence will be addressed. Our focus in the development of this particular digital test tool is on primary school pupils’ proficiency within the field of digital information processing, searching and seeking, validation, sharing and communication.

In a Nordic educational context and likewise at an international level several definitions on ICT competence have been developed. Similarly, the importance of the same has been acknowledged in a lot of policy documents like different international and national ICT curricula's. One example is the recommendation regarding eight key competences for lifelong learning. Digital competence, as one of the eight, is defined as “the confident and critical use of Information Society Technology (IST) for work, leisure and communication. It is underpinned by basic skills in ICT: the use of computers to retrieve, assess, store, produce, present and exchange information, and to communicate and participate in collaborative networks via the Internet” (European Parliament, 2006, p. 16).

The research group aims, in an overall project, at developing tests for different contexts within the area of digital literacy, using the same technology and methods for analyzing. The technology is a mobile laboratory, called IRT-laboratory and described further below. IRT refers to the statistical method used. The specific aim of this pilot study is to develop a test to measure secondary school pupils’ information literacy by using this mobile IRT-laboratory.

Aesaert, K., van Nijlen, D., Vanderlinde, R., & van Braak, J. (2014). Direct measures of digital information processing and communication skills in primary education: Using item response theory for the development and validation of an ICT competence scale. Computers & Education, 76, 168–181. doi:10.1016/j.compedu.2014.03.013 De Ayala, R. J. (2009). The theory and practice of item response theory (methodology in the social sciences series). New York: Guilford Publications. European Parliament. (2006). RECOMMENDATION OF THE EUROPEAN PARLIAMENT AND OF THE COUNCIL of 18 December 2006 on key competences for lifelong learning. Retrieved from http://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:32006H0962&from=EN Fraillon, J., Ainley, J., & Schulz, W. (2015). Preparing for life in a digital age: The IEA international computer and information literacy study international report. Switzerland: Springer International Publishing AG. Hambleton, R. K., & Swaminathan, H. (1984). Item response theory: Principles and applications (evaluation in education and human services). Boston: Distributors for North America, Kluwer Boston. Hambleton, R. K., Rogers, J. H., & Swaminathan, H. (1991). Fundamentals of item response theory (4th ed.). United States: SAGE Publications.