Measuring ICT competences in a valid way: a study on the accuracy of ICT self-efficacy.

Author(s):

Koen Aesaert(presenting / submitting)Johan van Braak

Conference:

ECER 2017

Network:

16. ICT in Education and Training

Format:

Paper

Session Information

16 SES 01 A, Research on K-12 Students' Digital Competence

Paper Session

Time:

2017-08-22

13:15-14:45

Room:

W4.23

Chair:

Anders D. Olofsson

Contribution

Problem statement

Most studies on the assessment of ICT competences are conducted from the perspective of self-perceived measures (Litt, 2013; Meelissen, 2008). One of the most commonly used self-perceived measures to assess students’ ICT competences is ICT self-efficacy. Although ICT self-efficacy measures are often deployed in large scale studies, they suffer from validity problems due to self-reported bias, i.e. students can over- and/or underestimate their own ICT competences (van Deursen & Van Dijk, 2011). Consequently, it can be assumed that the results of ICT self-efficacy measures are not always an accurate representation of students’ actual level of ICT competence (Siddiq, Hatlevik, Olsen, Throndsen, & Scherer, 2016). The general aim of this study is to explore whether ICT self-efficacy measures can be used to assess primary school students’ ICT competences in a valid way when taking the accuracy of students’ ICT self-efficacy into account.

Main theoretical concepts

ICT competences have been incorporated and defined in several educational frameworks on 21^st century skills and ICT literacy. For example, in the DIGCOMP framework, ICT competence is defined as “the confident, critical and creative use of ICT to achieve goals related to work, employability, learning, leisure, inclusion and/or participation in society” (Ferrari, 2013, p.2). The European Commission (2007) states that ICT competences concern information processing, problem solving, critical reasoning, and creative and innovative ICT use. According to these definitions, ICT competences are more concerned with higher cognitive capabilities in digital information processing and communication than with the mastery of basic ICT skills (Markauskaité, 2007; Siddiq et al., 2016). In this study, ICT competences are similarly defined as a student’s ability to search, process and communicate digital information using a computer and the Internet (Authors, 2015).

ICT self-efficacy originates from the construct of self-efficacy, which is derived from Bandura’s Social Cognitive Theory. According to Bandura (1986), self-efficacy is concerned with “people’s judgments of their capabilities to organize and execute courses of action required to attain designated types of performances” (p.391). In this context, ICT self-efficacy refers to peoples judgment about their abilities to successfully perform computer and Internet related behavior (Papastergiou, 2010). This study follows Tsai and Tsai’s (2010) specific conceptualization of ICT self-efficacy as a student’s individual judgment about his or her ability to explore digital information, and to communicate using the computer and the Internet. More specifically, it is about a student’s belief in his capability to successfully perform specific digital tasks with regard to 1) retrieving and processing appropriate digital information; and 2) communicating in a safe, sensible and appropriate way (Authors, 2014a).

Bias and accuracy are two measures that have not yet been studied within the context of ICT self-efficacy. Similar to self-efficacy in other academic domains, (Chen, 2003; Pajares & Graham, 1999) bias of ICT self-efficacy refers to the direction of the judgment error, i.e., bias indicates whether one is over- or underestimating his ICT competences. Accuracy of ICT self-efficacy refers to the magnitude of the judgment error, i.e., the extent to which the over- or underestimation of ICT competences is big or small.

Research objectives

The general aim of this study is to explore whether ICT self-efficacy measures can be used to assess primary school students’ ICT competences in a valid way when taking the accuracy of students’ ICT self-efficacy into account. This aim was divided into two research questions (RQ)

RQ1: To which degree does an ICT self-efficacy measure provides an accurate and non-biased representation of students’ ICT competences?

RQ2: To which degree does the accuracy of students’ ICT self-efficacy increases the predictive validity of an ICT self-efficacy measure?

Method

Sample Data were gathered from 378 sixth grade students from 58 primary schools in Flanders (the Dutch speaking part of Belgium). The students were randomly selected from each school. Of the students (mean age= 12.06; SD=0.46), exactly 50.0% were female and 50.0% were male. Instruments Students’ ICT competences were measured using a short version of the performance-based ICT competence test of Authors (2014a). The test measures the higher-order cognitive competences and technical ICT skills that students need for searching, processing and communicating digital information. All items have a dichotomous answer-format (4=correct, 1=incorrect). The short version of the ICT competence test contains nine items of which the content matches the content of the items of the ICT self-efficacy scale of Authors (2014b). The ICT competence scale is calibrated using IRT and has a theoretical range of -3 to 3. ICT self-efficacy was measured using the ICT self-efficacy scale of Authors (2014b). This scale was chosen because the items of this scale measure the same competences of the performance-based test. All items are rated on a 4-point Likert scale (1=not good at all, to 4=very good). Bias of ICT self-efficacy was calculated using the procedure of Pajares and Graham (1999), i.e., ICT competence scores were subtracted from the ICT self-efficacy scores. Consequently, the bias scale has a theoretical range from -3 to 3, with -3 indicating underestimation of ICT competences, 0 indicating no bias, and 3 indicating overestimation of ICT competences. Accuracy of ICT self-efficacy was calculated by subtracting the absolute value of each bias score from 3. As such, the accuracy of ICT self-efficacy scale has a theoretical range of 0 to 3, with 0 indicating complete inaccuracy and 3 indicating complete accuracy. Analysis First, a descriptive and correlational analysis was conducted to investigate the degree to which an ICT self-efficacy measure on average provides an accurate and non-biased representation of students’ ICT competences. Further, a one-way between groups ANOVA was used to find out whether the accuracy and bias in ICT self-efficacy was different for students with different levels of ICT competences. For this purpose, the sample was divided into 5 equal groups according to students’ score on the ICT competence test (RQ1). Second, a hierarchical regression analysis was conducted to investigate whether the accuracy of students’ ICT self-efficacy increases the predictive validity of an ICT self-efficacy measure (RQ2).

Expected Outcomes

With regard to the first research question, the results of the ANOVA illustrate that four of the five ICT competence groups overestimate their ICT competences, whereas only the group with a high level of ICT competences seems to express no bias (M=-0.03, SD=0.30). With regard to accuracy of ICT self-efficacy, the results indicate that the higher students’ level of ICT competence, the more they display accurate ICT self-efficacy. For example, whereas students with low ICT competence have an average accuracy of 1.51 (SD=0.55), students with medium ICT competences have a mean accuracy of 2.24 (SD=0.32) and students with high levels of ICT competence display almost complete accuracy (M=2.77; SD=0.17). This result illustrates that ICT self-efficacy measures can provide an accurate and non-biased representation of students’ ICT competences if the students have highly developed ICT competences. However, students with less developed ICT competences seem to misjudge their actual level of ICT competence more severely. This means that measures of ICT self-efficacy deliver accurate representations for students with highly developed ICT competences but not for students with less developed ICT competences. With regard to the second research question, the results of the regression analysis indicate that both ICT self-efficacy (β=.42) and accuracy of ICT self-efficacy (β=.87) significantly predict students’ level of ICT competence. While ICT self-efficacy explains only 1.53% of the variance in students’ ICT competences, the inclusion of accuracy of ICT self-efficacy into the model explained an additional 69.61% of the variance in students’ ICT competences. This evidence shows that accuracy of ICT self-efficacy increases the predictive validity of ICT self-efficacy measures. For this reason, accuracy of ICT self-efficacy should be taken into account when ICT self-efficacy measures are used as a proxy for students’ ICT competences.

References

Authors (2014a). Authors (2014b). Authors (2015). Bandura, A. (1986). Social foundations of thought and action: A social cognitive theory. Englewood Cliffs, NJ: Prentice Hall. Chen, P.P. (2003). Exploring the accuracy and predictability of the self-efficacy beliefs of seventh-grade mathematics students. Learning and Individual Differences, 14, 79-92. European Commission (2007). Key competences for lifelong learning. European reference framework. Luxembourg: Office for Official Publications of the European Communities. Ferrari, A. (2013). DIGCOMP: A framework for developing and understanding digital competence in Europe. JRC scientific and policy reports, Publications Office of the European Union, Luxembourg. Litt, E. (2013). Measuring users’ internet skills: A review of past assessments and a look toward the future. New Media & Society¸15(4), 612-630. Markauskaite, L. (2007). Exploring the structure of trainee teachers’ ICT literacy: the main components and relationships between general cognitive and technical capabilities. Educational Technology Research & Development, 55(6), 547-572. Meelissen, M. (2008). Computer attitudes and competencies among primary and secondary school students. In J. Voogt, & G. Knezek (Eds.), International Handbook of Information Technology in Primary and Secondary Education (pp. 381-395). New York: Springer. Pajares, F., & Graham, L. (1999). Self-Efficacy, Motivation Constructs, and Mathematics Performance of Entering Middle School Students. Contemporary Educational Psychology, 24, 124-139. Papastergiou, M. (2010). Enhancing Physical Education and Sport Science students’ self-efficacy and attitudes regarding Information and Communication Technologies through a computer literacy course. Computers & Education, 54, 298 – 308. Siddiq, F., Hatlevik, O.E., Olsen, R.V., Throndsen, I., & Scherer, R. (2016). Taking a future perspective by learning from the past - A systematic review of assessment instruments that aim to measure primary and secondary school students' ICT literacy. Educational Research Review, 19, 58-84. Tsai, M.J., & Tsai, C.C. (2010). Junior High School Students’ Internet Usage and Self-Efficacy: A Re-examination of the gender gap. Computers & Education, 54, 1182 – 1192. van Deursen, A., & van Dijk, J. (2011). Internet skills and the digital divide. New Media Society, 13(6), 893 – 911.

Author Information

Koen Aesaert (presenting / submitting)

Ghent University

Educational Studies

Ghent

Johan van Braak

Ghent University

Department of Educational Studies