Psychometric Validation of an Objective Measure of STEM School Achievement in Primary School

Author(s):

Dubravka Glasnovic Gracin(submitting)Toni Babarovic(presenting)Ivan Devic(presenting)Josip Burusic Mia Karabegovic

Conference:

ECER 2017

Network:

09. Assessment, Evaluation, Testing and Measurement

Format:

Paper

Session Information

09 SES 06 B, Investigating Outcomes in the STEM-field at Primary- and Lower Secondary School Level

Paper Session

Time:

2017-08-23

15:30-17:00

Room:

W5.18

Chair:

Mehmet İkbal Yetişir

Contribution

Many national governments and organizations around the world (e.g. American National Science Foundation, European Union countries etc.) within their educational policies recognized the problem of lower interest for careers in science, technology, engineering, mathematics and related fields (STEM). Therefore, a big effort is made in order to encourage the interest for these fields among students (Honey, Pearson & Schweingruber, 2014).
Some of the current international projects have focus on the students’ achievement in the fields of science and mathematics, such as PISA or TIMSS (OECD, 2003; Braš Roth, Gregurović, Markočić Dekanić & Markuš, 2008; Mullis & Martin, 2013). Their frameworks influenced changes in many national science or math curricula (e.g. KMK, 2003). PISA examines mathematical and science literacy on 15-year old students, while the TIMSS study encompasses students at the age of 10 and 14. This age interval is generally taken as an important for forming attitudes about future interests and careers (Lindahl, 2007) as well as acquiring knowledge and competences necessary for following educational levels (OECD, 2003).
Recent studies on STEM achievement refer to the problem of traditionally separated school subjects involved in STEM areas (mathematics, science, biology, physics, chemistry etc). “Most studies of STEM learning consider each discipline singly and do not measure students’ ability to make connections across disciplines” (Honey et al, 2014, p. 52). Also, “assessment instruments on integration in STEM are rare because theories and tests have generally focused on content area-specific concepts and procedures” (Honey et al, 2014, p. 52). Therefore Xie, Fang and Shauman (2015) discuss the term 'STEM education' which emphasizes logical and conceptual connections across different STEM fields. In this view, the STEM education is considered as a whole, not as a set of separated subjects. However, the idea of simply integrating STEM subjects in school practice is complex and should be considered carefully. According to meta-analysis about integrated mathematics and science instruction conducted by Hurley (2001), there are fewer positive benefits of integration for mathematics outcomes compared to science outcomes. The results indicate that it is difficult to enhance good mathematics achievement just by integrating the math into another discipline.
These findings imply that we should take the integration into account carefully, focusing on logical and conceptual connections across different STEM subjects. Cognitive abilities which connect STEM disciplines are problem-solving, spatial ability, numeracy, comprehension, reasoning, knowledge acquisition, abstract thought, and connection-making (Xie et al, 2015). Cognitive ability is strongly associated with students’ school performance, particularly in math and science tests (Deary, Strand, Smith & Fernandez, 2007).
The results of Croatian students on PISA and TIMSS assessments show the low or average results in science and mathematics (Braš Roth et al, 2008; NCVVO, 2012a; 2012b). Also, the achievement of Croatian students in science and mathematics is low in comparison to their achievement in other school subjects (Burušić, Babarović & Šakić, 2008). These findings motivated us for examining the STEM achievement of Croatian 10, 11 and 12-year-old students. This is the period when the Croatian students change from lower to middle compulsory education. Valid STEM achievement instruments and especially psychometric validated integrated STEM knowledge tests are still missing in Croatia.
With the purpose to adequately measure STEM knowledge in Croatian primary schools, we designed three STEM tests, following the above-mentioned outlines for STEM education and the analysis results of current curriculum resources (MZOS, 2006; 2010). The aim of this research is psychometric validation of this newly designed STEM tests for grades 4, 5 and 6.

Method

• Participants and procedure The participants in this study were 1798 primary school students attending grades 4 to 6 (age 10 - 12). They are equally distributed by gender (49.8% girls) and grades (N(4th grade) = 586, N(5th grade) = 580, N(6th grade) = 632). This study is a part of a broader research project JOBSTEM - STEM career aspirations during primary schooling (www.jobstem.eu). The study is conducted in 16 schools in the city of Zagreb and its surroundings. Students from sixteen primary schools participated in the study. Within each school, two classes of students within one generation were sampled by random, and joined the survey. Data used in this study was collected within regular classes in school. In addition, student school achievement data (school grades) was collected from schools. • Instruments Three tests were designed to measure knowledge in STEM area for grades 4, 5 and 6, following the outlines for STEM education and for task design (Sullivan, Clarke & Clarke, 2013; IDM, 2007; Smith & Stein, 1998) and current curriculum resources (MZOS, 2006; 2010). The final versions of the three knowledge tests were pretested with a larger number of items in pre-test versions (25 test items). The items with the best psychometric properties were retained in the final tests versions. The final tests for each grade contained 20 items. Each item in the test had only one correct answer and it was coded as 1 or 0. The test total score was calculated as sum of the item points ranging from 0 to 20, with higher scores indicating greater knowledge in STEM area. School achievement was measured by school grades (marks) scaling from 1 to 5 (5 - Excellent, best possible grade; 4 - Very Good, next highest grade; 3 - Good, indicates average performance; 2 - Sufficient, lowest passing grade; 1 - Insufficient, failing grade). In the Croatian school system and school curriculum, the STEM area in grades 4 to 6 is covered by the following school subjects: mathematics, nature, geography and technical education. STEM school achievement in our study was measured as an average of school marks in all STEM school subjects within different grades (4, 5, 6).

Expected Outcomes

The mean score in 4th grade was M=11.81, in 5th grade M=9.34 and in 6th grade M=10.93. The average item difficulty was 61%for 4th grade, 42%for 5th and 58% for 6th grade. Somewhat higher standard deviation (SD = 4.32) in 4th grade indicates slightly higher variations between students’ knowledge in 4th grade compared to 5th (SD = 3.66) and 6th grade (SD = 3.97). The distribution of scores in all three tests is symmetric but with higher tails. Cronbach’s alpha coefficients indicate adequate internal reliability, α4th grade = 0.78; α5th grade = 0.70; α6th grade = 0.79.To examine construct validity, principal component analysis (PCA) was performed. As expected, Scree test criterion revealed a single factor structure in all three tests, where all items were highly saturated with general factor. Single factor structures were confirmed by confirmatory factor analysis(CFA) for test in 4th grade (χ² (170)=239; p<0.05; χ²/df = 1.41; CFI = 0.951; TLI = 0.945; RMSEA = 0.026), 5th grade (χ² (170)=313.15; p<0.05; χ²/df = 1.84; CFI = 0.941; TLI = 0.935; RMSEA = 0.035) and 6th grade (χ² (170) = 248.8; p<0.05; χ²/df = 1.46; CFI = 0.938; TLI = 0.931; RMSEA = 0.027). Correlations (r) between test scores and achievement in STEM school subjects were all positive and moderate whereby all three test scores had highest correlations with achievement in math and slightly lower correlation with other STEM school subjects (geography, nature and technical education). The STEM knowledge tests proved to be reliable and valid measures of knowledge in STEM area in primary school. It seems that knowledge in STEM area, taught within distinctive school subjects, can be integrated in one comprehensive but concise measure. These tests could be used as objective, short and valid assessment of STEM knowledge in Croatian primary schools.

References

Braš Roth, M., Gregurović, M., Markočić Dekanić, A. & Markuš, M. (2008). PISA 2006. Prirodoslovne kompetencije za život. Zagreb: Nacionalni centar za vanjsko vrednovanje obrazovanja - PISA centar. Burušić, J., Babarović, T., & Šakić, M. (2008). Vanjsko vrednovanje obrazovnih postignuća u osnovnim školama Republike Hrvatske: Učenici 8. razreda, školska godina2007/2008., istraživački izvještaj. Zagreb: NCVVO/ Institut društvenih znanosti Ivo Pilar. Deary, I.J., Strand, S., Smith, P. & Fernandes, C. (2007). Intelligence and educational achievement. Intelligence, 35, p. 13–21. Honey, M., Pearson, G. & Schweingruber, H. (Eds.) (2014). STEM Integration in K-12 Education: Status, Prospects, and an Agenda for Research. Washington, DC: The National Academic Press. Hurley, M. M. (2001). Reviewing Integrated Science and Mathematics: The Search for Evidence and Definitions From New Perspectives. School Science and Mathematics, 101(5), 259-268. KMK - Beschlüsse der Kultusministerkonferenz. (2003). Bildungsstandards im Fach Mathematik für den Mittleren Schulabschluss. Beschluss vom 15. 10. 2004. Retrieved from http://www.kmk.org/fileadmin/Dateien/veroeffentlichungen_beschluesse/2003/2003_12_04-Bildungsstandards-Mathe-Mittleren-SA.pdf (1.1.2017) Lindahl, B. (2007). A longitudinal study of students' attitudes towards science and choice of career. In NARST annual conference, New Orleans. Mullis, I.V.S. & Martin, M.O. (Eds.). (2013). TIMSS 2015 Assessment Frameworks. Chestnut Hill, MA: TIMSS & PIRLS International Study Center, Boston College. MZOS - Ministarstvo znanosti, obrazovanja i športa (2006). Nastavni plan i program za osnovnu školu. HNOS. Zagreb: Ministarstvo znanosti, obrazovanja i športa Republike Hrvatske. MZOS - Ministarstvo znanosti, obrazovanja i športa (2010). Nacionalni okvirni kurikulum za predškolski odgoj i obrazovanje te opće obvezno i srednješkolsko obrazovanje. Zagreb: Ministarstvo znanosti, obrazovanja i športa Republike Hrvatske. NCVVO - Nacionalni centar za vanjsko vrednovanje obrazovanja (2012a). TIMSS 2011. Izvješće o postignutim razultatima iz matematike. Zagreb: Nacionalni centar za vanjsko vrednovanje obrazovanja. NCVVO - Nacionalni centar za vanjsko vrednovanje obrazovanja (2012b). TIMSS 2011. Izvješće o postignutim razultatima iz prirodoslovlja. Zagreb: Nacionalni centar za vanjsko vrednovanje obrazovanja. OECD - Organisation for Economic Co-operation and Development (2003). The PISA 2003 assessment framework – mathematics, reading, science and problem solving knowledge and skills. Preuzeto s http://www.pisa.oecd.org/dataoecd/46/14/33694881.pdf (1.6.2016.) Xie, Y., Fang, M. & Shauman, K. (2015). Stem Education. Annual Review of Sociology, Vol. 41, pp. 331-357, 2015.

Author Information

Dubravka Glasnovic Gracin (submitting)

University of Zagreb

Faculty of Teacher education

Zagreb

Toni Babarovic (presenting)

Ivo Pilar Institute of Social Sciences

Zagreb

Ivan Devic (presenting)

Institute of social sciences Ivo Pilar

Zagreb

Josip Burusic

Ivo Pilar Institute of Social Sciences

Zagreb

Mia Karabegovic

Central European University, Budapest, Hungary

Search the ECER Programme

Search for keywords and phrases in "Text Search"
Restrict in which part of the abstracts to search in "Where to search"
Search for authors and in the respective field.
For planning your conference attendance you may want to use the conference app, which will be issued some weeks before the conference
If you are a session chair, best look up your chairing duties in the conference system (Conftool) or the app.

Session Information

Contribution

Method

Expected Outcomes

References

Author Information

Search the ECER Programme

Navigation

Info for