Science inquiry practices and student outcomes on TIMSS: A cross-country comparison of the moderating role of teacher evaluation policies

Author(s):

Leslie Hawley (presenting / submitting) Gwen Nugent

Conference:

ECER 2016

Network:

09. Assessment, Evaluation, Testing and Measurement

Format:

Paper

Session Information

09 SES 02 B, Relating Student Attitudes and Teaching Practices to Science Achievement

Paper Session

Time:

2016-08-23

15:15-16:45

Room:

NM-F103a

Chair:

Trude Nilsen

Contribution

Guided scientific inquiry has been shown to be effective in promoting student achievement (Lynch, Kuipers, Pyke, & Szesze, 2005; Vandosdall, Klentschy, Hedges, Weisbaum, 2007). However, school climate can influence students’ motivation to learn and achieve (Deal & Peterson 2009). School climate, particularly as it relates to accountability, may lead teachers to alter their teaching practices, subsequently influencing student-level outcomes.

The goal of the proposed study is to use a multi-group multilevel structural equation model (MG-MSEM) to compare the relationship between school-level accountability policies, teachers’ science teaching practices and student-level outcomes (science achievement and affect towards science). Data from the 2011 Trends in Mathematics and Science Study (TIMSS) will be used to compare relationships across the United States, Australia and Finland.

Conceptual Approach

Science Inquiry

Guided scientific inquiry is student-centered, driven by student data collection and analysis, and leads to student formulation of an underlying science concept or principle. It is also teacher-facilitated, requiring extensive use of teacher questioning and scaffolding to guide students to greater understanding of science concepts, science content, and science practice skills. A comprehensive analysis of inquiry research also found that teaching strategies that actively engage students in the learning process through scientific investigations are more likely to increase conceptual understanding than strategies that rely on passive techniques (Minner, Levy, & Century, 2010).

Although students’ motivation and achievement have been shown to be positively related to teacher instructional practices, the positive influence of teacher practices may be influenced by a school’s climate (Deal & Peterson, 2009). In particular, the accountability methods used to evaluate teachers have the potential to influence both teachers and students.

Accountability Climate

Accountability methods in education present a complex inter-relationship between the pressures of reform efforts and unintended consequences for students, teachers, and schools. When schools have a strong emphasis on teacher accountability, teachers may feel pressured and motivation may become extrinsic as they act upon fear or potential consequences (Santiago & Benavides, 2009; Cruz & Brown, 2010). Research from across the globe has demonstrated some of the potential negative outcomes associated with an overemphasis on accountability including, but not limited to: a narrowing of the curriculum; test-centered rather than student-centered environments; heightened stress; and a marginalization of low-performing students (e.g., Jaeger, Merki, Oer, & Holmeier, 2012; Polesel, Rice, & Dulfer, 2014; Rustique-Forrester, 2005).

Test-based accountability systems are becoming increasing popular in educational reform efforts (Hamilton, 2003). These types of accountability systems are prevalent within countries such as the United States (U.S.), England and Australia (Rustique-Forrester, 2005). In contrast, the high achieving country, Finland, does not rely upon external standardized testing to evaluate the performance of schools or teachers (Darling-Hammond & McCloskey, 2008; Sahlberg, 2011).

Research Questions

Increased focus on accountability has the potential to alter teachers’ instructional practices and restrict students’ opportunities to engage in creative activities such as guided science inquiry. The purpose of the current study is to use a multi-group multilevel structural equation model (MG-MSEM) to compare the moderating role of accountability policies on the relationship between teachers’ science practices and students’ science affect on achievement across the U.S., Australia and Finland.

Method

Methods Data Data from the 2011 administration of TIMSS were collected from 4th grade students (mean age of at least 9.5 years) across more than 50 countries (Martin & Mullis, 2012). For our study, we selected students from the U.S., Australia, and Finland. These countries were chosen based on their accountability practices and science performance on the 2011 TIMSS. Finland was chosen as an example of a high-achieving country (average score 570), the U.S. as a medium-performing country (average score 544) and Australia as a lower-achieving country (average score 516) (Provasnik et al., 2012). Analyses Mplus v.7.1 software (Muthén & Muthén, 1998-2012) will be used to estimate the MG-MSEM. This modeling approach takes into account the nesting of children within schools (i.e., L1: student; L2: school) for each country. Appropriate sampling weights and centering will also be applied to the data. A latent MG-MSEM model will allow us to decompose the variance into between-school and within-school components for each country. This latent variable modeling framework will provide a means by which to evaluate the degree to which the latent constructs have the same meaning across groups. Prior to evaluating our complete model, we will test three levels of measurement invariance (configural, metric, and scalar) for each latent construct across countries. The advantage of this approach (invariant factor-loadings and intercepts) is that it allows for direct cross-country comparison. Variables Student Outcomes. Two student-level outcome variables will be included in our MG- MSEM model: students like learning science scale (SLS) and science achievement (ACHIEVE). The SLS is a 5-item scale that measure students’ degree of affect towards science (average reliability in our selected countries α = 0.89). Science achievement was measured using five plausible values. Plausible values will be incorporated into our model using the IMPUTATION function in Mplus. This will produce an average of the five plausible values using appropriate adjustments for standard errors. School and Teacher Variables. Analyses will include predictors of science inquiry practices (Teachers Emphasize Science Investigation scale; ESI) and a measure of accountability practices (ACCT). ESI is a six-item measure of science inquiry teaching practices (average reliability in our selected countries α = 0.75). School administrators provided information for the 4-item ACCT measure. Administrators indicated whether teachers were evaluated by aspects such as observations, achievement, and/or peer review.

Expected Outcomes

Our proposal represents a work in-progress. To date, we have selected our variables for inclusion in the models, completed data management for our analysis, and begun the early stages of the measurement invariance component of the project. Our team has extensive experience with large-scale international datasets and advanced statistical procedures in Mplus, so there is sufficient time to complete our proposed analyses in advance of the August conference. Expected Outcomes We hypothesize there will be significant cross-country differences in the moderating role of accountability policies on the relationship between teachers’ science practices and students’ science affect on achievement. Based on the literature, we assume there will be no significant moderating relationships in Finland. However, the results from U.S. and Australia are anticipated to paint a different picture. We anticipate the interactions of interest (Inquiry and ACCT; Affect and ACCT) will demonstrate a negative relationship with science achievement in the U.S. and Australia. Scientific Significance Results from our analyses will help contribute to understanding the influence teacher evaluation policies have on the relationship between science teacher practices and student outcomes. In particular, one of the benefits of using multilevel modeling is the ability to simultaneously separate between-school and within-school effects of evaluation policies across schools within a country. Consequently, this research will contribute to the substantive literature regarding evaluation policies with student achievement data across different contexts.

References

Darling-Hammond, L., & McCloskey, L. (2008). Assessment for learning around the world: What would it mean to be internationally competitive. Phi Delta Kappan, 90(4), 263-272. Deal, T. E., & Peterson, K. D. (2009). Shaping School Culture: Pitfalls Paradoxes and Promises.San Francisco: Jossey-Bass. Hamilton, L. (2003). Assessment as a Policy Tool. Review of Research in Education, 27, 25-68. Jaeger, D. J., Merki, K. M., Oer, B., & Holmeier, M. (2012). Statewide Low-stakes and a Teaching to the Test Effect? An Analysis of Teacher Survey Data from Two German States. Assessment in Education Principles Policy and Practice, 19, 451-467. Lynch, S., Kuipers, J., Pyke, C., & Szesze, M. (2005). Examining the effects of a highly rated science curriculum unit on diverse students: Results from a planning grant. Journal of Research in Science Teaching, 42, 912–946. doi:10.1002/tea.20080 Martin, M.O. & Mullis, I.V.S. (Eds). (2012). Methods and procedures in TIMSS and PIRLS 2011. Chestnut Hill, MA: TIMSS & PIRLS International Study Center, Boston College. Minner, D., Levy, A., & Century, J. (2010). Inquiry-based science instruction—what is it and does it matter? Results from a research synthesis years 1984–2002. Journal of Research in Science Teaching, 47, 474–496. doi:10.1002/tea.20347 Muthén, L. K., & Muthén, B. O. (1998-2012). Mplus User’s Guide. Seventh Edition. Los Angeles, CA: Muthén & Muthén. Polesel, J., Rice, S., & Dulfer, N. (2014). The impact of high-stakes testing on curriculum and pedagogy: a teacher perspective from Australia. Journal of Education Policy, 29, 640-657. Provasnik, S., Kastberg, D., Ferraro, D., Lemanski, N., Roey, S., and Jenkins, F. (2012). Highlights From TIMSS 2011: Mathematics and Science Achievement of U.S. Fourth- and Eighth-Grade Students in an International Context (NCES 2013-009 Revised). National Center for Education Statistics, Institute of Education Sciences, U.S. Department of Education. Washington, DC. Rustique-Forrester, E. (2005, April 8). Accountability and the pressures to exclude: A cautionary tale from England. Education Policy Analysis Archives, 13(26). Retrieved from http://epaa.asu.edu/epaa/v13n26/. Sahlberg, P. (2011). The Professional Educator: Lessons from Finland.American Educator, 35(2), 34-38. Vanosdall, R., Klentschy, M., Hedges, L. H., & Weisbaum, K. S. (2007). A randomized study of the effects of scaffolded guided-inquiry instruction on student achievement in science. Paper presented at the American Educational Research Association, Chicago, IL.

Author Information

Leslie Hawley (presenting / submitting)

Nebraska Center for Research on Children, Youth, Families and Schools

Lincoln

Gwen Nugent

Nebraska Center for Research on Children, Youth, Families and Schools, United States of America

Update Modus of this Database

The current conference programme can be browsed in the conference management system (conftool) and, closer to the conference, in the conference app.
This database will be updated with the conference data after ECER.

Search the ECER Programme

Search for keywords and phrases in "Text Search"
Restrict in which part of the abstracts to search in "Where to search"
Search for authors and in the respective field.
For planning your conference attendance, please use the conference app, which will be issued some weeks before the conference and the conference agenda provided in conftool.
If you are a session chair, best look up your chairing duties in the conference system (Conftool) or the app.