International large-scale assessment tests (ILSA) have been a source of criticism and praise since the beginning of their application in different countries, especially from the 21st century onwards (Gómez y Jiménez, 2018; Sanmartí i Puig & Sardà Jorge, 2007; Schleicher, 2016). These tests have served as a basis for the comparison of educational systems and results between countries, and as a justification for the development of educational policies. One of the most widely used and well-known ILSAs is the PISA test proposed by the OECD for 15-16 year-olds (OECD, 2016). This test aims to assess the level of acquisition of basic skills of these students (Ministerio de Educación y Formación Profesional, 2022).  Primarily, as it measures pupils' school performance, comparisons are made on the basis of these educational outcomes, which in turn serve as predictors of the school effectiveness of different schools.

This research is part of the R&D&I project "Efficient schools for the improvement of the system (EFI-SEIS), funded by the Ministry of Science, Innovation and Universities of the Government of Spain and with reference PGC2018-099174-B-I00.

We set out to find out which non-contextual variables measured in PISA 2018 influence school effectiveness in Spanish, Portuguese and Irish schools. To achieve this objective, we first developed a series of two-level hierarchical linear regression models (student and school) for each competency studied (reading comprehension, mathematics and science) in each country, thus obtaining the contextual factors associated with the academic performance of students participating in the 2018 edition of PISA in each of these countries. Our sample consisted of students participating in PISA 2018 from schools with more than 20 students in the three selected countries, in line with previous research (Gamazo et al., 2018; Martínez-Abad et al., 2017).

After controlling for contextual factors, we use multilevel models to calculate the school effectiveness of schools, understanding school effectiveness as "the good practices that schools carry out to achieve higher academic performance than expected based on their contextual characteristics" (Frade-Martínez et al., 2021, p. 733). This value referring to school effectiveness is obtained by means of the statistical residual of each school in each competence, that is, the difference between the scores obtained by a school and those expected according to its contextual characteristics.

We present here a comparison between the variables that influence school effectiveness in each of the three participating countries in order to propose a series of suggestions for improvement to be applied in each country.

The results of the binary logistic regression indicate that the variables influencing school effectiveness in each country are:

-      Spain: TCICTUSE from the teachers and METASUM, METASPAM and SCREADDIFF from students.

-      Portugal: METASUM and DISCLIMA from the students.

-      Republic of Ireland: SCREADCOMP, HOMESCH and AUTICT from the students.

These variables refer to:

-      TCICTUSE: Teacher's use of specific ICT applications.

-      METASUM: Meta-cognition: summarising.

-      METASPAM: Meta-cognition: assess credibility.

-      SCREADDIFF: Self-concept of reading: Perception of difficulty.

-      SCREADCOMP: Self-concept of reading: Perception of competence.

-      HOMESCH: Use of ICT outside of school (for school work activities).

-      AUTICT: Perceived autonomy related to ICT use.

-      DISCLIMA: Disciplinary climate in test language lessons.

The only similarity between the models refers to the METASUM variable in Spain and Portugar, which has a positive influence in their school effectiveness. Due to the diversity of the socio-economic and cultural composition of the three countries included in the analysis, there are a number of limitations to this study.

In order to achieve our objective, which was to find out the non-contextual variables with an influence on school effectiveness in Spanish, Portuguese and Irish schools with more than 20 students that participated in PISA 2018, we differentiated our study into two parts. Our sample consisted of 976 Spanish schools, 34411 students and 19991 teachers; 155 Irish schools and 5551 students; and 178 Portuguese schools, 4688 students and 4061 teachers; as we eliminated from the analyses those schools with less than 20 students as has been done in previous similar studies (Gamazo et al., 2018; Martínez-Abad et al., 2017; Meunier, 2011). First, using a Pearson Correlation we tested the relationship between the selected variables and the residuals of each competence in each country (difference between the score obtained by the students of a given school and the expected score based on their contextual characteristics) (Gamazo et al., 2019). The variables selected were 4 variables at school level, 22 variables at teacher level and 50 variables at student level; the latter two groups were aggregated according to the school to which they belong and all of them were selected based on the previous literature review. Next, based on the previous results, we performed a binary logistic regression in order to develop a model that would allow us to test the influence of the process variables that were previously found to be significant (on student performance in the three competences) on the school effectiveness of the schools (Gamazo et al., 2018), obtaining the results presented above.

The results of this research have two fundamental limitations: 1. When calculating the binary logistic regression between the variables that showed a significant relationship with the three competences studied, we find some missing values, so that the school to which this variable refers is eliminated from the analysis, making it impossible to apply the results obtained with complete certainty. In the case of Spain, this limitation means losing 10 schools and, in Portugal, one school, in this case with low effectiveness. 2. The samples of the three countries differ considerably in terms of the number of participants. In Portugal and Ireland we have a sample of around 4600 and 5500 students and 178 and 155 schools respectively, compared to 34411 students and 976 schools in Spain. In the case of Ireland, we don’t have data on the characteristics of teachers which limits the information available to us for our subsequent decision-making. In the cases of Portugal and Spain, the sample of teachers consists of 4061 and 19991 individuals respectively. This last limitation means that these models are not comparable with each other, but they provide us with an overview of the variables that most influence school effectiveness in each of the three countries, understood as the ability of schools to cope with the initial disadvantages in which they may find themselves depending on their contextual factors. These results suggest that, despite the exploitation of the ILSA tests, a better understanding of certain factors associated with academic performance and school effectiveness is necessary if we are to achieve an accurate portrait of education systems that facilitates international comparison and the improvement of education towards more effective schools that produce more competent and prepared students for the changing world in which we find ourselves.

Frade-Martínez, C., Olmos-Migueláñez, S., & Gamazo, A. (2021). Factors associated with the school performance of Spanish students: A study based on PISA 2018 data. Ninth International Conference on Technological Ecosystems for Enhancing Multiculturality (TEEM’21), 732-736. https://doi.org/10.1145/3486011.3486550 Gamazo, A., Martínez Abad, F., Olmos Migueláñez, S., & Rodríguez Conde, M. J. (2018). Evaluación de factores relacionados con la eficacia escolar en PISA 2015. Un análisis multinivel. Revista de educación, 379, 56-84. Gamazo, A., Olmos Migueláñez, S., & Martínez Abad, F. (2019). Factores asociados al rendimiento y a la eficacia escolar: Un estudio basado en métodos mixtos a partir de PISA 2015 [Universidad de Salamanca]. http://hdl.handle.net/10366/140406 Gómez y Jiménez, A. J. (2018). Desinforme Pisa. Publicaciones: Facultad de Educación y Humanidades del Campus de Melilla, 48(2), 279-299. Martínez-Abad, F., Lizasoain, L., Castro, M., & Joaristi, L. M. (2017). Selección de escuelas de alta y baja eficacia en Baja California (México). REDIE: Revista Electrónica de Investigación Educativa, 19(2), 38-53. Meunier, M. (2011). Immigration and student achievement: Evidence from Switzerland. Economics of Education Review, 30(1), 16-38. Ministerio de Educación y Formación Profesional. (2022). PISA. PISA. https://www.educacionyfp.gob.es/inee/evaluaciones-internacionales/pisa.html OECD. (2016). PISA 2015 Assessment and Analytical Framework: Science, Reading, Mathematic and Financial Literacy. Organisation for Economic Co-operation and Development. https://www.oecd-ilibrary.org/education/pisa-2015-assessment-and-analytical-framework_9789264255425-en Sanmartí i Puig, N., & Sardà Jorge, A. (2007). El caso PISA. Cuadernos de pedagogía, 370, 60-63. Schleicher, A. (2016). Desafíos para PISA. Relieve: Revista ELectrónica de Investigación y EValuación Educativa, 22(1), 19.