Improvements in system wide feedback to schools are both internationally important and country specific. Since 2007 Slovenia has implemented an interactive software solution for giving upper secondary schools back the information on students' achievement in schools (i.e. teacher's grade) and on external examinations of General and Vocational Matura. It is named Assessment of/for Learning Analytic Tool or shortly ALAT (Zupanc, Urank & Bren, 2009; Urank, Zupanc & Cankar, 2012).

Software enables interactive analysis of the data for each school with national benchmarks, calculated along same selection criteria (Brejc, Sardoč & Zupanc, 2011). In 2014 software was upgraded with new features: per item analysis for criterion-referenced interpretation and value added analysis.

All this new features were welcome but they have also stressed the necessity to include contextual factors into the analyses of achievement. Most information on contextual factors in Slovenia currently comes from an international research projects like PISA and TIMSS. However, they rely on student questionnaires and such methods cannot be economically applied on whole population.

This research is a foundation for inclusion of contextual factors into ALAT. As we plan to implement similar system in primary/lower secondary schools inclusion of contextual information is even more prudent. While SPFS cannot provide evaluation, they can provide reliable and valid data for the purpose.

Our work was inspired also through our cooperation in an international COMENIUS project (Improving Educational Effectiveness of Primary Schools), where University of Kragujevac in Serbia is working to establish a school performance feedback system (SPFS) with different partners from Belgium, Cyprus and Slovenia while all partners have also a chance to share experience and work on their own systems as well.

Objectives

We researched the validity and applicability of national administrative data for self-evaluation, educational improvement and quality assurance purposes that could be achieved through the ALA Tool.

Our main research problem was: Is it feasible to construct contextual variables from administrative data already in the system? Although research on background factors is very important, the task of data collection shouldn't burden the system in whole or the individuals in particular more than necessary. The extraction of variables from administrative sources would therefore be beneficial as it would provide data without any trouble to individuals involved. This would also facilitate research on effects of background factors on students' achievement in Slovenia and give example of good practice internationally.

Theoretical framework

Contextual factors of student's home and family background characteristics have well known effects on educational achievement. Since the Coleman’s report in 1966 (Hanushek, 2010) out of school determinants of students’ achievement have been extensively researched.   Hattie (2008) in his synthesis of meta-analyses relating to achievement, reports the effect size associated with socioeconomic status (SES) of d=0.57. This fairly large effect size serves as a warning that any valid interpretations about students’ achievement should account for student’s background. This is also evident in research surrounding Programme for international student assessment (PISA), where contextual factors are routinely applied to research equity and quality of education (OECD, 2013).

Our first research question was: Can we construct usable and valid indicator of SES or/and its main theoretical domains (education, occupation, wealth of parents) from administrative data? Such indicators should demonstrate construct validity through descriptive statistics, correlation patterns with other constructs and low proportion of missing data.

Second research question was: How much of the variance in achievement can be explained by including this newly constructed SES index? We would calculate this with multilevel modelling techniques and in the process examine the appropriateness of a two level (students within schools) and three level (students within classes within schools) hierarchical models.

Brejc, M., M. Sardoč & D. Zupanc (2011). Review on Evaluation and Assessment Frameworks for Improving School Outcomes: Country Background Report for Slovenia. National School for Leadership in Education in cooperation with the National Examinations Centre, http://www.oecd.org/education/school/48853911.pdf: 54–55. Hanushek, E. A. (2010). How well do we understand achievement gaps? Focus, 27(2), 5–12. Hattie, J. (2008). Visible learning : a synthesis of meta-analyses relating to achievement. London : New York: Routledge. Ganzeboom, H.B.G. De Graaf, P.M. & Treiman, D.J. (1992): A Standard International Socio-Economic Index of Occupational Status. Social Science Research 21 (1), 1-56. OECD, (2012). PISA 2009 Technical Report, PISA, OECD Publishing http://dx.doi.org/10.1787/9789264167872-en OECD (2013), PISA 2012 Results: Excellence Through Equity: Giving Every Student the Chance to Succeed (Volume II), PISA, OECD Publishing. http://dx.doi.org/10.1787/9789264201132-en Zupanc, D, Urank, M. & Bren, M. (2009). Variability analysis for effectiveness and improvement in classrooms and schools in upper secondary education in Slovenia: assessment of/for learning analytic tool. School Effectiveness and School Improvement, 20(1): 89–122. Urank, M., Zupanc, D. & Cankar G. (2012). Orodje za analizo izkazanega znanja ob zaključku srednje šole. [Assessment of/for Learning Analytic Tool]