Upgrade of Slovenian School performance feedback system (SPFS) with individual’s contextual data – feasibility study

Author(s):

Gašper Cankar(presenting / submitting)Darko Zupanc(presenting)

Conference:

ECER 2015

Network:

11. Educational Improvement and Quality Assurance

Format:

Paper

Session Information

11 SES 08 A, Educational Improvement: National Level (Part 1)

Paper Session to be continued in 11 SES 09 A

Time:

2015-09-10

09:00-10:30

Room:

102.Oktatóterem [C]

Chair:

Lynne Grant-McMahon

Contribution

Improvements in system wide feedback to schools are both internationally important and country specific. Since 2007 Slovenia has implemented an interactive software solution for giving upper secondary schools back the information on students' achievement in schools (i.e. teacher's grade) and on external examinations of General and Vocational Matura. It is named Assessment of/for Learning Analytic Tool or shortly ALAT (Zupanc, Urank & Bren, 2009; Urank, Zupanc & Cankar, 2012).

Software enables interactive analysis of the data for each school with national benchmarks, calculated along same selection criteria (Brejc, Sardoč & Zupanc, 2011). In 2014 software was upgraded with new features: per item analysis for criterion-referenced interpretation and value added analysis.

All this new features were welcome but they have also stressed the necessity to include contextual factors into the analyses of achievement. Most information on contextual factors in Slovenia currently comes from an international research projects like PISA and TIMSS. However, they rely on student questionnaires and such methods cannot be economically applied on whole population.

This research is a foundation for inclusion of contextual factors into ALAT. As we plan to implement similar system in primary/lower secondary schools inclusion of contextual information is even more prudent. While SPFS cannot provide evaluation, they can provide reliable and valid data for the purpose.

Our work was inspired also through our cooperation in an international COMENIUS project (Improving Educational Effectiveness of Primary Schools), where University of Kragujevac in Serbia is working to establish a school performance feedback system (SPFS) with different partners from Belgium, Cyprus and Slovenia while all partners have also a chance to share experience and work on their own systems as well.

Objectives

We researched the validity and applicability of national administrative data for self-evaluation, educational improvement and quality assurance purposes that could be achieved through the ALA Tool.

Our main research problem was: Is it feasible to construct contextual variables from administrative data already in the system? Although research on background factors is very important, the task of data collection shouldn't burden the system in whole or the individuals in particular more than necessary. The extraction of variables from administrative sources would therefore be beneficial as it would provide data without any trouble to individuals involved. This would also facilitate research on effects of background factors on students' achievement in Slovenia and give example of good practice internationally.

Theoretical framework

Contextual factors of student's home and family background characteristics have well known effects on educational achievement. Since the Coleman’s report in 1966 (Hanushek, 2010) out of school determinants of students’ achievement have been extensively researched. Hattie (2008) in his synthesis of meta-analyses relating to achievement, reports the effect size associated with socioeconomic status (SES) of d=0.57. This fairly large effect size serves as a warning that any valid interpretations about students’ achievement should account for student’s background. This is also evident in research surrounding Programme for international student assessment (PISA), where contextual factors are routinely applied to research equity and quality of education (OECD, 2013).

Our first research question was: Can we construct usable and valid indicator of SES or/and its main theoretical domains (education, occupation, wealth of parents) from administrative data? Such indicators should demonstrate construct validity through descriptive statistics, correlation patterns with other constructs and low proportion of missing data.

Second research question was: How much of the variance in achievement can be explained by including this newly constructed SES index? We would calculate this with multilevel modelling techniques and in the process examine the appropriateness of a two level (students within schools) and three level (students within classes within schools) hierarchical models.

Method

For the purpose of this research we established a research contract with our national statistics agency (Statistical Office of the Republic of Slovenia - SORS), where we joined the data from external examinations and national assessments at the end of grades 6, 9 and 13 (NA6, NA9, General Matura, Vocational Matura) with the national register of households. Through this link we achieved to associate students with their parents living in same household and through further connections with databases of income (year 2011), real-estate (2011), working population data (2011) and general census data (2011) for all their parents. This research involved data from various sources: 1. National assessment results for Grade 9 populations of 2012 and 2013 – Slovene and Maths achievement – population data obtained from National Examinations Centre (NEC). 2. External examination results on General and Vocational Matura (NEC) 3. National register of households (SORS) 4. Population census 2011 (SORS) 5. Real estate register (ownership, estimated value) 2011 (SORS) 6. Register of working population 2013 (SORS) 7. Income tax register 2011 (SORS) Databases were linked to associate student with her/his parents and their yearly income, values of any real-estates owned, levels of education and occupations. From linked databases several indexes were constructed: HISEI – ISCO-08 codes of occupations were transformed using procedures, developed by Ganzeboom, De Graaf & Treiman (1992) into International Socio-Economic Index of occupational status (ISEI) and highest ISEI score of both parents was recorded into HISEI variable. PARED – from both parents, highest level of education was recorded INCOME – yearly gross income of both parents for year 2011 was summed. POSESS – sum of estimated values of all real-estates from both parents From those variables, SES index was constructed. Since all newly constructed variables tried to reflect similar variables found in PISA (OECD, 2012, PISA 2009, Technical Report: 312), we can compare construct validity of them with similar variables in PISA databases. For comparison, PISA2009 and PISA 2012 databases for Slovenia were used together with variables about achievement on national assessment gathered through national option in Student's questionnaire.

Expected Outcomes

Results from research show that it is possible to link relatively large number of administrative records successfully and construct different variables about students’ background characteristics and join them in an ESCS index. This could be done on large scale with relatively small costs. Correlations between constructed background variables are similar to those that we can replicate on parallel variables in PISA 2009 and PISA 2012 datasets. This is an evidence of good construct validity. When national assessment results from Grades 6 and 9 in primary/lower secondary schools were analysed with multilevel modelling with three level model, only 0.1%-1.4% of variance was on the level of class, which makes the level of class indistinct and for later analysis only two level models were used. In upper secondary schools level of class would account for about 6%-8 % when the achievement was from General Matura and 9%-17% when the external assessment was Vocational Matura. On the level of upper secondary schools three level model was retained. Paper will present some more detailed results regarding constructed background variables, their effects on achievement and implications for Slovenian education. After thorough research evidence about feasibility and validity of new ESCS index, constructed from national administrative databases, comes the task of system wide implementation that requires consent of stakeholders, appropriate changes to legislation and upgrades to ALAT to provide contextual analyses to teachers, head teachers and experts in education.

References

Brejc, M., M. Sardoč & D. Zupanc (2011). Review on Evaluation and Assessment Frameworks for Improving School Outcomes: Country Background Report for Slovenia. National School for Leadership in Education in cooperation with the National Examinations Centre, http://www.oecd.org/education/school/48853911.pdf: 54–55. Hanushek, E. A. (2010). How well do we understand achievement gaps? Focus, 27(2), 5–12. Hattie, J. (2008). Visible learning : a synthesis of meta-analyses relating to achievement. London : New York: Routledge. Ganzeboom, H.B.G. De Graaf, P.M. & Treiman, D.J. (1992): A Standard International Socio-Economic Index of Occupational Status. Social Science Research 21 (1), 1-56. OECD, (2012). PISA 2009 Technical Report, PISA, OECD Publishing http://dx.doi.org/10.1787/9789264167872-en OECD (2013), PISA 2012 Results: Excellence Through Equity: Giving Every Student the Chance to Succeed (Volume II), PISA, OECD Publishing. http://dx.doi.org/10.1787/9789264201132-en Zupanc, D, Urank, M. & Bren, M. (2009). Variability analysis for effectiveness and improvement in classrooms and schools in upper secondary education in Slovenia: assessment of/for learning analytic tool. School Effectiveness and School Improvement, 20(1): 89–122. Urank, M., Zupanc, D. & Cankar G. (2012). Orodje za analizo izkazanega znanja ob zaključku srednje šole. [Assessment of/for Learning Analytic Tool]

Author Information

Gašper Cankar (presenting / submitting)

National Examinations Centre

Research & Development

Ljubljana

Darko Zupanc (presenting)