Exploring differences in Value Added measures of upper secondary education schools in Slovenia in light of contextual data

Author(s):

Gasper Cankar(presenting / submitting)Darko Zupanc(presenting)

Conference:

ECER 2016

Network:

11. Educational Improvement and Quality Assurance

Format:

Paper

Session Information

11 SES 08 A, Paper Session

Paper Session

Time:

2016-08-25

09:00-10:30

Room:

OB-E2.14

Chair:

Heidi Flavian

Contribution

Introduction

General description

This research builds on improvements in system wide feedback to schools that is happening in Slovenian upper secondary schools through an interactive software solution for giving back the information on students' achievement in schools (i.e. teacher's grade) and on external examinations of General and Vocational Matura. It is named Assessment of/for Learning Analytic Tool or shortly ALAT (Zupanc, Urank & Bren, 2009; Urank, Zupanc & Cankar, 2012). This software was in 2014 upgraded with data on population based national assessment results on students when they enter upper secondary schools and therefore allows calculation of value added data. National assessment in current form was established in 2006 which means there is Value Added (VA) data for upper secondary schools for past 5 years.

Software enables interactive analysis of the data for each school with national benchmarks, calculated along same selection criteria (Brejc, Sardoč & Zupanc, 2011).

In recent years a question of quality in education has gained prominence in Slovenia and this software is seen as one of providers of objective data for (self) evaluations of schools. But current implementation of software lacks contextual data. Most information on contextual factors in Slovenia currently comes from an international research projects like PISA and TIMSS. However, they rely on student questionnaires and such methods cannot be economically applied on whole population.

We explored the inclusion of contextual factors into ALAT before and we established that they cannot be neglected if we want to facilitate sound and meaningful interpretation. However, we didn’t explore any associations between SES and VA measures. Since some authors (e.g. Sanders, 1998) commenting on specific Value Added systems argue that inclusion of contextual factors is not necessary, we want to explore those associations on Slovenian data. As National Examinations Centre plans to implement similar software with similar value added cycle in primary/lower secondary schools inclusion of contextual information is even more prudent. While School Performance feedback Systems (SPFS) cannot provide evaluation, they can provide reliable and valid data for the purpose.

Our work was inspired through our cooperation in an international COMENIUS project (Improving Educational Effectiveness of Primary Schools), where University of Kragujevac in Serbia is working to establish a school performance feedback system (SPFS) with different partners from Belgium, Cyprus and Slovenia while all partners have also a chance to share experience and work on their own systems as well. We develop our own SPFS towards better information system that facilitates meaningful interpretation.

Objectives

We researched the range of VA measures and SES factor (as measured through PISA study and our own data) and more importantly the association between both measures within different types of upper secondary schools in Slovenia.

Our main research problems are:

- How big is variation in school’s VA measures over time

- How big is variation in school’s VA measures between compulsory subjects (Slovene language vs. Mathematics)

- Are there correlations between schools’ VA measures and SES data

Theoretical framework

Contextual factors of student's home and family background characteristics have well known effects on educational achievement. Since the Coleman’s report in 1966 (Hanushek, 2010) out of school determinants of students’ achievement have been extensively researched. Hattie (2008) in his synthesis of meta-analyses relating to achievement, reports the effect size associated with socioeconomic status (SES) of d=0.57. This fairly large effect size serves as a warning that any valid interpretations about students’ achievement should account for student’s background. This is also evident in research surrounding Programme for international student assessment (PISA), where contextual factors are routinely applied to research equity and quality of education (OECD, 2013).

Method

Methods For the purpose of this research we established a research contract with our national statistics agency (Statistical Office of the Republic of Slovenia - SORS), where we joined the data from external examinations and national assessments at the end of grades 6, 9 and 13 (NA6, NA9, General Matura, Vocational Matura) with the national register of households. Through this link we achieved to associate students with their parents living in same household and through further connections with databases of income (year 2011), real-estate (2011), working population data (2011) and general census data (2011) for all their parents. This research involved data from various sources: 1. National assessment results for Grade 9 populations of 2012 and 2013 – Slovene and Maths achievement – population data obtained from National Examinations Centre (NEC). 2. External examination results on General and Vocational Matura (NEC) 3. National register of households (SORS) 4. Population census 2011 (SORS) 5. Real estate register (ownership, estimated value) 2011 (SORS) 6. Register of working population 2013 (SORS) 7. Income tax register 2011 (SORS) Databases were linked to associate student with her/his parents and their yearly income, values of any real-estates owned, levels of education and occupations. From linked databases several indexes were constructed: HISEI – ISCO-08 codes of occupations were transformed using procedures, developed by Ganzeboom, De Graaf & Treiman (1992) into International Socio-Economic Index of occupational status (ISEI) and highest ISEI score of both parents was recorded into HISEI variable. PARED – from both parents, highest level of education was recorded INCOME – yearly gross income of both parents for year 2011 was summed. POSESS – sum of estimated values of all real-estates from both parents From those variables, SES index was constructed. Since all newly constructed variables tried to reflect similar variables found in PISA (OECD, 2012, PISA 2009, Technical Report: 312), we can compare construct validity of them with similar variables in PISA databases. For comparison, PISA2009 and PISA 2012 databases for Slovenia were used together with variables about achievement on national assessment gathered through national option in Student's questionnaire.

Expected Outcomes

Expected results Results from research will show the extent to which VA measure from National assessment to Matura results is usable as information for quality assurance purposes with regard to it’s variability over time and between different subjects within school. Wherever appropriate some solutions (such as rolling three year average) will be proposed. Any significant correlation between VA measures and SES would be a sign that perhaps the gap of achievement between high/low SES groups of students is widening. A method of multilevel analysis will help us understand the nature and possible reasons for such correlation. Results will also provide arguments to further facilitate f system wide implementation of contextual data within educational quality framework to provide contextual analyses to teachers, head teachers and experts in education.

References

Brejc, M., M. Sardoč & D. Zupanc (2011). Review on Evaluation and Assessment Frameworks for Improving School Outcomes: Country Background Report for Slovenia. National School for Leadership in Education in cooperation with the National Examinations Centre, http://www.oecd.org/education/school/48853911.pdf: 54–55. Hanushek, E. A. (2010). How well do we understand achievement gaps? Focus, 27(2), 5–12. Hattie, J. (2008). Visible learning : a synthesis of meta-analyses relating to achievement. London : New York: Routledge. Ganzeboom, H.B.G. De Graaf, P.M. & Treiman, D.J. (1992): A Standard International Socio-Economic Index of Occupational Status. Social Science Research 21 (1), 1-56. OECD, (2012). PISA 2009 Technical Report, PISA, OECD Publishing http://dx.doi.org/10.1787/9789264167872-en OECD (2013), PISA 2012 Results: Excellence Through Equity: Giving Every Student the Chance to Succeed (Volume II), PISA, OECD Publishing. http://dx.doi.org/10.1787/9789264201132-en Sanders, W. L. (1998). Value-added assessment. School Administrator, 55, 24–29. Zupanc, D, Urank, M. & Bren, M. (2009). Variability analysis for effectiveness and improvement in classrooms and schools in upper secondary education in Slovenia: assessment of/for learning analytic tool. School Effectiveness and School Improvement, 20(1): 89–122. Urank, M., Zupanc, D. & Cankar G. (2012). Orodje za analizo izkazanega znanja ob zaključku srednje šole. [Assessment of/for Learning Analytic Tool]

Author Information

Gasper Cankar (presenting / submitting)

National Examinations Centre

R&D

Ljubljana

Darko Zupanc (presenting)