This paper presents findings from a UK University Widening Participation funded project “Exploring value added measures for undergraduate students: the impact of admissions criteria (prior attainment) and other relevant student background, academic department and contextual factors on the progress and retention of disadvantaged and other students”. The objective is to employ multilevel modelling and large-scale administration and survey data to investigate the nature and extent of student progress in UK universities. The specific research questions are:

(1) What is the range and extent of undergraduate student progress within and across academic departments/subjects and UK universities? What student background, previous school, departmental, contextual/peer group or other factors may explain differences in student progress across departments/subjects and universities?  What are the university institutional effects and associated time trends when comparing disadvantaged, non UK and other students?

(2) What methodological tools can be developed to assist Higher Education (HE) Institutions and schools in monitoring and evaluating the impact of university Widening Participation policies, especially with regard to the reality of progress achieved by disadvantaged and other students with differing entrance qualifications?  What are the limitations of any possible methods identified?

The research draws on theories and paradigms developed within the fields of educational effectiveness and improvement research, including methods to evaluate students “value added” progress.  However, these powerful techniques have rarely been used in relation to higher education institutions and the academic progress of HE students in spite of recognition of the potential relevance of these approaches over 20+ years (Rodgers, 2005; Tam, 2001; Gallagher, 1991; McGeevor et al. 1990; Foley & Goldstein, 2012) and the need to better understand links between school performance and subsequent achievement at HE level (Gittoes & Thompson, 2007, Crawford, 2014, Chowdry et al., 2013).  Interestingly, it is suggested that some disadvantaged students may perform below their potential at A-level due to poor schooling quality and consequently may not be offered a university place in spite of the reality that they do have the ability to achieve very well at HE. This study seeks to examine this issue and in particular explore links between previous school performance and student progress at HE, for disadvantaged students to comparison to other students. Overall the research seeks to enhance understanding of the barriers to widening participation, retention and success at HE level and to examine the potential consequences of previous school attainment and context.

Value added measures of educational effectiveness have been widely used at the level of statutory schooling to examine the relative progress of students during their time at school and this method is adapted to examine the progress of HE students in UK universities. Previous studies have utilised longitudinal datasets, including students’ individual matched data records from different time points, and multilevel modelling statistical techniques to produce an estimate of the extra value that is added by schools and subject departments to student attainment over and above the progress or improvement that might normally be expected. The methodology involves comparing different models to separate out the effect of the school/department experience on individual student outcomes (what students achieve) and the extent to which student intake characteristics (such as their prior attainment, gender, ethnicity and social class) affect student outcomes (Goldstein et al 1993; Thomas & Mortimore, 1996; Scheerens et al, 2003, Thomas et al, 2007). Crucially these measures can be used to examine the progress of specific groups of students such as those from low income families, facilitating the targeting of additional support for disadvantaged students where progress may be less than expected (Thomas et al 1997; 2007). In this research quantitative data from administrative and student surveys collated by the UK Higher Education Funding Council for England (HEFCE) and Department for Education (DFE) are utilised in a variety of multilevel modelling analyses. The data comprises undergraduate degree outcomes for 948,776 full-time students’ in 159 universities who have completed a 3 or 4 year undergraduate programme over five consecutive cohorts (2011-15) matched to their prior attainment scores on entry to university (UCAS score, which creates a combined score from various eligible HE entry qualifications such as GCE Advanced level as well as GCSE scores (at age 16 years) and Key Stage 2 scores (at age 11 years)) and other student background, school and department data.

The findings from a variety of different cross classified and standard MLwin models indicate that there are substantial and statistically significant differences between the estimates of student outcomes and progress across academic departments/subjects and UK universities and these differences vary over time. The percentage of total variance in student’s unadjusted degree outcome scores attributable to differences between departments/subjects and universities is reduced after controlling for students previous attainment on entry to university, and other student background factors. The estimated impact of a variety of student admission criteria (prior attainment) and background factors (gender, age, SES, ethnicity, parents attended HE, disability) as well as University differences in learning outcomes for disadvantaged and other students are reported. Time trends over five consecutive cohorts (2011-2015) are also examined. The findings indicate systematic patterns for specific student groups (eg parents attended HE, ethnic minorities and disabled students) and this evidence will directly inform and support evidence-based WP policy and practice and self-evaluation at the institutional level and nationally. For example, in terms of recommending different intake requirements, recruitment and department/academic support and improvement strategies, especially for disadvantaged students. The findings are discussed in relation to the limitations and relevance of extending school effectiveness research methods to the HE context and to evaluate value added and learning outcomes of HE students. The implications of the findings are discussed in terms of higher educational policy and widening participation in UK and internationally, a key theme of ECER network 22.

Chowdry, H. et al., 2013. Widening participation in higher education: analysis using linked administrative data. Journal of the Royal Statistical Society, 176(2), pp.431–457. Crawford, Claire (2014) The link between secondary school characteristics and university participation and outcomes, CAYT Research Report. June 2014. DFE. Accessed 22/7/2015 https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/317276/RR353_-_The_link_between_secondary_school_characteristics_and_university_participation_and_outcomes_FINAL.pdf Foley, B. & Goldstein, H (2012) Measuring Success: League tables in the public sector. A British Academy Policy Centre report. Gallagher, A. (1991). Comparative value added as a performance indicator. Higher Education Review, 23(3), 19–29. Gittoes, M. & Thompson, J. (2007) Admissions to higher education: are there biases against or in favour of ethnic minorities? Teaching in Higher Education. 12:3: Goldstein, H, Rasbash, J, Yang, M, Woodhouse, G, Pan, H, Nuttall, D & Thomas, S (1993) A Multilevel Analysis of School Examination Results, Oxford Review of Education, 19, (4): 425-433. McGeevor, P. et al. (1990) The measurement of value added in higher education: a joint PCFC/CNAA project report. London: CNAA. Rodgers, T. (2005) Measuring value added in higher education: Do any of the recent experiences in secondary education in the United Kingdom suggest a way forward? Quality Assurance in Education. Volume: 25, Issue: 4, Pages: 469-106 Scheerens, J, Glas, C & Thomas, S (2003) Educational Evaluation, Assessment and Monitoring: A Systemic Approach. Swets & Zeitlinger: Lisse, Abingdon, Exton (PA), Tokyo. Tam, M. (2001) Measuring Quality and Performance in Higher Education. Quality in Higher Education. Volume 7:1: 47 – 54. Thomas, S & Mortimore, P (1996) Comparison of Value Added Models for Secondary School Effectiveness, Research Papers in Education, 11, (1): 5-33. Thomas, S, Sammons, P, Mortimore, P & Smees, R (1997b) Differential Secondary School Effectiveness: Examining the size, extent and consistency of school and departmental effects on GCSE outcomes for different groups of students over three years, British Educational Research Journal, 23, (4): 451-469. Thomas, SM, Peng, W-J, Gray, J. (2007) Value added trends in English secondary school performance over ten years. Oxford Review of Education, 33 (3: 261 - 295).