Excellence for Research in Australia from an Agency Perspective: Who Benefits and How in Research Quality Assessment?

Author(s):

Carmel DIezmann(presenting / submitting)James Watters

Conference:

ECER 2015

Network:

22. Research in Higher Education

Format:

Paper

Session Information

22 SES 10 E, Quality Assessment and Evaluation

Paper Session

Time:

2015-09-10

15:30-17:00

Room:

395. [Main]

Chair:

Patrick Baughan

Contribution

The Australian research quality assessment, Excellence in Research for Australia (ERA), commenced in 2010, was repeated in 2012, and will run again in 2015 (Australian Research Council (ARC), 2014). In brief, each discipline submission includes an Explanatory Statement, six years of publication data with 30% identified for review, and three years of grant income data. The purpose of ERA is “to progressively increase the number of research groups performing at world-class levels, as measured by international performance benchmarks” (Commonwealth of Australia, 2009, p. 33). The research quality score for each group discipline submission rated within the range of 5 to 1 with 5 and 1 being ‘well above’ and ‘well below’ world standard respectively. Elsewhere, research assessments have long been part of the higher education sector (Research Assessment Exercise, 2008).

Based on ERA results, there is considerable room for research improvement for some university discipline groups because only “40 per cent (sic) of Australian university research groups were rated as ‘above world standard” in ERA 2010 (Norton, 2013, p. 8). As a discipline, Education fared poorly in ERA 2010 and 2012. For example, in ERA 2012, 42 of the 85 units of evaluation (submissions of a discipline group) were assessed at ‘below’ or ‘well below’ world standard (ARC, 2012a, 2012b). The research questions are:

Which university discipline groups in Education require support to build research capacity?
How can ERA data be used to support the improvement of research performance in Education?

The ERA process uses the discipline norms for quality assessment. For example, Education is assessed by peer review (e.g., Ramsden, 1998) whereas science disciplines use citation data. In peer review disciplines, such as Education, multiple reviews of the selected ERA research outputs are sought from peer reviewers by discipline panels of the ERA Research Evaluation Committee (REC). These discipline panels then synthesise the reviews and evaluate the overall discipline group submissions relative to international benchmarks.

Research performance across Australian universities differs substantially across universities and at the discipline level (e.g., ARC, 2012b). This variation is not surprising because there are three broad phases of institutional research development (Bosch &Taylor, 2011). At the Instilling Phase, there is little or no research activity in the university. At the Broadening Phase, some discipline groups are performing highly and are recognised externally. At the Honing Phase, research is embedded in institutional life and high performance is part of the institutional culture.

Universities Australia, the peak member body of Australian universities, strongly supports the government’s agenda to improve research performance (Universities Australia, 2014). This accord between the government and Australian universities reflects the relationship of agency in which each assumes particular roles (Kivistö, 2008). The government acts as the principal in the relationship and delegates work in universities as its agents. There are multiple principal-agent relationships at work because each university has a performance agreement with the government (ARC, 2012b). Paralleling the relationship between the government, as principal, and the university, as agent, is the relationship between the ARC and academics. The ARC assumes the role of a principal when it delegates assessment and review work to academics in paid (i.e., REC) or unpaid capacities (i.e., peer reviewers). Despite the work undertaken by academics as reviewers or REC members, universities receive no feedback on the reviews nor are they privy to the rationale for decision making about the discipline group ratings. Instead, a national report is published by university and discipline codes together with benchmark data across the sector (e.g., ARC, 2012b). For more about ERA and educational research see Seddon, Bennett, Bennett, Bobis, Chan, Harrison, and Shore (2013).

Method

The research involved a document analysis of the ERA 2012 report (ARC, 2012b) within an agency framework. It had two foci. First, it focused on identifying which university discipline groups in Education require support to build research capacity and why. In the agency relationship, there is an assumption is informational asymmetry, a situation where the principal as delegator is less knowledgeable about the details of the tasks being undertaken than the agent as delegatee (Kivistö, 2008). Such asymmetry indicates that the agent has sufficient knowledge to undertake a particular task, that is, for ERA, to produce high quality research. If the assumption of asymmetry does not hold, the agent is vulnerable because they lack the requisite knowledge and skills expected by the principal. Similar to Dawson, Watson and Boudreau (2010), we produced a typology of information symmetry to examine this assumption. The four quadrants represent the four possible relationships between the principal and agents (university discipline groups) in terms of knowledge. In ERA, the principal is knowledgeable about what is required, that is world standard research (e.g., ARC, 2012a). Hence, only Quadrants 1 and 3 were of relevance. The results for these quadrants are presented shortly. Second, we examined how ERA data can be used to support the improvement of research performance in Education. Following the identification of what ERA data is and is not accessible to universities and researchers, the agency framework was used to examine the respective roles of the principal (ARC) and its agents (academics) and the reciprocity within their relationship. Often, there is a paid relationship between a principal and its agents. However, without a paid relationship, investigating reciprocity should be revelatory in terms of expected workload and anticipated mutual benefit.

Expected Outcomes

1. How can low performing research groups in Education build research capacity? Quadrant 3 revealed that some agents are unable to achieve the principal’s goal of world standard research performance. In Education in 2012, 21/37 university groups were in Quadrant 3 because they rated as ‘below’ or ‘well below world standard’ (ARC, 2012). A plausible explanation for why universities are classified as low performers, mixed performers (high and low discipline groups within a university), and high performers in ERA is that they are at the Instilling, Broadening or Honing Phases of research development respectively (Bosch & Taylor, 2011). Hence, to fulfil the relationship of agency in which government expects world class research performance, a university needs to at least be in the Broadening Phase where its high discipline performers support its low Education performers. This strategy might need to be supplemented with the strategic employment of high performing research staff in Education from elsewhere. 2. How can ERA data be used to support the improvement of research performance in Education? University access to ERA data is restricted to the Report (e.g., ARC, 2012b). Although the ARC introduced an open access policy that applies to all new ARC funding (ARC, 2013), it does not provide reciprocal open access to ERA data. Peer review feedback is provided on ARC grants. Hence, similarly the ARC could provide peer review feedback on selected ERA publications. Access to feedback on their ERA submission would provide guidance to Education groups aspiring to achieve a world standard rating. (Space precludes the inclusion of the ERA data that could be released; however, this will form part of presentation.) Further, the volume of information generated by experts through ERA provides opportunities to gain insights into quality publication in Education via a ‘big data’ set (Mayer-Schönberger & Cukier, 2013).

References

Australian Research Council (2012a). ERA 2012 evaluation handbook. Retrieved from http://www.arc.gov.au/pdf/era12/ERA%202012%20Evaluation%20Handbook_final%20for%20web_protected.pdf Australian Research Council (2012b). Excellence in Research for Australia 2012: National report. Retrieved from http://www.arc.gov.au/era/era_2012/outcomes_2012.htm Australian Research Council (2013). ARC Open Access Policy. Retrieved from http://www.arc.gov.au/applicants/open_access.htm Australian Research Council (2014). About ARC. http://www.arc.gov.au/about_arc/ Bosch, A. & Taylor, J. (2011). A proposed framework of institutional development phases. Journal of Higher Education Policy and Management, 33(45), 443-457. Commonwealth of Australia (2009). Powering ideas: An innovation agenda for the 21st century. Retrieved from http://sydney.edu.au/documents/about/higher_education/2012/20120308%20PoweringIdeas.pdf Kivistö, J. (2008). An assessment of agency theory as a framework for the government–university relationship. Journal of Higher Education Policy and Management, 30(4), 339-350. Mayer-Schönberger, V. & Cukier, K. (2013). Big data: A revolution that will transform how we live, work, and think. London: John Murray. Norton, A. (2013). Mapping Australian higher education, 2013 version, Grattan Institute. Retrieved from http://grattan.edu.au/static/files/assets/28a92f8b/184_2013_mapping_higher_education.pdf Ramsden, P. (1998). Learning to lead in higher education. London: Routledge Research Assessment Exercise (2008). RAE 2008 Submissions: UOA 45 Education. Retrieved from http://www.rae.ac.uk/submissions/submissions.aspx?id=45&type=uoa Seddon, T., Bennett, D., Bennett, S., Bobis, J., Chan, P., Harrison, N., & Shore, S. (2013). Educational research Australia: A changing ecology of knowledge and practice. Australian Educational Researcher, 40(4), 433-451. DOI 10.1007/s13384-013-0104-1 Universities Australia (2014). The voice of Australia’s universities. Retrieved from https://www.universitiesaustralia.edu.au/research-innovation-workforce

Author Information

Carmel DIezmann (presenting / submitting)

Australian Catholic University, Australia

James Watters

Queensland University of Technology, Australia

Search the ECER Programme

Search for keywords and phrases in "Text Search"
Restrict in which part of the abstracts to search in "Where to search"
Search for authors and in the respective field.
For planning your conference attendance you may want to use the conference app, which will be issued some weeks before the conference
If you are a session chair, best look up your chairing duties in the conference system (Conftool) or the app.