The Relevance Survey (RS) as Strategic Research Tool: Lessons From a Norwegian HEI Linking Quality Assurance to Quality Improvement Processes
Author(s):
Tone Cecilie Carlsten (presenting / submitting) Ingunn Hybertsen Lyso (presenting)
Conference:
ECER 2014
Network:
22. Research in Higher Education
Format:
Paper

Session Information

22 SES 06 B, Policy, Management and Governance in Higher Education

Paper Session

Time:
2014-09-03
15:30-17:00
Room:
B021 Anfiteatro
Chair:
Liudvika Leisyte

Contribution

Quality assurance is one of the areas where European cooperation has advanced significantly during the last decade. The European Commission recently published a report on issues related to quality and quality assurance in higher education stating that progress is reported in most areas. However, linking quality assurance to concrete quality improvement processes and strategic work of institutions is still considered to remain a main challenge (European Commission 2014). Similarly, in the 2013 Eurydice report on Education and Training in Europe 2020 member states emphasized the need for a renewed focus on strategic connections between educational relevance and quality in a quality assurance system. In fact, this aspect of higher education strategic competence management is highlighted as one of four areas relevant for the Education 2020 strategy (European Commission/EACEA/Eurydice 2013).

In this paper we present a survey for studying the relevance of leadership study programs at the Royal Norwegian Air Force Academy. The survey aims to meet the challenge described in the European context of higher education assurance and enhancement as a strategic institutional tool. The Royal Norwegian Air Force Academy is a highly prestigious institution accredited as a regular higher education institution since 2006. It has since 1949 educated leaders employed in a variety of sectors in Norway and internationally (Royal Norwegian Air Force Academy 2013). Its study programs respond to employability demands of both military and civilian competences needed to solve complex problems (Ministry of Defence 2011; 2012a; 2012b, 2013, Henriksen 2011). Responding to evaluation procedures and the common regulations of the Norwegian Agency for Quality Assurance in Education (NOKUT), the indicators for the Relevance Survey (RS) was developed in 2014 as a strategic institutional tool to study and evaluate the relevance of the Academy's study programs (cf. NOKUT 2008). The RS is based on the requirements in the Norwegian Qualification Framework, and is designed to be a strategic research tool for the Academy's internal quality assurance system meeting national quality standards.

The aim of this paper is not only to present the RS as a strategic tool, but to discuss its role as an institutional steering tool in light of the current move towards European expectations for linking quality assurance to quality improvement processes in systematic ways. The research question for our study based on a theory-based stakeholder evaluation model is:

How do national and allied institutions experience and assess the political, military-strategic, operational and tactical relevance of the Academy's study programs, and how may this information be collected systematically and strategically in order to improve teaching and learning at the Academy?

We link this research question to ongoing policy discussions at the European level. In higher education policy the concept of relevance has connotations to both employability and academic quality (European Commission/EACEA/Eurydice 2013). In higher education research the concept is considered elusive (Hovdhaugen & Carlsten 2013). In this study we frame the concept of relevance within an understanding occupational relevance. As such, it addresses the link between demands of academic quality and the interests of external stakeholders.

Theoretically, we approach the study with a descriptive and a functional analysis. The aim is to map the relations between curricular coherency and occupational competences. Our study rests on a cogent tradition of curricular administration theory and theory of organizational didactics (cf. Torgersen & Steiro 2009, Kvernbekk 2008). The analytical model is designed to address tensions between strategic competences related to academic excellence and operational competences related to employability (cf. Carlsten & Aamodt 2013). In the paper we discuss the model's validity in the field of higher education in general, and in the field of European policy research in particular. 

Method

Our study rests on empirical results from a qualitative pilot study for the Royal Norwegian Air Force Academy Fall 2013 - Spring 2014. The methodological approach is based on a theory-based stakeholder evaluation model. This model is developed to handle complex considerations in modern public administration (Hansen & Vedung 2010). This approach is aligned to the research question where we include experiences and assessments from national and allied institutions. The theory-based stakeholder model aims to describe a responsive relationship between curricular developments and competence demands moving beyond some of the dominant goal-oriented frameworks in research on higher education (e.g. Guba & Lincoln 1989, Vedung 2003, Hansen & Vedung 2010). Building on earlier insights in the field of higher education evaluations, our approach takes into account that theoretical and academic decisions not always function in practice in a concrete work situation (Watkins, Lysø & deMarrais 2011, Carlsten 2013). As mentioned, linking quality assurance to contextualized quality improvement processes is also deemed crucial in future higher education policy (European Commission/EACEA/Eurydice 2013). Data are collected from policy documents, reports, interviews and pilot version of the survey. Documents include Norwegian white papers and other national and international strategic government documents for the defence sector and the sector of higher education. Strategic documents from the Norwegian Agency for Quality Assurance in Education (NOKUT) as well as strategic documents and curricula from the Royal Norwegian Air Force Academy are naturally included along with reports from International Operations (INT-OPS) where Norwegian air force officers have participated during the last decade. We include semi-structured interviews with 12 informants from the Academy, the Ministry of Defence, the North Atlantic Treaty Organization (NATO), Air Operational Inspectorate/Ground Operations, the Air Force Inspectorate for Education, the Norwegian Defence University College and other relevant military career and civilian workplaces. The interview guide is based on the document analysis, and aligned with tools from similar studies we have conducted mapping both competence needs and competence strategies (e.g. Solberg et al, forthcoming). In the paper we include a brief discussion of indicators as these are considered crucial in the operationalization of the concept of relevance in higher education research. The guide is quality assured by representatives from the Air Force Academy, as well as from an expert group to our project. The preliminary analyses will serve as a foundation for further developing and implementing the RS-tool on a regular basis from Spring 2015.

Expected Outcomes

The notion of relevance in modern educational administration signals a crucial attention to the relationship between curricular coherency and competence demands, Still, this aspect of policy and practice is currently assessed as a main challenge in European higher education (European Commission 2014, European Commission/EACEA/Eurydice 2013). We present and discuss a case addressing ways of studying these new policy directions. The Relevance Survey (RS) for the Royal Norwegian Air Force Academy is an illustrative example of how a Norwegian higher education institution responds to new demands for strategic institutional work. Our analysis show how a relevance survey (RS) is designed and used as a strategic research tool. We also expect to indicate which aspects of curricular decisions that are crucial in a leadership program striving for excellence. Examining the relationship between curricula and demands for workplace competences for cadets of the Academy, we expect to map the field of strategic and operational competences in a way that has relevance for higher education research and development in general. Indicators developed in our pilot survey will illustrate some of the current dilemmas and opportunities institutions that aim to address both quality assurance and quality improvement face today. When it comes to novel contributions, this study is particularly relevant for the study of comparative international leadership programs including Air Force Academy programs educating future leaders. This is an area that represents a knowledge gap in higher education research. The introduction of a Relevance Survey (RS) at one of Norway's prestige higher education institutions to combine demands of educational relevance and quality in Norway may in fact be understood as part of the ongoing international discourse on how to ensure high academic quality and improved employability. We include a discussion of international scholarly relevance in our concluding remarks.

References

Carlsten, Tone Cecilie (2013). Student mellom to læringsarenaer. I: Kårstein, Asbjørn & Carlsten, Tone Cecilie: Evaluering av Internship-ordningen ved Universitetet i Tromsø. NIFU-rapport 26-2013. Carlsten, Tone Cecilie & Aamodt, Per Olaf (2013). Evaluering av etablering av ordning med Senter for fremragende utdanning (SFU). En kvalitativ analyse av aktørenes erfaringer og vurderinger. Oslo: NIFU-rapport 10/2013. European Commission (2014). Report from the Commission to the European Parliament, the Council, the European Economic and Social Committee and Committee of the Regions. Report on Progress in Quality Assurance in Higher Education. Brussels, 28.1.2014. European Commission/EACEA/Eurydice (2013). Education and Training in Europe 2020: Responses from the EU Member States. Eurydice Report. Brussels: Eurydice. Guba, E.G & Lincoln, Y. S. (1989). Fourth Generation Evaluation. Newbury Park. CA: SAGE. Hansen, Morten Balle & Vedung, Evert (2010). Theory-based stakeholder evaluation. In: American Journal of Evaluation 31(3) 295-313. Henriksen, Dag (2011) Airpower: The Need for More Analytical Warriors. In: Haug, Karl Erik and Maaø, Ole Jørgen (Eds.) Conceptualizing Modern War. London: Hurst & Company. Hovdhaugen, Elisabeth & Carlsten, Tone Cecilie (2013). Studiegjennomstrømning og studiekvalitet ved Universitetet for miljø- og biovitenskap. Fokusgruppeintervjuer og statistiske analyser. NIFU-rapport 39/2013. Kvernbekk, Tone (2008). The Nature of Military Pedagogical Theory. In: Kvernbekk. T, Simpson, H and Peters, M.A. (Eds.) Military Pedagogies and Why They Matter. London: Sense Publishers. Ministry of Defence (2011). Et forsvar for vår tid. Prop. 73 S Ministry of Defence (2012a). Et forsvar for vår tid. Iverksettingsbrev til forsvarssektoren for langtidsperioden 2013-2016. Ministry of Defence (2012b). Kompetanse for en ny tid. Meld. St. 14. NOKUT (2008). Evaluering av system for kvalitetssikring av utdanningen ved Luftkrigsskolen. Royal Norwegian Air Force Academy (2013). Luftkrigsskolens kvalitetsrapport for skoleåret 2012-2013. Solberg, Espen; Børing, Pål; Rørstad, Kristoffer & Carlsten, Tone Cecilie (forthcoming) The Competence Barometer. Annual survey of study and occupational competences for the Confederation of Norwegian Enterprise (NHO). NIFU-rapport xx/2014. Torgersen, Glenn-Egil & Steiro, Trygve J. (2009). Ledelse, samhandling og opplæring i fleksible organisasjoner. 1. utgave. Stjørdal: Læringsforlaget. Vedung, Evert (2003). Models of Evaluation. In: Haug, Peder and Schwandt, Thomas A. (eds) Evaluating Educational Reforms.Scandinavian perspectives. A volume in evaluation and society. Series Editors: Katherine Ryan and Thomas A. Schwandt. University of Illinois at Urbana-Champaign. CT: IAP. Watkins, Karen E; Lysø, Ingunn Hybertsen & deMarrais, Kathleen (2011). Evaluating Executive Leadership Programs: A Theory of Change Approach. In: Advances in Developing Human Resources. Volum 13 (2).

Author Information

Tone Cecilie Carlsten (presenting / submitting)
NIFU
Oslo
Ingunn Hybertsen Lyso (presenting)
Royal Norwegian Air Force Academy
Leadership and International Collaboration
Trondheim

Update Modus of this Database

The current conference programme can be browsed in the conference management system (conftool) and, closer to the conference, in the conference app.
This database will be updated with the conference data after ECER. 

Search the ECER Programme

  • Search for keywords and phrases in "Text Search"
  • Restrict in which part of the abstracts to search in "Where to search"
  • Search for authors and in the respective field.
  • For planning your conference attendance, please use the conference app, which will be issued some weeks before the conference and the conference agenda provided in conftool.
  • If you are a session chair, best look up your chairing duties in the conference system (Conftool) or the app.