A Challenge to Metrics as Evidence of Scholarity

Author(s):

Bonnie StelmachStuart von Wolff

Conference:

ECER 2009

Network:

22. Research in Higher Education

Format:

Paper

Session Information

22 SES 05 D, Research Quality Assessment in Higher Education

Paper Session

Time:

2009-09-29

08:30-10:00

Room:

HG, HS 48

Chair:

Contribution

Academies are increasingly joining in or falling prey to the market as their modus operandi. The consequence of this corporate positioning is that utility and practicality have become self-evident, uncompromising values (Ball, 2007; Brinkley, 2009; Donoghue, 2008). Accountability is now the symbolic bottom line to which academics are tethered. Because of the imperative to rank in a globally competitive academic market, universities articulate their efforts to validate their existence in business terms (e.g. league tables, benchmarking). Europe is particularly susceptible. Organisations such as the European Centre for Strategic Management of Universities (ESMU) issue statements admonishing academies: “universities still lack an entrepreneurial spirit, are too academically oriented and do not make relevance of their programmes to the needs of the labour market a sufficiently high priority” (2008, pp. 1-2). The EU modernization agenda for universities highlights endeavours that forward the European Union in becoming a “dynamic, knowledge-based economy” (European Commission, 2007, p. 5). Pan-European projects such as DAAD’s University-Enterprise Cooperation are lauded for assessing universities’ performance regarding higher education policy and professional skills development, even as SOCRATES supports an Accompanying Measure Project. In this corporate context, it is not surprising that statistical quantification has emerged as the sanctioned measure of scholarly output, perceived to be objective, factual and neutral. Publication in refereed academic journals is the scholarly output outlet par excellence, and the education scholar’s reputation is increasingly contingent upon quantified measurement. The digital age impels this measurement trend. What has hitherto been a practice largely limited to the sciences is now on the immediate horizon for education researchers: bibliometrics. This approach is consistent with central administration’s public relations concerns, since numerical, ‘objective’ ranking is easily and immediately understandable by, is acceptable to, and assuages the public. Thus, scholars are now also salespeople. Ball (2003) has argued that being “continually accountable and constantly recorded” (p. 221) has produced “new kinds of teacher subjects” (p. 217). Similarly, in this study we were concerned with: (a) how bibliometric tools that rank publication and quantify scholarity impact a redefinition of the identity of scholar; this ranking raises the added linguistic concern for European scholarship disseminated in non-English-language publications. Further, (b) how scholars’ preoccupation with providing quantifiable forms of evidence for continued membership in the academy impacts their ability to meet education research consumers’ (e.g. teachers, administrators, policy makers) needs and interests.

Method

Adapting Butler’s (2005; 2001) notion of normalizing categories as a framework brings to the surface the potential problems with a heavy reliance on automated, statistical markers of scholarship. Using document analysis we examined organisational missions and priorities for higher education (e.g. European universities), and interpreted through Butler’s feminism the potential impact on scholars’ identities and practices. Quantification is not only a European trend; therefore, we also examined documents of organisations such as the Social Sciences and Humanities Research Council of Canada, because external agencies tremendously impact the academy (Ball, 2007). We examined social sciences databases that are increasingly employing such measurement tools as: journal impact factor, citation half-life, and statistical ranking. We argue that bibliometrics discredit academia as a process, reducing it to a product. Re-norming academics as ‘producers’ serves as an instrument of a regulatory regime: the corporatizing academy. “Producing evidence” circumscribes and reconstitutes our identity category (Butler, 2001).

Expected Outcomes

Metrics-driven organisations describe their indicators as “factual and policy neutral” (American Academy of Arts and Sciences, n.d., p. 1). We question this by arguing that quantification changes scholars. Bartels (2003) concluded teachers reject the discourse of the academy, and therefore, its scholarship. Thus, needing to publish in venues that “count” potentially increases the fracture between practitioners and researchers. Also, Rhode (2006) claimed 75% of social sciences journals are never cited; thus, publishing is a questionable enterprise for improving knowledge and practice. Finally, there is evidence that privileging the journal article displaces/decomposes researcher creativity and aesthetics (Gibb, 2008; Kleijer & Tillekens, 2003). By making the humanities count (Solow, Oakley, Franklin, D’Arms, & Jones, 2002), much is dis-counted. Some claim naysayers to quantification are “fighting a losing battle” (Sadlak, Merisotis, & Liu, 2008, p. 195), but we contend European scholars must not embrace unquestioningly this new “research ideal” (Kronman, 2007, p. 7).

References

American Academy of Arts and Sciences. (n.d.). About the Humanities Indicators Prototype (HIP). Retrieved January 24, 2009, from http://www.humanitiesindicators.org/humanitiesData.aspx Ball, S.J. (2007). Education plc: Understanding private sector participation in public sector education. London: Routledge. Ball, S.J. (2003). The teacher’s soul and the terrors of performativity. Journal of Education Policy, 18(2), 214-228. Bartels, N. (2003). How teachers and researchers read academic articles. Teaching and Teacher Education, 19, 737-753. Brinkely, A. (2009). The landscape of humanities research and funding. Cambridge, MA: The American Academy of Arts and Sciences. Retrieved January 24, 2009, from http://www.humanitiesindicators.org/essays/brinkley.pdf Butler, J. (2005). From Gender trouble: Feminism and the subversion of identity. In W.K. Kolmar & F. Bartkowski (Eds.), Feminist theory: A reader (pp. 496-503). Boston: McGraw-Hill Higher Education Butler, J. (2001). Imitation and gender subordination. In S. Seidman & J.C. Alexander (Eds.), The new social theory reader (pp. 333-346). London: Routledge. Donoghue, F. (2008). The last professions: The corporate university and the fate of the humanities. New York: Fordham University Press. European Centre for Strategic Management of Universities. (2008). University-Enterprise Cooperation: Building on new challenges from past experiences. Key issues from a SOCRATES project. Accompanying measure project No 130023- AM-06-EMC. Brussels, Belgium: Author. Retrieved January 24, 2009, from http://www.esmu.be/fileadmin/documents/University_Enterprise/Executiv e_Summary_Nadine_Burquel.pdf European Commission. (2007). University-Enterprise Cooperation: Building on new challenges from the past. Project Report. SOCRATES Accompanying Measures: project 130023-AM-06-EMC. Brussels. Retrieved February 10, 2009, from http://eu.daad.de/imperia/md/content/eu/lllp/veranstaltungen/university_enterprise_web.pdf Gibb, C. (2008). Lured by fiction: Transitioning from academic to novelist. Academic Matters. Retrieved November 28, 2008, from http://www.ocufa.on.ca/AM_Feb08/FEB08-P8.pdf Kleijer, H., & Tillekens, G. (2003, February). Twenty-five years of Learning to Labour: Looking back at British cultural studies with Paul Willis. Soundscapes, 5. Retrieved November 30, 2008, from http://www.icce.rug.nl/~soundscapes/VOLUME05/Paul_WillisUK.shtml Kronman, A.T. (2007). Education’s end: Why our colleges and universities have given up on the meaning of life. New Haven: Yale University Press. Organisation for Economic Cooperation and Development (2008). Outcomes of higher education: Quality, relevance and impact. Retrieved January 25, 2009, from http://www.oecd.org/site/0,3407,en_21571361_38973579_1_1_1_1_1,00.html Rhode, D.L. (2006). In pursuit of knowledge: Scholars, status, and academic culture. Palo Alto, CA: Stanford University Press. Sadlak, J., Merisotis, J., & Liu, N.C. (2008). University rankings: Seeking prestige, raising visibility and embedding quality—the editors’ views. Higher Education in Europe, 33(2/3), 195-199. Solow, R.M., Oakley, F., Franklin, P., D’Arms, J., & Jones, CC. (2002). Making the humanities count: The importance of data. Cambridge, MA: The American Academy of Arts and Sciences. Retrieved January 24, 2009, from http://www.amacad.org/publications/monographs/Making_the_Humanities _Count.pdf Shore, C., & Wright, S. (1999). Audit culture and anthropology: Neo-liberalism in British higher education. The Journal of the Royal Anthropological Institute, 5(4), 557-575.

Author Information

Bonnie Stelmach

University of Saskatchewan

Educational Administration

Saskatoon

Stuart von Wolff

University of Saskatchewan, Canada

Search the ECER Programme

Search for keywords and phrases in "Text Search"
Restrict in which part of the abstracts to search in "Where to search"
Search for authors and in the respective field.
For planning your conference attendance you may want to use the conference app, which will be issued some weeks before the conference
If you are a session chair, best look up your chairing duties in the conference system (Conftool) or the app.