Boundaries and Potentials of Altmetrics in Educational Research
Author(s):
Christoph Schindler (presenting / submitting) Elisabeth Vogler Alexander Botte
Conference:
ECER 2017
Network:
22. Research in Higher Education
Format:
Paper (Copy for Joint Session)

Session Information

22 SES 11 F JS, Digital Scholarship, Metrics and Reputation

Joint Paper Session NW 12 and NW 22

Time:
2017-08-24
17:15-18:45
Room:
W3.15
Chair:
Britten Ekstrand

Contribution

 

This paper picks up some recent developments in scientometrics in terms of altmetrics and analyzes their relevance for monitoring educational research. Altmetrics aim at measuring scientific effects of social media communication and have been discussed as alternative metrics promising the extension of current citation-based impact factors (Priem et al., 2010). Concerning transdisciplinary research aspects (Koier & Horlings, 2015) of educational research, various potentials have been articulated. Altmetrics address societal progress (Bornmann, 2014a, Bornmann, 2014b) by involving the extra-scientific world and extend the range of research products for scientific evaluation (Bornmann, 2014b, Bornmann, 2015; Priem, 2014; Zahedi et al., 2014). On the other hand, a series of related limitations have been articulated ranging from coverage, consistency and traceability of data sources and aggregators (Chamberlain, 2013; Zahedi, Fenner, & Costas, 2014, Zahedi, Fenner, & Costas, 2015) to the dependency on communication and publication practices of different scientific communities (Costas, Zahedi, & Wouters, 2014; Peters et al., 2014; Zahedi et al., 2014). The transdisciplinary field of educational research is used as a case study to get a first insight into how current altmetric tools cover the field on the levels of its general publication output, and on the level of relevant journals. Additionally, we present an experimental approach targeting the Twitter mentions of a transdisciplinary research report.

Method

The survey on the coverage of publication practices in educational research was based on subject-specific databases that as far as possible portray disciplinary heterogeneity. For the level of publications (as of June 2016), the German Education Index (GEI) was used (2010 – 2015). For all GEI articles with the identifier DOI the altmetrics for the publications were retrieved via the Altmetric.com-API using R and the package rAltmetric on May 10, 2016. For the analysis of educational journals, the index of editing characteristics of educational research journals DEPOT was used. All ISSNs of the journals indexed as of February 26, 2016, were inserted in Altmetric.com to retrieve all journal articles indexed in the Altmetric.com database. A further collection of Twitter mentions (collected on June 16, 2016 + 7 days) addresses a research report on the situation of education in Germany (Autorengruppe Bildungsberichterstattung, 2016).

Expected Outcomes

21.2% of the publications with DOI (GEI 2010 - 2015were found to have altmetric data. 17.5% of these publications with DOI were mentioned by Twitter, followed by blogs (2.3%), Google+ (1.0%), and news (0.2%). Taking into account the restrictions of the needed identifier DOI , a profound limitation is apparent: concerning the full range of collected publications instead of high percentages of altmetric coverage (Mohammadi & Thelwall, 2014, Peters et al., 2014) only 2.5% (n=3,404) could be identified and connected to altmetric data. Additionally, the limited coverage of altmetrics aggregations is shown and the decreasing heterogeneity of publication practices (language, type …). The coverage of these ‘classical’ research outputs (publications) entails already profound shifts, which problematize fundamentally the usage of altmetrics in their current form for scientific evaluations and the measurement of societal impact. The studies deal with similar problems to bibliometric studies in terms of data collection and their limitation to journal articles. Nevertheless, potentials of social media analytics are revealed, which allow a monitoring of a much broader attention scope. The example of the report on education shows that extra-scientific discourse on research outputs, referenced by mainstream media and beyond, will be visible.

References

References Autorengruppe Bildungsberichterstattung (Ed.). (2016). Bildung in Deutschland 2016. (1. Auflage). Bielefeld: Bertelsmann, W. Bornmann, L. (2014a). Do altmetrics point to the broader impact of research? An overview of benefits and disadvantages of altmetrics. Journal of Informetrics, 8(4), 895–903. Bornmann, L. (2014b). Validity of altmetrics data for measuring societal impact: A study using data from Altmetric and F1000Prime. Journal of Informetrics, 8(4), 935–950. Bornmann, L. (2015). Usefulness of altmetrics for measuring the broader impact of research: a case study using data from PLOS and F1000Prime. Aslib Journal of Information Management, 67(3), 305–319. Bornmann, L. (2016). Measuring impact in research evaluations: A thorough discussion of methods for, effects of and problems with impact measurements. Higher Education. doi:10.1007/s10734-016-9995-x Chamberlain, S. (2013). Consuming Article-Level Metrics: Observations and Lessons. Information Standards Quarterly, 25(2), 4–13. Costas, R., Zahedi, Z., & Wouters, P. (2014). Do altmetrics correlate with citations? Extensive comparison of altmetric indicators with citations from a multidisciplinary perspective. doi:10.1002/asi.23309 Koier, E., & Horlings, E. (2015). How accurately does output reflect the nature and design of transdisciplinary research programmes? Research Evaluation, 24(1), 37–50. doi:10.1093/reseval/rvu027 Mohammadi, E., & Thelwall, M. (2014). Mendeley readership altmetrics for the social sciences and humanities. Journal of the Association for Information Science and Technology, 65(8), 1627–1638. Peters, I., Jobmann, A., Eppelin, A., Hoffmann, C. P., Künne, S., & Wollnik-Korn, G. (2014). Altmetrics for large, multidisciplinary research groups: a case study of the Leibniz Association. In Assessing libraries and library users and use. Part II Altmetrics - new methods in assessing scholarly communication and librariess. Zadar. Priem, J. (2014). Altmetrics. In B. Cronin & C. R. Sugimoto (Eds.), Beyond bibliometrics. Harnessing multidimensional indicators of scholarly impact (pp. 263–287). Cambridge, Massachusetts: The MIT Press. Priem, J., Taraborelli, D., Groth, P., & Neylon, C. (2010). altmetrics: a manifesto. Retrieved from http://www.altmetrics.org/manifesto Zahedi, Z., Costas, R., & Wouters, P. (2014). How well developed are altmetrics? A cross-disciplinary analysis of the presence of ‘alternative metrics’ in scientific publications. Scientometrics, 101(2), 1491–1513. Zahedi, Z., Fenner, M., & Costas, R. (2014). How consistent are altmetrics providers? Study of 1000 PLOS ONE publications using the PLOS ALM, Mendeley and Altmetric.com APIs. Zahedi, Z., Fenner, M., & Costas, R. (2015). Consistency among altmetrics data provider/aggregat ors: what are the challenges? In altmetrics15. the 2015 Altmetrics Workshop (pp. [3 S.]). Amsterdam.

Author Information

Christoph Schindler (presenting / submitting)
German Institute for International Educational Research (DIPF), Germany
Elisabeth Vogler
German Institute for International Educational Research (DIPF), Germany
Alexander Botte
German Institute for International Educational Research (DIPF), Germany

Update Modus of this Database

The current conference programme can be browsed in the conference management system (conftool) and, closer to the conference, in the conference app.
This database will be updated with the conference data after ECER. 

Search the ECER Programme

  • Search for keywords and phrases in "Text Search"
  • Restrict in which part of the abstracts to search in "Where to search"
  • Search for authors and in the respective field.
  • For planning your conference attendance, please use the conference app, which will be issued some weeks before the conference and the conference agenda provided in conftool.
  • If you are a session chair, best look up your chairing duties in the conference system (Conftool) or the app.