Abstract
Forward citations are widely recognized as a useful measure of the impact of scientific papers. However, an inherent characteristic of forward citations is that they take time to accumulate. This makes them valuable for retrospective impact evaluations, but less helpful for prospective forecasting exercises. To overcome this, it would be desirable to have an indicator that forecasts future citations close to the time a scientific paper is published. In this study, we discuss scientific paper usage (full-text PDF downloads and HTML views) as a useful indicator that precedes citations. This paper represents the first large scale study of usage for IEEE Xplore papers, which account for about one-third of all Electrical Engineering and Information Technology papers. We also resolve earlier contradictory results where some studies showed high correlation between usage and citations, while others highlighted low overlap between individual highly cited papers and high usage papers. We also show that usage significantly precedes citations, such that a usage indicator can predict the future citation impact of papers six months or less after publication. Finally, we develop a practical usage indicator that is relatively simple to calculate and easy to understand.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
Crossref was used in favor of other sources (Web of Science, SCOPUS etc.) at the recommendation of IEEE who supplied the usage data as well as the corresponding Crossref data.
To be precise, we use the term journal, but half (90 out of 180) are actually transactions. These are currently synonymous with journals, although historically transactions were slightly shorter than journal articles. The remainder of the journal set consists of journals (60) and magazines such as Spectrum and Computer (30).
References
Bollen, J., Van de Sompel, H., Hagberg, A., & Chute, R. (2009). A principal component analysis of 39 scientific impact measures. PLoS ONE. https://doi.org/10.1371/journal.pone.0006022.
Breitzman, A., & Narin, F. (1996). A case for patent citation analysis in litigation. The Law Works, 3(3), 10–27.
Brody, T., Harnad, S., & Carr, L. (2006). Earlier web usage statistics as predictors of later citation impact. Journal of the American Society for Information Science and Technology. https://doi.org/10.1002/asi.20373.
Chen, W. M. Y., Bukhari, M., Cockshull, F., & Galloway, J. (2020). The relationship between citations, downloads and alternative metrics in rheumatology publications: A bibliometric study. Rheumatology (Oxford, England). https://doi.org/10.1093/rheumatology/kez163.
Colledge, L. (2014). Snowball metrics recipe book. Snowball Metrics Program Partners. https://doi.org/10.1016/B978-1-4377-1454-8.00078-3.
Fan, K. W. (2015). Bias and other limitations affect measures of journals in integrative and complementary medicine. Journal of the Medical Library Association. https://doi.org/10.3163/1536-5050.103.3.009.
Garfield, E. (1964). Science citation index—A new dimension in indexing. Science. https://doi.org/10.1126/science.144.3619.649.
Garfield, E. (1986). Do nobel prize winners write citation classics? Current Contents, 23, 3–8.
Gorraiz, J., Gumpenberger, C., & Schlögl, C. (2014). Usage versus citation behaviours in four subject areas. Scientometrics. https://doi.org/10.1007/s11192-014-1271-1.
Guerrero-Bote, V. P., & Moya-Anegón, F. (2014). Relationship between downloads and citations at journal and paper levels, and the influence of language. Scientometrics. https://doi.org/10.1007/s11192-014-1243-5.
Harhoff, D., Narin, F., Scherer, F. M., & Vopel, K. (1999). Citation frequency and the value of patented inventions. Review of Economics and Statistics. https://doi.org/10.1162/003465399558265.
IEEE-Institute of Electrical and Electronics Engineers. (2020a). About IEEE Xplore. https://ieeexplore.ieee.org/Xplorehelp/#/overview-of-ieee-xplore/about-ieee-xplore. Accessed 16 March 2020.
IEEE-Institute of Electrical and Electronics Engineers. (2020b). IEEE at a Glance. https://www.ieee.org/about/today/at-a-glance.HTML?WT.mc_id=ab_lp_qui. Accessed 16 March 2020.
Khan, M. S., & Younas, M. (2017). Analyzing readers behavior in downloading articles from IEEE digital library: A study of two selected journals in the field of education. Scientometrics. https://doi.org/10.1007/s11192-016-2232-7.
Moed, H. F. (2005). Statistical relationships between downloads and citations at the level of individual documents within a single journal. Journal of the American Society for Information Science and Technology. https://doi.org/10.1002/asi.20200.
Moed, H. F., & Halevi, G. (2016). On full text download and citation distributions in scientific-scholarly journals. Journal of the Association for Information Science and Technology. https://doi.org/10.1002/asi.23405.
Nieder, C., Dalhaug, A., & Aandahl, G. (2013). Correlation between article download and citation figures for highly accessed articles from five open access oncology journals. SpringerPlus. https://doi.org/10.1186/2193-1801-2-261.
Patterson, D., Snyder, L., & Ullman, J. (1999). Evaluating computer scientists and engineers for promotion and tenure. Computing Research News, (September), A-B. https://cra.org/resources/best-practice-memos/evaluating-computer-scientists-and-engineers-for-promotion-and-tenure/.
Perneger, T. V. (2004). Relation between online “hit counts” and subsequent citations: Prospective study of research papers in the BMJ. BMJ. https://doi.org/10.1136/bmj.329.7465.546.
Rutgers University Libraries. (2020). Rutgers University Libraries/IEEE Xplore. https://www.libraries.rutgers.edu/indexes/ieee. Accessed 16 March 2020.
Schloegl, C., & Gorraiz, J. (2010). Comparison of citation and usage indicators: The case of oncology journals. Scientometrics. https://doi.org/10.1007/s11192-010-0172-1.
Schloegl, C., & Gorraiz, J. (2011). Global usage versus global citation metrics: The case of pharmacology journals. Journal of the American Society for Information Science and Technology. https://doi.org/10.1002/asi.21420.
Schlögl, C., Gorraiz, J., Gumpenberger, C., Jack, K., & Kraker, P. (2013). Download vs. citation vs. readership data: The case of an information systems journal. In Proceedings of ISSI 2013—14th International Society of Scientometrics and Informetrics Conference.
Schlögl, C., Gorraiz, J., Gumpenberger, C., Jack, K., & Kraker, P. (2014). Are downloads and readership data a substitute for citations? The case of a scholarly journal. In Libraries in the Digital Age (LIDA).
Vaughan, L., Tang, J., & Yang, R. (2017). Investigating disciplinary differences in the relationships between citations and downloads. Scientometrics. https://doi.org/10.1007/s11192-017-2308-z.
Watson, A. B. (2009). Comparing citations and downloads for individual articles at the Journal of Vision. Journal of Vision. https://doi.org/10.1167/9.4.i.
Acknowledgement
The author would like to acknowledge the IEEE for funding this project, as well as providing access to its Xplore data set and usage statistics. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Breitzman, A. The relationship between web usage and citation statistics for electronics and information technology articles. Scientometrics 126, 2085–2105 (2021). https://doi.org/10.1007/s11192-020-03851-5
Received:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11192-020-03851-5