Skip to main content
SpringerLink
Log in

A relevance ranking method for citation-based search results

  • Published:
Scientometrics Aims and scope Submit manuscript

Abstract

A growing number of researchers are exploring the use of citation relationships such as direct citation, bibliographic coupling, and co-citation for information retrieval in scientific databases and digital libraries. In this paper, I propose a method of ranking the relevance of citation-based search results to a set of key, or seed, papers by measuring the number of citation relationships they share with those key papers. I tested the method against 23 published systematic reviews and found that the method retrieved 87% of the studies included in these reviews. The relevance ranking approach identified a subset of the citation search results that comprised 27% of the total documents retrieved by the method, and 7% of the documents retrieved by these reviews, but that contained 75% of the studies included in these reviews. Additional testing suggested that the method may be less appropriate for reviews that combine literature in ways that are not reflected in the literature itself. These results suggest that this ranking method could be useful in a range of information retrieval contexts.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

References

  • Belter, C. W. (2016). Citation analysis as a literature search method for systematic reviews. Journal of the Association for Information Science and Technology, 67(11), 2766–2777. doi:10.1002/asi.23605.

    Article  Google Scholar 

  • Bichteler, J., & Eaton, E. A. (1980). The combined use of bibliographic coupling and cocitation for document retrieval. Journal of the American Society for Information Science, 31(4), 278–282. doi:10.1002/asi.4630310408.

    Article  Google Scholar 

  • Carevic, Z., & Schaer, P. (2014). On the connection between citation-based and topical relevance ranking: results of a pretest using I search. Paper presented at the proceedings of the first workshop on bibliometric-enhanced information retrieval, Amsterdam, 13 April.

  • Cribbin, T. (2014). Augmenting citation chain aggregation with article maps. Paper presented at the proceedings of the first workshop on knowledge maps and information retrieval, London

  • Eto, M. (2013). Evaluations of context-based co-citation searching. Scientometrics, 94(2), 651–673. doi:10.1007/s11192-012-0756-z.

    Article  Google Scholar 

  • Garfield, E. (1955). Citation Indexes for science: A new dimension in documentation through association of ideas. Science, 122(3159), 108–111. doi:10.1126/science.122.3159.108.

    Article  Google Scholar 

  • Garfield, E. (1964). “Science Citation Index”–A new dimension in Indexing. Science, 144(3619), 649–654. doi:10.1126/science.144.3619.649.

    Article  Google Scholar 

  • Heck, T., & Schaer, P. (2013). Performing informetric analysis on information retrieval test collections: Preliminary experiments in the physics domain. arXiv:1306.1743.

  • Hinde, S., & Spackman, E. (2015). Bidirectional citation searching to completion: An exploration of literature searching methods. Pharmacoeconomics, 33(1), 5–11. doi:10.1007/s40273-014-0205-3.

    Article  Google Scholar 

  • Horsley, T., Dingwall, O., & Sampson, M. (2011). Checking reference lists to find additional studies for systematic reviews. Cochrane database of systematic reviews (8), MR000026. doi:10.1002/14651858.MR000026.pub2.

  • Ingwersen, P. (1996). Cognitive perspectives of information retrieval interaction: elements of a cognitive IR theory. Journal of Documentation, 52(1), 3–50. doi:10.1108/eb026960.

    Article  Google Scholar 

  • Janssens, A. C. J. W., & Gwinn, M. (2015). Novel citation-based search method for scientific literature: Application to meta-analyses. BMC Medical Research Methodology, 15(1), 1–11. doi:10.1186/s12874-015-0077-z.

    Article  Google Scholar 

  • Larsen, B. (2002). Exploiting citation overlaps for information retrieval: Generating a boomerang effect from the network of scientific papers. Scientometrics, 54(2), 155–178. doi:10.1023/A:1016011326300.

    Article  Google Scholar 

  • Linder, S. K., Kamath, G. R., Pratt, G. F., Saraykar, S. S., & Volk, R. J. (2015). Citation searches are more sensitive than keyword searches to identify studies using specific measurement instruments. Journal of Clinical Epidemiology, 68(4), 412–417. doi:10.1016/j.jclinepi.2014.10.008.

    Article  Google Scholar 

  • Mayr, P., & Scharnhorst, A. (2015). Scientometrics and information retrieval: Weak-links revitalized. Scientometrics, 102(3), 2193–2199. doi:10.1007/s11192-014-1484-3.

    Article  Google Scholar 

  • National Research Council. (2011). Finding what works in health care: Standards for systematic reviews. Washington, DC: The National Academies Press.

    Google Scholar 

  • Ortuno, F., Rojas, I., Andrade-Navarro, M., & Fontaine, J.-F. (2013). Using cited references to improve the retrieval of related biomedical documents. BMC Bioinformatics, 14(1), 113. doi:10.1186/1471-2105-14-113.

    Article  Google Scholar 

  • Pao, M. L., & Worthen, D. B. (1989). Retrieval effectiveness by semantic and citation searching. Journal of the American Society for Information Science, 40(4), 226–235. doi:10.1002/(sici)1097-4571(198907)40:4<226:aid-asi2>3.0.co;2-6.

    Article  Google Scholar 

  • Robinson, K. A., Dunn, A. G., Tsafnat, G., & Glasziou, P. (2014). Citation networks of related trials are often disconnected: Implications for bidirectional citation searches. Journal of Clinical Epidemiology, 67(7), 793–799. doi:10.1016/j.jclinepi.2013.11.015.

    Article  Google Scholar 

  • Sci2 Team (2009). Science of science (Sci2) tool. Indiana University and SciTech strategies.

  • Shamseer, L., Moher, D., Clarke, M., Ghersi, D., Liberati, A., Petticrew, M., et al. (2015). Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015: Elaboration and explanation. British Medical Journal, 349, g7647. doi:10.1136/bmj.g7647.

    Article  Google Scholar 

  • Van Rijsbergen, C. J. (1979). Information retrieval. London: Butterworths.

    MATH  Google Scholar 

  • Wright, K., Golder, S., & Rodriguez-Lopez, R. (2014). Citation searching: A systematic review case study of multiple risk behaviour interventions. BMC Medical Research Methodology, 14, 73. doi:10.1186/1471-2288-14-73.

    Article  Google Scholar 

Download references

Acknowledgements

Thanks to Cindy Clark of the NIH Library for reviewing and editing a previous draft of this manuscript. Thanks also to two anonymous reviewers whose comments improved the manuscript.

Author information

Authors and Affiliations

  1. NIH Library, Office of Research Services, National Institutes of Health, Bethesda, MD, 20892, USA

    Christopher W. Belter

Authors
  1. Christopher W. Belter
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Christopher W. Belter.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (DOCX 15 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Belter, C.W. A relevance ranking method for citation-based search results. Scientometrics 112, 731–746 (2017). https://doi.org/10.1007/s11192-017-2406-y

Download citation

  • Received:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11192-017-2406-y

Keywords

Search

Navigation