Skip to main content
SpringerLink
Log in

An overview and evaluation of citation recommendation models

  • Published:
Scientometrics Aims and scope Submit manuscript

A Correction to this article was published on 07 June 2021

This article has been updated

Abstract

Recommendation systems assist web users with personalized suggestions based on past preferences for products, or items including documents, books, movies, and research papers. The plethora and variety of research papers on the Web and digital libraries make it challenging for researchers to find relevant publications to their scholarly interests. To cope with this inevitable challenge, various models and algorithms have been proposed to assist researchers with personalized citation recommendations. Nevertheless, so far, no research study has exploited the validity and suitability of evaluations conducted for these models to find the most promising among them. This study investigates and examines the existing citation recommendation algorithms based on the following criteria: evaluation methods adopted, comparative baselines employed, the complexity of the proposed algorithm, reproducibility of the experimental results, and consistency and universality of the evaluation methods. Besides this, our study presents a generic architecture and process of a typical citation recommendation system and provides a brief overview of information filtering methods used in the existing models. The findings of the study have implications for researchers and practitioners working on research paper recommendation and related areas.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Change history

Notes

  1. https://www.aminer.cn/citation.

  2. https://psu.app.box.com/v/refseer.

  3. http://www.citeulike.org/faq/data.adp.

  4. https://acl-arc.comp.nus.edu.sg/.

References

  • Abro, W. A., Qi, G., Gao, H. Khan, M. A., & Ali, Z. (2019). Multi-turn intent determination for goal-oriented dialogue systems. In 2019 international joint conference on neural networks (IJCNN) (pp. 1–8). IEEE.

  • Achakulvisut, T., Acuna, D. E., Ruangrong, T., & Kording, K. (2016). Science concierge: A fast content-based recommendation system for scientific publications. PLOS ONE, 11(7):1–11, 07.

  • Ali, Z., Kefalas, P., Muhammad, K., Ali, B., & Imran, M. (2020a). Deep learning in citation recommendation models survey. Expert Systems with Applications, 162, 113790.

    Google Scholar 

  • Ali, Z., Qi, G., Kefalas, P., Abro, W. A., & Ali, B. (2020b). A graph-based taxonomy of citation recommendation models. Artificial Intelligence Review, pp. 1–44.

  • Ali, Z., Qi, G., Muhammad, K., Ali, B., & Abro, W. A. (2020c). Paper recommendation based on heterogeneous network embedding. Knowledge-Based Systems, 210, 106438.

    Google Scholar 

  • Alotaibi, S., & Vassileva, J. (2018). Implicit social networks for social recommendation of scholarly papers. In Highlighting the importance of big data management and analysis for various applications (Vol. 27, pp. 79–92). Cham: Springer International Publishing.

  • Amami, M., Pasi, G., Stella, F., & Faiz, R. (2016). An LDA-based approach to scientific paper recommendation. In International conference on applications of natural language to information systems (pp. 200–210). Springer.

  • Amami, M., Pasi, G., Stella, F., & Faiz, R. (2016). An LDA-based approach to scientific paper recommendation. In Proceedings of natural language processing and information systems: 21st international conference on applications of natural language to information systems, NLDB 2016, Salford, UK, June 22–24, 2016 (pp. 200–210). Springer International Publishing.

  • Bai, X., Wang, M., Lee, I., Yang, Z., Kong, X., & Xia, F. (2019). Scientific paper recommendation: A survey. IEEE Access, 7, 9324–9339.

    Google Scholar 

  • Bansal, T., Belanger, D., & McCallum, A. (2016). Ask the gru: Multi-task learning for deep text recommendations. In Proceedings of the 10th ACM conference on recommender systems, RecSys’16, New York, NY, USA (pp. 107–114).

  • Beel, J., Gipp, B., Langer, S., & Breitinger, C. (2016). Research-paper recommender systems: A literature survey. International Journal on Digital Libraries, 17(4), 305–338.

    Google Scholar 

  • Beel, J., Langer, S., Genzmehr, M., Gipp, B., Breitinger, C., & Nürnberger, A. (2013). Research paper recommender system evaluation: A quantitative literature survey. In Proceedings of the international workshop on reproducibility and replication in recommender systems evaluation (pp. 15–22).

  • Bhagavatula, C., Feldman, S., Power, R., & Ammar, W. (2018) Content-based citation recommendation. In Proceedings of the 2018 conference of the North American chapter of the association for computational linguistics: Human language technologies, New Orleans, Louisiana (Vol. 1 (Long Papers), pp. 238–251).

  • Burke, R. (2002). Hybrid recommender systems: Survey and experiments. User Modeling and User-Adapted Interaction, 12(4), 331–370.

    MATH  Google Scholar 

  • Cai, X., Han, J., Li, W., Zhang, R., Pan, S., & Yang, L. (2018a). A three-layered mutually reinforced model for personalized citation recommendation. IEEE Transactions on Neural Networks and Learning Systems, 29(12), 6026–6037.

    Google Scholar 

  • Cai, X., Han, J., Pan, S., & Yang, L. (2018b). Heterogeneous information network embedding based personalized query-focused astronomy reference paper recommendation. International Journal of Computational Intelligence Systems, 11, 591–599.

    Google Scholar 

  • Cai, X., Han, J., & Yang, L. (2018c). Generative adversarial network based heterogeneous bibliographic network representation for personalized citation recommendation. In Thirty-second AAAI conference on artificial intelligence.

  • Cai, X., Zheng, Y., Yang, L., Dai, T., & Guo, L. (2019). Bibliographic network representation based personalized citation recommendation. IEEE Access, 7, 457–467.

    Google Scholar 

  • Caragea, C., Silvescu, A., Mitra, P., & Giles, C. L. (2013a). Can’t see the forest for the trees? A citation recommendation system. In Proceedings of the 13th ACM/IEEE-CS joint conference on digital libraries (JCDL) (pp. 111–114).

  • Caragea, C., Silvescu, A., Mitra, P., & Giles, C. L. (2013b). Can’t see the forest for the trees? A citation recommendation system. In Proceedings of the 13th ACM/IEEE-CS joint conference on digital libraries (pp. 111–114).

  • Chakraborty, T., Krishna, A., Singh, M., Ganguly, N., Goyal, P., & Mukherjee, A. (2016). Ferosa: A faceted recommendation system for scientific articles. In Proceedings, part II, of the 20th Pacific-Asia conference on advances in knowledge discovery and data mining (PAKDD) (Vol. 9652, pp. 528–541).

  • Chakraborty, T., Modani, N., Narayanam, R., & Nagar, S. (2015a). Discern: a diversified citation recommendation system for scientific queries. In 2015 IEEE 31st international conference on data engineering (pp. 555–566). IEEE.

  • Chakraborty, T., Modani, N., Narayanam, R., & Nagar, S. (2015b). Discern: A diversified citation recommendation system for scientific queries. In 31st IEEE international conference on data engineering, (ICDE) 2015, Seoul, South Korea, April 13–17, 2015 (pp. 555–566).

  • Chen, J., & Ban, Z. (2019). Academic paper recommendation based on clustering and pattern matching. In International CCF conference on artificial intelligence (pp. 171–182). Springer.

  • Chen, J., Liu, Y., Zhao, S., & Zhang, Y. (2019). Citation recommendation based on weighted heterogeneous information network containing semantic linking. In 2019 IEEE international conference on multimedia and expo (ICME) (pp. 31–36).

  • Chen, T., Zhang, W., Qiuxia, L., Chen, K., Zheng, Z., & Yong, Yu. (2012). Svdfeature: A toolkit for feature-based collaborative filtering. The Journal of Machine Learning Research, 13(1), 3619–3622.

    MathSciNet  Google Scholar 

  • Chen, X., Zhao, H., Zhao, S., Chen, J., & Zhang, Y. (2019). Citation recommendation based on citation tendency. Scientometrics, 121(2), 937–956.

    Google Scholar 

  • Cheng, G., Yang, C., Yao, X., Guo, L., & Han, J. (2018). When deep learning meets metric learning: Remote sensing image scene classification via learning discriminative cnns. IEEE Transactions on Geoscience and Remote Sensing, 56(5), 2811–2821.

    Google Scholar 

  • Cheng, G., Zhou, P., & Han, J. (2017). Duplex metric learning for image set classification. IEEE Transactions on Image Processing, 27(1), 281–292.

    MathSciNet  MATH  Google Scholar 

  • Cho, K., Van Merriënboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014). Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078.

  • Dai, T., Zhu, L., Cai, X., Pan, S., & Yuan, S. (2018). Explore semantic topics and author communities for citation recommendation in bipartite bibliographic network. Journal of Ambient Intelligence and Humanized Computing, 9(4), 957–975.

    Google Scholar 

  • Dai, T., Zhu, L., Wang, Y., & Carley, K. M. (2019a). Attentive stacked denoising autoencoder with BI-LSTM for personalized context-aware citation recommendation. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 28, 553–568.

    Google Scholar 

  • Dai, T., Zhu, L., Wang, Y., Zhang, H., Cai, X., & Zheng, Y. (2019b). Joint model feature regression and topic learning for global citation recommendation. IEEE Access, 7, 1706–1720.

    Google Scholar 

  • Dinesh, S. (2018). Real world evaluation of approaches to research paper recommendation. arXiv preprint arXiv:1802.06892.

  • Du, Z., Tang, J., & Ding, Y. (2019). Polar: Attention-based CNN for one-shot personalized article recommendation. In Machine learning and knowledge discovery in databases (pp. 675–690).

  • Ebesu, T., & Fang, Y. (2017). Neural citation network for context-aware citation recommendation. In Proceedings of the 40th international ACM SIGIR conference on research and development in information retrieval (pp. 1093–1096).

  • Ekstrand, M. D., Ludwig, M., Konstan, J. A., & Riedl, J. T. (2011). Rethinking the recommender research ecosystem: Reproducibility, openness, and lenskit. In RecSys’11 (pp. 133–140). New York, NY, USA: Association for Computing Machinery.

  • Färber, M., & Jatowt, A. (2020). Citation recommendation: Approaches and datasets. International Journal on Digital Libraries, pp. 1–31.

  • Färber, M., & Sampath, A. (2020). Hybridcite: A hybrid model for context-aware citation recommendation. arXiv preprint arXiv:2002.06406.

  • Färber, M., Klein, T., & Sigloch, J. (2020). Neural citation recommendation: A reproducibility study. In BIR@ECIR.

  • Färber, M., Thiemann, A., & Jatowt, A. (2018a). Citewerts: A system combining cite-worthiness with citation recommendation. In European conference on information retrieval (pp. 815–819).

  • Färber, M., Thiemann, A., & Jatowt, A. (2018b). To cite, or not to cite? Detecting citation contexts in text. In European conference on information retrieval (pp. 598–603).

  • Fu, T.-y., Lee, W.-C., & Lei, Z. (2017). Hin2vec: Explore meta-paths in heterogeneous information networks for representation learning. In Proceedings of the 2017 ACM on conference on information and knowledge management (pp. 1797–1806).

  • Galke, L., Mai, F., Vagliano, I., & Scherp, A. (2018). Multi-modal adversarial autoencoders for recommendations of citations and subject labels. In Proceedings of the 26th conference on user modeling, adaptation and personalization (pp. 197–205).

  • Ganguly, S., & Pudi, V. (2017). Paper2vec: Combining graph and text information for scientific paper representation. In Advances in information retrieval (pp. 383–395).

  • Grover, A., & Leskovec, J. (2016). node2vec: Scalable feature learning for networks. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining (pp. 855–864).

  • Guo, L., Cai, X., Hao, F., Dejun, M., Fang, C., & Yang, L. (2017a). Exploiting fine-grained co-authorship for personalized citation recommendation. IEEE Access, 5, 12714–12725.

    Google Scholar 

  • Guo, L., Cai, X., Hao, F., Mu, D., Fang, C., & Yang, L. (2017b). Exploiting fine-grained co-authorship for personalized citation recommendation. IEEE Access, 5, 12714–12725.

    Google Scholar 

  • Guo, L., Cai, X., Qin, H., Guo, Y., Li, F., & Tian, G. (2019). Citation recommendation with a content-sensitive deepwalk based approach. In 2019 international conference on data mining workshops (ICDMW) (pp. 538–543). IEEE.

  • Gupta, S., & Varma, V. (2017). Scientific article recommendation by using distributed representations of text and graph. In Proceedings of the 26th international conference on world wide web companion, WWW’17 companion, Republic and Canton of Geneva, Switzerland (pp. 1267–1268).

  • Han, J., Cheng, G., Li, Z., & Zhang, D. (2017). A unified metric learning-based framework for co-saliency detection. IEEE Transactions on Circuits and Systems for Video Technology, 28(10), 2473–2483.

    Google Scholar 

  • Han, J., Song, Y., Zhao, W. X., Shi, S., & Zhang, H. (2018) hyperdoc2vec: Distributed representations of hypertext documents. In Proceedings of the 56th annual meeting of the association for computational linguistics. Long papers (Vol. 1, pp. 2384–2394). Melbourne, Australia.

  • Hassan, H.A. M. (2017). Personalized research paper recommendation using deep learning. In Proceedings of the 25th conference on user modeling, adaptation and personalization (pp. 327–330).

  • He, Q., Pei, J., Kifer, D., Mitra, P., & Giles, L. (2010). Context-aware citation recommendation. In Proceedings of the 19th international conference on world wide web (pp. 421–430).

  • Hu, D., Ma, H., Liu, Y., & He, X. (2020). Scientific paper recommendation using author’s dual role citation relationship. In International conference on intelligent information processing (pp. 121–132). Springer.

  • Huang, E. H., Socher, R., Manning, C. D., & Ng, A. Y. (2012). Improving word representations via global context and multiple word prototypes. In Proceedings of the 50th annual meeting of the association for computational linguistics. Long papers (Vol. 1, pp. 873–882).

  • Huang, W., Wu, Z., Liang, C., Mitra, P., & Giles, C. L. (2015a). A neural probabilistic model for context based citation recommendation. In Proceedings of the twenty-ninth AAAI conference on artificial intelligence (pp. 2404–2410).

  • Huang, W., Wu, Z., Liang, C., Mitra, P., & Giles, C. L. (2015b). A neural probabilistic model for context based citation recommendation. In Twenty-ninth AAAI conference on artificial intelligence.

  • Huang, W., Wu, Z., Mitra, P., & Giles, C. L. (2014). Refseer: A citation recommendation system. In IEEE/ACM joint conference on digital libraries (pp. 371–374).

  • Jannach, D., Lerche, L., Gedikli, F., & Bonnin, G. (2013). What recommenders recommend—An analysis of accuracy, popularity, and sales diversity effects. In S. Carberry, S. Weibelzahl, A. Micarelli, & G. Semeraro (Eds.), User modeling, adaptation, and personalization (pp. 25–37). Berlin: Springer.

    Google Scholar 

  • Jeong, C., Jang, S., Shin, H., Park, E, & Choi, S. (2019). A context-aware citation recommendation model with Bert and graph convolutional networks. arXiv:1903.06464.

  • Jiang, Z., Liu, X., & Gao, L. (2015). Chronological citation recommendation with information-need shifting. In CIKM’15.

  • Jiang, Z., Lu, Y., & Liu, X. (2018). Cross-language citation recommendation via publication content and citation representation fusion. In Proceedings of the 18th ACM/IEEE on joint conference on digital libraries, JCDL’18 (pp. 347–348). New York, NY, USA: Association for Computing Machinery.

  • Jiang, Z., Yin, Y., Gao, L., Yao, L., & Liu, X. (2018). Cross-language citation recommendation via hierarchical representation learning on heterogeneous graph. SIGIR’18. The 41st international ACM SIGIR conference on research & #38; development in information retrieval, New York, NY, USA (pp. 635–644).

  • Joachims, T. (2002). Optimizing search engines using clickthrough data. In Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining (pp. 133–142).

  • Kataria, S., Mitra, P., & Bhatia, S. (2010). Utilizing context in generative Bayesian models for linked corpus. In Twenty-fourth AAAI conference on artificial intelligence.

  • Khadka, A., & Knoth, P. (2018). Using citation-context to reduce topic drifting on pure citation-based recommendation. In Proceedings of the 12th ACM conference on recommender systems (pp. 362–366).

  • Khusro, S., Ali, Z., & Ullah, I. (2016). Recommender systems: Issues, challenges, and research opportunities. In Information science and applications (ICISA) 2016 (pp. 1179–1189). Springer.

  • Kim, Y., Park, Y., & Shim, K. (2013). Digtobi: A recommendation system for digg articles using probabilistic modeling. In Proceedings of the 22Nd international conference on world wide web (WWW) (pp. 691–702).

  • Kobayashi, Y., Shimbo, M., & Matsumoto, Y. (2018). Citation recommendation using distributed representation of discourse facets in scientific articles. In Proceedings of the 18th ACM/IEEE on joint conference on digital libraries, JCDL’18, New York, NY, USA (pp. 243–251).

  • Kong, X., Mao, M., Wang, W., Liu, J., & Xu, B. (2019). Voprec: Vector representation learning of papers with text information and structural identity for recommendation. IEEE Transactions on Emerging Topics in Computing, 1–1.

  • Konstas, I., Stathopoulos, V., & Jose, J. M. (2009). On social networks and collaborative recommendation. In Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval (pp. 195–202).

  • Le Anh, V., Hai, V. H., Tran, H. N., & Jung, J. J. (2014). Scirecsys: A recommendation system for scientific publication by discovering keyword relationships. In Computational collective intelligence. Technologies and applications: 6th International conference, ICCCI 2014, Seoul, Korea, September 24–26, 2014. Proceedings (Vol. 8733, pp. 72–82). Springer International Publishing.

  • Le, Q., & Mikolov, T. (2014). Distributed representations of sentences and documents. In Proceedings of the 31st international conference on international conference on machine learning (Vol. 32. icml’14, pp. 1188–1196). jmlr. org.

  • Lee, J., Lee, K., & Kim, J. G. (2013) Personalized academic research paper recommendation system. CoRR.

  • Li, S., Brusilovsky, P., Su, S., & Cheng, X. (2018). Conference paper recommendation for academic conferences. IEEE Access, 6, 17153–17164.

    Google Scholar 

  • Li, X., Chen, Y., Pettit, B., & De Rijke, M. (2019). Personalised reranking of paper recommendations using paper content and user behavior. ACM Transactions on Information Systems (TOIS), 37(3), 1–23.

    Google Scholar 

  • Li, Y., Yang, M., & Zhang, Z. (2013). Scientific articles recommendation. In Proceedings of the 22nd ACM international conference on information & knowledge management (pp. 1147–1156).

  • Liu, H., Kong, X., Bai, X., Wang, W., Bekele, T. M., & Xia, F. (2015). Context-based collaborative filtering for citation recommendation. IEEE Access, 3, 1695–1703.

    Google Scholar 

  • Lops, P., Degemmis, M., & Semeraro, G. (2011a). Content-based recommender systems: State of the art and trends. In Recommender systems handbook.

  • Lops, P., de Gemmis, M., & Semeraro, G. (2011b). Content-based recommender systems: State of the art and trends (pp. 73–105). Boston, MA: Springer.

    Google Scholar 

  • Lu, Y., He, J., Shan, D., & Yan, H. (2011). Recommending citations with translation model. In Proceedings of the 20th ACM international conference on Information and knowledge management (pp. 2017–2020).

  • Ma, S., Zhang, C., & Liu, X. (2020a). A review of citation recommendation: From textual content to enriched context. Scientometrics, 1–28.

  • Ma, S., Zhang, C., & Liu, X. (2020b). A review of citation recommendation: From textual content to enriched context. Scientometrics, 122(3), 1445–1472.

    Google Scholar 

  • Ma, X., & Wang, R. (2019). Personalized scientific paper recommendation based on heterogeneous graph representation. IEEE Access, 7, 79887–79894.

    Google Scholar 

  • Ma, X., Zhang, Y., & Zeng, J. (2019). Newly published scientific papers recommendation in heterogeneous information networks. Mobile Networks and Applications, 24(1), 69–79.

    Google Scholar 

  • Meng, F., Gao, D., Li, W., Sun, X., & Hou, Y. (2013a). A unified graph model for personalized query-oriented reference paper recommendation. In Proceedings of the 22nd ACM international conference on information & knowledge management (pp. 1509–1512).

  • Meng, F., Gao, D., Li, W., Sun, X., & Hou, Y. (2013b). A unified graph model for personalized query-oriented reference paper recommendation. In Proceedings of the 22nd ACM international conference on information knowledge management (CIKM) (pp. 1509–1512).

  • Mikolov, T., Le, Q. V., & Sutskever, I. (2013a). Exploiting similarities among languages for machine translation. arXiv preprint arXiv:1309.4168.

  • Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S., & Dean, J. (2013b). Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems (pp. 3111–3119).

  • Mu, D., Guo, L., Cai, X., & Hao, F. (2018). Query-focused personalized citation recommendation with mutually reinforced ranking. IEEE Access, 6, 3107–3119.

    Google Scholar 

  • Nallapati, R. M., Ahmed, A., Xing, E. P., & Cohen, W. W. (2008). Joint latent topic models for text and citations. In Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining (pp. 542–550).

  • Nie, Z., Zhang, Y., Wen, J.-R., & Ma, W.-Y. (2005). Object-level ranking: Bringing order to web objects. In Proceedings of the 14th international conference on world wide web (pp. 567–574).

  • Nogueira, R., Jiang, Z., Cho, K., & Lin, J. (2020). Navigation-based candidate expansion and pretrained language models for citation recommendation. arXiv preprint arXiv:2001.08687.

  • Ohta, M., Hachiki, T., & Takasu, A. (2011). Related paper recommendation to support online-browsing of research papers. In Fourth international conference on the applications of digital information and web technologies (ICADIWT 2011) (pp. 130–136). IEEE.

  • Owen, S., Anil, R., Dunning, T., & Friedman, E. (2011). Mahout in action. Shelter Island: Manning Publications Co.

    Google Scholar 

  • Pan, L., Dai, X., Huang, S., & Chen, J. (2015). Academic paper recommendation based on heterogeneous graph. In Chinese computational linguistics and natural language processing based on naturally annotated big data (pp. 381–392). Springer.

  • Pan, S., Jia, W., Zhu, X., Zhang, C., & Wang, Y. (2016a). Tri-party deep network representation. Network, 11(9), 12.

    Google Scholar 

  • Pan, S., Wu, J., Zhu, X., Zhang, C., & Wang, Y. (2016b). Tri-party deep network representation. In Proceedings of the twenty-fifth international joint conference on artificial intelligence, IJCAI’16 (pp. 1895–1901). AAAI Press.

  • Pazzani, M. J., & Billsus, D. (2007). Content-based recommendation systems. In The adaptive web (pp. 325–341). Springer.

  • Peker, S., & Kocyigit, A. (2016). mrhr: A modified reciprocal hit rank metric for ranking evaluation of multiple preferences in top-n recommender systems. In C. Dichev & G. Agre (Eds.), Artificial intelligence: Methodology, systems, and applications (pp. 320–329). Cham: Springer International Publishing.

    Google Scholar 

  • Perozzi, B., Al-Rfou, R., & Skiena, S. (2014). Deepwalk: Online learning of social representations. In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining (pp. 701–710).

  • Polatidis, N., Kapetanakis, S., Pimenidis, E., & Kosmidis, K. (2018). Reproducibility of experiments in recommender systems evaluation. In AIAI.

  • Polatidis, N., Papaleonidas, A., Pimenidis, E., & Iliadis, L. (2019). An explanation-based approach for experiment reproducibility in recommender systems. Neural Computing and Applications, pp. 1–8.

  • Qiu, J., Dong, Y., Ma, H., Li, J., Wang, K., & Tang, J. (2018). Network embedding as matrix factorization: Unifying deepwalk, line, pte, and node2vec. In Proceedings of the eleventh ACM international conference on web search and data mining (pp. 459–467).

  • Raamkumar, S. A., Foo, S., & Pang, N. (2017). Using author-specified keywords in building an initial reading list of research papers in scientific paper retrieval and recommender systems. Information Processing and Management: An International Journal, 53(3), 577–594.

    Google Scholar 

  • Ren, X., Liu, J., Yu, X., Khandelwal, U., Gu, Q., Wang, L., et al. (2014). Cluscite: Effective citation recommendation by information network-based clustering. In Proceedings of the 20th ACM SIGKDD international conference on knowledge discovery and data mining (pp. 821–830).

  • Rendle, S. (2012). Factorization machines with LIBFM. ACM Transactions on Intelligent Systems and Technology (TIST), 3(3), 1–22.

    Google Scholar 

  • Rodriguez-Prieto, O., Araujo, L., & Martinez-Romo, J. (2019). Discovering related scientific literature beyond semantic similarity: A new co-citation approach. Scientometrics, 120(1), 105–127.

    Google Scholar 

  • Roy, D. (2017). An improved test collection and baselines for bibliographic citation recommendation. In Proceedings of the 2017 ACM on conference on information and knowledge management (CIKM) (pp. 2271–2274).

  • Said, A., & Bellogín, A. (2014). Rival: A toolkit to foster reproducibility in recommender system evaluation. In Proceedings of the 8th ACM conference on recommender systems, RecSys’14 (pp. 371–372). New York, NY, USA: Association for Computing Machinery.

  • Saier, T., & Färber, M. (2020). Semantic modelling of citation contexts for context-aware citation recommendation. In European conference on information retrieval (pp. 220–233). Springer.

  • Sakib, N., Ahmad, R. B., & Haruna, K. (2020). A collaborative approach toward scientific paper recommendation using citation context. IEEE Access, 8, 51246–51255.

    Google Scholar 

  • Sarwar, B. M., Karypis, G., Konstan, J. A., Riedl, J., et al. (2001). Item-based collaborative filtering recommendation algorithms. Www (Vol. 1, pp. 285–295).

  • Shi, C., Hu, B., Zhao, W. X., & Yu, P. S. (2019). Heterogeneous information network embedding for recommendation. IEEE Transactions on Knowledge and Data Engineering, 31, 357–370.

    Google Scholar 

  • Son, J., & Kim, S. B. (2017). Academic paper recommender system using multilevel simultaneous citation networks. Decision Support Systems, 105, 24–33.

    Google Scholar 

  • Sugiyama, K., & Kan, M.-Y. (2013). Exploiting potential citation papers in scholarly paper recommendation. In Proceedings of the 13th ACM/IEEE-CS joint conference on digital libraries (JCDL) (pp. 153–162).

  • Su, X., & Khoshgoftaar, T. M. (2009). A survey of collaborative filtering techniques. Advances in artificial intelligence,. https://doi.org/10.1155/2009/421425.

    Article  Google Scholar 

  • Sun, J., Ma, J., Liu, Z., & Miao, Y. (2014). Leveraging content and connections for scientific article recommendation in social computing contexts. The Computer Journal, 57(9), 1331–1342.

    Google Scholar 

  • Sun, Z., Yu, D., Fang, H., Yang, J., Qu, X., Zhang, J., et al. (2020). Are we evaluating rigorously? benchmarking recommendation for reproducible evaluation and fair comparison. In Fourteenth ACM conference on recommender systems, RecSys’20 (pp. 23–32). New York, NY, USA: Association for Computing Machinery.

  • Tang, J., Qu, M., Wang, M., Zhang, M., Yan, J., & Mei, Q. (2015). Line: Large-scale information network embedding. In Proceedings of the 24th international conference on world wide web (pp. 1067–1077).

  • Tang, J., & Zhang, J. (2009). A discriminative approach to topic-based citation recommendation. In Pacific-Asia conference on knowledge discovery and data mining (KDD) (pp. 572–579).

  • Tang, X., Wan, X., & Zhang, X. (2014). Cross-language context-aware citation recommendation in scientific articles. In SIGIR’14 (pp. 817–826). New York, NY, USA: Association for Computing Machinery.

  • Tanner, W., Akbas, E., & Hasan, M. (2019). Paper recommendation based on citation relation. In 2019 IEEE international conference on big data (big data) (pp. 3053–3059).

  • Tian, G., & Jing, L. (2013). Recommending scientific articles using bi-relational graph-based iterative RWR. In Proceedings of the 7th ACM conference on recommender systems (RecSys) (pp. 399–402).

  • Tian, H., & Zhuo, H. H. (2017). Paper2vec: Citation-context based document distributed representation for scholar recommendation. arXiv:1703.06587.

  • Waheed, W., Imran, M., Raza, B., Malik, A. K., & Khattak, H. A. (2019). A hybrid approach toward research paper recommendation using centrality measures and author ranking. IEEE Access, 7, 33145–33158.

    Google Scholar 

  • Wang, C., & Blei, D. M. (2011). Collaborative topic modeling for recommending scientific articles. In Proceedings of the 17th ACM SIGKDD international conference on knowledge discovery and data mining (KDD) (pp. 448–456).

  • Wang, H., Shi, X., & Yeung, D.-Y. (2017a). Relational deep learning: A deep latent variable model for link prediction. In Thirty-first AAAI conference on artificial intelligence.

  • Wang, H., & Li, W. (2015). Relational collaborative topic regression for recommender systems. IEEE Transactions on Knowledge and Data Engineering, 27(5), 1343–1355.

    MathSciNet  Google Scholar 

  • Wang, J., Song, D., Wang, Q., Zhang, Z., Si, L., Liao, L., et al. (2015). An entity class-dependent discriminative mixture model for cumulative citation recommendation. In Proceedings of the 38th international ACM SIGIR conference on research and development in information retrieval (pp. 635–644).

  • Wang, J., Zhu, L., Dai, T., & Wang, Y. (2020). Deep memory network with BI-LSTM for personalized context-aware citation recommendation. Neurocomputing, 410, 103–113.

    Google Scholar 

  • Wang, Z., Chen, C., & Li, W. (2017b). Predictive network representation learning for link prediction. In Proceedings of the 40th international ACM SIGIR conference on research and development in information retrieval (pp. 969–972).

  • West, J. D., Wesley-Smith, I., & Bergstrom, C. T. (2016). A recommendation system based on hierarchical clustering of an article-level citation network. IEEE Transactions on Big Data, 2(2), 113–123.

    Google Scholar 

  • Xia, F., Asabere, N. Y., Liu, H., Deonauth, N., & Li, F. (2014). Folksonomy based socially-aware recommendation of scholarly papers for conference participants. In Proceedings of the 23rd international conference on world wide web (WWW) (pp. 781–786).

  • Xia, F., Liu, H., Lee, I., & Cao, L. (2016). Scientific article recommendation: Exploiting common author relations and historical preferences. IEEE Transactions on Big Data, 2(2), 101–112.

    Google Scholar 

  • Yadav, P., Remala, N., & Pervin, N. (2019). Reccite: A hybrid approach to recommend potential papers. In 2019 IEEE international conference on big data (big data) (pp. 2956–2964).

  • Yang, C., Liu, Z., Zhao, D., Sun, M., & Chang, E. (2015). Network representation learning with rich text information. In Twenty-fourth international joint conference on artificial intelligence.

  • Yang, L., Zhang, Z., Cai, X., & Dai, T. (2019a). Attention-based personalized encoder-decoder model for local citation recommendation. Computational Intelligence and Neuroscience,. https://doi.org/10.1155/2019/1232581.

    Article  Google Scholar 

  • Yang, L., Zheng, Y., Cai, X., Dai, H., Mu, D., & Guo, L. (2018). A LSTM based model for personalized context-aware citation recommendation. IEEE Access, 6, 59618–59627.

    Google Scholar 

  • Yang, L., Zhang, Z., Cai, X., & Guo, L. (2019b). Citation recommendation as edge prediction in heterogeneous bibliographic network: A network representation approach. IEEE Access, 7, 23232–23239.

    Google Scholar 

  • Yang, L., Zheng, Y., Cai, X., Dai, H., Mu, D., Guo, L., et al. (2018). A LSTM based model for personalized context-aware citation recommendation. IEEE Access, 6, 59618–59627.

    Google Scholar 

  • Yang, Q., Li, Z., Liu, A., Liu, G., Zhao, L., Zhang, X., et al. (2019). A novel hybrid publication recommendation system using compound information. World Wide Web, 22(6), 2499–2517.

    Google Scholar 

  • Yin, J., & Li, X. (2017). Personalized citation recommendation via convolutional neural networks. In APWeb/WAIM.

  • Yu, X., Gu, Q., Zhou, M., & Han, J. (2012). Citation prediction in heterogeneous bibliographic networks. In Proceedings of the 2012 SIAM international conference on data mining (pp. 1119–1130). SIAM.

  • Zhai, C., & Lafferty, J. (2017). A study of smoothing methods for language models applied to ad hoc information retrieval. ACM SIGIR forum (Vol. 51, pp. 268–276). New York, NY, USA: ACM

  • Zhang, C., Zhao, X., Wang, K., & Sun, J. (2014). Content + attributes: A latent factor model for recommending scientific papers in heterogeneous academic networks. In Proceedings of the advances in information retrieval: 36th European conference on IR research (ECIR), Amsterdam, The Netherlands, April 13–16 (Vol. 8416, pp. 39–50). Springer International Publishing.

  • Zhang, Y., & Ma, Q. (2020). Citation recommendations considering content and structural context embedding. arXiv:2001.02344.

  • Zhang, Y., Yang, L., Cai, X., & Dai, H. (2018). A novel personalized citation recommendation approach based on GAN. In International symposium on methodologies for intelligent systems (pp. 268–278).

Download references

Author information

Authors and Affiliations

  1. School of Computer Science and Engineering Southeast University, Nanjing, China

    Zafar Ali

  2. Department of Computer Science, Shaheed Benazir Bhutto University, Sheringal, Pakistan

    Irfan Ullah

  3. School of Computer Science and Engineering, UESTC, Chengdu, 611731, China

    Amin Khan

  4. Department of Software, Sejong University, Seoul, 143-747, Republic of Korea

    Khan Muhammad

  5. Department of Computing, Abasyn University, Peshawar, Pakistan

    Asim Ullah Jan

Authors
  1. Zafar Ali
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Irfan Ullah
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Amin Khan
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Asim Ullah Jan
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Khan Muhammad
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

ZA: Conceptualization of this study, Methodology. IU: Data curation, Writing - Original draft preparation, Supervision. AK: Conceptualization of this study, Writing - Original draft preparation. AK: Conceptualization, Visualization. KM.: Writing - Original draft preparation, Supervision

Corresponding authors

Correspondence to Zafar Ali or Khan Muhammad.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ali, Z., Ullah, I., Khan, A. et al. An overview and evaluation of citation recommendation models. Scientometrics 126, 4083–4119 (2021). https://doi.org/10.1007/s11192-021-03909-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11192-021-03909-y

Keywords

Search

Navigation