Acessibilidade / Reportar erro

Brief comments on the 2020 Impact Factor of the AABC released by Journal Citation Reports

Despite several criticism of the Impact Factor (IF) made over the years (e.g., Wáng et al. 2014WÁNG YX ET AL. 2014. Implications of Web of Science journal impact factor for scientific output evaluation in 16 institutions and investigators’ opinion. Quant Imaging Med Surg 4 :453-461. DOI 10.3978/j.issn.2223-4292.2014.11.16.), the fact is that several authors and editors still rely on this scientometric index to get a sense of performance and influence of scientific production (e.g., Atallah et al. 2020ATALLAH AN, PUGA MES & AMARAL JLG. 2020. Web of Science Journal Citation Report 2020: the Brazilian contribution to the “Medicine, General & Internal” category of the journal impact factor (JIF) ranking (SCI 2019). Sao Paulo Med J 138: 271-274. DOI 10.1590/1516-3180.2020.138419092020.). At the same time, there are attempts to improve the evaluation of scientific contribution, addressing issues such as the number of authors in collaborations (e.g., Sivertsen et al. 2019Sivertsen G, Rousseau R & Zhang L. 2019. Measuring scientific contributions with modified fractional counting. J Infometr 13: 679-694. DOI 10.1016/j.joi.2019.03.010), publication trajectories in certain areas with focus on quantity of publications (e.g., Albertini et al. 2019ALBERTINI MK, BACKES AR & SÁ AL. 2019. A study of publication trajectories of the Brazilian Computer Science community. An Acad Bras Cienc 91: e20180559. DOI 10.1590/0001-3765201920180559.), and even how the language in which the article is published might influence citations (e.g., Abad et al. 2020ABAD JCS, ALENCAR RM, MARIMON JR BH, MARIMON B, SILVA ACC, JANCOSKI H, REZENDE RS & ALVES-SILVA E. 2020. Publishing in English is associated with an increase of the impact factor of Brazilian biodiversity journals. An Acad Bras Cienc 92: e20181263. DOI 10.1590/0001-3765202020181263.).

In a series of editorial notes published last year (e.g., Kellner 2020aKELLNER AWA. 2020a. Development of Agrarian Sciences at the AABC with comments on impact and performance evaluations. An Acad Bras Cienc 92: e202092S1. DOI 10.1590/0001-3765202092S1., b, c), I provided some background information on how papers from different fields published by the Annals of the Brazilian Academy of Sciences (AABC) are influencing the IF of this journal, the sole comprehensive multidisciplinary periodical published in Brazil. I prepared these brief analyses based on the AABC 2019 IF released by the Journal Citation Reports (JCR, Clarivate 2020CLARIVATE. 2020. 2019 Journal Impact Factor. Journal citation Reports.) focusing on the studies published during 2018 (AABC-90), as this gives the opportunity to compare how these articles perform in the subsequent JCR (Clarivate 2021CLARIVATE. 2021. 2020 Journal Impact Factor. Journal citation Reports.), which I will cover here.

In this process, I found out that articles could get missing from the JCR and Web of Science (WoS) databases, which happened with 17 manuscripts published in 2018 that were disregarded in the 2019 JCR (Clarivate 2020CLARIVATE. 2020. 2019 Journal Impact Factor. Journal citation Reports., Kellner 2020aKELLNER AWA. 2020a. Development of Agrarian Sciences at the AABC with comments on impact and performance evaluations. An Acad Bras Cienc 92: e202092S1. DOI 10.1590/0001-3765202092S1., bKELLNER AWA. 2020b. Development of Biological Sciences at the AABC. An Acad Bras Cienc 92: e202092S2. DOI 10.1590/0001-3765202092S2.). This led to the introduction of the Missing Article Index (MAI, see Kellner 2020bKELLNER AWA. 2020b. Development of Biological Sciences at the AABC. An Acad Bras Cienc 92: e202092S2. DOI 10.1590/0001-3765202092S2.), which for AABC-90 was 0.0567 considering the 2019 JCR (Kellner 2020cKELLNER AWA. 2020c. Impact and performances of different scientific fields at the AABC. An Acad Bras Cienc 92: e2020923. DOI 10.1590/0001-37652020923.). In the 2020 JCR, 15 of these 17 articles reappear and their citations were therefore considered for the AABC 2020 IF. I have examined the two remaining articles, both published in the Earth Sciences section (Angulo et al. 2018ANGULO RJ, GIANNINI PCF, SOUZA MC & LESSA GC. 2018. Reply to Castro et al. 2018 on “Holocene paleo-sea level changes along the coast of Rio de Janeiro, southern Brazil”. An Acad Bras Cienc 90: 1377-1380. DOI 10.1590/0001-3765201820180376., Castro et al. 2018CASTRO JWA, SEOANE JCS, CUNHA AM, MALTA JV, OLIVEIRA CA, VAZ SR & SUGUIO K. 2018. Comments to Angulo et al. 2016 on “Sea-level fluctuations and coastal evolution in the state of Rio de Janeiro, southeastern - Brazil” by Castro et al. 2014. An Acad Bras Cienc 90: 1369-1375. DOI 10.1590/0001-3765201820171010.). These papers discuss (and expand!) the results of a specific study published previously, but most likely were considered by Clarivate as special cases of not citable items for the JCR. While the exclusion of such articles may be understandable for JCR purposes, they were peer-reviewed and are not limited to a particular scientific argument, but rather broaden an interesting discussion of sea level variation. In addition, one of the papers (Castro et al. 2018CASTRO JWA, SEOANE JCS, CUNHA AM, MALTA JV, OLIVEIRA CA, VAZ SR & SUGUIO K. 2018. Comments to Angulo et al. 2016 on “Sea-level fluctuations and coastal evolution in the state of Rio de Janeiro, southeastern - Brazil” by Castro et al. 2014. An Acad Bras Cienc 90: 1369-1375. DOI 10.1590/0001-3765201820171010.) received two citations in 2019 and two again in 2020. So far, these two articles have received a total of nine citations (also considering 2021), a number that seems to be increasing. Even if in the AABC 2020 IF calculation their citations may have been considered in the nominator (number of citations), but not included in the denominator (number of articles), there are obvious differences for the IF calculation.

Table I
Citations in 2019 and 2020 of articles published by the Annals of the Brazilian Academy of Sciences in 2018 (AABC-90).

Although I will not analyze the contribution of the volume AABC-91 (all articles published by the AABC during 2019) regarding to the AABC 2020 IF, I would like to point out that, of the 301 articles, the 2020 JCR lists 300, leading to a MAI of 0.0033. This is the second consecutive year that papers published by the AABC get missing and were not made available for the JCR calculation. This situation has occurred previously (Kellner & Azevedo 2013KELLNER AWA & AZEVEDO RA. 2013. A closer look at the Impact Factor (JCR 2012): problems, concerns and actions needed. An Acad Bras Cienc 85: 859-862. DOI: 10.1590/S0001-37652013000300001.), suggesting that missing articles (evaluated by MAI) happens with some frequency and perhaps should be studied in other periodicals to get a sense of how this loss influences journal (and possible author) indexes.

Using the database presented by WoS published by Clarivate (2021)CLARIVATE. 2021. 2020 Journal Impact Factor. Journal citation Reports., I retrieved all items published in AABC-90, which includes articles (298 and the two that were not considered), editorials (4), letter (1), and erratum (2). To have a comparable basis, I have used only the 283 articles considered in the 2019 JCR (Kellner 2020bKELLNER AWA. 2020b. Development of Biological Sciences at the AABC. An Acad Bras Cienc 92: e202092S2. DOI 10.1590/0001-3765202092S2., cKELLNER AWA. 2020c. Impact and performances of different scientific fields at the AABC. An Acad Bras Cienc 92: e2020923. DOI 10.1590/0001-37652020923.). When the number of citations is compared between the datasets of the 2019 JCR and the latest version of WoS (Clarivate 2021CLARIVATE. 2021. 2020 Journal Impact Factor. Journal citation Reports.), differences are noted. A total of 39 papers showed higher citations in WoS: 33 articles received one, five articles two, and one article four more citations than in 2019 JCR (Clarivate 2020CLARIVATE. 2020. 2019 Journal Impact Factor. Journal citation Reports.). This means that 13.78% had fewer citations in the 2019 JCR compared to WoS, resulting in a total of 47 citations that were not considered in the AABC 2019 IF. For the calculation of the AABC 2019 IF, 2019 JCR found 305 citations for the volume AABC-90 (Clarivate 2020CLARIVATE. 2020. 2019 Journal Impact Factor. Journal citation Reports.); if the citations recovered in WoS were added, the AABC 2019 IF (1.280) would have been about 6.9% higher (1.368).

There is a difference between the number of citations for the citable items presented by JCR. The 283 articles of AABC-90 considered in 2019 JCR received a total of 293 citations, 12 less than the official number (305) employed to calculate the 2019 IF (Kellner 2020c). Using the WoS database (Clarivate 2021CLARIVATE. 2021. 2020 Journal Impact Factor. Journal citation Reports.), the four editorials published in AABC-90 received six citations in 2019. Even if the two citations obtained by the article by Castro et al. (2018)CASTRO JWA, SEOANE JCS, CUNHA AM, MALTA JV, OLIVEIRA CA, VAZ SR & SUGUIO K. 2018. Comments to Angulo et al. 2016 on “Sea-level fluctuations and coastal evolution in the state of Rio de Janeiro, southeastern - Brazil” by Castro et al. 2014. An Acad Bras Cienc 90: 1369-1375. DOI 10.1590/0001-3765201820171010. - which does not appear in the 2019 JCR-, would have been included, there are still four “ghost” citations whose origin remains to be established. In the 2020 JCR, the number of citations for items published in the AABC-90 used to calculate the 2020 IF of this journal was 674, differing slightly from the citation arrived by WoS (Clarivate 2021CLARIVATE. 2021. 2020 Journal Impact Factor. Journal citation Reports.). This is somewhat odd, as each time comparisons are made with JCR, WoS always recovers more citations. Again, there may be citations considered by the 2020 JCR whose origin, as in the 2019 JCR, is not clear.

Of the 283 papers published in AABC-90 considered by the 2019 JCR, less than half (134) were cited at least once, leading to an Article Citation Factor (ACF - see Kellner 2020bKELLNER AWA. 2020b. Development of Biological Sciences at the AABC. An Acad Bras Cienc 92: e202092S2. DOI 10.1590/0001-3765202092S2.) of 0.4735 (Table I). Using the WoS database, this number changes to 151 (ACF: 0.5333) for the same year. One year later, the number of articles cited retrieved by WoS has increased (193, ACF: 0.6820), showing that more than two-thirds of the articles published in AABC-90 were cited two years after publication and potentially contributed to the 2020 IF of this journal. If citations of both years (2019 and 2020) are considered, 225 articles from AABC-90 have received at least one citation (ACF: 0.7951). Although this increase is welcome, it should be noted that about 20% of the articles published in AABC-90 did not obtain one single citation during these two years and, therefore, did not contribute to the IF of this journal.

Regarding citations of individual articles, there are also differences in the number of citations between the JCR and WoS databases, being generally higher in the latter. Among the five most cited papers in 2019 JCR, four appear also in WoS of the same year. Regarding the WoS, four were repeated in 2020 with more citations. Just as a note, based on WoS, 31 articles were only cited in 2019 (and not in 2020), 74 only cited in 2020 (and not in 2019), and 122 in both years. In other words, only 43.1% (~4 out of 10) articles published in AABC-90 considered by the 2019 JCR have potentially (if citations were computed by JCR) contributed to the IF of the journal of both years. This number is not very impressive for an open access and free journal, showing that there is still a long way to go to improve citation strategies.

To get a sense how better selection of manuscripts (a subject that, despite being very complex and controversial, tends to increase due to the general lack of funding for publication, particularly in Brazil) might influence the IF of a journal, I have introduced the Reescalated Article Impact Factor (RAIF), which is obtained by dividing the total number of citations by the number of cited articles (Kellner 2020aKELLNER AWA. 2020a. Development of Agrarian Sciences at the AABC with comments on impact and performance evaluations. An Acad Bras Cienc 92: e202092S1. DOI 10.1590/0001-3765202092S1.). The IF essentially doubles in 2019 and triples in 2020 (Table I).

There are some preliminary conclusions that can be drawn from this brief exercise to assess the impact of articles. The most conspicuous one is to express concern about relying too heavily on the IF to infer scientific impact, and perhaps on other scientometric indices either, a caution that has been pointed out more and more in recent years. Not just because of the logic of how the IF is calculated, but also due to the fact that significant problems are being detected. It is at least worrisome to use an evaluation system that expresses the impact and importance of scientific production when articles may simply disappear and not be considered in the IF calculation, which might be happening more than one imagines. The citation discrepancies observed in the JCR and WoS databases, including the fact that papers may disappear in one year and reappear in others, leave an uneasy feeling about how to deal with this information.

If one would like to get an idea of the impact of articles, more than the journal itself where they are published, the best source is not the JCR database, in which the exclusion of articles happens with apparent frequency and seems to be random - you might never know who is the lucky one. Missing articles are only discovered post factum and regardless of what is done, corrections to citations and IF are infrequent (e.g., Kellner & Azevedo 2013KELLNER AWA & AZEVEDO RA. 2013. A closer look at the Impact Factor (JCR 2012): problems, concerns and actions needed. An Acad Bras Cienc 85: 859-862. DOI: 10.1590/S0001-37652013000300001., Kellner 2020cKELLNER AWA. 2020c. Impact and performances of different scientific fields at the AABC. An Acad Bras Cienc 92: e2020923. DOI 10.1590/0001-37652020923.). The database presented by WoS, while also susceptible to miss articles (Kellner 2020bKELLNER AWA. 2020b. Development of Biological Sciences at the AABC. An Acad Bras Cienc 92: e202092S2. DOI 10.1590/0001-3765202092S2.), has at least been corrected and papers (apparently also citations) can be (re)included. It should be noted, however, that it is still unclear how and if the citations added retroactively are indeed correct. There might be some a posteriori changes of values when the WoS database is updated, which seems to happen on an irregular basis. Although these changes are difficult to get around, the information retrieved by WoS apparently seems to better reflect the true number of citations, which does not appear to be the case with the JCR despite the obvious efforts that are being made to evaluate scientific output.

REFERENCES

  • ABAD JCS, ALENCAR RM, MARIMON JR BH, MARIMON B, SILVA ACC, JANCOSKI H, REZENDE RS & ALVES-SILVA E. 2020. Publishing in English is associated with an increase of the impact factor of Brazilian biodiversity journals. An Acad Bras Cienc 92: e20181263. DOI 10.1590/0001-3765202020181263.
  • ALBERTINI MK, BACKES AR & SÁ AL. 2019. A study of publication trajectories of the Brazilian Computer Science community. An Acad Bras Cienc 91: e20180559. DOI 10.1590/0001-3765201920180559.
  • ANGULO RJ, GIANNINI PCF, SOUZA MC & LESSA GC. 2018. Reply to Castro et al. 2018 on “Holocene paleo-sea level changes along the coast of Rio de Janeiro, southern Brazil”. An Acad Bras Cienc 90: 1377-1380. DOI 10.1590/0001-3765201820180376.
  • ATALLAH AN, PUGA MES & AMARAL JLG. 2020. Web of Science Journal Citation Report 2020: the Brazilian contribution to the “Medicine, General & Internal” category of the journal impact factor (JIF) ranking (SCI 2019). Sao Paulo Med J 138: 271-274. DOI 10.1590/1516-3180.2020.138419092020.
  • CASTRO JWA, SEOANE JCS, CUNHA AM, MALTA JV, OLIVEIRA CA, VAZ SR & SUGUIO K. 2018. Comments to Angulo et al. 2016 on “Sea-level fluctuations and coastal evolution in the state of Rio de Janeiro, southeastern - Brazil” by Castro et al. 2014. An Acad Bras Cienc 90: 1369-1375. DOI 10.1590/0001-3765201820171010.
  • CLARIVATE. 2020. 2019 Journal Impact Factor. Journal citation Reports.
  • CLARIVATE. 2021. 2020 Journal Impact Factor. Journal citation Reports.
  • KELLNER AWA. 2020a. Development of Agrarian Sciences at the AABC with comments on impact and performance evaluations. An Acad Bras Cienc 92: e202092S1. DOI 10.1590/0001-3765202092S1.
  • KELLNER AWA. 2020b. Development of Biological Sciences at the AABC. An Acad Bras Cienc 92: e202092S2. DOI 10.1590/0001-3765202092S2.
  • KELLNER AWA. 2020c. Impact and performances of different scientific fields at the AABC. An Acad Bras Cienc 92: e2020923. DOI 10.1590/0001-37652020923.
  • KELLNER AWA & AZEVEDO RA. 2013. A closer look at the Impact Factor (JCR 2012): problems, concerns and actions needed. An Acad Bras Cienc 85: 859-862. DOI: 10.1590/S0001-37652013000300001.
  • Sivertsen G, Rousseau R & Zhang L. 2019. Measuring scientific contributions with modified fractional counting. J Infometr 13: 679-694. DOI 10.1016/j.joi.2019.03.010
  • WÁNG YX ET AL. 2014. Implications of Web of Science journal impact factor for scientific output evaluation in 16 institutions and investigators’ opinion. Quant Imaging Med Surg 4 :453-461. DOI 10.3978/j.issn.2223-4292.2014.11.16.

Publication Dates

  • Publication in this collection
    01 Nov 2021
  • Date of issue
    2021
Academia Brasileira de Ciências Rua Anfilófio de Carvalho, 29, 3º andar, 20030-060 Rio de Janeiro RJ Brasil, Tel: +55 21 3907-8100 - Rio de Janeiro - RJ - Brazil
E-mail: aabc@abc.org.br