Alternative Measures of Scholarly Impact are Increasingly Adopted by Funders and Publishers

Pablo Markin
Published Online: 2017-07-29

While journal impact factor metrics have been and continue to be used to assess the quality of publications that scholars publish, it appears that the primarily digital format in which most scholarly articles are published and the attendant article-level data that can be retrieved via the Internet can make it possible to devise article-level impact measures. A relatively recent example of this is the Relative Citation Ratio (RCR) that is calculated by Public Library of Science (PLoS) for the National Institute of Health, the United States, in the domain of medical research. The supporters of this article-level metric argue that they can increase the visibility of high-quality publications regardless of the impact factor ranking that the journals in which they appear have, as the chart below illustrates. Consequently, this can also assist emerging scientific journals, such as in developing countries, to improve their reputation, even when they operate with limited financial support.

Journal impact factor vs citation counts, June 22, 2017 | © Courtesy of Erwin Irawan.

Journal impact factor vs citation counts, June 22, 2017 | © Courtesy of Erwin Irawan.

Readily available computing power allows the application of the RCR based on the ratio between the target article citation rate and that of subsequent articles that cite it, which arguably permits controlling for the citation rates specific to particular scientific fields, while enabling cross-field comparability of this metric. A recent Open Access (OA) article that had inquired into the performance of the RCR as an alternative metric vis-à-vis expert opinions has not found significant differences between these, which indicates that journal-level metrics can serve as fine-grained and relatively adequate measures of the academic quality that published articles have as compared to journal-level metrics that may fail to capture the possibly variable quality of the articles that scholarly journals publish. Though the algorithms behind the RCR are considered to be more complex than more traditional impact metrics, both these procedures and underlying data are made freely accessible to the general public. While the developers and investigators of the RCR are careful to qualify the discriminating power of this metric for the assessment of article-level impact, it is an important step in the direction of deploying multiple alternative influence measures in the field of science.

By Pablo Markin

The original  blog article and additional resources can be found at OpenScience blog.