EDF 600: Applied Research in Education and Behavioral Sciences

Introduction to Research Impact 

Research impact is loosely defined as how broadly scholarly research is being read, discussed, and used. This guide is designed to help researchers and students understand how research impact is currently measured, outline freely available resources for assessing impact, and what improvements to the current systems of measurements have been proposed.

While all of these tools can be revealing in helpful ways, they are also susceptible to misinterpretation and misuse. No one metric or tracking tool is best and not all of them are equally useful in all disciplines. However, researchers can explore options availables, the pros and cons of each, how to assess them critically, and how to use them for the benefit of their own careers and the careers of younger researchers they are mentoring.

"How Standards Proliferate" by xkcd is licensed under CC BY 4.0 

Because there are so many metrics and with notable differences between them, remember to include the following when citing them:

• the name of the metric you are using (not just the score)
• the source of the metric
• the date of the metric 

👍 Rule of thumb: The higher the score reported for a metric, the better. However, score ranges can vary by metric type and discipline.

Adapted from "Understanding Research Impact" by Duke University Libraries licensed under CC-BY-NC-SA.

Journal Metrics

Impact Factor

Impact factor is a measurement of the citation rate of a particular journal, individuals do not have an impact factor, publications do. Impact factor takes into account citations of the previous two years of published articles. Because citation practices vary widely between disciplines, impact factors also vary. While the number could differ depending on the tool used to measure the number of citations, Thompson Reuter's Journal Citation Reports tool is the general industry standard for measuring impact factor.

• Use Thomson Reuter's Journal Citation Reports (if your library subscribes to the database, UM does not.)
• Look for impact factor information on journal websites. Some, like The Journal of Educational Psychology, feature it. [1]

CiteScore

A journal's CiteScore is the total number of citations in a year to articles published in the four previous years, divided  by the total number of articles published in those four years. CiteScore is also limited to only 5 peer-reviewed document types (articles, reviews, conference papers, data papers, and book chapters). CiteScore is similiar to the Impact Factor but uses Scopus rather than Web of Science to gather its data and four years rather than two as the publication period.

While Scopus is a subscription product, you do not have to have a subscription to access the CiteScore and related metrics on the Scopus site. [2]

• Go to the Sources page of the Scopus website.
• Use the toggle at the top of the screen to search by Subject, Title, Publisher, or ISSN.
• The results default to the latest reporting year (2022), but you can change this by finding the drop-down by "View Metrics for Year" and selecting a different year.
• This Scopus report also includes the SJR and SNIP metrics for a journal.

SCImago Journal Rank (SJR)

SCImago is an open database that includes it's own journal metric, called the SJR (SCImago Journal Rank). This database uses Scopus data to derive its rankings and classifies journals using the same categories as Scopus. The SJR is freely accessible at SCImago's website or on the Scopus Sources page. It is weighted by the prestige of the journals from which the citations are derived.

SNIP (Source Normalized Impact per Paper)

Source Normalized Impact per Paper measures contextual citation impact by weighting ciations based on  the total number of citations in a subject field. The impact of a single citation is given higher value in subject areas where citations are less likely, and vice versa. Unlike the journal impact factor, SNIP corrects for differences in citation practices between scientific fields, thereby allowing for more accurate between-field comparisons for citation impact. 

SNIP was created by Professor Henk F. Moed at Centre for Science and Technology Studies (CWTS), University of Leiden. SNIP values are available from CWTS Journal indicators or in the SCOPUS database.

CWTS Journal Indicators currently provides four indicators:

• P: The number of publications of a source in the past three years.
• IPP: The impact per publication, calculated as the number of citations given in the present year to publications in the past three years divided by the number of publications in teh past three years. IPP is fairly similar to the well-known journal impact factor. Like the journal impact factor, IPP does not correct for differences in citation practices between scientific fields. IPP was previously known as RIP (raw impact per publication.
• SNIP: The source normalized impact per publication, calculated as the number of citations given in the present year to publications in the past three years divided by the total number of publications in the past three years. The difference with IPP is that in the case of SNIP citations are normalized in order to correct for differences in citation practices between scientific fields. Essentially, the longer the reference list of citing publication, the lower the value of a citation originating from that publication. (The SNIP score also reported on the open Scopus Sources page. [2]

Resurchify Impact Score (IS/JIS)

Resurchify is another open database that produces an alternative journal metric, called the impact score (IS) or journal impact score (JIS). It is an annual measure of the average citations of articles that have been recently published in a journal. Scopus data is also used for this metric.

Eigenfactor 

The Eigenfactor is similar to impact factor in that it attempts to measure "total importance" of a journal. It considers citations five years from the year of publication (more than the two of JCR's impact factor). It also weighs citations from more influential journals higher than those from less influential journals. The eigenfactors of all the journals in the eigenfactor index are scaled to a sum of 100, so a journal with 1.00 eigenfactor has 1% of the total importance of all the journals in the index. [1]

Example: The Journal of Educational Psychology

• EFn (normalized Eigenfactor Score) = 1.4 - The journal has 1.4 times more influence than the average JCR (Journal Citations Report) article
• AI (article influence score) = 2.2 - The average article in the journal has 2.2 times the influence of the mean JCR article (*an alternative to the JCR's Impact Score) 

H-index for Journals

As an alternative to impact factor, some databases and services report a h-index for journals. The h-index is the number (h) in which at least h articles in a publication were cited at least h times. For a journal with an h-index of 15 , this means that it had 15 articles that were cited at least 15 times.

• SCImago includes reports h-index along with the SJR for individual journals based on Scopus data.
• Google Scholar uses a slightly modified version, the h-5 index, which only includes works published within the last 5 years. To view this, go to the Top Publications page in Google Metrics (found in the Google Scholar menu). Then enter a journal title in the search field at the top of the page.

To see metrics for the journals in this guide, visit this page.

1. Adapted from "Understanding Research Impact" by Duke University Libraries licensed under CC-BY-NC-SA.

2. Adapted from "Journal Impact" by University of Arizona Libraries licensed under CC-BY 4.0.

Article Metrics

Cited By: Google Scholar and PubMed

The library does not subscribe to Scopus or Web of Science. Therefore, the best place to access article level metrics is Google Scholar. To view this metric, look underneath the article citation for "Cited by" which provides a number of times the article has been cited in other works. If you click this link, you will be taken a list of "Cited by" articles.

Also, PubMed has a "cited by" feature built in to their search results. In the search results, click on a title, then scroll to find the "Cited by" section.

Altmetrics

Article-level metrics: Instead of attempting to measure journals or individuals, article-level measure the impact of individual articles. PLoS pioneered this approach, and they measure usage (page views, downloads), citations (using Scopus, Web of Science, PMC, etc. data), and social networking mentions. Other publishers—primarily big, commercial, scientific publishers—have begun to use article-level metrics as well, often utilizing the Altmetric.com donut on their publisher site. Other publishers and institutional respositories use another provider, PlumX. These new article-level “altmetrics” make it easier to see the context of the impact. In other words, they show not only how many times an article was cited, but who was doing the citing (including in places other than scholarly literature) and what they said. Some of these metrics can indicate how the research is being used (i.e. how many times it has been downloaded, from where and by whom, and whether they are saving it for their own future reference, whether libraries are buying a copy of a book, and other indicators of value and impact). 

• PLoS: Article
• Springer (Altmetric): Article | Altmetric page 
• Alabama Libraries (Journal in University of South Alabama Institutional Repository): Article | PlumX 

Adapted from "Understanding Research Impact" by Duke University Libraries licensed under CC-BY-NC-SA.

Author Metrics

The h-index is a measurement of an individual researcher's impact. The index takes into consideration the number of citations of a researcher's publications (i.e. a researcher with with an index of h has published h papers which have been cited at least h times). The h-index was proposed in 2005 by physicist Jorge Hirsh and is alternatively called the "Hirsch index" or "Hirsch number". A number of services calculate h-index and the number can vary depending on the service used. Both Scopus and Google Scholar report h-index metrics.

Remember the h-index =  h # of articles that have been cited at least h number of times

                                          8 h-index = 8 articles published at least 8 times

Scopus Author Search 

While UM does not subscribe to Scopus, you can access a limited version of their Author Search page. Search by name or name and affliation (university or institution). This page provides an author's h-index as well as document and citation trends and article listing. (Example: Marshall Breeding's page).

Google Scholar 

Google Scholar has author metrics for all authors who have created an author profile. In addition to the h-index, Google Scholar reports the i10 index (# of author publications with a minimum of 10 citations). You can access an author's profile and accompanying metrics one of two ways. (Example: Marshall Breedings' page)

• Look under an citation in your Google Search result screen for the list of authors. If an author's name is linked, then a Google author profile exists. Click on the link to access the profile.
• Enter an author's name in the search field, look through the search results to find an article with the author's name linked. Click to visit the author page.
• The citation page is searchable. Look for the search field at the top and enter an author's name.

Adapted from "Understanding Research Impact" by Duke University Libraries licensed under CC-BY-NC-SA.

Book Citations

Book citations are much less mature that any kind of metrics of journal articles. Thomason Reuters offers a book citation tool, but it is less standardized than the measurement of the impact of journal articles. Researchers may use Google Scholar, Web of Science, or Scopus, considering them reliable, but they are not often used to assess impact in book-heavy disciplines.

Adapted from "Understanding Research Impact" by Duke University Libraries licensed under CC-BY-NC-SA.