Get the latest articles and downloads sent to your inbox in a monthly newsletter.

Get the latest articles and downloads sent to your inbox in a monthly newsletter.

Going beyond bibliometric and altmetric counts to understand impact

By Kristi Holmes, Northwestern University | May 27, 2014

Why do we need to talk about research impact?
Research impact and productivity are important considerations for promotion and tenure activities, but their influence and use within the academic community extend far beyond these specific applications. Researchers are increasingly called on to demonstrate the impact of their efforts in a variety of circumstances and settings: 
  • Discover how research findings are being used
  • Identify similar research projects
  • Identify possible collaborators
  • Confirm that research findings were properly attributed/credited
  • Discover community benefit as a result of research findings
  • Compile progress reports 
  • Justify funding requests
  • Quantify and document research impact 
Across research institutions, librarians are increasingly called upon to help researchers meet this need to better understand the greater impacts of research funding toward the advancement of science and (in the case of biomedical research) improved human health. Librarians are responding through traditional approaches, as well as through good old fashioned detective work.
Where do we start? 
We can’t escape it: counting is incredibly important. Papers, citations and grants are all widely recognized, easy-to-count artifacts of the scholarly process. We often find ourselves locked in a cycle around these objects (especially papers and citations), with the typical research workflow seeming to be a linear process. Going from “get grant” to “do research” to “write paper,” we may not take the time to reflect upon the complicated series of loops and relationships that span the research process and ecosystem. 
Publication data is a critical piece of the puzzle, and there is a great deal to be learned about researcher productivity from this information. Citations can facilitate a better understanding of the breadth and depth of dissemination and diffusion of research results globally, and they frequently play a role in individual and institutional ratings and rankings.
Like citation data, the rich publication metadata reveal deeper insights that may not be easily discernible otherwise. Some of the more interesting stories that can be teased from bibliographic metadata include identification of scientific networks in this era of team science, better understanding of funding opportunities and how they are being leveraged, and even a way to gauge the ebb and flow of research trends over time. While publication metadata contribute to the assessment of researcher productivity, alone they cannot tell the whole story. 
It can be tempting to use aggregate counts from various sources (publication data or alternative metric providers) as an easy way to assess value. Whether we look at citation counts or examine attention via social media, we must remember that the numbers are not themselves impact, per se. However, relative numbers that result from attention paid to a particular article, person, codebase, and so on can serve a valuable role, and help us find and more fully characterize and understand impactful efforts.
Going beyond the counts to assess research impact 
A number of alternative indicators and outputs (beyond journal articles) can help us to understand the more meaningful picture of research impact on the micro and macro levels. For example, building upon existing software code, reuse of research datasets, advancements in technology that lead to new devices and instrumentation or new experimental protocols, and even oral testimony which leads to policy change are all examples of going beyond the counts to find the interesting stories beneath the surface. These few examples (and many others!) can yield valuable insights into meaningful impact. By going beyond the counts, it becomes possible to see how research efforts helped to accelerate a discipline or improve human health or change the way society works. 
Where do you find this information? It isn’t always easy, but information can be found in a variety of places. Faculty profiling systems, institutional research information systems and even a researcher’s CV can disclose indicators of meaningful impact. These sources contain a richer representation of one’s scholarly profile than what can be captured by bibliographic metadata alone. Consider the typical curriculum vitae detailing research grants received and new lines of research, conference presentations, committee and society memberships, and awards — all of which can help tell a story. 
Let’s not be limited by these few resources. We should challenge ourselves to look for other good indicators such as policy documents, curriculum materials and guidelines, professional practice guidelines, and grey literature. We should also be open-minded, but maintain focus on issues, motivations, and the particular pathways of dissemination that are important in a particular discipline of interest. 
Specific pathways of dissemination have been identified within the biomedical research community which can facilitate a better understanding of impactful outputs:
  • Advancement of knowledge
  • Clinical implementation
  • Legislation and policy enactment 
  • Economic benefit
  • Community benefit
These and other pathways can help us uncover and characterize new and emerging indicators of impact, including new ways of carrying out diagnoses; improvements in clinical care or the actual physical clinical environment; application of cost-effective interventions that can benefit not only the economy, but the health of the community; and even materials and improvements related to how doctors are educated in the practice of medicine. The definition of these pathways can be easily adapted to cover other disciplines beyond biomedicine.
Figure 1: Website for The Becker Model, a library-based framework for assessing research impact.
The Becker Model
Cathy Sarli and I developed the Becker Model for the Assessment of Research Impact (2). This library-based framework for understanding meaningful research impact serves as a guide for libraries wishing to carry out this type of impact analysis. This evolving framework supplements publication analysis to provide a more robust and comprehensive perspective of biomedical research impact. We furnish tools to support this process, including reporting templates, glossaries, information about indicators, and suggested readings to help put things into context. The Model also provides guidance for quantifying and documenting research impact and suggested resources for tracking the diffusion of research outputs and activities and locating the evidence of impact. The framework can be applied at the micro (item or individual), meso (group or center), or macro (institution or greater) levels.  
An important aspect of our work is providing our research community with strategies to enhance the impact of research. These strategies focus on three areas:
  • Preparing for publication
  • Dissemination
  • Keeping track of your research
Repetition, consistency and an awareness of the intended audience underlie these strategies, and they can be incorporated into workshops to help researchers understand practical steps that they can take to maximize their impact.
How do we operationalize the process using research networking and discovery systems?  
Research information systems can play a big role in helping to operationalize this process. Librarians are heavily involved in these types of projects on campuses around the world — from data acquisition and clean up, to development and implementation work, to outreach, education, and ontology development efforts. 
Research networking and discovery systems (RNS), like VIVO, Profiles, and SciVal Experts, showcase achievements and expertise. They can be leveraged to uncover research impact across multiple sites en masse in an automated fashion by making VIVO-compatible data publicly available as linked open data, as recommended by the Clinical and Translational Science Award Consortium (3). The standard VIVO-ISF ontology (4) allows a wide variety of alternative research outputs and other information to be represented in the RNS in a platform-agnostic manner. Furthermore, the machine-readable data help to disseminate this information, as the results from these types of profiles tend to rank very highly on Google and other search engines.
The individual researcher profile page can also be an important discovery space for representing information about a person and can be enhanced by incorporating social media data from well-known and reputable services such as Plum Analytics and Altmetric. This altmetric data can be easily displayed on the individual researcher’s page in the form of a widget. Moreover, these services can leverage the richly structured data afforded by the RNS for information and visualizations. These counts aren’t impact, but they do allow a researcher to track and display the attention that s/he is receiving across a variety of social medial platforms and information services to augment content and value. The counts can also help raise awareness of the use and dissemination of research outputs. 
Perhaps, most importantly, not only do RNS provide visibility, they give the institution a way to accomplish higher-order tasks such as comply with open access mandates, leverage the data at the enterprise-level for benchmarking, perform analyses, and facilitate strategic planning efforts.
Figure 2: PlumX/VIVO profile of Kristi Holmes
Tying it all together
Going beyond gathering altmetric data to assessing research or researcher impact requires a sophisticated set of tools and expertise and a huge amount of tenacity, but the benefits are enormous. Libraries have a real role to play in this space and more libraries are working in this area every day. This work really reflects some significant strengths of libraries, namely our personnel and resources, and also our understanding of scholarly practices, long-standing tradition of service to all, and a commitment to privacy. Providing services and support in this area of assessing and understanding research impact can be a terrific launch pad for libraries to be involved in critical strategic efforts on campus as we begin to understand and appreciate the true impact of a single paper, individual, research center or an entire institution. 

This article is based on Kristi Holmes' Library Connect webinar presentation Going beyond counts: understanding impact.


1. Wells R., Whitworth A. 2007. Assessing outcomes of health and medical research: do we measure what counts or count what we can measure? Australia and New Zealand Health Policy, 4:14

Creative Commons License
Going beyond bibliometric and altmetric counts to understand impact by Kristi Holmes is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
Permissions beyond the scope of this license may be available at