- How do we apply article-level download counts to social media use?
- What are, and how do we measure the impact of, nontraditional scholarly outputs like datasets or software?
- How do we quantify the use or application of software?
Exploring common ground for alternative metrics
The National Information Standards Organization (NISO) is exploring a framework for alternative metrics (altmetrics). A US-based nonprofit with global members, NISO is a standards-setting body in the community of libraries, publishers and automation vendors.
How does NISO differentiate between a recommended practice and a standard?
In the language of standards there are a lot of “musts” and “shalls.” Recommended practices focus more along the lines of what community members “should” or “may” do, so they are less prescriptive. Because of this flexibility, recommended practices provide an opportunity to advance consensus in technology areas that are still developing. They are also meant as a starting point for something that could become more formal as practices develop.
NISO has a long history of being involved in assessment. One of NISO’s oldest standards, Z39.7 - Information Services and Use Metrics & Statistics for Libraries and Information Providers – Data Dictionary, dates back to the 1950s. It is a data dictionary that describes activities that can be measured for assessment purposes within a library. Over the last 15 years, we have been involved in the COUNTER and MESUR — a project led by Johan Bollen at Indiana University and Herbert Van de Sompel at Los Alamos National Laboratory — initiatives. For Project COUNTER, NISO maintains the data model and has supported the exchange of COUNTER data with the SUSHI standard. Within the MESUR initiative, NISO has supported exploration of new forms of measuring network behavior.
During the 2012 “unconference” Altmetrics 12, I asked the participants whether anyone would like to talk about standards around altmetrics. It turned out to be one of the largest discussion groups, and one of the action items was to continue the conversation. NISO formed an advisory steering committee and invited individuals who were at the meeting and having conversations around altmetrics to participate. Then we asked them to reach out to their communities for increased participation and input.
We approached the Alfred P. Sloan Foundation, which was involved in funding Altmetrics 12, with an idea about developing community consensus around alternative assessment metrics. Fortunately the Sloan Foundation decided to fund this work and last June we received a grant to undertake a two-phase project. We are just wrapping up Phase 1.
What did Phase 1 comprise?
Phase 1 was a brainstorming initiative consisting of three in-person meetings: San Francisco in October, Washington, DC in December, and in Philadelphia at the ALA Midwinter Meeting. We also conducted 30 one-on-one interviews.
The goal of the first phase was to get community input about what is necessary to create an environment where alternative metrics can be successful. Questions around altmetrics abound, including:
After gathering this input, we compiled a list of 9 major themes and 25 potential action items, culled out of more than 200. These have recently been made public in the Altmetrics White Paper Draft released on June 9, available on the NISO Altmetrics Project web page.
We are now asking for public input and feedback, comments and criticisms: “How come you have forgotten this?” “This sounds like a great idea, do this.” And we are interested in this input from a worldwide audience. The commenting feature will be available until July 18, 2014.
Are there any action items you are particularly interested in pursuing?
To be honest, there are. But I absolutely don’t want to prejudge the interests of the community. One of the challenges of being the head of NISO is people often ask me what I think. While I have my own opinions, it’s more important to get the opinions of the community. I’m not making funding decisions based on someone’s portfolio and what might that person need, or what might a dean need when reviewing for promotion and tenure. These use cases are much more important than what I think.
As a group, we have talked a lot about use cases. For example, have you shopped on Amazon? At the bottom of the page it says, “People who viewed this, also viewed this.” That’s an application of altmetrics. It’s tracking network behavior to expose what the system thinks you might be interested in based on your behavior. We could do similar things with library data to expose content to subsequent patrons.
What are you seeing in the library community? Is there widespread adoption of altmetrics?
First let me say that I’m not sure the term “altmetrics” is such a good thing; however, I think for now we are stuck with it. One day these will all be considered metrics … period.
So it’s a more difficult question than it appears as it gets to what is the definition of a traditional metric and an altmetric. Strictly speaking, a traditional metric would be citation use, citation tracking and the impact factor. People use that data all the time. Download counts and COUNTER reports are relatively new, but I would say most librarians are using them.
If we take a step further and look at article-level data instead of journal-level data, at what point does it start becoming commonplace? I think if librarians had access to article-level data, they would use it. I’m sure researchers would use it. Those publishers that have begun including more robust and granular data have had a positive response. On a broader scale, I think we’ll continue to move further along the lines of providing a suite of potential metrics; there is never going to be one number to rule them all. It’s all about context.
What happens next with the initiative?
Phase 2 involves a community effort to prioritize these 25 different potential action items since NISO won’t be able to take on 25 new projects. We want the community to tell us which ones they think which would make the most difference. With this input, the NISO leadership committee, the Business Information topic committee that oversees this work, and the NISO membership will decide which projects NISO moves forward with.
NISO will likely decide on three to five potential projects and move forward with them on the continued funding for Phase 2 from the Sloan Foundation. We anticipate by the end of 2015 or mid-2016 we will have some developed some recommended practices or standards.
We also might hand some of these project ideas off to another organization, such as a library or publisher association, for them to pursue. NISO is certainly not the only organization working in this space. We are always looking for opportunities to partner with and collaborate with other related organizations.
Are all the topics NISO looks at as broad as this?
This is bigger than most because it does touch on so many different things, but I would say one challenge NISO faces is that we touch on so many diverse communities. We are simultaneously at 30,000 feet and the microscopic level. For example, when you talk about metadata around ISBN numbers or research objects, you get nitty gritty very quickly. But if you talk about the broad application of DOIs across all digital objects, the discussion gets very big very quickly. It certainly provides an opportunity to engage at all levels of the publishing and scholarly communications chain.