Most scientific researchers know the agony of waiting to hear about the status of a submitted manuscript. They are eager to change the phrase “manuscript submitted” on a grant application or curriculum vitae to “in press” in advance of some crucial deadline. Publications in prestigious journals—not necessarily the articles themselves but the fact of their existence—are the established and universal, albeit imperfect, way of claiming credit for the scientific work you’ve done, and there’s always a delay.
But when sociologist Margarita Mooney of the University of North Carolina, Chapel Hill, recently applied for a grant, she was able to take instant credit for one aspect of her work: the readership of her blog, as documented by Google Analytics. When she told the review committee that her team blog, Black, White and Gray, had 15,000 page views in its first month, rising to 20,000 views in later months, they were impressed, she recalls. Blog readership is not a traditional measure of scholarship, but the committee, which was also evaluating public impact, rewarded her for it. She won the grant.
Internet-enabled media analytics have been around for decades, but they are just beginning to spread into academia. As new scientific metrics win wider acceptance, they may help tell a more complete story of researchers’ productivity than do traditional metrics alone.
Short-cited
Measuring science results isn’t new. Librarian Eugene Garfield began publishing indices of scientific journals and citations in the 1950s, and today Thomson Reuters and others carry on the tradition. Citations—the number of peer-reviewed papers that refer to a researcher’s peer-reviewed, published articles—is a handy proxy for the attention granted to a candidate’s work by colleagues. Funding bodies and university promotion committees, among others, use citations to evaluate scientists and the impact of their work.
But citation metrics have shortcomings. For one thing, citations are slow to appear, so the impact of a piece of work can’t really be evaluated until it has been around for a while; the same problem makes it difficult to compare researchers at different stages of their careers. When they finally start to trickle in, citations may not reflect the quality of the work; an article that includes a significant mistake could attract numerous refutations, for example, pumping up the number of citations. Another problem, says cybermetrics researcher Mike Thelwall of the University of Wolverhampton, City Campus, in the United Kingdom, is that citations fail to capture many research outputs, many of which—like data sets contributed to digital repositories—are new.
Alternative metrics
An example is Thomson Reuters’s Data Citation Index (DCI), which allows scientists to cite and take credit for their contributions to data repositories. DCI fills an unmet need, says Joshua Greenberg, program director of digital information technology at the Alfred P. Sloan Foundation in New York City. “We’d like to be able to assess the impact of these data sets, but citation practices aren’t yet standard, and there may well be valuable uses for data that don’t show up in the formal literature.”
But it’s not just new outputs that challenge traditional metrics. It’s also new kinds of attention. As science-specific social media services mature, and as scientists adopt social media tools such as Twitter to disseminate their work, a wide range of new metrics become possible. Sophisticated data-trawling tools can detect and collate a wide variety of different kinds of “citations” of work published online, beyond those listed in the references section of peer-reviewed scholarly articles: links to the article from elsewhere on the Web, tweets that mention it, the number of times it was read, and the number of comments in various Web-based media, from blogs to science-focused community sites.
One advantage that most of these new metrics share is speed. “I think the biggest change is you can get a faster idea of what impact your publications or even your ideas are having,” says computer scientist Paul Groth of Vrije University Amsterdam, who studies scientific metrics.
“Nobody really knows exactly what it means,” says Euan Adie, a founder of Altmetric, which offers new-metrics services to its clients—”but what we’re scoring is attention, it’s not quality. It’s literally how much attention it’s getting online.” Altmetric, Adie says, is not a substitute for peer review. “At this stage it’s more about collecting the data and seeing what’s in there,” he says.
There is, however, a way to inject an assessment of quality into Internet-based metrics. Large institutions such as Harvard University are experimenting with metrics sold by Mendeley, an online research-collaboration platform and academic database. Mendeley’s metrics include how often papers by certain researchers, or in certain journals, are downloaded, shared with colleagues, and commented on. Restricting metrics to platforms frequented by scholars ensures at least some of the credibility of traditional metrics that count only peer-reviewed sources.
Jason Priem, an information scientist at the University of North Carolina, Chapel Hill, and an author of altmetrics: a manifesto, thinks measuring the attention of scientists is the best available tool for measuring quality. “Some people say, ‘I don’t care about popular science; I only care about quality science,’ ” Priem says. “The only measure we have [of science quality] is the consensus of the scientific community. One could call that popularity; one could call it expert consensus.”
Complement, not competition
Research institutions are testing a variety of metrics alongside traditional ones. Thelwall is working on an E.U.-funded project called Acumen to construct a template for a kind of super-CV for researchers, which would include measurements of a researcher’s impact on the Web. In June, the U.S. National Institutes of Health issued a call for improvements in the biographical sketch that it requires grant applicants to submit. A future sketch, the NIH notice suggested, could document a wide range of research outputs including “peer-reviewed publications or other types of scientific output such as data sets, videos, crystal coordinates, patents, licenses, or documented changes to standard medical practice or government policies.” In October, the U.S. National Science Foundation revised its grant proposal guidelines along similar lines. Greenberg, who oversees a $125,000 grant to a project called ImpactStory, which was founded by Priem along with Heather Piwowar, says, “In the near term at least I’d think of these metrics as complementary to traditional metrics, rather than replacements.”
Meanwhile, officials at the University of North Carolina, Chapel Hill, have encouraged Mooney to include in her tenure package both traditional metrics and measures of her Web-based impact—including the measure of page views her blog receives. The university is reviewing its promotion procedures to incorporate more expansive measures of research output, but Mooney doesn’t expect it to abandon traditional metrics.
Alternative metrics organizations and proponents often use the word “story” to describe their final, ideal product, instead of words such as “metric” or “score.” It’s a recognition that even an instant array of numbers will always be more persuasive embedded in a seductive story. “I really like the idea of using a suite of metrics, together with a narrative, to try and persuade the world what you’re doing, why it’s important, and why it’s successful,” Thelwall says.