Why don’t negative results get published?

On a recent AskMe thread discussing a Science article on gender and collective intelligence, someone commented:

I read an article not too long ago about how studies that find fewer/no gender differences are significantly less likely to be published, and are often actively discouraged from publication. I thought I’d saved it, but I didn’t. Anyone know what I’m talking about?

Well, I don’t know the specific article, but there’s little doubt that this is true throughout science. Publishing negative results just doesn’t happen very often. Historically, I suppose there were reasons for this. As I’m banging my head against a problem, I may try 10 different approaches before finding one that works well. If each of those failures was a paper or even a paragraph, it would have made old paper journal subscriptions rather unwieldy and much less useful.

Now that everything is online, though, a handful of scientists are starting to stand up and say “hey, we should be announcing our failures as well, so others aren’t doomed to make the same mistakes”. In my opinion, these people have an excellent point.

So there are two major ways that this can come about. The first is to be encourage more openness when publishing papers. In the methods, or at least the supplement, authors could include a decent description of what techniques turned out not to be helpful and why they might have failed. This isn’t common practice now, mostly because for reasons of communication and reputation. Journal articles are always written as though the experiments were a nice, linear process. We did A, then B, then got result C. This isn’t a very accurate description of the process, and everyone knows it, but it makes those involved look smart. (I suppose if you’re clawing your way towards tenure or angling to land a good post-doc position, you don’t necessarily want to broadcast your failures). The more valid claim is that writing articles this way makes for a nice, easy to communicate story. Still, there’s no reason why more comprehensive supplements shouldn’t be added.

The second way to better announce negative results is to practice open-notebook science, where methods and raw data are published to the web in real time (or after a short delay). What’s holding this back is that scientists worry that by revealing too much, their competitors will get a leg up and publish the next big paper before they can. In this era of crushingly low paylines, where less than 20% of NIH grant applications get funded, things have gotten pretty cutthroat. Stories of being “scooped” abound, and although some people feel that these claims are exaggerated, it can happen, sometimes with career-tanking results.

So to make a long story short, no, negative results aren’t often published, even though doing so would probably be a boon to scientific enterprise as a whole. The good news is that there’s a pretty strong movement underway that is slowly making science more open, transparent, and reproducible.

Scientific Software

Two fairly interesting articles about scientific software appeared in Nature News this week. The first is an exhortation to scientists that can be summed up as: “Your code is good enough – publish it already!” (sounds familiar)

The second is a nice piece by Zeeya Merali warning about the dangers that lurk in scientific programs. The article gives a few examples of papers that had to be retracted because of buggy software, and then does a decent job of summing up many of the problems: untested code, poor documentation, and journals that don’t require code release along with publications.

While she talks about some potential solutions, including openness, better training, and collaborations with trained computer scientists, I feel like she glosses over a crucial point: The reason that we don’t have better scientific software is because it isn’t well-incentivized by the scientific community.

Grants are awarded for work on sexy diseases, not for reliable and robust software engineering. Don’t get me wrong – efforts like BioPerl and Bioconductor are fantastic community resources, but I’d argue that they’re examples of how good things can be despite a lack of systematic support. Allocating serious funding to groups that can produce platforms of solid, well-tested bioinformatics code would go a long way towards helping data science keep pace with the deluge of biological data that’s surging towards us.