Why don’t negative results get published?

On a recent AskMe thread discussing a Science article on gender and collective intelligence, someone commented:

I read an article not too long ago about how studies that find fewer/no gender differences are significantly less likely to be published, and are often actively discouraged from publication. I thought I’d saved it, but I didn’t. Anyone know what I’m talking about?

Well, I don’t know the specific article, but there’s little doubt that this is true throughout science. Publishing negative results just doesn’t happen very often. Historically, I suppose there were reasons for this. As I’m banging my head against a problem, I may try 10 different approaches before finding one that works well. If each of those failures was a paper or even a paragraph, it would have made old paper journal subscriptions rather unwieldy and much less useful.

Now that everything is online, though, a handful of scientists are starting to stand up and say “hey, we should be announcing our failures as well, so others aren’t doomed to make the same mistakes”. In my opinion, these people have an excellent point.

So there are two major ways that this can come about. The first is to be encourage more openness when publishing papers. In the methods, or at least the supplement, authors could include a decent description of what techniques turned out not to be helpful and why they might have failed. This isn’t common practice now, mostly because for reasons of communication and reputation. Journal articles are always written as though the experiments were a nice, linear process. We did A, then B, then got result C. This isn’t a very accurate description of the process, and everyone knows it, but it makes those involved look smart. (I suppose if you’re clawing your way towards tenure or angling to land a good post-doc position, you don’t necessarily want to broadcast your failures). The more valid claim is that writing articles this way makes for a nice, easy to communicate story. Still, there’s no reason why more comprehensive supplements shouldn’t be added.

The second way to better announce negative results is to practice open-notebook science, where methods and raw data are published to the web in real time (or after a short delay). What’s holding this back is that scientists worry that by revealing too much, their competitors will get a leg up and publish the next big paper before they can. In this era of crushingly low paylines, where less than 20% of NIH grant applications get funded, things have gotten pretty cutthroat. Stories of being “scooped” abound, and although some people feel that these claims are exaggerated, it can happen, sometimes with career-tanking results.

So to make a long story short, no, negative results aren’t often published, even though doing so would probably be a boon to scientific enterprise as a whole. The good news is that there’s a pretty strong movement underway that is slowly making science more open, transparent, and reproducible.

The problem of prioritization

Dan Kobolt just threw up a great post talking about sifting through the hundreds of mutations that we’re finding in each genome to find those that actually, y’know, mean something.

we are facing two significant challenges. First, identifying the subset of variants have functional significance – separating the wheat from the chaff, if you will. Second, understanding how these functional variants contribute to a phenotype. This is soon to be the frontier in genetics and genomics.

I couldn’t agree more, and since my comment on his post got to be a little longer than I intended, I decided to reproduce it (edited slightly) over here.

I’ve been tackling similar ideas as part of my thesis work. Specifically, we’ve been developing tools that go beyond simple recurrence and look at mutational patterns that can give insight into the significance and functional role of mutations.

The easiest one to think about is mutual exclusivity. If I have part of an oncogenic pathway with two genes (A and B), then we expect that mutations in either one may be enough to disrupt the system, and there will be no selective pressure for mutation in the other. So if we assay a panel of tumors and see that half the tumors have a mutation in gene A, and the other half have a mutation in gene B, with no overlap, it’s quite likely that the mutations play similar functional roles. By detecting these patterns, we can create testable hypotheses about how genes interact, even if they’re not represented in functional databases.

It’s also important to remember that pathways can be disrupted in multiple ways. Exome sequencing to find point mutations may not be enough, as we know that copy-number alterations may lead to altered expression levels, or aberrant methylation may cause dysregulation. A integrative approach is going to be key as we move forward.

So my point is, yeah, there are absolutely people working on improving this process and doing a better job of prioritizing these mutations for in vivo validation. It’s an exciting place to be working right now, as it’s a major bottleneck preventing us from translating ubiquitous sequencing into personalized medicine. I’m glad to be working here in the thick of it.

Scientific Software

Two fairly interesting articles about scientific software appeared in Nature News this week. The first is an exhortation to scientists that can be summed up as: “Your code is good enough – publish it already!” (sounds familiar)

The second is a nice piece by Zeeya Merali warning about the dangers that lurk in scientific programs. The article gives a few examples of papers that had to be retracted because of buggy software, and then does a decent job of summing up many of the problems: untested code, poor documentation, and journals that don’t require code release along with publications.

While she talks about some potential solutions, including openness, better training, and collaborations with trained computer scientists, I feel like she glosses over a crucial point: The reason that we don’t have better scientific software is because it isn’t well-incentivized by the scientific community.

Grants are awarded for work on sexy diseases, not for reliable and robust software engineering. Don’t get me wrong – efforts like BioPerl and Bioconductor are fantastic community resources, but I’d argue that they’re examples of how good things can be despite a lack of systematic support. Allocating serious funding to groups that can produce platforms of solid, well-tested bioinformatics code would go a long way towards helping data science keep pace with the deluge of biological data that’s surging towards us.

Bug Report

This is a moth that shorted out a relay in one of the world’s first computers.

An professor of mine once flashed this image up on the screen and told us that when a moth causes problems, it’s a bug. Those mistakes in your program? Those are errors, and they’re your fault.

image via the Wikimedia commons

Dumping my code: the good, the bad, and the ugly

Writing code for use in biological research is a lot different from writing code for a software company. First of all, 99% of the time, I’m the only person who will ever see it. Lots of my work involves one-off scripts to parse data that I’ll never deal with again. Furthermore, cleaning up code and testing it prior to release is hard, time-consuming work. If it’s not moving me towards meaningful results or a publication, there just isn’t much incentive to do it.

The irony of all this is that reproducibility is one of the foundations of science, and better code release would go a long way towards making that a reality in today’s high-throughput world. Furthermore, without great open-source projects like R/Bioconductor and BioPerl, bioinformatics would just be crawling along as everyone wasted valuable time trying to build everything up from foundations.

All of this had been floating around in the back of my mind for some time when I stumbled across Matt Might’s post describing the CRAPL. He proposes that we stop worrying about quality and release even our ugly, hacky, code, provided that it works. The CRAPL license itself is a parody of more established licenses and contains gems like these:

- The Author” probably refers to the caffeine-addled graduate student that got the Program to work moments before a submission deadline.

. . .

- You agree to hold the Author free from shame, embarrassment or ridicule for any hacks, kludges or leaps of faith found within the Program.

So in the spirit of the CRAPL, I’m starting to dump some of the useful code snippets I’ve written over the years in various places. I’ll package up a few of the larger ones and post them on my GitHub page, but most of the little one-off scripts will just be slapped up here on the blog, under the codedump tag.

I’ll just go ahead and throw it all under the MIT License so you can use it and abuse it any way you want. If I can save someone else a few hours of work, it’s worth it. Happy Hacking!

Microbial Medicine

Today Jonathan Eisen makes some good points about probiotics being sold like snake oil miracle cures.

I’ll agree that it’s a growing problem. I often find people on message boards advocating probiotic treatments, despite the lack of evidence-based support for such actions. I always tell them the same thing: Popping some Lactobacillus acidophilus isn’t likely to hurt, but there’s little proof that it will help you, either.

Lactobacillus acidophilus

Someone was even asking about fecal transplant bacteriotherapy the other day, after reading Carl Zimmer’s excellent article about how it can treat severe Clostridium difficile.

Look, I totally understand why people grasp at straws and search for anything that might help. Living with a chronic condition that has no cure sucks. Wasting your money on treatments that don’t help, though, will only make you lighter in the wallet and still sick. The problem with probiotics in this case are two-fold:

The first problem is that we just don’t understand the composition of the gut well enough to start trying to theraputically change the composition of your microflora. It’s an enormously complex ecosystem, and we’re just now dipping our toes into the water with initiatives like the Human Microbiome Project. Scientists are studying this with a great deal of interest though. In fact, he other day I stumbled across a study showing that by treating rats with antibiotics, then doing fecal transplant, a team could create long-term alterations in the microbiota of rats. This is pretty exciting stuff, but it’s only the first of many advances that will be needed to prove the feasibility of bacteriotherapy. The next steps will be to figure out which microbes we want in our guts, which we don’t, and how we can create a stable environment for those beneficial species. This will undoubtedly take a while.

The other problem with using probiotics to treat gut conditions is that most of these disorders (IBS/IBD/Crohn’s) have both genetic and environmental factors that contribute to the aetiology. I’m most familiar with Crohn’s Disease, which occurs when cells in the intestinal walls aren’t able to keep microbes out properly, so the immune system kicks into overdrive. So far, it hasn’t been linked to any specific species, which seems to make it a poor target for microbial therapy. Add in the fact that there are often systemic inflammatory problems like arthritis and back problems, and it’s not at all clear that we can treat this by just swapping around the microbes in the gut.

So to sum things up, using probiotics to seriously treat disease will have to wait until we better understand both the diseases in question and host/microbe interactions. That said, research into the microbiome is enormously promising, and I do believe that microbial medicine will someday be commonplace. It’s just that we still have a long way to go.

image source

Transparent histograms redux

The other day, I posted an R function that simulates transparency where histograms overlap. In the comments, Alice left this tip:

You could also use transparent colors, which could mean less programming effort:
e.g. col=rgb(0, 1, 0,0.5)) ..the 4th argument is the degree of transparency alpha

*smacks forehead* That’s a lot simpler than my solution. I didn’t know R supported transparency like that. So instead of using the function I wrote, I could have just done something like this:

a=rnorm(1000, 3, 1)
b=rnorm(1000, 6, 1)
hist(a, xlim=c(0,10), col="red")
hist(b, add=T, col=rgb(0, 1, 0, 0.5)

And gotten the following output:

Much simpler.

A friendly reminder

Correlation != Causation

Heilmeier’s Catechism

George Heilmer (somewhat) famously proposed a list of questions that you should be able to answer before starting any new research or development project:

What are you trying to do? Articulate your objectives using absolutely no jargon.

How is it done today, and what are the limits of current practice?

What’s new in your approach and why do you think it will be successful?

Who cares?

If you’re successful, what difference will it make?

What are the risks and the payoffs?

How much will it cost?

How long will it take?

What are the midterm and final “exams” to check for success?

Every second year graduate student should have a worksheet containing this list as they prepare to propose their thesis project. It’s equally applicable to grant applications, product design in industry – damn near everything you do in science really.

quoted from Wikipedia, after being mentioned by Titus Brown

Transparent overlapping histograms in R

Today I wanted to compare the histograms from two data sets, but it’s hard to see the differences when the plots overlap. Here’s an example of the problem:

a = rnorm(10000,5)
b = rnorm(10000,3)
hist(a,xlim=c(0,10))
hist(b,col="gray20",add=T)

Output:

I want to know what’s going on behind all that solid grey!

So I coded up a little function to add pseudo-transparency to pairs of histograms:

plotOverlappingHist <- function(a, b, colors=c("white","gray20","gray50"),
                                breaks=NULL, xlim=NULL, ylim=NULL){

  ahist=NULL
  bhist=NULL

  if(!(is.null(breaks))){
    ahist=hist(a,breaks=breaks,plot=F)
    bhist=hist(b,breaks=breaks,plot=F)
  } else {
    ahist=hist(a,plot=F)
    bhist=hist(b,plot=F)

    dist = ahist$breaks[2]-ahist$breaks[1]
    breaks = seq(min(ahist$breaks,bhist$breaks),max(ahist$breaks,bhist$breaks),dist)

    ahist=hist(a,breaks=breaks,plot=F)
    bhist=hist(b,breaks=breaks,plot=F)
  }

  if(is.null(xlim)){
    xlim = c(min(ahist$breaks,bhist$breaks),max(ahist$breaks,bhist$breaks))
  }

  if(is.null(ylim)){
    ylim = c(0,max(ahist$counts,bhist$counts))
  }

  overlap = ahist
  for(i in 1:length(overlap$counts)){
    if(ahist$counts[i] > 0 & bhist$counts[i] > 0){
      overlap$counts[i] = min(ahist$counts[i],bhist$counts[i])
    } else {
      overlap$counts[i] = 0
    }
  }

  plot(ahist, xlim=xlim, ylim=ylim, col=colors[1])
  plot(bhist, xlim=xlim, ylim=ylim, col=colors[2], add=T)
  plot(overlap, xlim=xlim, ylim=ylim, col=colors[3], add=T)
}

The results are much easier to interpret:

a = rnorm(10000,5)
b = rnorm(10000,3)
plotOverlappingHist(a,b)

I’m sure this could be improved upon and generalized to multiple histograms, but this solves my problem, so I’m calling it quits for now. Let me know if you make improvements or find it useful.