Cleaning up after yourself

August 27th, 2015

Whoops! It turns out that when you sort bams with sambamba, it uses /tmp/ instead of the working directory (who knew!). So, long story short, I ended up with temp dirs filled or half-filled on a bunch of the blades in our LSF-managed cluster.

To clean up the mess, my first instinct was just to write a little bash script that removed the appropriate directories owned by me, then loop through each blade like so:

bhosts | awk '{print $1}' | while read i;do
ssh $i bash ~/cleanup.sh
done

This approach failed for a number of reasons. First of all, I’ve never logged into most of the hundreds of blades, so I was being prompted for input by ssh:

The authenticity of host 'xxxxxx' can't be established.
RSA key fingerprint is ...
Are you sure you want to continue connecting (yes/no)?


Secondly, even if the warning hadn’t popped up, STDIN is being gobbled up and only the first blade in the list was being accessed before the loop exited.

I was able to figure out that adding the following to my ssh command skips the auth warning:

-o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no

And thanks to some help from stack exchange, using a different file descriptor prevented the input from being eaten. The final command looks like:

bhosts | awk '{print $1}' >hostlist;
while read -u10 HOST;do
  echo $HOST;
  ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no $HOST bash ~/cleanup.sh;
done 10<hostlist

Twenty minutes later, all was right with the world and no admins were cursing my name.

Tags: | Comments Off