Cleaning up after yourself
August 27th, 2015
Whoops! It turns out that when you sort bams with sambamba, it uses /tmp/ instead of the working directory (who knew!). So, long story short, I ended up with temp dirs filled or half-filled on a bunch of the blades in our LSF-managed cluster.
To clean up the mess, my first instinct was just to write a little bash script that removed the appropriate directories owned by me, then loop through each blade like so:
bhosts | awk '{print $1}' | while read i;do
ssh $i bash ~/cleanup.sh
done
This approach failed for a number of reasons. First of all, I’ve never logged into most of the hundreds of blades, so I was being prompted for input by ssh:
The authenticity of host 'xxxxxx' can't be established.
RSA key fingerprint is ...
Are you sure you want to continue connecting (yes/no)?
Secondly, even if the warning hadn’t popped up, STDIN is being gobbled up and only the first blade in the list was being accessed before the loop exited.
I was able to figure out that adding the following to my ssh command skips the auth warning:
-o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no
And thanks to some help from stack exchange, using a different file descriptor prevented the input from being eaten. The final command looks like:
bhosts | awk '{print $1}' >hostlist;
while read -u10 HOST;do
echo $HOST;
ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no $HOST bash ~/cleanup.sh;
done 10<hostlist
Twenty minutes later, all was right with the world and no admins were cursing my name.
Tags: | Comments Off