LSF Job Array

A useful function of the LSF cluster management system is the use of job arrays. They enable you to submit many jobs at the same time that differ by only one parameter, such as input file or something. In this case all the jobs are actually identical as they are randomly permuting an input file and scoring the results. So I am using this as a way of making more iterations of the job run in the same time.

bsub -q qname -J "jobName[1-1000]%240" -o /dev/null '~/script/script_name.pl inputFile $(echo $LSB_JOBINDEX).txt option1 option2'

You could also use them to run the same script on many input files, or run with a range of parameters too. I have another use of them where I compare many position weight matrices to a sequence using a job array, one job for each PWM.

Bash Scripting: for loop

Sometimes you want to run a program over and over with different parameters, you can do this with bash script wrapped around your program to cover a range of numbers:

for ((i=5;i<=15;i++));do echo ${i};done

So you could do something like this:

for ((i=0;i<=10;i++));do fancy_program --option ${i} --output ${i}_output.out;done

Or even something like this:

for f in *.fas;
do for ((i=0;i<=10;i++));
do fancy_program --input ${f} -n ${i} -o $(basename ${f} .fas)_${i}_output.out; done;
done

To loop through all files and all numbers. Of course you probably want to do this by submitting the jobs to a computing cluster, using bsub for example with LSF, otherwise thing might get a bit slow.