Perl One Liner: Delete files with wrong number of lines

Just a quick perl one liner for future reference. I needed to delete some text files that didn't have the correct number of lines as they would break a downstream R script to parse the results

for f in *.txt;do perl -ne 'END{unlink $ARGV unless $.==200}' ${f} ;done

I always forget that $ARGV is the variable for the input file name in a one-liner.

Things I would tell a budding bioinformatician to learn.

I recently read Ewan Birney's blog post, which I found echoed a lot of my own thoughts about the use of statistical in computational biology. I thought I would compile my own similar list but for bioinformatics  / computational biology in general. I have not been and in the field as long as Ewan and I certainly still have a lot to learn, particularly about statistics due to my biological background, but I have learnt some things over the last ten years, that like Ewan, I wish someone had told me long ago.  The points are in no particular order.

Perl one liner: Command Line Google

I stumbled across this really cool perl one liner here while looking for something else.

function google () {     u=`perl -MURI::Escape -wle 'print "http://google.com/search?q=".
    uri_escape(join " ",  @ARGV)' $@`;     links $u; }

Add this to your .bashrc file and then you can just google at the command line!
 For example :

google cambridge weather

You may have to change links to be w3m or lynx depending on which text browsers you have on your system.

I will probably never ever use this, but I feel happier knowing that I can

Perl One Liner: Remove whitespace from file name

It is often a pain to have spaces in file names when working at the command line, so remove them, or at least replace then with underscores. In this case it works on all bed files in the current directory.

ls *.bed |perl -ne 'use File::Copy;chomp;$old=$_;s/\s+/_/g;move($old,$_);'

You could replace the ls with a find for more control. 

Perl One Liner: Filtering Files

Perl one liners can be a really useful way of filtering a file and only returning lines that pass some criteria. For example removing lines with low quality or something. There first examples are all assuming numeric data.


This will only show lines from test.txt where the values in the second column are less than 6.

perl -ane 'print if $F[1] < 6' test.txt

This will only show lines where the sum of the columns one and two is less than five.

perl -ane 'print if $F[0]+$F[1] < 5' test.txt

This one will return the number of lines in test.txt where the first column is less than the second column, but not actually return the lines. You could of course.

perl -ane '$sum++ if $F[0] < $F[1];END{print "$sum\n"}' test.txt

These next examples are to filter character, or text files. Maybe you only want to return lines that have your name in them:

perl -ne 'print if m/Stew/' test_names.txt

 Or maybe you want the lines with your name in but you also want to know how many didn't have your data in:

perl -ne 'if (m/Stew/) {print} else {$notMe++};END{print "$notMe\n"}' test_names.txt

Or how about only showing those lines that have two columns

perl -ane 'print if @F == 2' test.txt

The possibilities are endless. 

Perl one liner: Rename a file with some of its contents

I had some microarrays files that the scanner had named something not very useful. I wanted them renamed with their chip barcode which was in side the file.

perl -ne '`mv $ARGV $1.txt` if m/(1234567890(\d+_\d+_\d+))/;' *.txt

This little one liner does just that. The current file name is stored in $ARGV by default then I simply rename it from that to the extracted text I want.

Perl one liner: Random Lines from a File

I have some bed files that are too large to process in a reasonable time, so I need to randomly sample lines from them to create files of a workable size.

I used some bash and perl magic for this.

for f in *.bed;do export WC=`wc ${f} -l |cut -f 1 -d " "`;perl -i -ne 'srand;print if rand() <1500/$ENV{'WC'}' ${f} ;done

Basically, it checks the length of the file and stores the result in the environment variable WC, then it reads in the file line by line and only prints out the line if a random number between 0 and 1 is less than the proportion of our required size (1500 in this case) of our length (WC).

This is looped round all bed files in the current directory.

Edit:
You could also do something like this:

perl -ne 'print rand;print "\t";print;' FILENAME |sort |head -n 100 |cut -f 2 >NEWFILENAME

Which will return a random 100 lines from the file.

RSPerl : Using R from within Perl

Some things I write in perl some in R, sometime I use perl to write R and run R. One thing that I find very useful is the functionality of RSPerl which enables you to call R functions from within perl and on perl variables. It can also call perl from R, though I have no idea why you would want to do this.

My main use is to carry out statistical tests on the results of things carried out in. For example I use perl to run patser to count the number of hits to a position weight matrix in a test sequence and a background sequences, then I use RSPerl to calculate the p-value via the binomial test (binom.test function).

It was a pain to setup, as I had to recompile R and install various modules in the correct places and setup some environment variables. Once working though it is a great tool. You can even use R's great graphical capabilities to automatically generate figures from data in perl variables.

perl -e 'use R;&R::initR("--silent","--vanilla");&R::eval("r <- rnorm(100);plot(r,pch=20)");'

This one plots a histogram of the length of perl scripts!

for f in *.pl; do wc -l ${f}|cut -f 1 -d " "; done | perl -ne 's/\n/,/g;print;' |perl -ne 'use R;&R::initR("--silent","--vanilla");chop;&R::eval("hist(c($_),main=\"File Lengths\",xlab=\"Number of Lines\")");sleep 10'

RSPerl

Perl One Liner: Lower case file names

If you want to make file names lower case this will make it so.

find . -type f -maxdepth 1| while read seq;do mv ${seq} $(echo ${seq} |perl -ne 'tr/A-Z/a-z/;print'); done

or with the easier ls

ls | while read seq;do mv ${seq} $(echo ${seq} |perl -ne 'tr/A-Z/a-z/;print'); done

You can add, for example, -name "*.txt" to the find part to only rename files that end in .txt. Also changing the type to -type d then it will work on directories.

Edit:
You could also move the file within perl rather than in the shell, using File::Copy

ls | perl -ne 'use File::Copy;chomp;$old=$_;tr/A-Z/a-z/;move($old,$_)'

You could replace the ls with a find statement as well for more control.

One Liner: Convert PDF to PNG

I needed to convert all the pdf files in a few directories to png files, so I used the following one liner:

find . -name "*.pdf" -type f -maxdepth 2|while read file;do convert ${file} $(echo ${file} |perl -ne "chomp;s/pdf/png/;print");done

It requires that you have ImageMagick installed.

One Liner: Remove whitespace from filenames

Spaces in filenames can make all kinds of things break, it is good practice not to use them, but people do. So here is a good fix.

find . -type f -name "* *" -maxdepth 1| while read src; do mv "$src" `echo $src | tr " " "_"`; done

This script looks for all files in the current directory with a space in and moves them (which is how to rename things in unix) with spaces replaced by underscores. 

You could do the same with with perl too, which includes a check to stop overwriting of any existing files. 

find . -type f -maxdepth 1| perl -ne 'chomp; $o = $_; s/ /_/g; next if -e; rename $o, $_'; 

You could replace the find part with a simple ls but  this method makes it easy to change the -type f into a -type d to do the same thing on directories, or change the -maxdepth to work a a whole tree of directories. 

Perl one liner: lowercase a file

If you want to make a text file all lowercase, this perl one liner will do the job.

perl -i -ne 'tr/A-Z/a-z/;print;' file

The -i part modifies the file in place, so only use that if you are sure.

Quick one liner: Remove empty lines from files

This quick bash and perl one liner will remove any empty lines from all the files in the current directory.

for f in *;do perl -i -ne 'print unless /^$/' ${f};done

It will not remove lines that contain only white space however. For that you would need:

for f in *;do perl -i -ne 'print if /\S/' ${f};done