Organizing log files into date based directories

Mariano — Thu, 10 Apr 2008 16:01:26 +0000

I recently had to grep through a few hundred thousand log files in a single directory. The logs dated back several years and had never been organized into directories. In fact, there were so many log files that grep’ing the directory resulted in an “argument list too long” error. This error is described clearly here, and the quickest solution is to use find to pipe the filenames into grep using xargs:

find . -name "*.log" -print0 | xargs -0 grep "xyz"

I still wanted to clean things up so I decided to organize the logs into date based directories. I wrote a Perl script that scans files in a directory, parses a date from each filename (YYYY_MM_DD_…), and moves the file to the appropriate directory. For example, the script moves 2008_03_01_foo.txt to 2008/03/01/2008_03_01_foo.txt and creates the directories if necessary.

#!/usr/bin/perl -w
use strict;
use File::Path;
 
my $year;
my $month;
my $day;
my $dir;
 
while (<*>) 
{
    if (($year, $month, $day) = ($_ =~ /(\d+)_(\d+)_(\d+)/))
    {
        $dir = sprintf("%04d/%02d/%02d" ,$year, $month, $day);
        mkpath($dir) if (!-d $dir);
        rename ($_, "$dir/$_") if (!-d $_) || print "Unable to move $_:  $!\n";
    }
}

Mariano Peterson » perl code

Organizing log files into date based directories