Find a file that was created within five days of another file in shell
Asked Answered
A

3

4

i'm still pretty new to scripting so stick with me and if you have any questions please feel free to ask.

Okay, so: I have a file let's say file.txt

file.txt exists in a directory /this/is/the/directory/file.txt

In a separate directory .log files exist that tell me what happens when file.txt was created.

fuubar.log, fuu.log, bar.log, this.log, that.log, some.log, other.log...there is an unknown number of these logs.

I need to gather all the log files that occurred +-5 days of the file.txt file being created.

For example: file.txt was created on 7 July 2013 (don't pay any attention to date format) I need the log files that occurred on and between 2 July 2013 and 12 July 2013.

Any ideas?

EDIT: I'm more confused about comparing the dates of the files to get the correct ones, i know how to copy files.

Perl and stat are not available to me

Artois answered 18/7, 2013 at 13:57 Comment(1)
The standard way to do this is create files with the date range (using touch) and then use them as -newer and ! -newer arguments to find.Seal
S
2

Since you are on QNX you can't use existing POSIX/Linux/Unix/BSD scripts off the shelf in your approach to this. If you could either install programming languages that can help you do this or install updated system utilities (from QNX one would hope) or shell tools to achieve the same thing it might be easier, but I realize this is not always possible.

You could try the following horrid hackish approach for a quick "works now" solution though:

First use find /path/to/log/file.txt -printf %Cj to give you the day of the year from 1-366 in which the log file was created and -printf %CY to give you the year value. Next use expr +/- 5 as you did before to find values for $day_of_year_start and $day_of_year_end which you can then use to set the $start and $end variables just as in JP's script or my variation on it.

Now you need a way to get epoch time for use with date -s so you can convert from epoch to a time format touch -t will use. Without a date that can convert to and from epoch time and no strftime that outputs epoch time to make this easy, you need a "hack". One messy approach would be to set a constant for the epoch time at the beginning of a year (say 1356998400 for Jan 1 2013 ; plus values for other years if needed) and then add ($day_of_year_start x 86400) and ($day_of_year_end x 86400) seconds to that constant to get two epoch time values to run through date -s on QNX and set a date/timestamp for $start and $end respectively.

Messy but it might actually work :-) Let us know if you try this.

Cheers, ps: here's the start of a script you'll need to check command arguments and date formats for QNX to make sure they are correct. It works on BSD boxes I have with gfind (GNU find) and date -r (instead of date -s for QNX).

#!/bin/sh
# Find files created within 5 days before/after
# the creation of another file on QNX.
# 
#  ./rangesearch /path/to/logfile.txt
# 
# NB: 
# QNX shell environment lacks some POSIX/GNU features:
# 1. 'stat', 'date -r' are not available (use find to get file ACM dates)
# 2. use GNU 'find' extensions (since -printf is needed for output of dates)
# 3. 'date' cannot modify dates using day/month/year values so we convert
#    to epoch values in a roundabout way. 'find' cannot output epoch dates
#    due to limitations of QNX strftime so we compare year of file creation
#    against a set of constants and then add/subtract daily amounts of 
#    seconds from this value and use QNX 'date -s' to convert these dates to 
#    formats usable by QNX 'touch'. 
#
# TODO: improve and extend year <--> epoch time constant matching with
#       an array and for loop (Bourne shell 'sh' does not really have
#       an "array" feature per se). 
#
#       detect version of find/gfind date/gdate so this runs on Linux OSX BSD 
#
#       add more precise units (hours minutes seconds) to epoch time
#       calculation (exercise for reader)
#

year2013="1357016400"
year2012="1325394000"
year2011="1293858000"
year2010="1262322000"
# .... etc etc,  some kind of vector or array would be nice ...

fileyear=`find $1 -printf %CY`
filedoyr=`find $1 -printf %Cj`

if [ $fileyear -eq "2013" ]; then
  logfile_epoch=$(( $year2013 + ($filedoyr * 86400) ))
  echo "file is from $fileyear or $logfile_epoch SSL"
elif [ $fileyear -eq "2012" ]; then
  logfile_epoch=$(( $year2012 +($filedoyr * 86400) ))
  echo "file is from $fileyear or $logfile_epoch SSL"
elif [ $fileyear -eq "2011" ]; then
  logfile_epoch=$(( $year2011 + ($filedoyr *86400) ))
  echo "file is from $fileyear or $logfile_epoch SSL"
elif [ $fileyear -eq "2010" ]; then
  logfile_epoch=$(( $year2010 + ($filedoyr *86400) ))
  echo "file is from $fileyear or $logfile_epoch SSL"
else
  echo "there is a problem"
  exit
fi

echo "file is from day $filedoyr of $fileyear"

epochal_start=$(( $logfile_epoch - (5*86400) ))
epochal_end=$(( $logfile_epoch + (5*86400) ))
start=`date -s $epochal_start +%Y%m%d%H%S`
end=`date -s $epochal_end +%Y%m%d%H%S`

echo "epochal_start $epochal_start"
echo "epochal_end $epochal_end"

echo -e "Searching for files from $start to $end in age \n...\n"

touch -t "$start" /tmp/s$$
touch -t "$end" /tmp/e$$

# -print0 and xargs -0 allow for file names with whitepace
find . -type f -newer /tmp/s$$ -and ! -newer /tmp/e$$ -print0 |xargs -0 ls -tal

# clean temporary files:
rm /tmp/s$$
rm /tmp/e$$

SSL here means Seconds Since Long ago :-)

Seal answered 31/7, 2013 at 1:33 Comment(2)
Sorry it took so long for me to get back. I'll be trying this approach shortly and will let you know of the results.Artois
This works!! Thank you so much. I will probably venture further and try and attempt some of the 'TODO' at some point but for now I have other projects on my plate. Thanks again! Cheers!Artois
G
4

Well something to get your started unless someone posts a one-liner!

# Get the date in you want to start listing files from using your
# sample file. This will assign the variable in the format of MMDDYYYY
$ start=$(date -d "-5 days" '+%m%d%Y' < <(date -r /this/is/the/directory/file.txt))

# Get the date in you want to end listing files from. using your
# sample file. This will assign the variable in the format of MMDDYYYY
$ end=$(date -d "+5 days" '+%m%d%Y' < <(date -r /this/is/the/directory/file.txt))

# Create two temp files. touch -t will force the timestamp on
# these files to match the content of variables
$ touch -t "$start" /tmp/s$$
$ touch -t "$end" /tmp/e$$

# Use these temp files timestamp as a pivot mechanism by find commands
# newer option. This will list files whose timestamp is in between our 
# temp files timestamp which we captured from your sample file
$ find /path/to/files -type f -newer /tmp/s$$ -and -not -newer /tmp/e$$
Gonfanon answered 18/7, 2013 at 15:52 Comment(8)
I hate to sound ignorant, but, could you please give me an explanation of this. Rather than just copying it i'm trying to understand it so that i may implement it in the format which will best suits my project.Artois
@js very nice, but don't forget to remove your tmpfiles s$$ and e$$ after you're doneHut
@umläute Yep either delete them manually or reboot the machine so /tmp gets cleared! ;)Gonfanon
This is written for GNU so won't work "out of the box" on POSIX BSD OS/X AIX Solaris without GNU coreutils ... just for info of people cutting and pasting on those platforms :)Seal
@umläute JS I tried to make a version that runs under plain sh with (mostly) POSIX tools. If you have an OSX or Solaris system give it a try. CheersSeal
I just double check and date does exist in my bin and the path to the bin is correct.Artois
Is there a different way to get the start and end date? My date does not have the -d or -r commands supported...Artois
Marked useful because in most situation this should have worked. I had not fully explained that I was using QNX so there was no way for you to know.Artois
S
2

Since you are on QNX you can't use existing POSIX/Linux/Unix/BSD scripts off the shelf in your approach to this. If you could either install programming languages that can help you do this or install updated system utilities (from QNX one would hope) or shell tools to achieve the same thing it might be easier, but I realize this is not always possible.

You could try the following horrid hackish approach for a quick "works now" solution though:

First use find /path/to/log/file.txt -printf %Cj to give you the day of the year from 1-366 in which the log file was created and -printf %CY to give you the year value. Next use expr +/- 5 as you did before to find values for $day_of_year_start and $day_of_year_end which you can then use to set the $start and $end variables just as in JP's script or my variation on it.

Now you need a way to get epoch time for use with date -s so you can convert from epoch to a time format touch -t will use. Without a date that can convert to and from epoch time and no strftime that outputs epoch time to make this easy, you need a "hack". One messy approach would be to set a constant for the epoch time at the beginning of a year (say 1356998400 for Jan 1 2013 ; plus values for other years if needed) and then add ($day_of_year_start x 86400) and ($day_of_year_end x 86400) seconds to that constant to get two epoch time values to run through date -s on QNX and set a date/timestamp for $start and $end respectively.

Messy but it might actually work :-) Let us know if you try this.

Cheers, ps: here's the start of a script you'll need to check command arguments and date formats for QNX to make sure they are correct. It works on BSD boxes I have with gfind (GNU find) and date -r (instead of date -s for QNX).

#!/bin/sh
# Find files created within 5 days before/after
# the creation of another file on QNX.
# 
#  ./rangesearch /path/to/logfile.txt
# 
# NB: 
# QNX shell environment lacks some POSIX/GNU features:
# 1. 'stat', 'date -r' are not available (use find to get file ACM dates)
# 2. use GNU 'find' extensions (since -printf is needed for output of dates)
# 3. 'date' cannot modify dates using day/month/year values so we convert
#    to epoch values in a roundabout way. 'find' cannot output epoch dates
#    due to limitations of QNX strftime so we compare year of file creation
#    against a set of constants and then add/subtract daily amounts of 
#    seconds from this value and use QNX 'date -s' to convert these dates to 
#    formats usable by QNX 'touch'. 
#
# TODO: improve and extend year <--> epoch time constant matching with
#       an array and for loop (Bourne shell 'sh' does not really have
#       an "array" feature per se). 
#
#       detect version of find/gfind date/gdate so this runs on Linux OSX BSD 
#
#       add more precise units (hours minutes seconds) to epoch time
#       calculation (exercise for reader)
#

year2013="1357016400"
year2012="1325394000"
year2011="1293858000"
year2010="1262322000"
# .... etc etc,  some kind of vector or array would be nice ...

fileyear=`find $1 -printf %CY`
filedoyr=`find $1 -printf %Cj`

if [ $fileyear -eq "2013" ]; then
  logfile_epoch=$(( $year2013 + ($filedoyr * 86400) ))
  echo "file is from $fileyear or $logfile_epoch SSL"
elif [ $fileyear -eq "2012" ]; then
  logfile_epoch=$(( $year2012 +($filedoyr * 86400) ))
  echo "file is from $fileyear or $logfile_epoch SSL"
elif [ $fileyear -eq "2011" ]; then
  logfile_epoch=$(( $year2011 + ($filedoyr *86400) ))
  echo "file is from $fileyear or $logfile_epoch SSL"
elif [ $fileyear -eq "2010" ]; then
  logfile_epoch=$(( $year2010 + ($filedoyr *86400) ))
  echo "file is from $fileyear or $logfile_epoch SSL"
else
  echo "there is a problem"
  exit
fi

echo "file is from day $filedoyr of $fileyear"

epochal_start=$(( $logfile_epoch - (5*86400) ))
epochal_end=$(( $logfile_epoch + (5*86400) ))
start=`date -s $epochal_start +%Y%m%d%H%S`
end=`date -s $epochal_end +%Y%m%d%H%S`

echo "epochal_start $epochal_start"
echo "epochal_end $epochal_end"

echo -e "Searching for files from $start to $end in age \n...\n"

touch -t "$start" /tmp/s$$
touch -t "$end" /tmp/e$$

# -print0 and xargs -0 allow for file names with whitepace
find . -type f -newer /tmp/s$$ -and ! -newer /tmp/e$$ -print0 |xargs -0 ls -tal

# clean temporary files:
rm /tmp/s$$
rm /tmp/e$$

SSL here means Seconds Since Long ago :-)

Seal answered 31/7, 2013 at 1:33 Comment(2)
Sorry it took so long for me to get back. I'll be trying this approach shortly and will let you know of the results.Artois
This works!! Thank you so much. I will probably venture further and try and attempt some of the 'TODO' at some point but for now I have other projects on my plate. Thanks again! Cheers!Artois
S
1

There's generally no simple portable (POSIX) way to get file modification times.

Note that if your Unix has a version of find that includes GNU extensions (like -printf) you can use find to get the date of the original logfile. If your version of date does not include the -v for adjusting the date string forward and backward in time, then you have to find some way to convert the date to Epoch dates (seconds since long ago) and adjust the dates using expr (+/- 86400*5) and convert that to a format usable with touch.

You've told us you're using QNX so with the QNX version of find you will have the -printf extension. This means you can create your values with find but not able to adjust them +/- 5 days with date -v or convert them to Epoch times for modification with expr. The QNX documents seems silent on how you might do this in a simple obvious way, illustrating why a simple portable (POSIX shell) way to get file modification times, set the date, convert date formats, etc. etc. would be so nice to have in the real world. You'll need to ask QNX gurus for more help. Sorry!

To complement Jaypal Singh's GNU coreutils solution BSD/POSIX and Perl approaches follow.

For BSD derived systems (FreeBSD, DragonFly, possibly OS/X) replacing the commands that create the $start and $end variables in @JS's solution with the following might work:

#!/bin/sh
#
middle=`stat -r file.txt | cut -d" " -f10` # file.txt could 
                                           # come from a script 
                                           # positional argument like $1



# the number date range could also come from a positional argument like $2

start=`date -v-5d -r $middle +%Y%m%d%H%S`
end=`date -v+5d -r $middle +%Y%m%d%H%S`

touch -t "$start" /tmp/s$$
touch -t "$end" /tmp/e$$

find . -type f -newer /tmp/s$$ -and ! -newer /tmp/e$$

The last part of the script as @JS already described.

Another alternative involves someone riding a cross-platform camel to the rescue ... like this :-)

This could be much nicer of course but here's a perl approach cribbed from find2perl output:

#!/usr/bin/env perl  
# findrange  Usage: cd /dir/to/search ; findrange 5 /file/to/compare.txt

use strict;  use warnings;                                            
use File::Find ();                                                          
use vars qw/*name/;                                                         

*name   = *File::Find::name;                                                

sub wanted;                                                                 

# some values to use                                                        
my $RANGE = $ARGV[0] ;  # get date range                                    
my $REF_FILE = $ARGV[1] ;  # get file to compare date                       
my $AGE_REF = -M $REF_FILE ;                                                
my $start = $AGE_REF - $RANGE ; # +/- days from @ARGV[0] 
my $end = $AGE_REF +  $RANGE ;                                              

# Descend file system searching/finding.                                     
# from current directory  "."                                               
File::Find::find({wanted => \&wanted}, '.');                                
exit;                                                                       

sub wanted {                                                                
  ( lstat($_)) &&  #caches results in "_" for -M to use   
    -f _ &&                                                                 
    (-M _ > $start) &&                                                      
    ! (-M _ > $end)                                                         
    && print("$name\n");                                                    
}                                                                           

If it's for interactive use you could add:

if ($#ARGV != 1) {  # require 2 args (last array index +1)
 usage() ;  
 }

above/before the sub wanted runs and something like sub usage { print "whatever \n"; exit;} to make it more fancy.

Cheers,

Seal answered 18/7, 2013 at 21:21 Comment(10)
My issue with this is that stat is not recognized by the VM i'm using and i don't know if it is installed on the machines that this script will be deployed on.Artois
That's because, as mentioned, there's generally no simple portable (POSIX) way to get file modification times. Your stat is probably GNU which does (almost) entirely different things. I'll post a perl approach - at least that should work everywhere.Seal
I moved on to something else and finished it, so now this issue has come back to haunt me again. Perl and stat are not options for me. Is there anyway other way to get the date of the file and then go about using the touch compare the files?Artois
If stat and date -r and perl are not available, what UNIX are you using? Linux uses GNU user space for the most part and stat there is a different beast (something to with file systems) so that's understandable. If you do have Linux then you should have GNU coreutils which does have a date -r command ... (boggle). I suppose you could try parsing the output of ls which is generally frowned upon ... You need to tell us more about your platform/OS etc.Seal
It's QNX; a Unix-like OS. The script itself is going to be on multiple devices (one is a control center and the other is a supporting device for the control center)...both of these use the same OS and have nearly the same bin files, however -like previously mentioned, for some reason, does not include anything useful -_-Artois
Also, i've been trying to figure out how to parse the ls output, however, it doesn't include the year. this is troublesome because touch needs the year for its formatArtois
With QNX's version of find you have -printf so you should be able to use that to get the date. I put a sample command in my answer which you can can accept (click the the check mark) if it works and vote up since we Canadians love QNX :-)Seal
That would work fine as long as the day is above the 5th. If it occurs 1, 2, 3, 4, 5 of the month then when the subtraction occurs the month changes however the date becomes skewed. Example: Middle 2013 07 02 Start 2013 06 97 1840 End 2013 07 071840Artois
It acts this way because it reads the entire date 20130702 as one integer and then subtracts 5 from that entire number. It will act the same way if i were to try to get 30 days after the time it would give me : 20130632....hmmmmArtois
@Artois - I meant to say if the year and month don't matter you could split out the days and subtract from that. Then just mess on through by manually setting the responses for Jan 1-5 (days 1 to 5) to be days 360-365 of the previous year etc ... find -print %Cj gives day of the year from 1-366 and you still have xargs and other useful tools :-) You need a way to get epoch time for use with date -s but see my second response for a hackish approach. Cheers!Seal

© 2022 - 2024 — McMap. All rights reserved.