gawk / awk: piping date to getline *sometimes* won't work

Question

I'm attempting to convert dates from one format to another: From e.g. "October 29, 2005" to 2005-10-29. I have a list of 625 dates. I use Awk.

The conversion works -- most of the time. Hovewer, sometimes the conversion won't happen at all, and the variable supposed to hold the (converted) date remains undefined.

This always happens with the exact same rows. Running `date' explicitly (from the Bash shell) on the dates of those weird rows works fine (the dates are properly converted). -- It's not the textual contents of those rows that matters.

Why this behavior, and how can I fix my script?
Her it is:

awk 'BEGIN { FS = "unused" } { 
  x = "undefined";
  "date \"+%Y-%m-%d\" -d " $1 | getline x ;
  print $1 " = " x
}' uBXr0r15.txt \
 > bug-out-3.txt

If you want to reproduce this problem:

1. Download this file: uBXr0r15.txt.
2. Run the Awk skript.
3. Search for "undefined" in bug-out-3.txt.
   ("undefined" found 122 times, on my computer.)

Then you could run the script again, and (on my computer) bug-out-3.txt remains unchanged -- exactly the same dates are left undefined.

(Gawk version 3.1.6, Ubuntu 9.10.)

Kind regards, Magnus

Letters

Answer 1

Whenever you open a pipe or file for reading or writing in awk, the latter will first check (using an internal hash) whether it already has a pipe or file with the same name (still) open; if so, it will reuse the existing file descriptor instead of reopening the pipe or file.

In your case, all entries which end up as undefined are actually duplicates; the first time that they are encountered (i.e. when the corresponding command date "..." -d "..." is first issued) the proper result is read into x. On subsequent occurrences of the same date, getline attempts to read a second, third etc. lines from the original date pipe, even though the pipe has been closed by date, resulting in x no longer being assigned.

From the gawk man-page:

NOTE: If using a pipe, co-process, or socket to getline, or from print or printf within a loop, you must use close() to create new instances of the command or socket. AWK does not automatically close pipes, sockets, or co-processes when they return EOF.

You should explicitly close the pipe every time after you have read x:

close("date \"+%Y-%m-%d\" -d " $1)

Incidentally, would it be OK to sort and uniq uBXr0r15.txt before piping into awk, or do you need the original ordering/duplication?

Kissee

This solves my problem, thanks. I don't need the original ordering and if I reorder my indata the problem is also gone -- and I save some CPU I suppose. (The real world indata also contains non-date rows, so I can't use sort' and uniq', and the real Awk script is somewhat different.) – Letters

Thanks, this also solves my problem - I was getting an error "Too many open files" and wondered how to close these "files" because I didn't know awk uses files for those pipe operations. – Tenace

I noticed this problem on subsequent calling of a function to reseed random. – Firecrest

Answer 2

Though I love awk it is not necessary for this.

tr -d '"' < uBXr0r15.txt | date +%Y-%m-%d -f -

Houseyhousey

Thanks, didn't know I could do that. The example in my original post is a simplified example. My real Awk script is somewhat longer, and the real input file also contains non-date rows. – Letters

Answer 3

 gawk 'BEGIN{
       m=split("January|February|March|April|May|June|July|August|September|October|November|December",d,"|")
       for(o=1;o<=m;o++){
          months[d[o]]=sprintf("%02d",o)
       }
       FS="[, ]"
    }
    {
      gsub(/["]/,"",$1)
      gsub(/["]/,"",$4)
      t=mktime($4" "months[$1]" "$2" 0 0 0")
      print strftime("%Y-%m-%d",t)
    }' uBXr0r15.txt

doing everything inside gawk will be faster than calling external commands.

Mcclanahan

This was an ambitious solution :-) Works fine. It's alright for me to wait those fractions of a second longer the other solutions take though :-) – Letters