jpegoptim on ASP.Net - "error opening temporary file"

Question

I suspect I'm failing to understand where jpegoptim tries to write its temp files.

I have IIS 7.5 running an ASP.Net 4 AppDomain. In it I have a process that optimizes JPEGs with jpegoptim like so:

FileHelper.Copy(existingPath, optimizerPath);
var jpegOptimResult = await ImageHelper.JpegOptim(optimizerPath, 30);

Running locally I get an optimized image. Running on the above server I get:

D:\www\hplusf.com\b\pc\test.jpg 4096x2990 24bit N Adobe [OK] jpegoptim: error opening temporary file.

I can show the code for FileHelper.Copy(), but it's basically just File.Copy() that overwrites if the file already exists.

Here's ImageHelper.JpegOptim:

public static async Task<string> JpegOptim(string path, int quality)
{
    string jpegOptimPath = Path.GetDirectoryName(new Uri(Assembly
            .GetExecutingAssembly().CodeBase).LocalPath)
        + @"\Lib\jpegoptim.exe";

    var jpegOptimResult = await ProcessRunner.O.RunProcess(
        jpegOptimPath,
        "-m" + quality + " -o -p --strip-all --all-normal \"" + path + "\"",
        false, true
    );

    return jpegOptimResult;
}

jpegOptimResult is what you're seeing there as the error message it's producing. And here's ProcessRunner.RunProcess:

public async Task<string> RunProcess(string command, string args,
    bool window, bool captureOutput)
{
    var processInfo = new ProcessStartInfo(command, args);

    if (!window)
        makeWindowless(processInfo);

    string output = null;
    if (captureOutput)
        output = await runAndCapture(processInfo);
    else
        runDontCapture(processInfo);

    return output;
}

protected void makeWindowless(ProcessStartInfo processInfo)
{
    processInfo.CreateNoWindow = true;
    processInfo.WindowStyle = ProcessWindowStyle.Hidden;
}

protected async Task<string> runAndCapture(ProcessStartInfo processInfo)
{
    processInfo.UseShellExecute = false;
    processInfo.RedirectStandardOutput = true;
    processInfo.RedirectStandardError = true;

    var process = Process.Start(processInfo);

    var output = process.StandardOutput;
    var error = process.StandardError;

    while (!process.HasExited)
    {
        await Task.Delay(100);
    }

    string s = output.ReadToEnd();
    s += '\n' + error.ReadToEnd();

    return s;
}

So:

• jpegOptim runs properly on my local machine, and optimizes the file, so it's not how I'm calling jpegOptim.

• The Copy operation succeeds without Exception, so it's not a Permissions issue with the ASP.Net user reading/writing from that directory

• jpegOptim just optimizes and overwrites the file, so if it is in fact running under the same ASP.Net user, it should have no problem writing this file, but...

• It's unclear where jpegOptim attempts to write its temp file, so perhaps the underlying issue is where this temporary file is being written.

However, judging by the Windows source:

http://sourceforge.net/p/jpegoptim/code/HEAD/tree/jpegoptim-1.3.0/trunk/jpegoptim.c

jpegOptim's "temporary file" appears to just be the destination file when used with the above options. Relevant lines of jpegOptim source:

int dest = 0;

int main(int argc, char **argv) 
{
    ...

There's some code here looking for the -d argument that sets dest=1 - meaning here dest remains 0. It then hits an if branch, and the else clause, for dest == 0, does this:

if (!splitdir(argv[i],tmpdir,sizeof(tmpdir))) 
    fatal("splitdir() failed!");
strncpy(newname,argv[i],sizeof(newname));

That's copying the directory name portion of the input image filename to the variable tmpdir - so like C:\Blah\18.jpg would assign tmpdir="C:\Blah\". Then it dumps the entire input image filename to newname, meaning it's just going to overwrite it in place.

At this point in the code the variables it's using should be:

dest=0
argv[i]=D:\www\hplusf.com\b\pc\test.jpg
tmpdir=D:\www\hplusf.com\b\pc\
newname=D:\www\hplusf.com\b\pc\test.jpg

It then in fact opens the file, and there's an opportunity to error out there, suggesting jpegoptim is successfully opening the file. It also decompresses the file further confirming it's successfully opening it.

The specific error message I'm seeing occurs in these lines - I'll confess I don't know if MKSTEMPS is set or not for a default build (which I'm using):

    snprintf(tmpfilename,sizeof(tmpfilename),
        "%sjpegoptim-%d-%d.XXXXXX.tmp", tmpdir, (int)getuid(), (int)getpid());
#ifdef HAVE_MKSTEMPS
    if ((tmpfd = mkstemps(tmpfilename,4)) < 0) 
        fatal("error creating temp file: mkstemps() failed");
    if ((outfile=fdopen(tmpfd,"wb"))==NULL) 
#else
    tmpfd=0;
    if ((outfile=fopen(tmpfilename,"wb"))==NULL) 
#endif
        fatal("error opening temporary file");

So snprintf is like C# String.Format(), which should produce a path like:

D:\www\hplusf.com\b\pc\jpegoptim-1-2.XXXXXX.tmp

Judging by what I can find it's likely MKSTEMPS is not defined meaning fopen is being called with "wb" meaning it's writing a binary file, and it's returning null meaning it failed to open, and out comes the error message.

So - possible causes:

• Bad path in tmpdir It's possible I'm following the C++ poorly (likely), but, from the looks of it it should be identical to the source path of the image. But perhaps it's mangled for tmpdir, by jpegoptim? The input path is clearly clean because jpegoptim actually emits it cleanly in the error message.

• Permissions issue Seems fairly unlikely. The ASP.Net user this is running under can clearly read and write because it copies to the dir before jpegoptim fires, and the only user on the machine with any permissions to this dir is that user, so jpegoptim should have failed prior to this point if it were permissions. It could be attempting to access a different dir, but that would really be the Bad tmpdir scenario.

• Something else I've not thought of.

Ideas?

Note: This question is similar:

Using jpegtran, jpegoptim, or other jpeg optimization/compression in C#

However, that question is asking about a shared env on GoDaddy, causing answers to spiral around the likelihood he can't spin up processes. We have full control over our server, and as should be clear from the above, the jpegoptim Process is definitely starting successfully, so it's a different scenario.

Ecospecies

Answer 1

As it turns out my reading of jpegoptim was incorrect. The tmpdir it uses is where the executable's Working Directory points to, not where the input images are, and not where the executable sits. So, the solution was 2-fold:

1. Give the exe permissions to write to its own directory* (but deny it access to modify itself)
2. Modify ProcessRunner to run processes in-place - set the Working Directory to where the exe resides.

The second modification looks like this:

var processInfo = new ProcessStartInfo(command, args);

// Ensure the exe runs in the path where it sits, rather than somewhere
// less safe like the website root
processInfo.WorkingDirectory = (new FileInfo(command)).DirectoryName;

*Note: I happen to have jpegoptim.exe isolated on the server to its own dir to limit risk. If you had it someplace more global like Program Files, you definitely should not do this - instead set the Working Directory as above, but to someplace isolated/safe like a tmp dir or even better a scratch disk. If you've got the RAM for it a RAMdrive would be fastest.

**Second Note: Because of how hard drives and jpegoptim work if the tmp location is not the same disk as the ultimate destination of the output there is a potential, partial race condition introduced between jpegoptim and other code you might be using that depends on its outputs. In particular if you use the same disk, when jpegoptim is done the output JPEG is complete - the OS changes the entry in its file table but the data for the image on the hard drive has already been written to completion. When tmp and destination are separate disks, jpegoptim finishes by telling the OS to move from the tmpdir to the output dir. That's a data move that finishes sometime after jpegoptim is done running. If your waiting code is fast enough it will start its work with an incomplete JPEG.

Ecospecies