Why is this process crashing as soon as it is launched?

Question

We have an IIS WCF service that launches another process (app.exe) as a different user. I have complete control over both applications (and this is a dev environment for now). The IIS app pool runs as me, a domain user (DOMAIN\nirvin), who is also a local administrator on the box. The second process is supposed to run as a local user (svc-low). I am using System.Diagnostics.Process.Start(ProcessStartInfo) to launch the process. The process launches successfully - I know because there are no exceptions thrown, and I get a process ID. But the process dies immediately, and I get an error in the Event Log that looks like:

Faulting application name: app.exe, version: 1.0.3.0, time stamp: 0x514cd763

Faulting module name: KERNELBASE.dll, version: 6.2.9200.16451, time stamp: 0x50988aa6

Exception code: 0xc06d007e

Fault offset: 0x000000000003811c

Faulting process id: 0x10a4

Faulting application start time: 0x01ce274b3c83d62d

Faulting application path: C:\Program Files\company\app\app.exe

Faulting module path: C:\Windows\system32\KERNELBASE.dll

Report Id: 7a45cd1c-933e-11e2-93f8-005056b316dd

Faulting package full name:

Faulting package-relative application ID:

I've got pretty thorough logging in app.exe (now), so I don't think it's throwing errors in the .NET code (anymore).

Here's the real obnoxious part: I figured I was just launching the process wrong, so I copied my Process.Start() call in a dumb WinForms app and ran it on the machine as myself, hoping to tinker around till I got the parameters right. So of course that worked the very first time and every time since: I'm able to consistently launch the second process and have it run as intended. It's only launching from IIS that doesn't work.

I've tried giving svc-low permission to "Log on as a batch job" and I've tried giving myself permission to "Replace a process level token" (in Local Security Policy), but neither seem to have made any difference.

Help!

Environment Details

• Windows Server 2012
• .NET 4.5 (all applications mentioned)

Additional Details

At first app.exe was a Console Application. Trying to launch was making conhost.exe generate errors in the Event Log, so I switched app.exe to be a Windows Application. That took conhost out of the equation but left me the situation described here. (Guided down that path by this question.)

The ProcessStartInfo object I use looks like this:

new ProcessStartInfo
{
    FileName = fileName,
    Arguments = allArguments,
    Domain = domainName,
    UserName = userName,  
    Password = securePassword,
    WindowStyle = ProcessWindowStyle.Hidden,
    CreateNoWindow = true,  
    UseShellExecute = false,
    RedirectStandardOutput = false
    //LoadUserProfile = true  //I've done it with and without this set
};

An existing question says I should go down to the native API, but a) that question addresses a different situation and b) the success of the dumb WinForms app suggests that Process.Start is a viable choice for the job.

Chacma

Have you tried a simpler case running the child as the same (domain admin) account? – Longwise

It might be worth trying an app.exe that is completely blank, to rule out all the code inside it. – Unhouse

For what it's worth that exception code appears to mean 'module not found'. If this were me I would probably run Process Monitor (Russinovich et al) to get an idea what the application is trying to load. – Bookish

Those module not found errors often end up in the application event log. They even tell you what module wasn't found. – Vu

Yeah it sort of feels like a path problem, agree sysinternals is the way to go. – Longwise

So I am pretty confident it is not a path problem - the path was wrong once before, and Process.Start() throws a pretty specific exception in that case. I will double-check though. I have tried running with me as both the app pool and the 'child', and that works for dev, but it won't fly in production - at best it would be risky, and at worst it would be outright rejected by our security team. – Chacma

@romkyns, good thought: I've now tried that, I created a "Hello World" win forms .NET 4.5 app.exe that references only System and does pretty much nothing. I still get the same error in the Event Log. It feels like there's some resource(s) svc-low doesn't have access to, but I can't figure out what. – Chacma

Answer 1

I ended up opening a case with Microsoft, and this is the information I was given:

Process.Start internally calls CreateProcessWithLogonW(CPLW) when credentials are specified. CreateProcessWithLogonW cannot be called from a Windows Service Environment (such as an IIS WCF service). It can only be called from an Interactive Process (an application launched by a user who logged on via CTRL-ALT-DELETE).

(that's verbatim from the support engineer; emphasis mine)

They recommended I use CreateProcessAsUser instead. They gave me some useful sample code, which I then adapted to my needs, and now everything works great!

The end result was this:

using System;
using System.ComponentModel;
using System.Diagnostics;
using System.Runtime.InteropServices;
using System.Security;

public class ProcessHelper
{
    static ProcessHelper()
    {
        UserToken = IntPtr.Zero;
    }

    private static IntPtr UserToken { get; set; }

    public int StartProcess(ProcessStartInfo processStartInfo)
    {
        LogInOtherUser(processStartInfo);

        Native.STARTUPINFO startUpInfo = new Native.STARTUPINFO();
        startUpInfo.cb = Marshal.SizeOf(startUpInfo);
        startUpInfo.lpDesktop = string.Empty;

        Native.PROCESS_INFORMATION processInfo = new Native.PROCESS_INFORMATION();
        bool processStarted = Native.CreateProcessAsUser(UserToken, processStartInfo.FileName, processStartInfo.Arguments,
                                                         IntPtr.Zero, IntPtr.Zero, true, 0, IntPtr.Zero, null,
                                                         ref startUpInfo, out processInfo);

        if (!processStarted)
        {
            throw new Win32Exception(Marshal.GetLastWin32Error());
        }

        uint processId = processInfo.dwProcessId;
        Native.CloseHandle(processInfo.hProcess);
        Native.CloseHandle(processInfo.hThread);
        return (int) processId;
    }

    private static void LogInOtherUser(ProcessStartInfo processStartInfo)
    {
        if (UserToken == IntPtr.Zero)
        {
            IntPtr tempUserToken = IntPtr.Zero;
            string password = SecureStringToString(processStartInfo.Password);
            bool loginResult = Native.LogonUser(processStartInfo.UserName, processStartInfo.Domain, password,
                                                Native.LOGON32_LOGON_BATCH, Native.LOGON32_PROVIDER_DEFAULT,
                                                ref tempUserToken);

            if (loginResult)
            {
                UserToken = tempUserToken;
            }
            else
            {
                Native.CloseHandle(tempUserToken);
                throw new Win32Exception(Marshal.GetLastWin32Error());
            }
        }
    }

    private static String SecureStringToString(SecureString value)
    {
        IntPtr stringPointer = Marshal.SecureStringToBSTR(value);
        try
        {
            return Marshal.PtrToStringBSTR(stringPointer);
        }
        finally
        {
            Marshal.FreeBSTR(stringPointer);
        }
    }

    public static void ReleaseUserToken()
    {
        Native.CloseHandle(UserToken);
    }
}

internal class Native
{
    internal const int LOGON32_LOGON_INTERACTIVE = 2;
    internal const int LOGON32_LOGON_BATCH = 4;
    internal const int LOGON32_PROVIDER_DEFAULT = 0;

    [StructLayout(LayoutKind.Sequential)]
    internal struct PROCESS_INFORMATION
    {
        public IntPtr hProcess;
        public IntPtr hThread;
        public uint dwProcessId;
        public uint dwThreadId;
    }

    [StructLayout(LayoutKind.Sequential)]
    internal struct STARTUPINFO
    {
        public int cb;
        [MarshalAs(UnmanagedType.LPStr)]
        public string lpReserved;
        [MarshalAs(UnmanagedType.LPStr)]
        public string lpDesktop;
        [MarshalAs(UnmanagedType.LPStr)]
        public string lpTitle;
        public uint dwX;
        public uint dwY;
        public uint dwXSize;
        public uint dwYSize;
        public uint dwXCountChars;
        public uint dwYCountChars;
        public uint dwFillAttribute;
        public uint dwFlags;
        public short wShowWindow;
        public short cbReserved2;
        public IntPtr lpReserved2;
        public IntPtr hStdInput;
        public IntPtr hStdOutput;
        public IntPtr hStdError;
    }

    [StructLayout(LayoutKind.Sequential)]
    public struct SECURITY_ATTRIBUTES
    {
        public System.UInt32 nLength;
        public IntPtr lpSecurityDescriptor;
        public bool bInheritHandle;
    }

    [DllImport("advapi32.dll", EntryPoint = "LogonUserW", SetLastError = true, CharSet = CharSet.Unicode, CallingConvention = CallingConvention.StdCall)]
    internal extern static bool LogonUser(String lpszUsername, String lpszDomain, String lpszPassword, int dwLogonType, int dwLogonProvider, ref IntPtr phToken);

    [DllImport("advapi32.dll", EntryPoint = "CreateProcessAsUserA", SetLastError = true, CharSet = CharSet.Ansi, CallingConvention = CallingConvention.StdCall)]
    internal extern static bool CreateProcessAsUser(IntPtr hToken, [MarshalAs(UnmanagedType.LPStr)] string lpApplicationName, 
                                                    [MarshalAs(UnmanagedType.LPStr)] string lpCommandLine, IntPtr lpProcessAttributes,
                                                    IntPtr lpThreadAttributes, bool bInheritHandle, uint dwCreationFlags, IntPtr lpEnvironment,
                                                    [MarshalAs(UnmanagedType.LPStr)] string lpCurrentDirectory, ref STARTUPINFO lpStartupInfo, 
                                                    out PROCESS_INFORMATION lpProcessInformation);      

    [DllImport("kernel32.dll", EntryPoint = "CloseHandle", SetLastError = true, CharSet = CharSet.Auto, CallingConvention = CallingConvention.StdCall)]
    internal extern static bool CloseHandle(IntPtr handle);
}

There are some pre-requisites to making this code work. The user running it must have the user right to 'Replace a process level token' and 'Adjust memory quotas for a process', while the 'other user' must have the user right to 'Log on as a batch job'. These settings can be found under the Local Security Policy (or possibly through Group Policy). If you change them, a restart will be required.

UserToken is a property that can be closed via ReleaseUserToken because we will call StartProcess repeatedly and we were told not to log the other user on again and again.

That SecureStringToString() method was taken from this question. Using SecureString was not part of Microsoft's recommendation; I did it this way so as not to break compatibility with some other code.

Chacma

Yes this code has been running in production under very high load with no errors since late May 2013. – Chacma

This is very helpful - maybe you have some insight here: #27362904 ? – Koziarz

By the way, I later did have modify this code a little bit, as documented by this SO question (be sure to read the comments): #23687515 – Chacma

Is there an easy way to capture the output of this without rewriting half of the pipes in the Process class? – Ideation

Is there a reason you used the Wide (Unicode) method call for the LogonUser call and the Ansi method call for the createprocess call? – Ideation

@Doug, it's been a while, so I can't remember for sure, but I'm pretty sure that I just used the method calls recommended by the Microsoft Support engineer. The code above was later expanded to capture the standard output, and it did take a while to get right, but the final code wasn't too horrible (see this question) – Chacma

@Chacma is there any chance that you post the whole class to Gist (or a fake question here) as it's been something I've been working on for 4 days, and still can't get it quite right. I can't get the process to start successfully and I can't get it to pipe/event. Trying to lift as much as possible from the source code but failing (feel like i'm only inches away). – Ideation

@Doug, This code was written at a previous job and I no longer have direct access to it. I'll contact an old co-worker and see if he can/will send it to me, though. – Chacma

I think I'm facing a similar issue but using start-process commandlet from a powershell script run by a service. Could it be using CreateProcessWithLogin internally ? For reference, here is my question thread : #30942521 – Vogue

The code is working. But my exe can't get the arguments. Have you a solution for it? – Gamogenesis

@SerdarDidan I think it would be best for you to post a question of your own so you can lay out all the details. – Chacma

Answer 2

  Exception code: 0xc06d007e

This is an exception that's specific to Microsoft Visual C++, facility code 0x6d. The error code is 0x007e (126), ERROR_MOD_NOT_FOUND, "The specified module could not be found". This exception is raised when a delay-loaded DLL cannot be found. Most programmers have the code that generates this exception on their machine, vc/include/delayhlp.cpp in the Visual Studio install directory.

Well, it is the typical "file not found" mishap, specific to a DLL. If you have no idea what DLL is missing then you can use SysInternals' ProcMon utility. You'll see the program search for the DLL and not finding just before it bombs.

A classic way to get poorly designed programs to crash with Process.Start() is by not setting the ProcessStartInfo.WorkingDirectory property to the directory in which the EXE is stored. It usually is by accident but won't be when you use the Process class. Doesn't look like you do so tackle that first.

Antione

Setting ProcessStartInfo.WorkingDirectory doesn't change anything for me. Reflecting through System.Diagnostic.Process tells me that when that value is not set, .NET will use Environment.CurrentDirectory. Is there a specific working directory that you feel I should be setting the value to? – Chacma

Same as where the EXE is located. Have you found the DLL yet? If you have no clue then be sure to contact the owner of the program for support. – Antione

I am the program owner. I haven't been able to see any errors through ProcMon: it actually says it's loading KERNELBASE.DLL fine and I don't see any other LoadImage errors. – Chacma

No, it is not kernelbase.dll and you won't see "LoadImage errors". You'll see it searching for a file and not finding it. If you are the owner of the program then surely you remember using the linker's /delayload option? – Antione

Process Monitor does report NAME NOT FOUND for several DLLs when the process fails. However, it reports those same error events for when the process works exactly as desired. The program is written in C#.NET, so I've not directly worked with the linker, I'm just building through Visual Studio. I'm not loading any DLLs at runtime in my own code, though .NET certainly might be under the hood. I'll research the /delayload option to see if Visual Studio is using it. – Chacma