Trying to build an XLL "This program cannot run in DOS mode"

Question

I've been trying to build a simple XLL that will just create a process with a hard coded command line. Here is my code:

#include "pch.h"

BOOL APIENTRY DllMain( HMODULE hModule,
                       DWORD  ul_reason_for_call,
                       LPVOID lpReserved
                     )
{
    switch (ul_reason_for_call)
    {
    case DLL_PROCESS_ATTACH: {
        LPSTARTUPINFOA si = new STARTUPINFOA();
        LPPROCESS_INFORMATION pi = new PROCESS_INFORMATION();
        CreateProcessA(NULL, (LPSTR)"notepad.exe", NULL, NULL, TRUE, CREATE_SUSPENDED, NULL, NULL, si, pi);
    }
    case DLL_THREAD_ATTACH:
    case DLL_THREAD_DETACH:
    case DLL_PROCESS_DETACH:
        break;
    }
    return TRUE;
}

Basically creating and DLL and then just renaming it as an XLL. I am using Visual Studio 2019 and here are my flags for compiling:

/JMC /permissive- /Yu"pch.h" /ifcOutput "x64\Debug\" /GS /W3 /Zc:wchar_t /ZI /Gm- /Od /sdl /Fd"x64\Debug\vc142.pdb" /Zc:inline /fp:precise /D "_DEBUG" /D "EVIL_EXPORTS" /D "_WINDOWS" /D "_USRDLL" /D "_WINDLL" /D "_UNICODE" /D "UNICODE" /errorReport:prompt /WX- /Zc:forScope /RTC1 /Gd /MT /FC /Fa"x64\Debug\" /EHsc /nologo /Fo"x64\Debug\" /Fp"x64\Debug\Evil.pch" /diagnostics:column 

I am also using Professional Excel 2013 64 bits. What could be the problem? Been trying even with Excel 2016 64 bits and Office 365. Version 16.01.

UPDATE: Essentially yes that statement probably should appear in the Excel spreadsheet but at least I expect "notepad.exe" to run, but that doesn't happen.

Here are the linker flags:

/OUT:"C:\Users\Cristi Cretan\source\repos\XLLCalc\Release\XLLCalc.xll" /MANIFEST /LTCG:incremental /NXCOMPAT /PDB:"C:\Users\Cristi Cretan\source\repos\XLLCalc\Release\XLLCalc.pdb" /DYNAMICBASE "kernel32.lib" "user32.lib" "gdi32.lib" "winspool.lib" "comdlg32.lib" "advapi32.lib" "shell32.lib" "ole32.lib" "oleaut32.lib" "uuid.lib" "odbc32.lib" "odbccp32.lib" /IMPLIB:"C:\Users\Cristi Cretan\source\repos\XLLCalc\Release\XLLCalc.lib" /DEBUG /DLL /MACHINE:X86 /OPT:REF /SAFESEH /INCREMENTAL:NO /PGD:"C:\Users\Cristi Cretan\source\repos\XLLCalc\Release\XLLCalc.pgd" /SUBSYSTEM:WINDOWS /MANIFESTUAC:NO /ManifestFile:"Release\XLLCalc.xll.intermediate.manifest" /LTCGOUT:"Release\XLLCalc.iobj" /OPT:ICF /ERRORREPORT:PROMPT /ILK:"Release\XLLCalc.ilk" /NOLOGO /TLBID:1 

Treadle

Some guesses: Maybe a mixture of 32 bit DLL and 64 bit office, or vice versa? Otherwise, according to the documentation, XLLs are supposed to have at least an xlAutoOpen export. So maybe Excel is missing that one and thus bails with the confusing error message? – Clair

@Clair The DLL is 64 bit also, However I got a friend and we found this github project (github.com/CuckooEXE/PopCalc) We both cloned it and built it, and for him it works and now I got his exact version of Excel (Office 365 16.01 64 bit) and for me it's still not working. This github also has a test.exe which also works for me so the DLL must work, it's probably something from the excel? – Treadle

@Clair However, all I'm doing to convert DLL to XLL is rename it. but from what I saw on the internet that shouldn't be a problem – Treadle

@Clair on his PC, it still shows the message in the spreadsheet but at least it opens "calc.exe" – Treadle

Sorry, I fear I cannot help any further since I have never attempted to write an XLL. But some general thing to try would be to try ProcMon (Process Monitor), and see if it can give you hints what is going wrong while loading the DLL. Or maybe some Excel security setting that is causing problems? Or some anti virus? I don't know... Sorry. – Clair

Can you build it in release mode and try again? The short reason is that debug DLLs can only be loaded by applications that were built in debug mode. The This program cannot run in DOS mode is confusing because that is what windows would tell you if you tried to run an executable with a corrupted PE – Rozellarozelle

@Rozellarozelle Hello, yes I've tried to build in Release as well, did it work for you? – Treadle

@C.Cristi I haven't worked on this before so that was just my assumption. There is someone who is also working on XLL, might that be relevant? Build and Debug Excel add-in using XLW Project and Visual Studio 2022 C++ – Rozellarozelle

@Rozellarozelle I saw that, I dont even know how he got to install the XLW nuget on VS 2022. It doesn't fetch for me. Also in the github project they said it should only be working on VS 2020 – Treadle

Where do you get "This program cannot run in DOS mode"? – Ojibwa

@Ojibwa in the excel rows when opening the XLL – Treadle

What about link flags? Not sure if CreateProcess fits well in DllMain. What happens if the line is commented out? – Assurbanipal

@Assurbanipal Nothing, I will update linker flags – Treadle

Try interpreting the return code and also GetLastError. Also what are the exact steps to reproduce the error (cause the title is a bit misleading), and how exactly is it manifesting? – Assurbanipal

@Assurbanipal Build a simple DLL with the given code, rename it with ".xll" instead of ".dll" and run it – Treadle

Run it how? That's what I meant by detailed. [SO]: How to create a Minimal, Reproducible Example (reprex (mcve)). Is it reproducible with any executable (passing the full path makes any difference)? Also note that for the other project, POPCALC_ON_LOAD is only defined by default for Debug|Win32. – Assurbanipal

Also (and this might be it):, STARTUPINFOA is not initialized: ZeroMemory(si, sizeof(STARTUPINFOA));, si->cb = sizeof(STARTUPINFOA). On a different topic, why dynamically allocating the structures (and thus having memory leaks)? – Assurbanipal

@Assurbanipal Double click on the XLL and it will open excel, that's how I run it. I tried some things and now I think it might be a problem of a linker, maybe not having linked Xlcall32.lib – Treadle

Does it work if you build the official samples: learn.microsoft.com/en-us/office/client-developer/excel/… – Ownership

By calling the file an XLL, Excel will assume it is an add-in, and be looking for a minimum number of entry points (see Excel SDK). This may be a poorly-worded error because Excel does not know what to do with the file that is being opened. I am unclear though ... if you comment out the three lines that try to create the process, do you still get the same error? – Clamp

Related: https://mcmap.net/q/1473993/-c-excel-add-in-loading-error-xll-file-is-loaded-by-excel-as-text-file – Pendulous

Answer 1

There's a number of things missing:

• STARTUPINFO initialization (hat tip @CristiFati)
• xlAutoOpen entry point, with C external linkage

This code compiles (for me at least), and displays Notepad when the xll is loaded (I do get a security warning for a missing certificate, but that is likely my locked-down Excel settings).

//MyXll.cpp    
#include <windows.h>

BOOL APIENTRY DllMain(HMODULE hModule,
                       DWORD  ul_reason_for_call,
                       LPVOID lpReserved
                     )
{
    switch (ul_reason_for_call)
    {
    case DLL_PROCESS_ATTACH:
    {     
        STARTUPINFO si;
        PROCESS_INFORMATION pi;

        ZeroMemory(&si, sizeof(si));
        si.cb = sizeof(si);
        ZeroMemory(&pi, sizeof(pi));

        if( CreateProcess(TEXT("c:\\windows\\system32\\notepad.exe"), NULL, 
                  NULL, NULL, FALSE, CREATE_NEW_CONSOLE, NULL, NULL, &si, &pi) )
        {
            CloseHandle(pi.hProcess);
            CloseHandle(pi.hThread);
        }
    }
    case DLL_THREAD_ATTACH:
    case DLL_THREAD_DETACH:
    case DLL_PROCESS_DETACH:
        break;
    }
    return TRUE;
}

extern "C" __declspec(dllexport) int WINAPI xlAutoOpen(void)
{
    return 1;
}

I've statically linked with the C-runtime (under Code Generation, select "Multi-threaded Debug") and have _DEBUG,_WINDOWS,UNICODE and _USRDLL preprocessor definitions. In the Advanced properties, "Target File Extension" is set to .xll. I'm not using pre-compiled headers.

Compiling with MSVC Pro 22 and loading into Excel 16 64-bit.

Excel will expect to find the xlAutoOpen entry point, and will not recognize the xll unless it is present. The function doesn't have to do anything (although in an actual xll this is where function registration takes place).

It is not necessary to include or link with any of the Excel SDK elements (eg xlcall.h, xlcall32.lib) if you are not providing any Excel functionality.

Link to docs.

Clamp