How to use Rundll32 to execute DLL Function?

Question

Using the ShellExecute documentation as a reference:

I run the following from the command line:

C:\>RUNDLL32.EXE SHELL32.DLL,ShellExecute handle,"open","C:\Documents and Settings\admin\Desktop\tmp",NULL,NULL,SW_SHOWNORMAL

This results in an exception error.

I don't know what this means:

HINSTANCE ShellExecute(
  __in_opt  HWND hwnd,
  __in_opt  LPCTSTR lpOperation,
  __in      LPCTSTR lpFile,
  __in_opt  LPCTSTR lpParameters,
  __in_opt  LPCTSTR lpDirectory,
  __in      INT nShowCmd
);

But in the description, a handle (HWND), and a pointer to a null-terminated string (LPCTSTR), are mentioned, but it is very confusing.

Any help would be greatly appreciated. I would also like to learn more, so any references (book, web links, etc) would also be great!

Gleesome

It would be helpful if you explained what you're trying to accomplish since there may be better ways than using RUNDLL32. Regrdless, you are not calling RUNDLL32 correctly. For example, parameters must be separated by spaces (comma only separates entry point from DLL), hwnd and nShowCmd expect integer values, etc. See support.microsoft.com/kb/164787 for more info. – Hexane

I am a wandering head. I read: vlaurie.com/computers2/Articles/rundll32.htm. Then I started looking at different dll files, and found this: msdn.microsoft.com/en-us/library/bb776426%28v=VS.85%29.aspx. I started to look at the individual functions, and wondered if they could be used with rundll32.exe. I picked the ShellExecute function because I understand what it does (open a folder). I am mainly trying to learn how these things work (in MSDN). I don't even know if it is C, C++, C#, etc. – Gleesome

Answer 1

Rundll32 only supports running DLL exports with the following signature:

void CALLBACK
  EntryPoint(HWND hwnd, HINSTANCE hinst, LPSTR lpszCmdLine, int nCmdShow);

It does not support running arbitrary entry points. Since ShellExecute does not have that signature, clearly bad things will happen.

INFO: Windows Rundll and Rundll32 Interface has more info on the rundll32 interface.

If you want to do the equivelent of ShellExecute from the command line, just use start:

C:\>start "C:\Documents and Settings\admin\Desktop\tmp"

Hawthorne

How do I know which DLL exports have the proper signature? – Gleesome

You shoudln't be calling Rundll32 with random functions. Either it is documented that you can use Rundll32 (for install, InstallHinfSection) or you provide the export your self. – Hawthorne

If you're brave, you can call other functions - which will accept these 4 parameters (through cdecl calling convention). This is possible for some argumentless functions, or others that would just accept a meaningless handle or two as arguments. You can also easily write your own DLLs, with entry points (=dll exports) adhering to this signature, and call them with rundll32. And the functions in WinAPI are documented in MSDN. You'll see that really few can be used directly with rundll32... – Ankylostomiasis

It seems like it doesn't have to be void CALLBACK EntryPoint(HWND hwnd, HINSTANCE hinst, LPSTR lpszCmdLine, int nCmdShow). I compiled my own DLL with extern "C" void Example() and it ran properly (extern "C" was there so I could get a clean function name, void Example() works as well as long as you use Dependency Walker to find the name of the function). So it seems like it just has to be void for it to work. However, I'm not necessarily encouraging you to use this. – Suiting

If you just have extern "C" void Example(), it's most probably __cdecl (the default calling convention in which the caller cleans up the stack). CALLBACK is __stdcall calling convention, where the callee cleans up the stack. Your function will ignore the 4 parameters, and that's fine, but it won't clean up the stack, which means the stack will be corrupted when it returns, potentially crashing rundll32 process. – Crooked

FYI C:\> rundll32 user32.dll, MessageBeep -MB_ICONEXCLAMATION will play a sound. – Tokharian