Writing a Windows NT subsystem [closed]

Question

I'd like to try writing my own minimal NT subsystem on Windows 7 for purely educational purposes -- something like a bare-bones equivalent of the posix.exe in Microsoft's Subsystem for Unix-based Applications.

But I can't seem to find any public documentation on this topic. What API does a subsystem need to implement? How does it get registered with Windows? How does the subsystem image need to be built (what flags need to be set in the PE header, etc.)?

I'd most like to find a book or web site with an overview of the entire subject, or even the source code for a "hello world" NT subsystem that someone else has written. But anything at all would be appreciated if you can point me in the right direction here...

Anechoic

The native API is not documented. – Kilimanjaro

-1? This is an interesting question. Just because no one is likely to know the answers doesn't make it a bad question. – Charybdis

A lot of the Win32 "subsystem" support is hardcoded into the kernel, and I don't think there are any special system calls for custom subsystems. All I know is that smss can be configured to start subsystems. What are you trying to do though? A "subsystem" will simply consist of an API, which you put in a DLL. This may use LPC to invoke your subsystem server. What else is there? – Vacuity

@Hans: No offense, but that is one of the most annoying, least constructive, and most incorrect comments that people give regarding the native API. See, for example, here. – Junket

@wj32: Ah, there's more to it -- if you can use the a native API, you can make native programs like AutoChk that run during boot time, for example. So a subsystem that can help with that would be useful. – Junket

@Lambert - that's the WDK API, not the same thing as the usermode API. Careful throwing those stones. – Kilimanjaro

@Hans, from that MSDN page: Note: If the call to this function occurs in user mode, you should use the name "NtOpenFile" instead of "ZwOpenFile". That looks like documentation for a usermode API to me... although it's unclear whether it was intended to be used as such :) – Accede

@bdonlan: Thanks for copying that here, it was exactly what I was referring to. :) There's also pages like this one and others, with documentation on a lot of the said "undocumented" APIs (although obviously not every single native function is documented). – Junket

@Lambert: Of course you'll be using the Native API. I just mean, why call it a "subsystem"? You'll just end up making a server executable and a client DLL that communicates with the server using LPC. – Vacuity

@wj32: Yes, okay, I guess the word "subsystem" is overkill, since no one will probably implement a new subsystem like Win32 or POSIX. :) – Junket

@Hans: I agree with you that the Native API is supposed to be undocumented. But I don't think it's fair to make a distinction between the "WDK API" and the "usermode API". What makes the user-mode entry points so different that the WDK documentation doesn't apply? – Vacuity

@Lambert: My problem isn't with the word "subsystem", it's with the question. Chris Becke claims "no one is likely to know the answers"; I disagree. We all know about the Native API and SMSS, it's just that the question is too vague. What does the OP mean by "subsystem"? Is the OP simply asking for ways to create native programs? Is the OP asking for information about the Session Manager? – Vacuity

@wj32: I assumed the OP's looking for a way to make something like kernel32.dll from scratch, and then have it load with processes just as how kernel32.dll loads with regular user processes. But I agree, it's vague. – Junket

@Lambert: If you link an executable to a DLL, it will get loaded automatically, won't it? – Vacuity

@wj32: Well... yeah, but I'm not sure what you mean...? – Junket

@Lambert: If an executable imports a function from foobar.dll, then that DLL will load with the executable. That's how you do a kernel32-like DLL. – Vacuity

@wj32: Not really, because kernel32 is special in that even if you don't link to it, it's still loaded because your exe is marked as using the Win32 subsystem. – Junket

@Lambert: But that's not what's required for what the OP is trying to do. Importing from the DLL works fine... – Vacuity

@wj32: But if his subsystem DLL is linked like a normal DLL, then it will load Kernel32 no matter what, and hence it'll just be an extension of Win32 rather than a new subsystem... – Junket

@Lambert: I thought we were assuming that the executables would be native? – Vacuity

@wj32: Well, if he's trying to make a new subsystem, it would be neither native nor Win32 -- it would be something new. But he would have to link the DLL natively, yes. :) – Junket

Midipix is a project that includes an open-source Windows subsystem called ntctty. – Estoppel

Answer 1

Here are the major components of a subsystem:

• User-mode server. The server creates a (A)LPC port and listens for and handles client requests.
• User-mode client DLL. In the DLL_INIT_ROUTINE, you can connect to the port set up by the server. This DLL will expose your subsystem's API, and some functions will require communication with the server.
• Kernel-mode support driver (you might not need this).

You will want to store process or thread state in either your server or driver. If you're storing it in the server, you might need something like NtRegisterThreadTerminatePort to ensure you get to clean up when a process or thread exits. If you're using a driver, you need PsSetCreateProcessNotifyRoutine.

And lastly, if you're on XP and below, you can add new system calls. You can do this by calling KeAddSystemServiceTable. To invoke the system calls from user-mode, you need to create stubs like this (for x86):

; XyzCreateFooBar(__out PHANDLE FooBarHandle, __in ACCESS_MASK DesiredAccess, ...)
mov     eax, SYSTEM_CALL_NUMBER
mov     edx, 0x7ffe0300
call    [edx]
retn    4

On Vista and above you can no longer add new system service tables because there is only room for two: the kernel's system calls and win32k's system calls.

After a bit of Googling I found this: http://winntposix.sourceforge.net/. I think it's very similar to what you're looking for, and uses a lot of the things I have mentioned.

Vacuity

Answer 2

I'm also obsessed with the native API. :)

And I'm glad to say that it's nowhere near as dangerous or as undocumented as some people make it seem. :]

There's no source code for "Hello, world" because the native API doesn't interact so easily with the console, since it's part of the Win32 subsystem and requires client/server communication with ports. If you need to write a console application, you need to communicate directly with CSRSS, whose message formats are undocumented (although some of its format can be found in ReactOS's source -- it would do you many benefits if you get familiar with ReactOS).

I'll post an example here soon that you might find interesting; for now, do be aware that your only option ever is to link with NTDLL.dll, and that, for that, you need the Driver Development Kit (since you need the lib file).

---

Update: Check this out!

(I have a feeling no one else will post something quite as rebellious as this. Showing GUI with the native API?! I must be crazy!)

#include <Windows.h>

typedef DWORD NTSTATUS;

//These are from ReactOS
typedef enum _HARDERROR_RESPONSE_OPTION
{
    OptionAbortRetryIgnore,
    OptionOk,
    OptionOkCancel,
    OptionRetryCancel,
    OptionYesNo,
    OptionYesNoCancel,
    OptionShutdownSystem
} HARDERROR_RESPONSE_OPTION, *PHARDERROR_RESPONSE_OPTION;

typedef enum _HARDERROR_RESPONSE
{
    ResponseReturnToCaller,
    ResponseNotHandled,
    ResponseAbort,
    ResponseCancel,
    ResponseIgnore,
    ResponseNo,
    ResponseOk,
    ResponseRetry,
    ResponseYes,
    ResponseTryAgain,
    ResponseContinue
} HARDERROR_RESPONSE, *PHARDERROR_RESPONSE;

typedef struct _UNICODE_STRING {
    USHORT  Length;
    USHORT  MaximumLength;
    PWSTR  Buffer;
} UNICODE_STRING, *PUNICODE_STRING;

//You'll need to link to NTDLL.lib
//which you can get from the Windows 2003 DDK or any later WDK
NTSYSAPI VOID NTAPI RtlInitUnicodeString(IN OUT PUNICODE_STRING DestinationString,
    IN PCWSTR SourceString);
NTSYSAPI NTSTATUS NTAPI NtRaiseHardError(IN NTSTATUS ErrorStatus,
    IN ULONG NumberOfParameters, IN ULONG UnicodeStringParameterMask,
    IN PULONG_PTR Parameters,
    IN HARDERROR_RESPONSE_OPTION ValidResponseOptions,
    OUT PHARDERROR_RESPONSE Response);
#define STATUS_SERVICE_NOTIFICATION_2 0x50000018

int main()
{
    HARDERROR_RESPONSE response;
    ULONG_PTR items[4] = {0};
    UNICODE_STRING text, title;
    RtlInitUnicodeString(&text,
        L"Hello, NT!\r\nDo you like this?\r\n"
        L"This is just about as pretty as the GUI will get.\r\n"
        L"This message will self-destruct in 5 seconds...");
    RtlInitUnicodeString(&title, L"Native Message Box!");
    items[0] = (ULONG_PTR)&text;
    items[1] = (ULONG_PTR)&title;
    items[2] = (ULONG_PTR)OptionYesNo;
    items[3] = (ULONG_PTR)5000;
    NtRaiseHardError(STATUS_SERVICE_NOTIFICATION_2, ARRAYSIZE(items),
        0x1 | 0x2 /*First two parameters are UNICODE_STRINGs*/, items,
        OptionOk /*This is ignored, since we have a custom message box.*/,
        &response);
    return 0;
}

If you have any questions, feel free to ask! I'm not scared of the native API! :)

---

Edit 2:

If you're trying to make your own DLL version of Kernel32 and have it load like Kernel32 does with every process (hence a new subsystem), I just wanted to let you know that I don't think it's possible. It's rather similar to this question that I asked a couple of days ago, and it seems that you can't extend the NT PE Loader to know about new subsystems, so I don't think it'll be possible.

Junket

ReactOS was my first thought, too. – Wallas

@Uwe: Yeah, except you can't find this particular example in there. :) (By the way, does anyone know how to use <pre> to force C++ syntax highlighting?) – Junket

An interesting example, but it seems a bit tangential to creating an actual subsystem, rather than just a native subsystem application – Accede

Haha yeah I guess I agree, it doesn't really answer the question I guess. It was fun to write this answer, though! :) – Junket

I know this is pretty dead, but since it came up in a search for me I wanted to add my two cents - it is possible to replace the PE loader in Windows, so while you can't extend it you could replace it and redirect calls to regular PE files to the original (I have done this before to cause 16-bit EXE/COM files to be loaded in DOSBox). There are some odd glitchy behaviors, but it does work. Also, you probably could use the ReactOS PE loader, or more questionably try to compile the leaked NT4 bits for it. – Renoir

@Yushatak: Interesting, thanks for sharing. – Junket