Can I create a file in Windows that only exists in memory - and if so, how?

Question

_{This question is not a duplicate of any of these existing questions:}

• _{How can I store an object File that only exists in memory as a file inside of my storage system? - This question is not about Java's File API.}
• _{Temp file that exists only in RAM? - This is close to what I'm asking, except the OP isn't asking how to create files from memory for the purposes of sharing passing them to child-processes}
• _{I'm not asking about Win32's Memory-mapped File either - as they're essentially the opposite of what I'm after: a memory-mapped file is a file-on disk that's mapped to a process' virtual memory space - whereas what I want is a file that exists in the OS' filesystem (but not the disk's physical filesystem) like a mount-point and that file's data is mapped to an existing buffer in memory.}
  • _{I.e., with Memory Mapped Files, writing/writing to a byte at a particular buffer address and offset in memory will cause the byte at the same offset from the start of the file to be modified - but the file physically exists on-disk, which isn't what I want.}

To elaborate and to provide context:

• I have an ASP.NET Core server-side application that receives request streams sized between 1 and 10MB on a regular basis. This program will run only on Windows / Windows Server, so using Windows-specific functionality is fine.
• 75% of the time my application just reads through these streams by itself and that's it.

• But a minority of the time it needs to have a separate applications read the data which it starts using Process.Start and passing the file-name as a command-line argument.

  • It passes the data to these separate applications by saving the stream to a temporary file on-disk and passing the filename of that stream.
  • Unfortunately it can't write the content to the child-process's stdin because some of the those programs expect a file on-disk rather than reading from stdin.

• Additionally, while the machine it's running on has lots of RAM (so keeping the streams buffered in-memory is fine) it has slow spinning-rust HDDs, which is further reason to avoid temporary files on-disk.

• I'd like to avoid unnecessary buffering and copies - ideally I'd like to stream the entire 1-10MB request into a single in-memory buffer, and then expose that same buffer to other processes and use that same buffer as the backing for a temporary file.

• If I were on Linux, I could use tmpfs - it isn't perfect:

  • To my knowledge, an existing process can't instruct the OS to take an existing region of its virtual-memory and map a file in tmpfs to that memory region, instead tmpfs still requires that the file be populated by writing (i.e. copying) all of the data to its file-descriptor - which is counter to the aim of having a zero-copy system.

• Windows' built-in RAM-disk functionality is limited to providing the basis for a RAM-disk implementation via a third-party device-driver - I'm surprised that Microsoft never shipped Windows with a built-in RAM-disk GUI or API, especially given their relative simplicity.

  • The ImDisk program is an implementation of a RAM-disk using Microsoft's RAM-disk driver platform, but as far as I can tell while it's more like tmpfs in that it can create a file that exists only in-memory, it doesn't allow the file's data to be backed by a buffer directly accessible to a running process (or a shared-memory buffer).

Surgeon

Possible duplicate of C - create file in memory, C++/Win32 Create a file in virtual memory, etc – Anaemia

There seems no such user application level API for this purpose. It can be done via implementing a driver. Could you mind sharing the reason of using RAM? For better read/write speed? – Panayiotis

@RitaHan-MSFT I want to pass 1GB of data from one process to a new child-process: but that process can only accept a filename instead of passing it through stdin - and I'd like to avoid copying that 1GB around in-memory. – Surgeon

Answer 1

CreateFileMapping with hFile = INVALID_HANDLE_VALUE "creates a file mapping object of a specified size that is backed by the system paging file instead of by a file in the file system".

From Raymond Chen's The source of much confusion: “backed by the system paging file”:

In other words, “backed by the system paging file” just means “handled like regular virtual memory.”

If the memory is freed before it ever gets paged out, then it will never get written to the system paging file.

Inexorable

Ah, I didn't know you could create a page-file backed memory-mapped file - however how can I map that file to an existing shared-memory buffer? I see I can use MapViewOfFileEx to map a memory-mapped file (including a pagefile-backed mapped file) into my process' memory space, but I guess I'd have to use MapViewOfFileEx first and store the buffer into there - but I wonder what would happen if I passed-in a pointer to a buffer already allocated. – Surgeon

Update: turns out you can't. The documentation for MapViewOfFileEx states: "the function succeeds if the specified memory region is not already in use by the calling process". Bummer. – Surgeon

@Surgeon Depending on the use-case, you may be able to do it the other way around and use the mapping for sharing, with Creating Named Shared Memory and OpenFileMapping. – Inexorable

A paging-file backed mapping isn't a regular file that a program can open, so it doesn't satisfy the requirements. The program could create a regular file with the attribute FILE_ATTRIBUTE_TEMPORARY to avoid writing to disk as much as possible. – Ynes

@ErykSun isn't a regular file Right, of course. On the other hand, no file "that only exists in memory" (as the title says) can be a "regular" file. A named mapping can, however, be opened even though not with CreateFile. – Inexorable

A file can exist just in memory if it's on a ramdisk. Dai mentions this, but Microsoft doesn't distribute a ramdisk driver with Windows. If Dai is willing to use a 3rd party tool such as ImDisk, a file on the ramdisk could be mapped into the process virtual memory. – Ynes

The OP introduces another constraint, though: The file contents need to be mapped into memory at a predefined location (that presumably aliases an existing buffer). – Morman