How to call DeviceIoControl to retrieve the amount of memory it needs?

Question

I'm trying to call DeviceIoControl(IOCTL_VOLUME_GET_VOLUME_DISK_EXTENTS) API, as shown here, but I need it to first "tell me" how much memory it needs (unlike the code I linked to.)

So I call it as such:

//First determine how much data do we need?
BYTE dummyBuff[1];
DWORD bytesReturned = 0;
if(!::DeviceIoControl(hDevice, dwIoControlCode, lpInBuffer, nInBufferSize, 
    dummyBuff, sizeof(dummyBuff), &bytesReturned, NULL))
{
    //Check last error
    int nError = ::GetLastError();
    if(nOSError == ERROR_INSUFFICIENT_BUFFER ||
        nOSError == ERROR_MORE_DATA)
    {
        //Alloc memory from 'bytesReturned' ...
    }
}

but it always returns error code 87, or ERROR_INVALID_PARAMETER and my bytesReturned is always 0.

So what am I doing wrong?

Dinger

Do you care to explain downvotes? – Dinger

The buffer size is documented as sizeof(VOLUME_DISK_EXTENTS), no need to query. Have you read the docs? – Disenthrone

We can't see all of the code needed to know what you are doing. A minimal reproducible example is very easy to make. Might I ask why you did not spend that extra time to do so. – Paratrooper

@JonathanPotter: Have you? Extents member of VOLUME_DISK_EXTENTS is defined with the size of ANYSIZE_ARRAY, which defaults to 1, which works only if NumberOfDiskExtents == 1, that will obviously fail if there's more than 1 extent. I don't think I need to explain this, do I? This is uncommon and that is why most code can get away with what you thought. – Dinger

@DavidHeffernan: To get hDevice for my code above, do this: CreateFile(L"\\\\.\\C:", GENERIC_READ, FILE_SHARE_READ | FILE_SHARE_WRITE, NULL, OPEN_EXISTING, 0, NULL); and set dwIoControlCode to IOCTL_VOLUME_GET_VOLUME_DISK_EXTENTS, and lpInBuffer and nInBufferSize to 0. – Dinger

Please make a minimal reproducible example. Don't ask the question in comments. – Paratrooper

You need to pass a VOLUME_DISK_EXTENTS structure with a sizeof(VOLUME_DISK_EXTENTS) output buffer size. On return, the call either succeeds, or "[...] the error code ERROR_MORE_DATA is returned. You should call DeviceIoControl again, allocating enough buffer space based on the value of NumberOfDiskExtents after the first DeviceIoControl call." This is documented. – Essayist

Answer 1

The instructions for getting all disk volume extents are documented under the VOLUME_DISK_EXTENTS structure:

When the number of extents returned is greater than one (1), the error code ERROR_MORE_DATA is returned. You should call DeviceIoControl again, allocating enough buffer space based on the value of NumberOfDiskExtents after the first DeviceIoControl call.

The behavior, if you pass an output buffer, that is smaller than sizeof(VOLUME_DISK_EXTENTS) is also documented at IOCTL_VOLUME_GET_VOLUME_DISK_EXTENTS control code:

If the output buffer is less than sizeof(VOLUME_DISK_EXTENTS), the call fails, GetLastError returns ERROR_INSUFFICIENT_BUFFER, and lpBytesReturned is 0 (zero).

While this explains the returned value in lpBytesReturned, it doesn't explain the error code 87 (ERROR_INVALID_PARAMETER)¹⁾.

The following code will return the disk extents for all volumes:

VOLUME_DISK_EXTENTS vde = { 0 };
DWORD bytesReturned = 0;
if ( !::DeviceIoControl( hDevice, IOCTL_VOLUME_GET_VOLUME_DISK_EXTENTS, NULL, 0, 
                         (void*)&vde, sizeof(vde), &bytesReturned, NULL ) )
{
    // Check last error
    int nError = ::GetLastError();
    if ( nError != ERROR_MORE_DATA )
    {
        // Unexpected error -> error out
        throw std::runtime_error( "DeviceIoControl() failed." );
    }

    size_t size = offsetof( VOLUME_DISK_EXTENTS, Extents[vde.NumberOfDiskExtents] );
    std::vector<BYTE> buffer( size );
    if ( !::DeviceIoControl( hDevice, IOCTL_VOLUME_GET_VOLUME_DISK_EXTENTS, NULL, 0, 
                             (void*)buffer.data(), size, &bytesReturned, NULL ) )
    {
        // Unexpected error -> error out
        throw std::runtime_error( "DeviceIoControl() failed." );
    }
    // At this point we have a fully populated VOLUME_DISK_EXTENTS structure
    const VOLUME_DISK_EXTENTS& result =
        *reinterpret_cast<const VOLUME_DISK_EXTENTS*>( buffer.data() );
}
else
{
    // Call succeeded; vde is populated with single disk extent.
}

---

Additional references:

• Why do some structures end with an array of size 1?
• offsetof Macro

---

¹⁾ At a guess I would assume, that BYTE[1] begins at a memory address, that is not sufficiently aligned for the alignment requirements of VOLUME_DISK_EXTENTS.

Essayist

When posting the answer I didn't have access to a disk with multiple extents. The documentation is unclear (to me anyway) whether it returns an error or success code, if the first DeviceIoControl call succeeds, but there are additional disk extents. @c00000fd: If you can test the code, I would be thankful, if you posted the result, so I can update the answer as necessary. – Essayist

Yes, thank you. Your method does work for IOCTL_VOLUME_GET_VOLUME_DISK_EXTENTS with more than one disk extent. What I didn't see is that this API is totally a$#backwards. Most WinAPIs return the size of required buffer in bytes if you didn't provide a buffer long enough. Anyway, this still doesn't solve the issue for me. My goal was to write a "shim" method that would allocate memory dynamically independent of dwIoControlCode (like it should've been done in this API in the first place.) – Dinger

@c00000fd: there is no method that works for every I/O control code, any more than there is a method that works for every Win32 API function. (How do you expect Windows to know how much buffer space the device driver is going to require?) – Ganoid

@HarryJohnston: Well, I just wrote one. How would Windows expect to know the size of the buffer? It'd ask the driver to provide it. Although this API is probably seriously old, going back to 1995 or even older, so knowing how sloppy Microsoft used to write their code back then, it is probably not even in the documentation for the device driver. So yes, in that case, they can't know... – Dinger

@c00000fd: you mean trial and error? Sure, that will work, sort of - but I assumed you wanted something a bit more elegant. (Better make sure you thoroughly document the fact that the IO control code may be issued multiple times, since that might have side-effects.) It's true that the device driver API could have required every driver to provide a guess as to the output buffer size needed, but it didn't, because it would be inefficient and because you're not supposed to be issuing I/O codes you don't understand anyway. – Ganoid

(Ideally, of course, the OS would allocate the output buffer for you. On modern computers, that might even be practical. But when Windows was originally written, it would have been too slow, and it isn't an important enough API to warrant re-implementation.) – Ganoid

@HarryJohnston: What side-effects are you referring to? Can you specify. – Dinger

I mean "side-effects" in the programming sense of the word, i.e., the control code might actually do something apart from returning data, and it might not be something the caller wanted to happen multiple times. I didn't have any particular control code in mind, so I'll make up an example out of thin air - oh, I don't know - a control code for a computerized toilet, which makes the toilet flush and returns information about the water flow rates during the flush. Calling your function might flush the toilet ten times instead of once. :-) (Which is fine, so long as it's documented!) – Ganoid

Answer 2

Following @IInspectable's advice, here's what I came up with for a more general case:

BYTE* DeviceIoControl_Dynamic(HANDLE hDevice, DWORD dwIoControlCode, DWORD dwszCbInitialSuggested, LPVOID lpInBuffer, DWORD nInBufferSize, DWORD* pncbOutDataSz)
{
    //Calls DeviceIoControl() API by pre-allocating buffer internally
    //'dwIoControlCode' = control code, see DeviceIoControl() API
    //'dwszCbInitialSuggested' = suggested initial size of the buffer in BYTEs, must be set depending on the description of 'dwIoControlCode'
    //'lpInBuffer' = input buffer, see DeviceIoControl() API
    //'nInBufferSize' = size of 'lpInBuffer', see DeviceIoControl() API
    //'pncbOutDataSz' = if not NULL, receives the size of returned data in BYTEs
    //RETURN:
    //      = Data obtained from DeviceIoControl() API -- must be removed with delete[]!
    //      = NULL if error -- check GetLastError() for info
    BYTE* pData = NULL;
    int nOSError = NO_ERROR;

    DWORD ncbSzData = 0;

    if((int)dwszCbInitialSuggested > 0)
    {
        //Initially go with suggested memory size
        DWORD dwcbMemSz = dwszCbInitialSuggested;

        //Try no more than 10 times
        for(int t = 0; t < 10; t++)
        {
            //Reserve mem
            ASSERT(!pData);
            pData = new (std::nothrow) BYTE[dwcbMemSz];
            if(!pData)
            {
                //Memory fault
                nOSError = ERROR_NOT_ENOUGH_MEMORY;
                break;
            }

            //And try calling with that size
            DWORD bytesReturned = 0;
            if(::DeviceIoControl(hDevice, dwIoControlCode, lpInBuffer, nInBufferSize, 
                pData, dwcbMemSz, &bytesReturned, NULL))
            {
                //Got it
                ncbSzData = bytesReturned;
                nOSError = NO_ERROR;

                break;
            }

            //Check last error
            nOSError = ::GetLastError();

            //Knowing how badly Windows drivers are written, don't rely on the last error code!

            //Alloc more memory (we'll just "wing it" on the amount)
            dwcbMemSz += 1024;

            //Free old mem
            delete[] pData;
            pData = NULL;
        }
    }
    else
    {
        //Bad initial size
        nOSError = ERROR_INVALID_MINALLOCSIZE;
    }

    if(pncbOutDataSz)
        *pncbOutDataSz = ncbSzData;

    ::SetLastError(nOSError);
    return pData;
}

and then to call it, say for IOCTL_VOLUME_GET_VOLUME_DISK_EXTENTS:

DWORD bytesReturned;
VOLUME_DISK_EXTENTS* p_vde = (VOLUME_DISK_EXTENTS*)DeviceIoControl_Dynamic(hDsk, 
    IOCTL_VOLUME_GET_VOLUME_DISK_EXTENTS, sizeof(VOLUME_DISK_EXTENTS), NULL, NULL, &bytesReturned);

which can be later used as such:

//Ensure that driver returned the correct data
if(p_vde &&
    offsetof(VOLUME_DISK_EXTENTS, Extents[p_vde->NumberOfDiskExtents]) <= bytesReturned)
{
    //All good
    for(int x = 0; x < p_vde->NumberOfDiskExtents; x++)
    {
        DWORD diskNumber = p_vde->Extents[x].DiskNumber;
        //...
    }
}

//Remember to free mem when not needed!
if(p_vde)
{
    delete[] (BYTE*)p_vde;
    p_vde = NULL;
}

Dinger

You've got the arithmetics wrong. sizeof(VOLUME_DISK_EXTENTS) + (p_vde->NumberOfDiskExtents - 1) * sizeof(p_vde->Extents) does not take alignment into account. I posted a link in my answer, that specifically explains, why your calculation is wrong. Plus, your wrapper isn't generic. If you pass dwszCbInitialSuggested that is smaller than sizeof(VOLUME_DISK_EXTENTS) together with IOCTL_VOLUME_GET_VOLUME_DISK_EXTENTS, it'll fail. Not good. – Essayist

And now that you've committed yourself to one particular allocator, you can no longer safely put the code in a DLL. It's also unusual to impose manual memory management on the client, when using C++. You could return a std::vector, or at least a std::unique_ptr. – Essayist

@IInspectable: Yeah, good point about alignment. How do you use offsetof macro with a struct pointer? – Dinger

As for memory allocation, you can choose whatever pleases you. I just posted my production code. – Dinger

offsetof( VOLUME_DISK_EXTENTS, Extents[p_vde->NumberOfDiskExtents] ); – Essayist

@IInspectable: Although you know, I just checked "my math" and it holds true. Did you see that I used <= bytesReturned which should account for alignment discrepancy. Also your other point about specifying a smaller original size in dwszCbInitialSuggested. Even if you pass 1 byte, it will take several passes to call DeviceIoControl but the overall method will eventually succeed. So I'm not sure what you meant there in your first comment? – Dinger

Take a disk with 100 extents. That makes your math accumulate 100 times the error. The expected size (which is a lot smaller than the required size) could now be lower than bytesReturned, although the real number of bytes (accounting for alignment) is beyond bytesReturned. The math is wrong, and works by coincidence. Occasionally. – Essayist

@IInspectable: Sorry, you didn't understand how my method works. If there's 100 disk extents, it will require about 2400 bytes (for a 32-bit process.) If I set dwszCbInitialSuggested as 1 byte, the first pass will fail, the second pass with 1025 bytes will fail as well, the third pass with 2049 bytes will also fail, but it will succeed on the 4th pass with 3073 bytes of allocated memory. 4 passes is not 100. – Dinger

I'm referring to the sanity check in the //Ensure that driver returned the correct data part. The math is off, and it can produce false positives, as described in my previous comment. – Essayist

@IInspectable: In that case where are you taking this "the real number of bytes (accounting for alignment) is beyond bytesReturned"? My calculation method in the sanity check part can only lower the number of bytes required because it doesn't account for the alignment. bytesReturned is returned by the driver, or how much data it filled out. I'm checking that the number of DISK_EXTENT structs it claims to have found fits the minimum required size. – Dinger

You are checking, if a number of bytes, or a smaller number have been returned by the driver.. Using offsetof returns the true size in bytes, regardless of the alignment rules of any given platform. – Essayist

@IInspectable: OK, we're arguing about something that is unrelated to my question. I agree your method is more acceptable for a general case (although it won't matter in my situation) so I changed it to offsetof in the example above. – Dinger

It might perform better if you allocate a big buffer and then shrink it afterwards rather than looping; many control codes are quite slow, and they'll already have done most of the work by the time they notice the buffer isn't big enough. Or have a big permanent buffer and then copy the data to a new buffer of the exact size. – Ganoid

@HarryJohnston: OK, good point. Although I would probably increment it by 4K instead of 1K, or maybe even 8K. For a modern Windows PC that would not be an issue but that buffer size would definitely be long enough for the driver to fill it with what it needs to. – Dinger

Answer 3

You are getting error code ERROR_INVALID_PARAMETER when you have invalid parameter, like its name says. In your case it should be bad handle because all others looks fine, if we expect that dwIoControlCode argument is IOCTL_VOLUME_GET_VOLUME_DISK_EXTENTS, lpInBuffer and nInBufferSize are ignored.

In insufficient buffer you will get another error code mentioned in above comments.

Lets check what is saying documentation:

DeviceIoControl can accept a handle to a specific device. For example, to open a handle to the logical drive A: with CreateFile, specify \.\a:. Alternatively, you can use the names \.\PhysicalDrive0, \.\PhysicalDrive1, and so on, to open handles to the physical drives on a system.

In other words, when you open handle with "C:\" instead of "\\.\c:" argument in CreateFile and use it in DeviceIoControl, the result is ERROR_INVALID_PARAMETER.

Antecedent