Unable to read final few Kb of logical drives on Windows 7 64-bit

Question

I'm having some problems addressing logical drives. For clarity, my definition of 'Physical Disk' (PD) is the raw disk regardless of partitioning. 'Logical Drive' (LD) refers to a volume such as Drive E:, Drive F: etc.

Using the examples from RRUZ (my hero SO member) and implementing the WMI Class I have created a Freepascal program for reading disks. I address PD by \.\PhyscialDiskX and that works fine by the examples created by RRUZ (here). I can read all the bytes no problem for PD's.

I use the same handle technique for logical volumes, which are \?\E: or \?\F: etc. I then use IOCTL_DISK_GET_LENGTH_INFO to get the length of the PD or LV and then read the byte range until ReadBytes = TotalLength. I read on the MSDN website that it will automatically retrieve the size of whatever device handle it is passed - PD or LD alike. And indeed I have checked the szie values returned by my program againzt WinHex, FTK Imager, HxD and several other low level disk tools. With the exception of 1 byte variances caused by zero or 1 starting positions, they match.

However, for some reason, my program is failing to acquire the final 32Kb on Windows 7 Pro 64-bit, despite running the program as administrator. It reads the whole disk and then on the final buffer read (which is done as 64Kb buffers) BytesRead returns -1. Using the debugger I worked out the following values :

493,846,527 exact LV size of Drive F:
493,813,760 total bytes read at time of failure
32,767 bytes missing

The result of the following

BytesRead    := FileRead(hDiskHandle, Buffer, (DiskSize - TotalBytesRead));

is -1 on the final buffer read. THis the line that tests for the end of the disk by saying "if the amount left to read is less than the size of the buffer size, only try to read what is left". So, the value of bytes being asked to be stored by FileRead at the end is 32,767 (because DiskSize - TotalBytesRead at that point is 32,767, meaning that's how many bytes are left to read of the disk). The designated size of buffer is 64Kb. My understanding is that you can put less in a buffer than it is capable of holding but not more (FileRead states : "Buffer must be at least Count bytes long. No checking on this is performed"? IS that correct? If it is not then this may be (and probably is) the issue.

I don't know if it's due to IOCTL_DISK_GET_LENGTH_INFO, the buffer storage or something else? Hoping someone can help? I have also posted along with some screenshot at the Lazarus Freepascal forum. Here is my relevant code sections:

The handle:

// Create handle to source disk. Abort if fails
  hSelectedDisk := CreateFileW(PWideChar(SourceDevice), FILE_READ_DATA,
                   FILE_SHARE_READ, nil, OPEN_EXISTING, FILE_FLAG_SEQUENTIAL_SCAN, 0);

  if hSelectedDisk = INVALID_HANDLE_VALUE then
  begin
    RaiseLastOSError;
  end 

Compute the size ion bytes of the given device:

ExactDiskSize := GetDiskLengthInBytes(hSelectedDisk);

Now read the device and store the input as a flat file

ImageResult   := WindowsImageDisk(hSelectedDisk, ExactDiskSize, HashChoice, hImageName);

Functions for the above:

function GetDiskLengthInBytes(hSelectedDisk : THandle) : Int64;
const
  // These are defined at the MSDN.Microsoft.com website for DeviceIOControl
  // and https://forum.tuts4you.com/topic/22361-deviceiocontrol-ioctl-codes/
  {
  IOCTL_DISK_GET_DRIVE_GEOMETRY      = $0070000
  IOCTL_DISK_GET_PARTITION_INFO      = $0074004
  IOCTL_DISK_SET_PARTITION_INFO      = $007C008
  IOCTL_DISK_GET_DRIVE_LAYOUT        = $007400C
  IOCTL_DISK_SET_DRIVE_LAYOUT        = $007C010
  IOCTL_DISK_VERIFY                  = $0070014
  IOCTL_DISK_FORMAT_TRACKS           = $007C018
  IOCTL_DISK_REASSIGN_BLOCKS         = $007C01C
  IOCTL_DISK_PERFORMANCE             = $0070020
  IOCTL_DISK_IS_WRITABLE             = $0070024
  IOCTL_DISK_LOGGING                 = $0070028
  IOCTL_DISK_FORMAT_TRACKS_EX        = $007C02C
  IOCTL_DISK_HISTOGRAM_STRUCTURE     = $0070030
  IOCTL_DISK_HISTOGRAM_DATA          = $0070034
  IOCTL_DISK_HISTOGRAM_RESET         = $0070038
  IOCTL_DISK_REQUEST_STRUCTURE       = $007003C
  IOCTL_DISK_REQUEST_DATA            = $0070040
  IOCTL_DISK_CONTROLLER_NUMBER       = $0070044
  IOCTL_DISK_GET_PARTITION_INFO_EX   = $0070048
  IOCTL_DISK_SET_PARTITION_INFO_EX   = $007C04C
  IOCTL_DISK_GET_DRIVE_LAYOUT_EX     = $0070050
  IOCTL_DISK_SET_DRIVE_LAYOUT_EX     = $007C054
  IOCTL_DISK_CREATE_DISK             = $007C058
  IOCTL_DISK_GET_LENGTH_INFO         = $007405C  // Our constant...
  SMART_GET_VERSION                  = $0074080
  SMART_SEND_DRIVE_COMMAND           = $007C084
  SMART_RCV_DRIVE_DATA               = $007C088
  IOCTL_DISK_GET_DRIVE_GEOMETRY_EX   = $00700A0
  IOCTL_DISK_UPDATE_DRIVE_SIZE       = $007C0C8
  IOCTL_DISK_GROW_PARTITION          = $007C0D0
  IOCTL_DISK_GET_CACHE_INFORMATION   = $00740D4
  IOCTL_DISK_SET_CACHE_INFORMATION   = $007C0D8
  IOCTL_DISK_GET_WRITE_CACHE_STATE   = $00740DC
  IOCTL_DISK_DELETE_DRIVE_LAYOUT     = $007C100
  IOCTL_DISK_UPDATE_PROPERTIES       = $0070140
  IOCTL_DISK_FORMAT_DRIVE            = $007C3CC
  IOCTL_DISK_SENSE_DEVICE            = $00703E0
  IOCTL_DISK_INTERNAL_SET_VERIFY     = $0070403
  IOCTL_DISK_INTERNAL_CLEAR_VERIFY   = $0070407
  IOCTL_DISK_INTERNAL_SET_NOTIFY     = $0070408
  IOCTL_DISK_CHECK_VERIFY            = $0074800
  IOCTL_DISK_MEDIA_REMOVAL           = $0074804
  IOCTL_DISK_EJECT_MEDIA             = $0074808
  IOCTL_DISK_LOAD_MEDIA              = $007480C
  IOCTL_DISK_RESERVE                 = $0074810
  IOCTL_DISK_RELEASE                 = $0074814
  IOCTL_DISK_FIND_NEW_DEVICES        = $0074818
  IOCTL_DISK_GET_MEDIA_TYPES         = $0070C00
  }
  IOCTL_DISK_GET_LENGTH_INFO  = $0007405C;
type
  TDiskLength = packed record
    Length : Int64;
  end;

var
  BytesReturned: DWORD;
  DLength: TDiskLength;
  ByteSize: int64;

begin
  BytesReturned := 0;
  // Get the length, in bytes, of the physical disk
  if not DeviceIOControl(hSelectedDisk,  IOCTL_DISK_GET_LENGTH_INFO, nil, 0,
         @DLength, SizeOf(TDiskLength), BytesReturned, nil) then
           raise Exception.Create('Unable to determine byte capacity of disk.');
  ByteSize := DLength.Length;
  ShowMessage(IntToStr(ByteSize));
  result := ByteSize;
end;

The disk reader function

function WindowsImageDisk(hDiskHandle : THandle; DiskSize : Int64; HashChoice : Integer; hImageName : THandle) : Int64;
var
  Buffer                   : array [0..65535] of Byte;   // 1048576 (1Mb) or 262144 (240Kb) or 131072 (120Kb buffer) or 65536 (64Kb buffer)

  BytesRead                : integer;

  NewPos, SectorCount,
  TotalBytesRead, BytesWritten, TotalBytesWritten : Int64;
  ... 
 // Now to seek to start of device
      FileSeek(hDiskHandle, 0, 0);
        repeat
          // Read device in buffered segments. Hash the disk and image portions as we go
          if (DiskSize - TotalBytesRead) < SizeOf(Buffer) then
            begin
              // Read 65535 or less bytes
              BytesRead    := FileRead(hDiskHandle, Buffer, (DiskSize - TotalBytesRead));
              BytesWritten := FileWrite(hImageName, Buffer, BytesRead);
            end
          else
            begin
              // Read 65536 (64kb) at a time
              BytesRead     := FileRead(hDiskHandle, Buffer, SizeOf(Buffer));
              BytesWritten  := FileWrite(hImageName, Buffer, BytesRead);
            end;
          if BytesRead = -1 then
            begin
              ShowMessage('There was a read error encountered. Aborting');
              // ERROR IS THROWN AT THIS POINT ONLY WITH LD's - not PD's
              exit;
            end
          else
          begin
          inc(TotalBytesRead, BytesRead);
          inc(TotalBytesWritten, BytesWritten);
          NewPos := NewPos + BytesRead;   
...
  until (TotalBytesRead = DiskSize);

Downbow

What does GetLastError return? How do you open hDiskHandle? – Attendant

Why read 64k if there's only 32k there. Perhaps that's the issue. – Player

It reads in 64Kb segments until there's 64Kb or less to read. If that's the case, it only reads the remainder, whatever that may be. At least that is how I designed it, unless it's worked previously only by fluke and all the physical disks have just happened to be a perfect division of exactly 64Kb. See if (DiskSize - TotalBytesRead) < SizeOf(Buffer) then begin // Read 65535 or less bytes BytesRead := FileRead(hDiskHandle, Buffer, (DiskSize - TotalBytesRead)); BytesWritten := FileWrite(hImageName, Buffer, BytesRead); – Downbow

Try adding some error checking to your code, something like if BytesRead = -1 then RaiseLastOSError; – Huldahuldah

@Gizmo You definitely should be checking for errors. That's critically important. FWIW, I'd use the raw Win32 api calls for such low level code. – Player

I'm also curious to know why you packed that record. – Player

Using the RaiseLastOSError I get : "System Error: (OS Code 1784): The supplied buffer is not valid for the requested operation.". I will Google that, but meanwhile, if that means anything to anyone do let me know! – Downbow

@Downbow It's telling you that your buffer size is not a multiple of the sector size – Player

David - I get the distinct impression you know the answer and am leading me to the answer in the hope I will realise it for myself, but I don't get what you mean. Sectors are typically 512 bytes. My buffer capacity is 64Kb, which is a multiple of 512. So if you know what it should be, please do tell me. – Downbow

It's as I stated in my answer. You are calling ReadFile requesting to read a buffer whose length is not a multiple of the sector size. The fact that you allocated a buffer of length 65,536 is of no concern to ReadFile since all it knows is you asked to read 32,767 bytes. Which is not a multiple of the sector size. Anyway, you don't seem to believe me. Especially as you seem to think that 32,767 is a multiple of 512. – Player

Answer 1

Probably, it is a boundary check error. Quote from MSDN (CreateFile, note on opening physical drives and volumes, which you call logical drives):

To read or write to the last few sectors of the volume, you must call DeviceIoControl and specify FSCTL_ALLOW_EXTENDED_DASD_IO

Watering

This may be it. I did try that before but in relation to an earlier infinite loop problem I was having. I forgot about it for this issue. It certainly makes sense but can't try it until Sunday. I will let you know then. Thanks. – Downbow

I have tried to add FSCTL_ALLOW_EXTENDED_DASD_IO by first creating the handle as stated in my post, followed immediately after the exception check with : if not DeviceIOControl(hSelectedDisk, FSCTL_ALLOW_EXTENDED_DASD_IO, nil, 0, @DLength, SizeOf(TDiskLength), BytesReturned, nil) then raise Exception.Create('Unable to initiate EXTENDED_DASD_IO.'); but same problem remains. Is that the right to use it? What code is it supposed to be assigned? I have had to assign it as a const the same as IOCTL_DISK_GET_LENGTH_INFO = $0007405C. Is that right? – Downbow

Answer 2

I suspect that the problem stems from the use of 64-bit integers and arithmetic to calculate a value passed as a 32-bit Integer:

FileRead(hDiskHandle, Buffer, (DiskSize - TotalBytesRead));

I cannot explain with certainty why this might affect only LD's and not PD's, except to speculate that there may be some difference in the reported DiskSize that somehow avoids the Int64 arithmetic/32-bit problem in that case.

e.g. If the 32-bit truncated result of the Int64 arithmetic is NEGATIVE (which requires only that the high bit be set, i.e. 1 not 0) then FileRead() will return -1 since a negative value for "bytes to read" is invalid.

But if the high-bit in the result is NOT set, resulting in a positive value, then even if that value is significantly greater than 64KB this will not cause an error since this invocation is called only when you have already determined that there are fewer than 64K bytes to be read. The 32-bit truncated Int64 arithmetic may result in a request to read 2 BILLION bytes but FileRead() is only going to read the actual 32K bytes that remain anyway.

However, this very fact points to a solution (assuming that this diagnosis is correct).

As noted, FileRead() (which is just a wrapper around ReadFile(), on Windows) will read either the number of bytes specified or as many bytes remain to be read, whichever is lower.

So if you specify 64KB but only 32KB remain, then only 32KB will be read.

You can replace all of this code:

if (DiskSize - TotalBytesRead) < SizeOf(Buffer) then
begin
  // Read 65535 or less bytes
  BytesRead    := FileRead(hDiskHandle, Buffer, (DiskSize - TotalBytesRead));
  BytesWritten := FileWrite(hImageName, Buffer, BytesRead);
end
else
begin
  // Read 65536 (64kb) at a time
  BytesRead     := FileRead(hDiskHandle, Buffer, SizeOf(Buffer));
  BytesWritten  := FileWrite(hImageName, Buffer, BytesRead);
end;

With simply:

BytesRead     := FileRead(hDiskHandle, Buffer, SizeOf(Buffer));
BytesWritten  := FileWrite(hImageName, Buffer, BytesRead);

This eliminates the 64-bit arithmetic and any possibility for errors resulting from 64-bit results being passed in 32-bit values. If the final segment contains only 32KB, then only 32KB will be read.

You can also simplify your loop termination (removing the 64-bit arithmetic, unless you need the accumulated values for other purposes). Instead of accumulating the total bytes read, you can terminate your loop simply when your FileRead() reads less than the number of bytes specified. i.e. BytesRead < 64KB:

Even if your disk is an exact multiple of 64KB blocks, your penultimate FileRead() will return a full buffer of 64KB, and the very next FileRead() will read 0 bytes, which is < 64KB, terminating the loop. :)

Godless

Your code is simpler but the first half of your answer is just wrong. There's no problem with the arithmetic. – Player

Maybe not in this specific case (I did mention that this was speculative and not proven). But mixing 64-bit arithmetic with 32 bit types (especially with implicit casting as in this case, passing the result of an expression to a parameters) absolutely can cause the problems described. Problem solving 101 is reducing unnecessary complexity and eliminating the potential for such problems. Once you have reduced the problem to its simplest possible form the cause of the problem then often becomes obvious. – Godless

Since the disk size won't fit in 32 bit type, and the file api accepts 32 bit types, it is sometimes essential to mix 32 and 64 bit arithmetic. One just needs to understand it. Whilst your code is cleaner, it behaves in exactly the same way as that in the question. – Player

Who mentioned "mixing 32-bit and 64-bit arithmetic" ? Not me (you will see that I am very careful to ensure this distinction). But truncating the result of a 64-bit arithmetic expression to 32-bits absolutely is an issue and is precisely what happens in the call to FileRead() in this case (silently in Delphi, btw). In this case the condition on that branch of the code should avoid the potential for error but this depends very much on the Int64 codegen for that condition in the compiler. Bottom line: #101 - remove unnecessary complexity and solve what's left. – Godless

(DiskSize - TotalBytesRead) cannot be negative. You should delete this non-answer. – Player

If I remove the "if buffer is less than 64Kb" error check loop, it get to the end of the disk and then just loops infinately at 100 CPU utilisation. I think I just need to start again from scratch because this is getting all very confusing now. I thought the answer would be simple but know I've tried so many things I've lost count of what I had before :-( – Downbow

Answer 3

32,767 is an odd number and not a multiple of the sector size. That means that the final part sector is simply not readable.

Player

But the whole disk is read fine as a physical disk. No errors, exact match to other tools etc. – Downbow

Direct disk access requires access to be sector aligned – Player

Counting from zero, 32,767 is exact sectors of 512 (512 * 2 * 2 * 2 * 2 * 2* 2 = 32,768) – Downbow

You are counting wrong. It's an odd number! That's like saying, counting from zero, I have zero heads! – Player

:-). I'm sure you're correct but I am sure the error is my code, not the disk. – Downbow

I'm not saying that the error is on the disk. I'm saying that the error is in your code. – Player

The error may be with the disk (volume) after all, the missing byte is due to disk size not being a multiple of 512 (or a power of 2 for that matter) in the question (493,846,527). – Hugely

@SertacAkyuz My point is that the code is wrong to attempt a read with a buffer size not a multiple of sector size. I don't know whether or not there is a problem with the disk, but the code needs to follow sector alignment rules. – Player

I commented because the poster might be chasing the wrong thing. – Hugely

@SertacAkyuz Yes, that seems likely – Player

Answer 4

OK, I've done it.

Credit to user2024154 as that was the first major thing. So based on that I've granted the answer there.

However, what was not clear was how to properly assign its const value. After many hours of Googling I stumbled across this . It was the only Delphi example I could find where it actually shows FSCTL_ALLOW_EXTENDED_DASD_IO defined, allbeit one had to look through much of it to pull the values together.

For the benefit of anyone else, the values that I needed, which now work, is :

const
   FILE_DEVICE_FILE_SYSTEM = $00000009;
   FILE_ANY_ACCESS = 0;
   METHOD_NEITHER = 3;
   FSCTL_ALLOW_EXTENDED_DASD_IO = ((FILE_DEVICE_FILE_SYSTEM shl 16)
                                    or (FILE_ANY_ACCESS shl 14)
                                    or (32 shl 2) or METHOD_NEITHER);   

I then used FSCTL_ALLOW_EXTENDED_DASD_IO after first creating the handle and then :

 if not DeviceIOControl(hSelectedDisk, FSCTL_ALLOW_EXTENDED_DASD_IO, nil, 0,
             nil, 0, BytesReturned, nil) then
               raise Exception.Create('Unable to initiate FSCTL_ALLOW_EXTENDED_DASD_IO disk access.');

This works with freepascal and with some minor adjustment should work with Delphi.

Thanks to you all for your continued help, especially user2024154 and, as always, David, for his continued assistance.

UPDATE : Except now physcial disk access doesn't work at all! But I'll work something out.

Downbow

"After many hours of Googling ..." - Install Windows SDK and you'll be able to find how it's defined in no time. – Hugely

This is why, as I always say, that you should have a working knowledge of C and C++. printf("%u\n", FILE_DEVICE_FILE_SYSTEM) does the job in seconds. – Player