This question was very important for the project I'm working on, so I finally got it (almost) 100% working.
Note : all of the below code snippets are in C#
Thanks to the previous answers from Hannes de Jager who pointed me to the right direction and documentations, I can now read an USN journal from a VSS snapshot or any other special device that the regular API cannot work with ; in my case, I mean VMware snapshots mounted using the VDDK (VMware SDK for VM disks).
I also reused or imported code coming from great projects :
In case anyone else is interested, I share the code I use righ now, still in a rather crude state, but working.
How does it work?
First, you must acess the required Usn journal componenents. They are at the root of the device as ADS (alternate data streams) in an hidden entry. They cannot be accessed using the standard System.IO namespace, that's why I previously told I used the AlphaFS project, but pinvoking CreateFile() and ReadFile() should be enough.
1/2
The entry \$Extend\$UsnJrnl:$Max has global information about the current state of the journal. The most important parts are the usn journal ID (that you can use to check that the journal has not been reset if you want to compare multiple VSS snapshots) and the lowest valid USN journal sequence number.
USN journal structure:
// can be directly extracted from $MAX entry using Bitconverter.ToUint64
public struct USN_JOURNAL_DATA{
public UInt64 MaximumSize; //offset 0
public UInt64 AllocationDelta; // offset 8
public UInt64 UsnJournalID; // offset 16
public Int64 LowestValidUsn; // offset 24
}
2/2
The entry \$Extend\$UsnJrnl:$J contains the journal records. It is a sparse file so its disk usage is much lower than it size.
To answer the initial question, how can one know the Max used USN sequence from a previous VSS snapshot and compare it with that of another snapshot?
Well, the NextUsn value is simply equals to the size of the $Usnjrnl:$J entry.
On your "new" vss snapshot USN journal, you can seek to the "reference" VSS snapshot max USN before starting parsing records, if you want to parse the records changed between the two snapshots.
Generally speaking, each USN journal entry as a unique ID (USN number) which is the offset inside $J at whch the journal entry itself is located.
Each entry has a variable size, so to sequentially read then we have to calculate:
next entry offset inside $J =
offset of current entry (or its USN sequennce number + length of current entry
Fortunately the record length is a also a field of the USN entry record. Enough said, here is the USN record class:
public class UsnEntry : IComparable<UsnEntry>{
private const int FR_OFFSET = 8;
private const int PFR_OFFSET = 16;
private const int USN_OFFSET = 24;
private const int REASON_OFFSET = 40;
private const int FA_OFFSET = 52;
private const int FNL_OFFSET = 56;
private const int FN_OFFSET = 58;
public UInt32 RecordLength {get; private set;}
public Int64 USN {get; private set;}
public UInt64 FileReferenceNumber {get;private set;}
public UInt64 ParentFileReferenceNumber {get; private set;}
public UInt32 Reason{get; set;}
public string Name {get; private set;}
public string OldName{get; private set;}
private UInt32 _fileAttributes;
public bool IsFolder{
get{
bool bRtn = false;
if (0 != (_fileAttributes & Win32Api.FILE_ATTRIBUTE_DIRECTORY))
bRtn = true;
return bRtn;
}
}
public bool IsFile{
get{
bool bRtn = false;
if (0 == (_fileAttributes & Win32Api.FILE_ATTRIBUTE_DIRECTORY))
bRtn = true;
return bRtn;
}
}
/// <summary>
/// USN Record Constructor
/// </summary>
/// <param name="p">Buffer pointer to first byte of the USN Record</param>
public UsnEntry(IntPtr ptrToUsnRecord){
RecordLength = (UInt32)Marshal.ReadInt32(ptrToUsnRecord); //record size
FileReferenceNumber = (UInt64)Marshal.ReadInt64(ptrToUsnRecord, FR_OFFSET);
ParentFileReferenceNumber = (UInt64)Marshal.ReadInt64(ptrToUsnRecord, PFR_OFFSET);
USN = (Int64)Marshal.ReadInt64(ptrToUsnRecord, USN_OFFSET);
Reason = (UInt32)Marshal.ReadInt32(ptrToUsnRecord, REASON_OFFSET);
_fileAttributes = (UInt32)Marshal.ReadInt32(ptrToUsnRecord, FA_OFFSET);
short fileNameLength = Marshal.ReadInt16(ptrToUsnRecord, FNL_OFFSET);
short fileNameOffset = Marshal.ReadInt16(ptrToUsnRecord, FN_OFFSET);
Name = Marshal.PtrToStringUni(new IntPtr(ptrToUsnRecord.ToInt32() + fileNameOffset), fileNameLength / sizeof(char));
}
public int CompareTo(UsnEntry other){
return string.Compare(this.Name, other.Name, true);
}
public override string ToString(){
return string.Format ("[UsnEntry: RecordLength={0}, USN={1}, FileReferenceNumber={2}, ParentFileReferenceNumber={3}, Reason={4}, Name={5}, OldName={6}, IsFolder={7}, IsFile={8}", RecordLength, USN, (int)FileReferenceNumber, (int)ParentFileReferenceNumber, Reason, Name, OldName, IsFolder, IsFile);
}
}
I tried to isolate the smallest part of code that can parse an USN journal and extract its entries, starting from the lowest valid one. Remember, a record has a variable length; also note that some records point to a next record that is empty (the first 4 bytes, which are normally the record length, are zeroed). In this case I seek 4 bytes and retry parsing, until I get the next record. This behavior also have been reported by people having written similar parsing tools in Python so I guess I'n not too wrong here.
string vol = @"\\?\path_to_your_VSS_snapshot";
string maxHandle = vol + @"\$Extend\$UsnJrnl:$Max";
string rawJournal= vol + @"\$Extend\$UsnJrnl:$J";
// cannot use regular System.IO here, but pinvoking ReadFile() should be enough
FileStream maxStream = Alphaleonis.Win32.Filesystem.File.OpenRead(maxHandle);
byte[] maxData = new byte[32];
maxStream.Read(maxData, 0, 32);
//first valid entry
long lowestUsn = BitConverter.ToInt64(maxData, 24);
// max (last) entry, is the size of the $J ADS
IntPtr journalDataHandle = Win32Api.CreateFile(rawJournal,
0,
Win32Api.FILE_SHARE_READ| Win32Api.FILE_SHARE_WRITE,
IntPtr.Zero, Win32Api.OPEN_EXISTING,
0, IntPtr.Zero);
Win32Api.BY_HANDLE_FILE_INFORMATION fileInfo = new Win32Api.BY_HANDLE_FILE_INFORMATION();
Win32Api.GetFileInformationByHandle(journalDataHandle, out fileInfo);
Win32Api.CloseHandle(journalDataHandle);
long lastUsn = fileInfo.FileSizeLow;
int read = 0;
byte[] usnrecord;
byte[] usnraw = new byte[4]; // first byte array is to store the record length
// same here : pinvoke ReadFile() to avoid AlphaFS dependancy
FileStream rawJStream = Alphaleonis.Win32.Filesystem.File.OpenRead(rawJournal);
int recordSize = 0;
long pos = lowestUsn;
while(pos < newUsnState.NextUsn){
seeked = rawJStream.Seek(pos, SeekOrigin.Begin);
read = rawJStream.Read(usnraw, 0, usnraw.Length);
recordSize = BitConverter.ToInt32(usnraw, 0);
if(recordSize == 0){
pos = pos+4;
continue;
}
usnrecord = new byte[recordSize];
rawJStream.Read(usnrecord, 4, recordSize-4);
Array.Copy(usnraw, 0, usnrecord, 0, 4);
fixed (byte* p = usnrecord){
IntPtr ptr = (IntPtr)p;
// here we use the previously defined UsnEntry class
Win32Api.UsnEntry entry = new Win32Api.UsnEntry(ptr);
Console.WriteLine ("entry: "+entry.ToString());
ptr = IntPtr.Zero;
}
pos += recordSize;
}
Here are the pinvokes I use:
public class Win32Api{
[StructLayout(LayoutKind.Sequential, Pack = 1)]
public struct BY_HANDLE_FILE_INFORMATION{
public uint FileAttributes;
public FILETIME CreationTime;
public FILETIME LastAccessTime;
public FILETIME LastWriteTime;
public uint VolumeSerialNumber;
public uint FileSizeHigh;
public uint FileSizeLow;
public uint NumberOfLinks;
/*public uint FileIndexHigh;
public uint FileIndexLow;*/
public FileID FileIndex;
}
[DllImport("kernel32.dll", SetLastError = true)]
[return: MarshalAs(UnmanagedType.Bool)]
public static extern bool
GetFileInformationByHandle(
IntPtr hFile,
out BY_HANDLE_FILE_INFORMATION lpFileInformation);
[DllImport("kernel32.dll", SetLastError = true)]
public static extern IntPtr
CreateFile(string lpFileName,
uint dwDesiredAccess,
uint dwShareMode,
IntPtr lpSecurityAttributes,
uint dwCreationDisposition,
uint dwFlagsAndAttributes,
IntPtr hTemplateFile);
}
This is definitely not the best piece of code on earth, but I think it will provide a good starting point for anyone having to do the same thing.
