How to create a virtual file system with offline files (similar to OneDrive) in .NET?

Question

I need to create a virtual file system that syncs files from my Azure app service to local drive and backward. I need only some files and folders to be synchronized to my local file system and be available offline. The files/folders should synchronize during first access or when the user selects the "Always keep on this device" menu. Similar to how OneDrive is doing. I found the Microsoft CloudMirror C++ sample which has some basic OneDrive functionality, but it syncs all files one time during start and no listing/loading on first access.

Is it possible to build virtual file system with above functionality completely in .NET?

Manos

Answer 1

To build an OneDrive-like client you need to create the Sync Provider using cloud filter API, which is mostly programmable via Win32 (with some exceptions, see below), but you can build a complete OneDrive analog in .NET, including "Always keep on this device" menu and on-demand loading.

Here is the functionality that is available via .NET, so you do not need to PInvoke it:

• Sync root registration can be done in .NET, via the StorageProviderSyncRootManager and StorageProviderSyncRootInfo classes.
• You can create properties in .NET using StorageProviderItemProperty class and IStorageProviderItemPropertySource interface.
• Icons in Windows File Manager on files and folders can be set via StorageProviderItemProperties.SetAsync() method and StorageFile/StorageFolder class.
• You can read and set file attributes in .NET using File.GetAttributes() / File.SetAttributes(). You will need them to read offline attribute, pinned and unpinned attributes.
• File system path to URI mapping can be done via implementing IStorageProviderUriSource interface.

Here is functionality available via Win32 functions only, you will need to call it via PInvoke:

• Folders on-demand enumeration, on-demand file content loading is available via Win32 callback only. You will need to register callback using CfConnectSyncRoot().
• Files hydration/dehydration via Windows Explorer menus. You will need to monitor the file system and call CfHydratePlaceholder() and CfUpdatePlaceholder().
• Download progress reporting, error reporting, notifications to Windows, sync provider status reporting. To report progress use the CfReportProviderProgress(). Errors are reported via CfExecute().
• Detection is the file is a regular file/folder or a placeholder file/folder. Can be done via CfGetPlaceholderStateFromFileInfo() call.
• Creation of placeholders and update of placeholder's info. Can be done using CfCreatePlaceholders() and CfUpdatePlaceholder() functions.
• Converting files/folders into placeholders and back. Can be done using CfConvertToPlaceholder() and CfRevertPlaceholder().
• Setting and reading placeholder statuses, such as in-sync/not in-sync. Can be done using CfGetPlaceholderStateFromFileInfo() and CfGetPlaceholderInfo().

Here is what you can do to write all code in .NET:

• One option is to import all required Win32 functions using extern to .NET, for example:

   [DllImport("cldapi.dll", SetLastError = true, ExactSpelling = true)]
   public static extern int CfGetPlaceholderStateFromFileInfo(IntPtr infoBuffer, FILE_INFO_BY_HANDLE_CLASS infoClass);
  
   [DllImport("cldapi.dll", SetLastError = true, ExactSpelling = true)]
   public static extern HRESULT CfSetPinState(IntPtr fileHandle, int pinState, int pinFlags, IntPtr overlapped);
  

• Another option could be using this sample.

Some more notes on building an OneDrive-like file system:

• The "Always keep on this device"/"Free up space" shows automatically on sync root registration. But it only sets Pinned and Unpinned file attributes. You need to monitor the file system, examine the pinned/unpinned attributes, and hydrate/dehydrate each file.
• The on-demand folders listing is done inside the CF_CALLBACK_TYPE_FETCH_PLACEHOLDERS callback. You need to register callback using CfConnectSyncRoot() and listen to it in .NET code.
• The on-demand file content download (hydration) is done inside the CF_CALLBACK_TYPE_FETCH_DATA callback. You need to register and listen to it in .NET code.

Eluvium

what callback does clicking "Always keep on this device"/"Free up space" should I listen to? I am not getting any FileSystemWatcher callback or CF callback – Holdback

If adding attributes to notifyfilter sends that event, how can I differentiate between last write time vs attributes in callback – Holdback