How to write a shell extension in C++?

Question

This seemed like a common question but after doing some searching, I wasn't really able to find my answers. There is an article on this here:

http://www.codeproject.com/KB/shell/shellextguide1.aspx

But it's for a very old version of Visual Studio. I'm using VS 2008, so the instructions and interfaces don't seem to match what I'm seeing.

I want to create a simple shell extension using C++ that creates a context menu for files with extension .GZ. When right clicking on these files, I should be able to click my context menu item and have a callback in code to do some sort of operation on that file.

Other context menu items would do things like spawn modless dialogs to accept user input before executing some action.

From what I've seen, ATL is used for this but I have never used ATL, so all of the object types and interfaces are very confusing to me. It wouldn't be so bad if I had a proper tutorial or documentation to read.

Can anyone help me out? Isn't there some sort of tutorial out there that isn't 10 years old?

Magnetism

This is actually a particularly difficult thing to do. I had developed a context menu item for when the user right-clicked on printers. Even though my code was only 40 or so lines, there was about 1,000 lines of generated code. I wish I could be more helpful but I can't share any of the code I have. As a tip, your shell extension must reside in a DLL, and it must have its own GUID, and the DLL needs to be registered with regsvr32. You want to implement BOTH the IContextMenu and IShellExtInit COM interfaces. There is a way to get VS to generate most of it for you, but I can't remember how. – Lovett

IContextMenu doc, and also IShellExtInit doc. – Lovett

@Lovett I used the project wizard for ATL project, but when I create a new class it also creates an interface for it -- I don't want an interface, but it doesn't look easy to remove that functionality. Plus I'm not sure why there need to be generated code. – Magnetism

Shell extensions almost always are interface-based, which is why the project wizard generates them as such. You're also implementing two interfaces in order to provide the shell extension (that's required for context menu extensions) and must be registered as doing so. You can't really avoid interfaces, so you might as well resign yourself to the fact you're going to have to use them. And the generated code saves you from manually writing all of the housekeeping tasks yourself by generating the code for them. Trust what everyone is saying - let the wizard do the gruntwork. :) – Leannleanna

@KenWhite: Can I create a regular DLL project, give it a GUID, and then register it manually with my installer? Do I need to use ATL? – Magnetism

Not a regular DLL project, unless you want to write all the required housekeeping code to implement IContextMenu and IShellExtInit interfaces yourself (which all Shell Extensions are required to implement, as previously stated). If you look in the list of Related topics to the right of this thread, and read a few of the linked questions, you'll get an idea of what's required. (Scrub the recommended article I provided; the link to the accepted answer is now dead (the reason I abhor link only answers)). – Leannleanna

@KenWhite: I'm confused then, does ATL implement IContextMenu and IShellExtInit for me? I didn't see those anywhere... – Magnetism

I don't recall the specific ATL implementation details; I haven't used ATL or C++ for ages. :) I remember the headaches involved, though. You might find this link helpful; it's specifically for VS 2010 C++, and demonstrates a context menu extension. Here's an ATL example for VS 2008 C++ – Leannleanna

10 year old tutorials are fine. Nothing's changed in that time. Anyway, you don't even need a shell extension for this. You can add items to the context menu that spawn executables passing the file name as a parameter. You can do that with just registry settings. Easily the simplest way to get going. – Tye

@DavidHeffernan: There are limitations to that though. Only a Shell Context menu can provide dynamic information about the object in question, for example, if you wanted to write a shell extension that created a submenu containing a zip's contents, you can't do that with just registry settings. – Lovett

@Lovett Indeed so. But if you don't need anything more advanced then why suffer the pain of writing a shell extension? – Tye

Well using the registry is what I initially wanted to do. All I'm doing is executing a script on a file right clicked in the explorer. However, if I select multiple files, it runs the script 1 time for each, when I'd rather have the list of files passed into the script so it can run only 1 time on those files. I don't know if this can be done with just the registry. There's the issue of file associations too. How do I know the right key under HKCR to add my shell commands? I need a way to set this up with a script or installer. – Magnetism

@RobertDailey: Don't worry about file associations, if you have a context menu handler it will work regardless of what application the file is associated with. Writing a shell extension will give you greater flexibility, especially with regards to multiple files (because you have the chance to check all selected files before even showing your context menu items), but it comes at the cost of greater maintenance requirements and overall more headaches. – Lovett

¤ You do not need to write a shell extension to get your custom context menu item. Just define the item and let it invoke a handler program. (1) Fire up regedit. (2) Go to [HKEY_CLASSES_ROOT\.gz]. (3) Add a subkey [shell]. (4) In that key, add subkey [Wow, it works!]. (5) In that key, add subkey [Command]. (6) Set that key's default value to [cmd /k echo doh]. (7) Create a gz file, right click and try it out. Cheers & hth., – Werner

Information does not automatically become stale after 10 years. The ASCII code was defined in the 1960's. It's still valid and relevant. – Werner

@AlfP.Steinbach: Did you read all of the comments? The OP doesn't want his script/program to run 10 times if the user has selected 10 files, which is what the registry-only approach will do. Similarly, if he wants to create menu items or submenus based off of the data inside the GZ file, he'll need to use a shell extension as it's the only way to be able to add menu items dynamically. – Lovett

@dreamlax, Please consolidate your comments into an answer so I can check it :) Thanks to everyone who provided insight into this issue. – Magnetism

Answer 1

I can't tell you exactly how to write a shell extension, but I will provide a number of tips. Writing a Shell Extension offers some significant advantages over the much simpler “registry-only” method:

• With a Shell Extension, you can dynamically create a context menu item (or submenu) that is more relevant to the selected file(s). For example, if you are writing a Shell Extension for zip files, it is possible to create a submenu within the context menu that shows the entire contents of the zip.
• You can handle multiple files simultaneously, which may be more beneficial not just for performance purposes but also so that you can work out what to do based on the selection as a whole rather than just for each file.

Some of the downfalls to Shell Extensions are:

• Substantially increased complexity. Be prepared to spend a lot of effort on this to get it working. Have a home-espresso machine installed next to your computer and/or hire someone to make you coffee.

• Substantially increased difficulty in debugging. Ditto about coffee.

It's difficult to write a Shell Extension because they can be very hard to debug.

• Shell Extensions are loaded by the explorer.exe process, and without specific configuration of Explorer, you need to force-quit the explorer.exe process so that you can install a newer version of your Shell Extension. There is a way to get Explorer to unload DLLs that it is no longer using, but you should only do this on a development machine and not on a deployment target:

  1. In RegEdit, browse to the following key:

     HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Explorer

  2. Add a new DWORD key called “AlwaysUnloadDLL” and set its value to 1.

  3. Restart explorer.

  This works most of the time, but there may still be times where you need to close Explorer because the Shell Extension was not unloaded.

• Keep in mind that your Shell Extension may be loaded by other applications, for example, if you right-click on a file with an applications “open file” dialog, then your Shell Extension will be loaded into that application, and not Explorer.

• If your Shell Extension causes a runtime error, quite often the result will simply be that your context menu item does not show, very rarely will you be told that your Shell Extension failed to load or that it caused a runtime error.

• Configuration can be hard, even with an installation, registry data needs to be created in several places, and depending where you want your context menu to show, the places in the registry may differ between different versions of Windows.

What you'll need to do:

• Visual Studio offers some shortcuts to creating Shell Extensions, but basically you'll need to create a COM DLL. A Shell Extension for context menu items must implement both the IContextMenu interface and the IShellExtInit interface.
• In the IShellExtInit::Initialize() method, you can obtain the selected files from the IDataObject parameter. From memory, the data is in “Drag-n-Drop” format, so you need to get an HDROP handle from the IDataObject and query the files from there (this is from memory, it may actually be different than as I described here, so proceed with caution).
• Once your DLL is ready to be “installed”, you must copy it somewhere, and then run regsvr32 to make sure it is registered.
• Follow this guide to know where to put registry keys.
• There may be issues with 64-bit Windows, if you build a 32-bit DLL it may not load in 64-bit Explorer… so keep this in mind if you are having trouble with 64-bit Windows.
• Your DLL will actually have two GUIDs associated with it. I can't remember exactly how it works, but one GUID refers to the DLL itself and the other refers to the actual Shell Extension. Make sure you use the GUID of the actual Shell Extension when creating keys in the registry where a GUID is required.

---

All things considered… (tl;dr)

Weigh up the costs of whether a Shell Extension is worth it. If you want to create menu items dynamically based on the selected files, then a Shell Extension may be the only way. If you want to handle all files simultaneously then you'll probably need a Shell Extension as well.

An alternative to the context menu method, could be to have a drag-n-drop target on the user's desktop or something. Explore other ways that you could have the user submit your files to your application, because a Shell Extension is often far more effort than it is worth. I found this out the hard way and I think everyone else has too.

Lovett

+1 Awesome Post, Things have gotten a little easier with .NET 4 you can write shell extensions in managed code now. Mentioning it here so people don't get completely dissuaded. – Spin

@Spin That seems to be discouraged. – Messere