VCL Style from DLL is affecting TMenuItem in Application

Question

I am using Delphi XE6 and VCL styles. I have main application and dlls. My main application has enabled runtime themes and I am using vcl style files. I did quite similar thing to my DLLs. I enabled runtime themes and added VCL.Themes, VCL.Styles under uses and resource file with VCL style file within it. When DLL is loaded I load VCL style from resources and set it for DLL gui. Main app and DLL are not built with runtime packages.

Now I have main app GUI styled with own style and DLL gui styled with own style too. This seems to work fine until...

When I click on button in my main app which event opens TPopupMenu it's styled with same style as DLL GUI instead of main app style. If I navigate through menu I get AV too and program crashes. Take a look at the attached image.

What am I doing wrong? The only workaround I currently see would be to make my own customized TPopupMenu derived from some other control.

---

As I promised I prepared simple demo program which is similar to my application. It consists of host application with own style and DLL with style added to resource. Run it and click on button Popup then try select something from popup. It will crash and stop in some StdWindowProc or something like that. Also if you go to window system menu (left top corner) when you try to select something from that menu you will notice that system menu is styled as DLL gui and crashes too. Link to rar file: dropbox.com/sh/f2jmbsmw18akpyg/AAA6SWdBmVhf6n6K-mvYLLmua?dl=0

Thanks for your help.

Silverfish

The most likely explanation is that you are passing VCL objects across the boundary between your modules, which is not allowed unless you use runtime packages. – Vandalism

The other possibility that I can think of is that VCL styles code perhaps flubs resource enumeration and enumerates resources across the entire process. A quick scan of the code suggests that could be a problem if AutoDiscoverStyleResources is True. That call to EnumModules looks a bit iffy to me. – Vandalism

If you can't debug this then I guess you are asking us to dig into it. To do so we may need to have code that demonstrates the issue. – Vandalism

I don't pass any VCL objects accross modules. But I have on both sides included VCL.Stlyes and VCL.Themes but I think that's not the problem. – Silverfish

I will prepare example for debugging. I think I can reproduce that problem. – Silverfish

VCL styles are terribly unstable. In Tokyo they fixed lots of bugs but there are still about 200 known bugs! – Oleson

Answer 1

This is a fundamental problem with VCL styles and the way that they style menus. The styling is implemented with a process wide hook. Specifically a CBT hook installed by a call to SetWindowsHookEx from TCustomStyleEngine.CreateSysHook in the Vcl.Themes unit. In fact, the hook applies just to the GUI thread, but that is process wide in the sense that there is exactly one GUI thread in the process.

Since you have multiple instances of the VCL in your application (one in the DLL and one in the application), two hooks are installed. That is one too many. The hook installed most recently (the DLL as it happens) wins, and that's why the DLL menu styling infects your executable. And why you encounter an access violation. The DLL is trying to operate on a menu that belongs to the executable. And so, in spite of your best efforts, you've ended up with the DLL code accessing VCL objects from the host executable.

There's no simple way to work around this and support styles fully in both modules. What we have here is a fundamental consequence of the design. The system was not designed to support multiple VCL instances. If you wish to use VCL styles in multiple modules, then the designers expect you to use runtime packages.

I suppose that you might be able to get some traction by operating the DLL out of a completely different thread. That would involve loading the DLL from that different thread so that the VCL is initialized in the thread. And all calls to the DLL would have to be from that thread. And you'd need to run a message loop in that thread. It's possible that you might be able to make that work, but I doubt it. Even with all the provisos mentioned you still have to handle the fact that you have two GUI threads which presents all sorts of issues with the input queue handling.

Perhaps another approach would be to uninstall the hook from the DLL. So long as your DLL is not showing menus then you may well be able to get away with uninstalling that hook. It would disable styling for menus shown by the DLL, but perhaps that's acceptable.

This version of your DLL (after I simplified it somewhat also) uninstalls the hook.

library VCLStyleDLL;

{$R 'Style.res' 'Style.rc'}

uses
  VCL.Styles,
  VCL.Themes,
  VCL.SysStyles; // to gain access to TSysPopupStyleHook

{$R *.res}

begin
  TStyleManager.TrySetStyle('Glossy', false);
  TCustomStyleEngine.UnRegisterSysStyleHook('#32768', TSysPopupStyleHook);
end.

With this version of the DLL, the host executable does not suffer the problems your describe in your question.

Vandalism

Thank you David for your explanation. Since now we used third party AlphaControls to skin our app but now we decided to use Delphi VCL styles and I am just discovering technology behind it. Thanks also for editing/reviewing my post and make this topic more readable. I really appreciate it. – Silverfish

You are welcome. Thank you for doing a great job of providing info and working with us to make a good question and solve your problem. Btw, you should, when you are ready, accept the answer you deem to be the best. Your choice. – Vandalism

Answer 2

As David says this is caused because each VCL instance install a hook to detect when a popup menu (#32768) is created. So there is two hook instances working at the same time.

As workaround you can disable the popupmenu style hook in the dll (or in the app) using the UnRegisterSysStyleHook function defined in the Vcl.SysStyles unit.

TCustomStyleEngine.UnRegisterSysStyleHook('#32768', TSysPopupStyleHook);

Gemmule

+1 for showing the official way to unregister this hook – Vandalism

I wasn't aware of that but because I just started with VCL styles and I am not that familiar with them yet. Since now we used third party Alpha controls but we decided to go with delphi VCL styles. Thank you. – Silverfish