A plethora of deployment problems can be caused by custom action use - most of them very serious. If your setup fails to complete or crashes, it is a fair bet that a erroneous custom action is at fault.
Pardon the "trifles" listed below. A lot of this may be banality, but maybe use the arguments listed to make a case for yourself to avoid custom actions and to work on better solutions - usually involving smarter application launch sequence and application self-configuration. Trust us, application coding will be more interesting than dealing with Windows's Installer's complex conditioning, sequencing and impersonation for custom actions.
The below has been updated so many times that sections are a little bit out of sequence or repetitive here and there. Will be cleaned up as time allows.
Developers are good at coding - so with all due respect - you tend to overuse custom actions to do things that are better done using built-in MSI features or ready-made MSI extension solutions such as those available in WiX for advanced things such as XML file updates, IIS, COM+, firewall rules, driver installation, custom permissioning for disk and registry entries, modify NT privileges, etc... Such support is also found in commercial tools such as Installshield and Advanced Installer.
It is also possible to do a lot of what is done in custom action as part of an application's launch sequence. The prime example is initializing user data and copying settings file to each user profile. There is a little summary of this issue here: Create folder and file on Current user profile, from Admin Profile.
It is obvious, but very often custom actions are used out of ignorance of what is already available "out of the box" by means of ready-made solutions. This happens to all of us all the time, doesn't it? There is always some smarter way to do things that would have saved you lots of grief? In general anyway - though sometimes you are breaking new ground. Don't break new ground in your setup - unless you absolutely have to! Save it for your application.
I want to emphasize that these built-in Windows Installer solutions and extensions from WiX and commercial tools have been written by the best deployment specialists available. And moreover, and even more importantly, they have been used and tested by thousands, millions - heck even billions of people for built-in MSI features. Do you really think you can do any better? Moral of the story: pick your battles and use your great coding skills to solve new problems, and let deployment be as dumb as possible. Use what already works, and don't reinvent the wheel. There are too many unknowns in deployment, too many variables that can't be controlled - you are dealing with "any machine" in "any state" and in "any language". See the Complexity of Deployment section here if you want examples - a bit down the page - just a short dump of all the ways your target systems may differ in their states when your package hits it. Each variable is a new bear trap for custom code - from OS version, to application estate to malware situation. The list goes on and on. As the conclusion reads in the linked content: "Deployment is a simple concept, with a complicated mix of variables that can cause the most mysterious errors - including the developer favorite: the intermittent bug. As we all know the seriousness of such bugs can not be overstated as they are often impossible to debug properly."
Certain advanced things must be done in your setup - since they require elevated rights and your application should not request this whilst running. This is what a setup is for, advanced, elevated system configuration - so embrace this complexity, but use ready made solutions! Don't roll your own script and solutions, use built-in, well tested stuff. Will your script run correctly in Korea? Will your custom action be blocked by a major anti-malware solution of caliber that you never had the time to test your script with? Do you have time to write your own check for whether a certain prerequisite runtime is installed on the computer - which works in all locales and across all OS versions? The potential for bugs here is staggering - and sometimes impossible to test properly. Do you have test machines in Arabic, Chinese, Korean or Japanese? Maybe you do. But do you have a terminal server to test on? How about a Korean terminal server? Did you test your setup with MSI advertisement? For advanced system management features to work, you have to make your setup as dumb and standard as possible. Spend several days looking for ready-made solutions before you write anything on your own I'd say. Remember, with ready-made solutions you are not just borrowing their code, but crucially their creator's QA, testing and UAT - which you can almost never hope to repeat.
Real-world testing is the only yardstick that matters - there is no substitute. Don't take on this world of pain! A slight change in Windows deployed via a Windows Update and your custom action breaks in any number of ways. Choose a better battle to make use of your skills. If you fight the design, then Windows installer fights back.
And there is a lot going on with deployment that makes it complex - and not dumb like we want it (just copy some darn files), your fight is to make it as simple as possible, but no simpler. Here is a list of deployment tasks showing why it is hard to get your package "dumb enough": What is the benefit and real purpose of program installation? Use only ready-made constructs whenever you can - it is the first easy win. All you need to do is to read and search a little.
Finally I will add that all custom actions are supposed to support rollback to put the system back to the original state if the install fails. In the real-world this is almost never done in an ad-hoc custom action (in my experience). Believe me it is complicated to deal with - I fear the word "rollback" as a Siberian husky fears the word "bath" after I implemented MSI rollback in C++. Not the most fun I ever had, but it eventually worked properly. If you want your setup to be without too many dependencies, C++ is the way to go, and as we all know it is not trivial to deal with. The ready-made solutions support rollback out of the box - it is an easy-win for only a bit of reading and searching to enable them. Complexity yes, but you are on more solid ground. And crucially: once an obscure error happens on a Japanese machine we can work for a community fix, or a third party vendor needs to sort it.
After all those "general observations", on a purely technical level, custom actions in Windows Installer are very complex with regards to implementation, scheduling, conditioning and rollback, and should hence be used when absolutely necessary (often for early-adopter stuff). A further complexity is runtime requirements - for example dependencies on specific versions of the .NET runtime, or PowerShell, or Installscript runtimes (scripting language of Installshield - it had a runtime dependency, but this is largely solved now, it used to be a problem).
Errors from missing runtime requirements can be a enormous hairball to deal with - for zero gain. For this reason I only use minimal dependency C++ dll's or Installscript when making custom actions. It happens that I use VBScript or JavaScript for read-only custom actions in the user interface sequence. These just retrieve data, and make no system changes. These are the only types of custom actions that are not that error prone. They should however be set to run without checking exit codes (to prevent them from triggering rollback or abort in a setup that is being run - often in major upgrade mode).
Also: Windows Installer hosts its own scripting runtime engine, so you know your active scripts (VBScript, Javascript, etc...) can run unless your target system is actually broken (there is no missing runtime on a normal system). This is in contrast to managed code custom actions - it is entirely possible for target systems to have no .NET runtime at this point (Jan 2018). Now this will change in the future, as .NET becomes a really built-in and required feature in Windows (or some minimal version of it hosted by Windows Installer itself). There are still problems with managed custom action code failing for strange reasons - such as the wrong .NET version being used to run it, or the wrong version of the CLR being loaded and being used, etc... I have limited experience here, but the problems are serious in my opinion. Eventually, though, we will all write managed code custom actions I think. I would still use C++ for worldwide distribution scenarios though.
Powershell scripts are particularly hairy for custom actions as they apparently run out of process and can't access the MSI's session object (source: MSI expert Chris Painter). Powershell also requires the .NET framework to be installed. I would never use Powershell script for custom actions - not even for internal, corporate deployment. Your call. Just as a "sample", here is an expert in the field stating his opinion (aging blog item, but still relevant if you ask me): Don’t use managed code to write your custom actions!
I tried to write a summary of pros and cons of different custom action types. Frankly I am not too happy with it, but here it is: Windows Installer fails on Win 10 but not Win 7 using WIX. What I am not happy with? The recommendation of JavaScript for one thing - I have used it seldomly, and although it is a better language than VBScript - particularly for error handling - I have had lots of problems using Javascript with MSI. It may be a case of the problem existing between the keyboard and the chair :-), but I think there are some real gotchas with JavaScript as well. Try to avoid complex use of scripts. For simple things like retrieving properties they seem to work OK for me. However, the experts in the field are merciless on script custom actions: Don’t use vbscript/jscript to write your custom actions! (Aaron Stebner), VBScript (and Jscript) MSI CustomActions suck (Rob Mensching - WiX benevolency).
Let me also add that custom action complexity is "silly, conspiratory complexity" not fun-to-deal-with stuff. It bites you. Gotchas. You will discover all the mistakes you have made in due course - as you move along (and then you have some explaining to do) - they will often not be immediately obvious (problems suddenly arise when upgrading, patching, uninstalling, your custom action runs erroneously during repair because it is not conditioned properly and wipes out certain registry settings, silent install doesn't work properly - it leaves the install incomplete because the custom action only exists in the user interface sequence, there are runtime errors on Windows XP that you never tested your custom action code with, there are broken things on target systems that you didn't shield yourself from, someone calls you and tells you your package doesn't work with active directory, your package can't be advertised, it fails on all Korean and Japanese systems, self-repair triggers runtime warnings and access denied for normal users, etc...). You will have no time to fix this properly once it hits you, and you will likely have to proceed with sub-standard solutions to get yourself ahead over the next hurdle. And no, I didn't experience all of this myself. In fact I discovered most of these problems when fixing up third party vendor MSI files for corporate deployment. You really discover a lot of potential error sources when you see, compare, review and adapt hundreds of packages to a corporate standard for large scale deployment. In all honesty there are serious shortcomings and errors in all but the simplest of packages. And obviously you get to see what you did wrongly when you made such vendor setups a few years earlier. And guess what tops the list? Overuse of custom actions when built-in constructs were available.
And to add to it all, the inherent complexity of deployment with its requirement to work on any machine, anywhere in any state, there is the issue of MSI technology borderline anti-patterns. Parts of the MSI technology that are causing repeated deployment problems because they are poorly understood, and sometimes fragile in real-world use. There is a brief summary of this in this answer (towards the bottom): How to make better use of MSI files (along with a list of the very important corporate benefits of MSI - and a random dump of common technical problems seen in real-world packages). Particularly the latter link has struck me as less than ideal after I wrote it. Take it for what it is: messy, real-world advice without much else going for it. It identifies too many problems without showing much in the line of fixes in places.
A substantial proportion of a software's support calls tend to come from problems seen during the deployment process. As the story goes: "...failing to properly install your great software may be close to being the most expensive error to make in software development. You can never hope to sell a software that was never possible to test" (from one of the linked answers above). A bad custom action is very often behind such problems.
The most common unnecessary custom action uses we see are in my opinion:
You install Windows Services via custom actions. This is much better done in the MSI itself using built-in constructs.
You install .NET assemblies to the GAC via a custom action. This is fully supported by Windows Installer itself without a line of (risky) code.
You run custom .NET assembly installer classes. These are to be used for development and testing only. They should never be run as part of deployment. Rather your MSI should use built-in constructs to deploy and register your assembly.
You run prerequisite setups and runtime installers via a custom action in your own MSI. This should be done entirely differently. What you need is a bootstrapper that can launch your setup and its prerequisites in a sequence. Commercial deployment tools such as Advanced Installer and Installshield have features for this, but free frameworks such as WiX has support via its Burn feature, and there are also free GUI applications such as DOTNetInstaller (untested by me) with these kind of features.
Please take my word for it in this particular case: running embedded setups via a custom action will fail eventually - usually rather immediately. MSI is too complicated in its scheduling, impersonation, transaction, rollback and overall runtime architecture to make this possible. Only one MSI transaction can run at a time by design, and this makes things very complicated. It used to be you could run embedded MSI files as a concept in MSI, but this was deprecated - it didn't work properly. You must run each setup in sequence. The correct sequence. There are bootstrappers / chainers available that will allow you to define such an installation sequence. Here is an answer which describes some of them (don't let the question title throw you off - it is about bootstrappers / chainers and other deployment considerations): Wix - How to run/install application without UI.
