I believe the answer can be found in this MSDN post: VSTO, ClickOnce and auto update
An excerpt:
This is True: VSTO applications are ClickOnce applications
This is not True: The ClickOnce API is supported by VSTO applications
Why: While VSTO Applications are ClickOnce applications, they require functionality that extends the base implementation of ClickOnce. A product of this requirement is that not everything within ClickOnce (for Windows Forms) applies to VSTO. One of these specific areas is the Runtime API.
This is True: Some parts of the API will work
Why: Because the VSTO Runtime Uses the core part of ClickOnce, some parts actually will function. What is not known is where exactly this line is drawn. I have found as very loose general rule of thumb: anything that doesn't change the state of the application (anything that provides you with "information") will likely work. This is why my blog post describes how to use the API to "check" for the update but uses the VSTOInstaller exe to do the actual act of updating.
This is not True: You can use the API to Download an update
Why: This goes back to the ClickOnce/VSTO difference. If you imagine ClickOnce as this sort of generic technology, you can think of VSTO as a specific implementation of it. In most cases (specifically Winforms applications) the generic technology does everything that is required. For VSTO though, we needed to extend the technology to make it do stuff it had never done before (specifically register customizations with office and maintain some data need to setup entrypoints and whatnot). As such the generic technology doesn't provide all of the functionality we need. In this specific case incurring an update changes the state of the application in such a way that we have to change some of the registration information for Office. ClickOnce "doesn know" enough to update these values, and as such isn't capable (in its current state) of doing a "correct" update of a VSTO application. It is the VSTO Runtime that does these steps.
He mentions a blog post, which I believe is this one: Click-Once forced updates in VSTO: Some things we don’t recommend using, that you might consider anyway.
An excerpt:
//Call VSTOInstaller Explicitely in "Silent Mode"
string installerArgs = " /S /I \\\\GenericServer\\WordDocument2.vsto";
string installerPath = "C:\\Program Files\\Common Files\\microsoft
shared\\VSTO\\9.0\\VSTOINSTALLER.exe";
System.Diagnostics.Process VstoInstallerProc = new System.Diagnostics.Process();
VstoInstallerProc.StartInfo.Arguments = installerArgs;
VstoInstallerProc.StartInfo.FileName = installerPath;
VstoInstallerProc.Start();
VstoInstallerProc.WaitForExit();
It's not exactly production-ready code, but you get the idea.