The available approaches depend on the .NET version. Prior to .NET 4.0, the CLR enforced CAS policy, which made it possible to restrict permissions for any assembly. However, as of .NET 4.0, the CLR no longer applied CAS policy unless you enable it via the NetFx40_LegacySecurityPolicy element in any given application's app.config file. While this approach is currently available, you might want to keep in mind that it might not be supported in future .NET Framework versions.
The new style of permission limitation is via restriction of permissions by a host other than the "naked" CLR. This is what ASP.NET does, and it's what your alternate hosting platform would need to do as well in order to implement this properly. The simplest mechanism for such permission limitations is via a sandboxed appdomain. I am unfamiliar with Topshelf, and their intro docs don't make it clear whether they support this. However, it would presumably be a relatively simple enhancement, so it might be worth requesting the feature.