I was curious about this from the point of view of a best practice or recommendation in Topshelf's documentation, but couldn't find anything. I did, however, find two separate comments from phatboyg...
Best comment... how to stop service on exception, via this issue (I cut out some of the detail):
If your service's Start method throws an exception, the service will fail to start.
Once the service is running, if an unhandled exception is thrown, the service will stop, and report it as a crash to the service control manager.
If you need to Stop your service programatically, use the HostControl method Stop.
So I think the easiest answer is to throw an exception.
You were doing that, and you mention "the windows service stays around as a process after the exception". That seems like an unrelated bug somewhere in your code, or perhaps you somehow had multple instances running? I've been testing these scenarios this morning and have not seen my service running after throwing an Exception in the start method.
Also, relevant to checking before HostFactory.Run, mentioned in the accepted answer, via https://groups.google.com/forum/embed/#!topic/topshelf-discuss/nX97k3yOhJU:
"Your application should do nothing more than configure NLog/Log4Net before calling the HostFactory.Run() method."
FileNotFoundException
clr will do it for you. – Ticklish