All answers are good here. But I would mention one more option.
The intention of author to show some fancy message is understandable.
Also, default Windows error message is really ugly. Besides, if application is not submitted to "Windows Excellence Program" the developer will not receive information about this problem. So what is the point to use default runtime handler if it does not help?
The thing here is that default exception handler of CLR host ( https://learn.microsoft.com/en-us/previous-versions/visualstudio/visual-studio-2008/9x0wh2z3(v=vs.90)?redirectedfrom=MSDN ) works in a very safe way. The purpose of it is clear: log the error, send it to developer, set the return code of your process and kill it. The general way of how to change that is to write your own host. In this case you can provide your own way of handling exceptions.
Still, there is an easy solution which satisfies CA1031 and still most of your needs.
When catching the exception, you can handle it your own way (log, show the message etc) and at the end you can set the process result code and do the exit (using the mix of Thread.Abort and "Exit" methods, for example). Still, at the end of your catch block you can just put "throw;" (which will never be called because of ThreadAbortedException, but will satisfy the rule). Still there are some cases, like StackOverflowException, which can't be handled like that and you will see that default message box, for fixing which you need to fallback to custom CLR host option.
Additionally, just for your information, you application can run several threads (besides that one which execute Main method). To receive exceptions from all of them you can use AppDomain.UnhandledException. This event does not allow you to "mark" the exception as handled, still you can freeze the thread using Thread.Join() and then do the job (log, msgbox, exit) using another (one more) thread.
I understand all this looks a little tricky and may be not right, but we have to deal with the implementation of AppDomain.UnhandledException, ThreadAbortException, CorruptedState exceptions and default CLR host. All of this eventually does not leave us much of choice.