The rewrite rule will not return the proper status code. Meaning that a browser will see your custom error page but the response code will still be 200, and the browser (or whatever client is talking to the server) will think that the specific bad request is actually a good request.
The ErrorDocument statement will show the custom error page but will also return a 404 response code, letting the browser (or whatever client) know that what it requested was not found on the server.
Note that mod_rewrite has an R flag that you can give a response code to:
RewriteRule ^(.*)$ /not-found.html [L,R=404]
This will return a 404 response code but it will not display the /not-found.html page. It will display the custom error document if you use the ErrorDocument statement:
RewriteRule ^(.*)$ - [L,R=404]
ErrorDocument 404 /not-found.html
The rewrite rule will force a 404, its target is essentially ignored, then the ErrorDocument makes sure the custom error page is served up as well as the proper 404 response.
So you want ErrorDocument, pretty much always. The other thing to note is that mod_rewrite is in a different place in the URI to File mapping pipeline than ErrorDocument. If you're relying on mod_rewrite to determine what should be 404 and what shouldn't, it's likely to be affected by other modules that may come after (like mod_proxy, for example).
To be clear, the ErrorDocument is used after the URL-to-file mapping pipeline concludes with a "resource not found" (i.e. a 404), then the custom error document statement, ErrorDoucment 404, serves up a custom error page instead of the default one. When you use mod_rewrite like you do in your question, it completely circumvents the natural way the pipeline arrives at a 404.
