Rack Middleware and ActionController Filters are really quite different.
Rack is the standard Ruby webserver interface. It is designed to work in such a way that Rack applications or "Middlewares" can be chained together, each transforming the request / response in a particular way. If you create/use a Rack Middleware you are getting a chance to transform the request prior to it actually reaching the Rails app.
ActionController Filters are simply before/after hooks that execute before or after your immediate controller methods in Rails. These will be invoked immediately before or after your controller method, but after the entire remainder of the Rails stack.
Therefore there are significant differences in what is possible via a Rack Middleware and an ActionController filter, namely, because a Rack Middleware is executing prior to your application code it will not be executed in the same scope as your application code -- e.g. you won't be able to use your app models etc. unless you explicitly require them and perform the necessary initialization (like establishing a database connection).
If you're looking for rules of thumb, off the top of my head here's what I would tell you:
If you want to do something with the request before methods only in a specific controller, use a before filter in that controller.
If you want to do something with the request before all controller methods in your app, and what you want to do is very specific to your application or relies on your application code, use a filter on your ApplicationController.
If you want to do something generic with the request, not tied to your application code at all, and you imagine it would be nice to be able to re-use that in another app, a Rack Middleware would be a better fit.
Hope that helps.