authentication is not done via a filter. authentication is done before filter processing.
as soon as request arrives OSGi HttpService calls handleSecurity of the HttpContext associated with the servlet/resource. In case of Sling this calls into SlingMainServlet.handleSecurity which calls SlingAuthenticator.authentication.
SlingAuthenticator selects an authenticationHandler for the request and forwards the authenticate call.
authentication handler implements extractCredentials method that (based on the auth scheme e.g. Authorization header based authentication, session based authentication or cookie based authentication) is responsible for reading credentials from cookies (or header or session).
It would return AuthenticationInfo after successful authentication, if authentication fails either an anonymous session is acquired (if anonymous is allowed per configuration) or requestCredentials method is called, which would render(or redirect to) a login form.
after handleSecurity execution is done, HttpService would either terminate the request (if handleSecurity returned false) or call SlingMainServlet.service which would be the entry point for Sling Request Processing.
Request level filters would be processed after that. see https://sling.apache.org/documentation/the-sling-engine/filters.html