Why does Spring Security 6 not create sessions when authenticating with curl and basic auth?

Question

I recently upgraded to Spring Security 6, and have found that authenticating using basic auth from JS or from curl no longer works but authenticating with basic auth using Java's HttpClient does work. My goal is to be able to authenticate with all approaches.

The app uses Java 17, Spring Security 6, and Spring Session 3. It has a "login" endpoint which is just a convenience endpoint that is expected to be hit with basic auth and create a session, and it returns a User object. The session id should be used for subsequent requests to other endpoints.

The curl command is like so:

 curl -kv --user admin:admin "https://localhost:9000/login"

VS the HttpClient is configured like so and calling HttpClient.get(loginUrl)

HttpClient.newBuilder()
       .connectTimeout(Duration.ofSeconds(300))
       .cookieHandler(new CookieManager())
       .authenticator(new BasicAuthenticator(username, password))
       .sslContext(createSsl())
       .build();

public class BasicAuthenticator extends Authenticator {

   private PasswordAuthentication authentication;

   public BasicAuthenticator(String username, String password) {
       authentication = new PasswordAuthentication(username, password.toCharArray());
   }

   @Override
   public PasswordAuthentication getPasswordAuthentication() {
       return authentication;
   }
}

The security configuration is the block below... In upgrading to SpringSecurity 6 I added the requireExplicitSave() method, I have suspicions around this because my trouble is around saving sessions, but the added code is supposed to have spring security using the old functionality.

http
   .securityContext( securityContext -> securityContext.requireExplicitSave(false))
   .authorizeHttpRequests((authz) -> authz
           .requestMatchers(openEndpoints).permitAll()
           .anyRequest().authenticated()
   )
   .httpBasic()
       .and()
   .csrf()
       .disable()
   .exceptionHandling()
       .accessDeniedHandler((req, resp, e) -> e.printStackTrace() )
       .and()
   .logout()
       .invalidateHttpSession(true)
       .clearAuthentication(true);

I turned on request logging, security logging, and SQL logging. The SQL is all the same, and the basic auth request is always authenticated for all scenarios. The headers are different, but I can't see the headers for the HttpClient preflight call, and of the headers I do see, I don't know why authentication or session creation would work for one set of headers but not the other.

The core of the problem seems to be that the login request from the HttpClient ends with a session being created and the request from curl does not. Note that the big difference in the server logs when using curl is "Failed to create a session, as response has been committed. Unable to store SecurityContext." However even stepping through the spring security code I can't tell what is causing the difference. 

See logs here:

CURL

2022-12-14T16:38:07.594-05:00 DEBUG 92726 --- [nio-9000-exec-1] o.s.security.web.FilterChainProxy        : Securing GET /login
2022-12-14T16:38:07.597-05:00 DEBUG 92726 --- [nio-9000-exec-1] s.s.w.c.SecurityContextPersistenceFilter : Set SecurityContextHolder to empty SecurityContext
2022-12-14T16:38:07.704-05:00 DEBUG 92726 --- [nio-9000-exec-1] org.hibernate.SQL                        : select u1_0.id,u1_0.display_name,u1_0.email,u1_0.enabled,u1_0.password,u1_0.registration_time,r1_0.user_id,r1_0.role_id,u1_0.username from app_user u1_0 join user_role r1_0 on u1_0.id=r1_0.user_id where u1_0.username=?
2022-12-14T16:38:07.797-05:00 DEBUG 92726 --- [nio-9000-exec-1] o.s.s.a.dao.DaoAuthenticationProvider    : Authenticated user
2022-12-14T16:38:07.799-05:00 DEBUG 92726 --- [nio-9000-exec-1] o.s.s.w.a.www.BasicAuthenticationFilter  : Set SecurityContextHolder to UsernamePasswordAuthenticationToken [Principal=org.springframework.security.core.userdetails.User [Username=admin, Password=[PROTECTED], Enabled=true, AccountNonExpired=true, credentialsNonExpired=true, AccountNonLocked=true, Granted Authorities=[ROLE_ADMIN, ROLE_USER]], Credentials=[PROTECTED], Authenticated=true, Details=WebAuthenticationDetails [RemoteIpAddress=127.0.0.1, SessionId=null], Granted Authorities=[ROLE_ADMIN, ROLE_USER]]
2022-12-14T16:38:07.801-05:00 DEBUG 92726 --- [nio-9000-exec-1] o.s.security.web.FilterChainProxy        : Secured GET /login
2022-12-14T16:38:07.805-05:00 DEBUG 92726 --- [nio-9000-exec-1] o.s.w.f.CommonsRequestLoggingFilter      : Before request [GET /login, headers=[host:"localhost:9000", authorization:"Basic YWRtaW46YWRtaW4=", user-agent:"curl/7.84.0", accept:"*/*"]]
2022-12-14T16:38:07.816-05:00 DEBUG 92726 --- [nio-9000-exec-1] horizationManagerBeforeMethodInterceptor : Authorizing method invocation ReflectiveMethodInvocation: public com.seebie.dto.User com.seebie.server.controller.UserController.login(java.security.Principal); target is of class [com.seebie.server.controller.UserController]
2022-12-14T16:38:07.822-05:00 DEBUG 92726 --- [nio-9000-exec-1] horizationManagerBeforeMethodInterceptor : Authorized method invocation ReflectiveMethodInvocation: public com.seebie.dto.User com.seebie.server.controller.UserController.login(java.security.Principal); target is of class [com.seebie.server.controller.UserController]
2022-12-14T16:38:07.826-05:00 DEBUG 92726 --- [nio-9000-exec-1] org.hibernate.SQL                        : select u1_0.id,u1_0.display_name,u1_0.email,u1_0.enabled,u1_0.password,u1_0.registration_time,u1_0.username from app_user u1_0 where u1_0.username=?
2022-12-14T16:38:07.832-05:00 DEBUG 92726 --- [nio-9000-exec-1] org.hibernate.SQL                        : select r1_0.user_id,r1_0.role_id from user_role r1_0 where r1_0.user_id=?
2022-12-14T16:38:07.836-05:00 DEBUG 92726 --- [nio-9000-exec-1] org.hibernate.SQL                        : select a1_0.user_id,a1_0.id,a1_0.city,a1_0.line1,a1_0.state,a1_0.zip from address a1_0 where a1_0.user_id=?
2022-12-14T16:38:07.840-05:00 DEBUG 92726 --- [nio-9000-exec-1] org.hibernate.SQL                        : select s1_0.principal_name,s1_0.primary_id,s1_0.session_id from spring_session s1_0 where s1_0.principal_name=?
2022-12-14T16:38:07.871-05:00 DEBUG 92726 --- [nio-9000-exec-1] o.s.w.f.CommonsRequestLoggingFilter      : REQUEST DATA : GET /login, headers=[host:"localhost:9000", authorization:"Basic YWRtaW46YWRtaW4=", user-agent:"curl/7.84.0", accept:"*/*"]]
2022-12-14T16:38:07.873-05:00  WARN 92726 --- [nio-9000-exec-1] w.c.HttpSessionSecurityContextRepository : Failed to create a session, as response has been committed. Unable to store SecurityContext.
2022-12-14T16:38:07.873-05:00 DEBUG 92726 --- [nio-9000-exec-1] s.s.w.c.SecurityContextPersistenceFilter : Cleared SecurityContextHolder to complete request

HttpClient

2022-12-14T06:31:28.390-05:00 DEBUG 85610 --- [o-auto-1-exec-1] o.s.security.web.FilterChainProxy        : Securing GET /login
2022-12-14T06:31:28.420-05:00 DEBUG 85610 --- [o-auto-1-exec-1] s.s.w.c.SecurityContextPersistenceFilter : Set SecurityContextHolder to empty SecurityContext
2022-12-14T06:31:28.913-05:00 DEBUG 85610 --- [o-auto-1-exec-1] org.hibernate.SQL                        : select u1_0.id,u1_0.display_name,u1_0.email,u1_0.enabled,u1_0.password,u1_0.registration_time,r1_0.user_id,r1_0.role_id,u1_0.username from app_user u1_0 join user_role r1_0 on u1_0.id=r1_0.user_id where u1_0.username=?
2022-12-14T06:31:29.102-05:00 DEBUG 85610 --- [o-auto-1-exec-1] o.s.s.a.dao.DaoAuthenticationProvider    : Authenticated user
2022-12-14T06:31:29.103-05:00 DEBUG 85610 --- [o-auto-1-exec-1] o.s.s.w.a.www.BasicAuthenticationFilter  : Set SecurityContextHolder to UsernamePasswordAuthenticationToken [Principal=org.springframework.security.core.userdetails.User [Username=admin, Password=[PROTECTED], Enabled=true, AccountNonExpired=true, credentialsNonExpired=true, AccountNonLocked=true, Granted Authorities=[ROLE_ADMIN, ROLE_USER]], Credentials=[PROTECTED], Authenticated=true, Details=WebAuthenticationDetails [RemoteIpAddress=127.0.0.1, SessionId=19ab0971-5fb3-47fd-a4f9-cdde1ad24883], Granted Authorities=[ROLE_ADMIN, ROLE_USER]]
2022-12-14T06:31:29.108-05:00 DEBUG 85610 --- [o-auto-1-exec-1] o.s.security.web.FilterChainProxy        : Secured GET /login
2022-12-14T06:31:29.136-05:00 DEBUG 85610 --- [o-auto-1-exec-1] o.s.w.f.CommonsRequestLoggingFilter      : Before request [GET /login, headers=[authorization:"Basic YWRtaW46YWRtaW4=", content-length:"0", host:"localhost:64723", user-agent:"Java-http-client/17.0.2", cookie:"SESSION=MTlhYjA5NzEtNWZiMy00N2ZkLWE0ZjktY2RkZTFhZDI0ODgz", Content-Type:"application/json;charset=UTF-8"]]
2022-12-14T06:31:29.274-05:00 DEBUG 85610 --- [o-auto-1-exec-1] horizationManagerBeforeMethodInterceptor : Authorizing method invocation ReflectiveMethodInvocation: public com.seebie.dto.User com.seebie.server.controller.UserController.login(java.security.Principal); target is of class [com.seebie.server.controller.UserController]
2022-12-14T06:31:29.332-05:00 DEBUG 85610 --- [o-auto-1-exec-1] horizationManagerBeforeMethodInterceptor : Authorized method invocation ReflectiveMethodInvocation: public com.seebie.dto.User com.seebie.server.controller.UserController.login(java.security.Principal); target is of class [com.seebie.server.controller.UserController]
2022-12-14T06:31:29.373-05:00 DEBUG 85610 --- [o-auto-1-exec-1] org.hibernate.SQL                        : select u1_0.id,u1_0.display_name,u1_0.email,u1_0.enabled,u1_0.password,u1_0.registration_time,u1_0.username from app_user u1_0 where u1_0.username=?
2022-12-14T06:31:29.392-05:00 DEBUG 85610 --- [o-auto-1-exec-1] org.hibernate.SQL                        : select r1_0.user_id,r1_0.role_id from user_role r1_0 where r1_0.user_id=?
2022-12-14T06:31:29.409-05:00 DEBUG 85610 --- [o-auto-1-exec-1] org.hibernate.SQL                        : select a1_0.user_id,a1_0.id,a1_0.city,a1_0.line1,a1_0.state,a1_0.zip from address a1_0 where a1_0.user_id=?
2022-12-14T06:31:29.413-05:00 DEBUG 85610 --- [o-auto-1-exec-1] org.hibernate.SQL                        : select s1_0.principal_name,s1_0.primary_id,s1_0.session_id from spring_session s1_0 where s1_0.principal_name=?
2022-12-14T06:31:29.678-05:00 DEBUG 85610 --- [o-auto-1-exec-1] o.s.w.f.CommonsRequestLoggingFilter      : REQUEST DATA : GET /login, headers=[authorization:"Basic YWRtaW46YWRtaW4=", content-length:"0", host:"localhost:64723", user-agent:"Java-http-client/17.0.2", cookie:"SESSION=MTlhYjA5NzEtNWZiMy00N2ZkLWE0ZjktY2RkZTFhZDI0ODgz", Content-Type:"application/json;charset=UTF-8"]]
2022-12-14T06:31:29.680-05:00 DEBUG 85610 --- [o-auto-1-exec-1] w.c.HttpSessionSecurityContextRepository : Stored SecurityContextImpl [Authentication=UsernamePasswordAuthenticationToken [Principal=org.springframework.security.core.userdetails.User [Username=admin, Password=[PROTECTED], Enabled=true, AccountNonExpired=true, credentialsNonExpired=true, AccountNonLocked=true, Granted Authorities=[ROLE_ADMIN, ROLE_USER]], Credentials=[PROTECTED], Authenticated=true, Details=WebAuthenticationDetails [RemoteIpAddress=127.0.0.1, SessionId=19ab0971-5fb3-47fd-a4f9-cdde1ad24883], Granted Authorities=[ROLE_ADMIN, ROLE_USER]]] to HttpSession [org.springframework.session.web.http.SessionRepositoryFilter$SessionRepositoryRequestWrapper$HttpSessionWrapper@7ba784f2]
2022-12-14T06:31:29.680-05:00 DEBUG 85610 --- [o-auto-1-exec-1] s.s.w.c.SecurityContextPersistenceFilter : Cleared SecurityContextHolder to complete request

Note the session in headers for the HttpClient call... I think HttpClient makes a preflight auth call that gets a 401 and then makes the "real" call with the credentials, at least that's how it was in Java 11.

I think if I understood the difference in how these calls are being made/handled that causes one technique to work but not the other, I would be able to solve the problem. So that really is the question: What is the difference in spring security 6 (along with spring session) handling session creation when using Java 17's HttpClient vs curl?

[UPDATE] to anyone who read this far: the behavior is actually expected behavior for Spring Security. A full discussion and explanation are in the spring security issue that I had opened here

Nonoccurrence

This definitely seems like a bug, but to be sure, a minimal reproducible sample would be helpful. Have you been able to isolate this behavior from your larger application and can you reproduce it with a fresh spring boot app? – Behlke

@SteveRiesenberg as I was putting together the question I realized there is potentially a lot going on that could make it hard to pinpoint what's going on. Isolating the issue in a fresh app sounds like my next step, thanks. I'll post a link here once I get a chance to put that together. – Nonoccurrence

does enabling Force Eager Session Creation help? FYI, according to logs curl and HttpClient are consuming different urls: localhost:9000 and localhost:64723 – Afterglow

+ docs.spring.io/spring-security/reference/migration/servlet/… – Afterglow

@AndreyB.Panfilov using eager session creation fixed it! I knew about SessionCreationPolicy but hadn't tried changing it for this issue - I am interested in understanding what's going on instead of just patching it. But that is a good find, thank you. The migration guide is something I have already gone over, that's where my addition of requireExplicitSave() came from. – Nonoccurrence

@AndreyB.Panfilov The different ports are because 9000 is from running the app standalone and hitting it with curl from the command line, and the other is a random port assigned when running the @ SpringBootTest where HttpClient is called from a test method. – Nonoccurrence

@Nonoccurrence your guess about preflight calls performed by HttpClient seems to be correct, old versions had a special flag: hc.apache.org/httpclient-legacy/… and if my memory serves me right I was doing something very similar for new versions. – Afterglow

This is about java.net.http.HttpClient, yeah the names are confusing :/ – Nonoccurrence

The same idea in java.net.http.HttpClient – Afterglow

I think that when using the HttpClient it is always sending the credentials in the request, while in JS and Curl you are sending only once. The basic authentication has a stateless nature and needs the credentials to be sent on every request unless you are using some kind of Remember me feature – Perspire

@SteveRiesenberg here is a simple fresh project that demonstrates the behavior: github.com/thinkbigthings/curl-test The behavior is exactly the same as what I see in my larger project. I actually was able to replicate the behavior of HttpClient using curl and craft my own preflight call. – Nonoccurrence

The question is now better phrased: "why does spring security not create a session linked to the authenticated user on initial authentication?" or "How can I configure spring security to link a session to the authenticated user on initial authentication?" I think it worked like that before I upgraded to spring security 6 - all of my curl test commands worked before but now don't work without a separate preflight call to create the session. – Nonoccurrence

Answer 1

Well, if we are not going to investigate why preflight requests make sense (IMO, that seems to be a bug), the explanation of what has been changed in spring 6 is following:

as was mentioned in Session Management Migrations now Spring does not enable SecurityContextPersistenceFilter by default, however in Spring 5 SecurityContextPersistenceFilter was responsible for saving SecurityContext in http session (and hence creating it) unless that was explicitly disabled. Now in order to return previous behaviour you desire you need to setup SecurityContextRepository via:

http.securityContext(securityContext -> securityContext.
      securityContextRepository(new HttpSessionSecurityContextRepository())
)

Afterglow

That does indeed restore the old behavior! I didn't know I could restore the old repository directly, this is an interesting find and seems like it should have been included in the migration guide. Thank you sir! – Nonoccurrence

Out of curiosity, what do you mean "investigate why preflight requests make sense"? Does it make more sense to use preflight requests than to make a basic auth request directly? – Nonoccurrence

@Nonoccurrence Preflight requests cause spring to handle AccessDenied exceptions and save failed requests in http session. Such implementation seems to be weird: it does not create sessions by default for authenticated users and does create session for unauthenticated users, who are potential attackers. – Afterglow

So, attacker may potentially consume a lot of server memory via sending malicious http requests, that is definitely a DoS. – Afterglow

I just replicated it explicitly: it does NOT create sessions for calls that pass authentication and DOES create sessions for calls that fail authentication - which seems backwards. Another aspect is that it does create a session for authenticated users if we use SessionCreationPolicy.IF_REQUIRED ... I read online that this is the default policy, but the behavior clearly changes when I set it, and stepping into the debugger I see that the policy is actually null unless I set it. – Nonoccurrence

@andrey-b-panfilov I created a spring security issue, let's see what they say github.com/spring-projects/spring-security/issues/12431 – Nonoccurrence

Adding above helped with fixing a issue with Async Type APIs. An API Which returns a CompletableFuture was giving a 403 even after being authenticated with new securityFilterChain method in Spring. – Excrete