Actuator endpoints returning 500 error after upgrade to spring boot 3

Question

I am upgrading a service from spring boot 2.7 to 3.2.0 and bumping into an issue where I get an HTTP Status 500 – Internal Server Error error when accessing actuator endpoints.

Looking at the stack trace I can see that when accessing an actuator endpoint via port 9090 the org.springframework.security.config.annotation.web.AbstractRequestMatcherRegistry.DispatcherServletDelegatingRequestMatcher#matcher method fails to find the dispatcherServletRegistration in the this.servletContext.getServletRegistration(name); call.

When accessing normal app endpoints on port 8080, I can see that it finds it just fine.

This is my actuator configuration, besides the defaults update it is unchanged from my spring-boot 2.7 config.

management:
  defaults:
    metrics:
      export:
        enabled: true
  endpoints:
    web:
      exposure:
        include: '*'
  info:
    build:
      enabled: true
  endpoint:
    health:
      show-details: when_authorized
  server:
    port: 9090

During start-up logs seem to indicate that things are setup ok

2023-12-23T21:36:28.729 [main] [] INFO  o.a.coyote.http11.Http11NioProtocol.log - Starting ProtocolHandler ["http-nio-8080"]
2023-12-23T21:36:28.755 [main] [] INFO  o.s.b.w.e.tomcat.TomcatWebServer.start - Tomcat started on port 8080 (http) with context path ''
HOTSWAP AGENT: 21:36:28.756 INFO (org.hotswap.agent.plugin.spring.SpringPlugin) - Spring plugin initialized - Spring core version '6.1.1'
2023-12-23T21:36:28.813 [main] [] INFO  o.s.b.w.e.tomcat.TomcatWebServer.initialize - Tomcat initialized with port 9090 (http)
2023-12-23T21:36:28.813 [main] [] INFO  o.a.coyote.http11.Http11NioProtocol.log - Initializing ProtocolHandler ["http-nio-9090"]
2023-12-23T21:36:28.813 [main] [] INFO  o.a.catalina.core.StandardService.log - Starting service [Tomcat]
2023-12-23T21:36:28.813 [main] [] INFO  o.a.catalina.core.StandardEngine.log - Starting Servlet engine: [Apache Tomcat/10.1.16]

(No difference if starting it up without the hotswap agent)

Quite a lot of other spring-boot 2->3 migration done, perhaps I missed something (perhaps spring-security related)?

Any advice on what the issue might be or how to troubleshoot it greatly appreciated.

Edit1: this is the stack trace when accessing e.g. localhost:9090/actuator

java.lang.IllegalArgumentException: Failed to find servlet [dispatcherServletRegistration] in the servlet context
    org.springframework.util.Assert.notNull(Assert.java:172)
    org.springframework.security.config.annotation.web.AbstractRequestMatcherRegistry$DispatcherServletDelegatingRequestMatcher.matcher(AbstractRequestMatcherRegistry.java:529)
    org.springframework.security.web.access.intercept.RequestMatcherDelegatingAuthorizationManager.check(RequestMatcherDelegatingAuthorizationManager.java:79)
    org.springframework.security.web.access.intercept.RequestMatcherDelegatingAuthorizationManager.check(RequestMatcherDelegatingAuthorizationManager.java:48)
    org.springframework.security.authorization.ObservationAuthorizationManager.check(ObservationAuthorizationManager.java:63)
    org.springframework.security.web.access.intercept.AuthorizationFilter.doFilter(AuthorizationFilter.java:95)
    org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)
    org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)
    org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)
    org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:126)
    org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:120)
    org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)
    org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)
    org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)
    org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:100)
    org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)
    org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)
    org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)
    org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:145)
    org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:101)
    org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)
    org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)
    org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)
    org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:179)
    org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)
    org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)
    org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)
    org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63)
    org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)
    org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)
    org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)
    org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:227)
    org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:221)
    org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)
    org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)
    org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)
    org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:107)
    org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:93)
    org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)
    org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)
    org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)
    org.springframework.security.web.csrf.CsrfFilter.doFilterInternal(CsrfFilter.java:117)
    org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
    org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)
    org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)
    org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)
    org.springframework.web.filter.CorsFilter.doFilterInternal(CorsFilter.java:91)
    org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
    org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)
    org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)
    org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)
    org.springframework.security.web.header.HeaderWriterFilter.doHeadersAfter(HeaderWriterFilter.java:90)
    org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:75)
    org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
    org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)
    org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)
    org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)
    org.springframework.security.web.context.SecurityContextHolderFilter.doFilter(SecurityContextHolderFilter.java:82)
    org.springframework.security.web.context.SecurityContextHolderFilter.doFilter(SecurityContextHolderFilter.java:69)
    org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)
    org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)
    org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)
    org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:62)
    org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
    org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)
    org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)
    org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)
    org.springframework.security.web.session.DisableEncodeUrlFilter.doFilterInternal(DisableEncodeUrlFilter.java:42)
    org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
    org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)
    org.springframework.security.web.ObservationFilterChainDecorator$AroundFilterObservation$SimpleAroundFilterObservation.lambda$wrap$0(ObservationFilterChainDecorator.java:323)
    org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:224)
    org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)
    org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:233)
    org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:191)
    org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:352)
    org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:268)

Edit2:

Created a brand new Spring Initilizer project and see the following differences which look relevant, but not yet sure why.

Spring Initilizer new project:

2023-12-26T18:23:07.912Z  INFO 43176 --- [2)-192.168.0.31] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring DispatcherServlet 'dispatcherServlet'
2023-12-26T18:23:07.912Z  INFO 43176 --- [2)-192.168.0.31] o.s.web.servlet.DispatcherServlet        : Initializing Servlet 'dispatcherServlet'
2023-12-26T18:23:07.913Z  INFO 43176 --- [2)-192.168.0.31] o.s.web.servlet.DispatcherServlet        : Completed initialization in 0 ms
2023-12-26T18:23:13.567Z  INFO 43176 --- [nio-9090-exec-1] o.a.c.c.C.[Tomcat-1].[localhost].[/]     : Initializing Spring DispatcherServlet 'dispatcherServletRegistration'
2023-12-26T18:23:13.567Z  INFO 43176 --- [nio-9090-exec-1] o.s.web.servlet.DispatcherServlet        : Initializing Servlet 'dispatcherServletRegistration'
2023-12-26T18:23:13.567Z  INFO 43176 --- [nio-9090-exec-1] o.s.web.servlet.DispatcherServlet        : Completed initialization in 0 ms

My app

2023-12-26T18:24:40.958 [RMI TCP Connection(4)-192.168.0.31] [] INFO  o.a.c.c.C.[Tomcat].[localhost].[/].log - Initializing Spring DispatcherServlet 'dispatcherServlet'
2023-12-26T18:24:40.958 [RMI TCP Connection(4)-192.168.0.31] [] INFO  o.s.web.servlet.DispatcherServlet.initServletBean - Initializing Servlet 'dispatcherServlet'
2023-12-26T18:24:40.960 [RMI TCP Connection(4)-192.168.0.31] [] INFO  o.s.web.servlet.DispatcherServlet.initServletBean - Completed initialization in 1 ms

Cheers, Mike

Centurion

I think all relevant information you will find Spring-Boot-3.0-Migration-Guide – Napoli

Not been able to figure this out yet via migration docs – Centurion

Answer 1

Short-short version

For actuator do not use .requestMatcher("/actuator/health") instead use MvcRequestMatcher.Builder#pattern

Slightly longer version

After creating a small poc to troubleshoot; my issue boiled down to changes in how to add request matchers to the SecurityFilterChain.

The old code was simply using something like .requestMatchers("/actuator/health").permitAll() which isn't the correct way anymore.

org.springframework.security.web.access.intercept.RequestMatcherDelegatingAuthorizationManager#check(java.util.function.Supplier, jakarta.servlet.http.HttpServletRequest) Iterates over list of configured request matchers.

When it comes across a requestMatcher("/path") like matcher the underlying request matcher used is of type org.springframework.security.config.annotation.web.AbstractRequestMatcherRegistry.DispatcherServletDelegatingRequestMatcher.

The matcher logic then does the following where it fails (registration is null)

@Override
public MatchResult matcher(HttpServletRequest request) {
    String name = request.getHttpServletMapping().getServletName();
    ServletRegistration registration = this.servletContext.getServletRegistration(name); <-- Doesn't find a registration
    Assert.notNull(registration, "Failed to find servlet [" + name + "] in the servlet context");
    if (isDispatcherServlet(registration)) {
        return this.mvc.matcher(request);
   }
    return this.ant.matcher(request);
}

I solved my problem by defining this bean to create a different Matcher

@Bean
MvcRequestMatcher.Builder mvc(HandlerMappingIntrospector introspector) {
    return new MvcRequestMatcher.Builder(introspector);
}

Which I inject into my SecurityFilterChain bean creation method and use like this

.requestMatchers(mvc.pattern("/actuator/health"), mvc.pattern("/actuator/info")).permitAll() 

I've also had all sorts of other issues

• For static resources I use the antMatcher, for example antMatcher("/css/**").
• Don't think old .requestMatchers(PathRequest.toStaticResources().atCommonLocations()).permitAll() is working anymore.

I'm still wrapping my head around this and after having tried a ridiculous amount of variations, I've not yet fully grokked the different ways of specifying request matchers but hopefully above will be of use to someone else.

Centurion

Just wanted to add a minor clarifying note - you don't have to use the MvcRequestMatcher, you can also use the AntPathRequestMatcher and it will resolve the issue. The problem starts when the delegate has to determine which type to build from a String (e.g., .requestMatchers("...")). So long as you choose one or the other the bug won't get triggered. – Genu

Answer 2

Having actuator endpoints on a different port (e.g. 9090) will create additional Servlet context (dispatcherServletRegistration), for which you cannot use the spring security that you defined for the rest of the application (port 8080) as the request matchers are strictly bound to its own servlet context (dispatcherServlet).

To solve this issue, instruct the SecurityFilterChain for the main application to not be used for the actuator endpoint

public SecurityFilterChain webFilterChain(HttpSecurity http) {
    http.securityMatcher(new NegatedRequestMatcher(new AntPathRequestMatcher("/actuator/**")));
    ...
    return http.build();
}

And define new SecurityFilterChain just for the actuator

@Bean
public SecurityFilterChain actuatorFilterChain(HttpSecurity http) throws Exception {
    http.securityMatcher(EndpointRequest.toAnyEndpoint());
    http.authorizeHttpRequests((requests) -> requests.anyRequest().permitAll());
    return http.build();
}

Please note that this configuration will allow anyone to access the actuator without any credentials.

Twitch

Answer 3

You can use:

.requestMatchers(new AntPathRequestMatcher("/actuator/health")).permitAll()                   

Libyan

Answer 4

Issue

We have custom configuration for HttpSerurity where we are able to provide custom endpoints, e.g. actuator endpoints. Application has multiple servlets and management endpoints are set up with custom port. This was working in previous version of the application. After the upgrade of Spring Boot to 3.3.0 we started getting following exceptions in actuator endpoint when changing values of loggers at runtime.

2024-08-05 15:08:01,808 [,] [99] [,] ERROR juli.logging.DirectJDKLog - Servlet.service() for servlet [dispatcherServletRegistration] in context with path ] threw exception java.lang.IllegalArgumentException: Failed to find servlet [dispatcherServletRegistration] in the servlet context
at org.springframework.util.Assert.notNull(Assert.java:172) ~[spring-core- 6.3.0.jar!/:6.3.0]
at org.springframework.security.config.annotation.web.AbstractRequestMatcherRegistry$DispatcherServletDelegatingRequestMatcher.matches(AbstractRequestMatcherRegistry.java:518) ~[spring-security-config-6.3.0.jar!/:6.3.0]
...

DispatcherServletDelegatingRequestMatcher can't resolve proper servlet by its name for actuator endpoints that are running on specific port.

Solution

This was the issue in spring-security itself. See the opened issue DispatcherServletDelegatingRequestMatcher causes errors when there is more than one ServletContext

After upgrading spring-security dependencies to latest version 6.3.1 everything was working as expected, servlet can be resolved out of the box.

As a hint, before I was able to upgrade and test whole Spring Boot from 3.3.0 to 3.3.1 or 3.3.2, I just forced versions of spring-security like:

dependencies {
  implementation(
    'org.springframework.security:spring-security-core:6.3.1',
    'org.springframework.security:spring-security-config:6.3.1',
    'org.springframework.security:spring-security-web:6.3.1'
  )
  ...
}

or BOM

// Spring Boot dependency management
dependencyManagement {
  imports {
    mavenBom "org.springframework.security:spring-security-bom:6.3.1"
    ..
  }
}

Anastassia