How to exclude some uri to be observed using springboot3/micrometer

Question

Hy

I am using springboot 3 with the new micrometer observation. Is there a way to prevent generating a trace_id/span_id for some paths like /actuator/prometheus? Observation add a trace id for each call to /actuator/*.

Thank you

Rejoin

Answer 1

I supplied one answer using ObservationPredicate, which filters out tracing and metrics.

Here is another answer using SpanExportingPredicate, which only filters out traces, and leaves metrics in place.

import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.http.server.observation.ServerRequestObservationContext;
@Configuration
public class TracingConfig {

@Bean
SpanExportingPredicate noActuator() {
    return span -> span.getTags().get("uri") == null || !span.getTags().get("uri").startsWith("/actuator");
}
@Bean
SpanExportingPredicate noSwagger() {
    return span -> span.getTags().get("uri") == null || !span.getTags().get("uri").startsWith("/swagger");
}
@Bean
SpanExportingPredicate noApiDocs() {
    return span -> span.getTags().get("uri") == null || !span.getTags().get("uri").startsWith("/v3/api-docs");
}

}

Headstone

Answer 2

I managed to find a half solution to the problem, by defining the ObservationRegistry this way:

@Bean
@ConditionalOnMissingBean
ObservationRegistry observationRegistry() {
    PathMatcher pathMatcher = new AntPathMatcher("/");
    ObservationRegistry observationRegistry = ObservationRegistry.create();
    observationRegistry.observationConfig().observationPredicate((name, context) -> {
        if(context instanceof ServerRequestObservationContext) {
            return !pathMatcher.match("/actuator/**", ((ServerRequestObservationContext) context).getCarrier().getRequestURI());
        } else {
            return true;
        }
    });
    return observationRegistry;
}

This doesn't completely ignore the actuator requests, only the first span. So if you have for example Spring Security on your classpath, those spans are left intact.

EDIT: Looks like you don't need to redefine the entire observation registry, you can use a bean like the one show here: https://docs.spring.io/spring-security/reference/servlet/integrations/observability.html

EDIT2: From what I can tell, you need to include these 2 beans, and no actuator call will be traced (completely disables tracing for spring security too):

    @Bean
    ObservationRegistryCustomizer<ObservationRegistry> skipActuatorEndpointsFromObservation() {
        PathMatcher pathMatcher = new AntPathMatcher("/");
        return (registry) -> registry.observationConfig().observationPredicate((name, context) -> {
            if (context instanceof ServerRequestObservationContext observationContext) {
                return !pathMatcher.match("/actuator/**", observationContext.getCarrier().getRequestURI());
            } else {
                return true;
            }
        });
    }

    @Bean
    ObservationRegistryCustomizer<ObservationRegistry> skipSecuritySpansFromObservation() {
        return (registry) -> registry.observationConfig().observationPredicate((name, context) ->
                !name.startsWith("spring.security"));
    }

Also, you might want to keep an eye out on this issue: https://github.com/spring-projects/spring-framework/issues/29210

Cotswold

I've seen that is the first span if filtered out (return false by the predicate) the parentOsevation of the subsequent span is a NoopObservation. So when the parent is Noop, I'm also filtering the next spans ... See response below for full code. – Implement

Answer 3

I've managed to do some filtering with a ObservationPredicate that filters some URLS and filters the next spans too.

I've seen that is the first span if filtered out (return false by the predicate) the parentOsevation of the subsequent span is a NoopObservation.

So when the parent is Noop, I'm also filtering the next spans ...

package fr.common.micrometer.observers;

import java.util.Collection;
import java.util.List;
import java.util.Optional;
import java.util.function.Predicate;
import org.springframework.http.server.observation.ServerRequestObservationContext;
import org.springframework.security.web.util.matcher.AntPathRequestMatcher;
import io.micrometer.observation.Observation.Context;
import io.micrometer.observation.ObservationPredicate;
import io.micrometer.observation.ObservationRegistry;
import io.micrometer.observation.ObservationView;
import jakarta.servlet.http.HttpServletRequest;
import lombok.extern.slf4j.Slf4j;

@Slf4j
public class UrlObservationPredicate implements ObservationPredicate {

  private static final String REQUEST_NAME = "http.server.requests";

  private static final List<String> DEFAULT_FILTERED_URLS = List.of("/actuator/**");

  private List<AntPathRequestMatcher> filteredUrls;

  public UrlObservationPredicate() {
    this(DEFAULT_FILTERED_URLS);
  }

  public UrlObservationPredicate(List<String> filteredPaths) {
    this.filteredUrls = Optional.ofNullable(filteredPaths)
                                .filter(Predicate.not(Collection::isEmpty))
                                .orElse(DEFAULT_FILTERED_URLS)
                                .stream()
                                .map(AntPathRequestMatcher::new)
                                .toList();
  }

  @Override
  public boolean test(String name, Context context) {
    if (parentIsNoop(context)) {
      log.trace("Parent is Noop so filtering: {}", name);
      return false;
    }
    if (!REQUEST_NAME.equals(name)) {
      return true;
    }
    if (context instanceof ServerRequestObservationContext requestContext) {
      HttpServletRequest httpRequest = requestContext.getCarrier();
      if (filteredUrls.stream()
                      .anyMatch(matcher -> matcher.matches(httpRequest))) {
        log.trace("Filtering uri: {}", httpRequest.getRequestURI());
        return false;
      }
    }
    return true;
  }

  private boolean parentIsNoop(Context context) {
    return Optional.ofNullable(context.getParentObservation())
                   .map(ObservationView::getObservationRegistry)
                   .filter(ObservationRegistry::isNoop)
                   .isPresent();
  }
}

Implement

Answer 4

You can register multiple ObservationPredicate beans, one for each thing you want to filter out. I was surprised to find some context on my URIs so I found I had to use 'contains' rather than 'startsWith' style matching.

import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.http.server.observation.ServerRequestObservationContext;

import io.micrometer.observation.ObservationPredicate;

@Configuration
public class TracingConfig {

    /*
     * Spring Micrometer tracing configuration
     * Includes predicates to skip tracing on non-business endpoints such as actuator and swagger
     */

    @Bean
    public ObservationPredicate noSpringSecurity() {
        return (name, context) -> !name.startsWith("spring.security.");
    }

    @Bean
    public ObservationPredicate noActuator() {
        return (name, context) -> {
            if (context instanceof ServerRequestObservationContext srCtx) {
                return !srCtx.getCarrier().getRequestURI().contains("/actuator/");
            }
            return true;
        };
    }

    @Bean
    public ObservationPredicate noSwagger() {
        return (name, context) -> {
            if (context instanceof ServerRequestObservationContext srCtx) {
                return !srCtx.getCarrier().getRequestURI().contains("/swagger");
            }
            return true;
        };
    }

    @Bean
    public ObservationPredicate noApiDocs() {
        return (name, context) -> {
            if (context instanceof ServerRequestObservationContext srCtx) {
                return !srCtx.getCarrier().getRequestURI().contains("/v3/api-docs");
            }
            return true;
        };
    }
}

Headstone

Answer 5

You need to give more information about a problem, but i think you have set this line in application.propeties like:

management.endpoints.web.exposure.include=/actuator/*

But exists option like:

management.endpoints.web.exposure.exclude=/actuator/prometheus

Middlebrow

hy dr34mer, thank you for the answer. But i need to include the /actuator/prometheus, i use it to monitor the application. I am using the new observability provided by springboot/micrometer. When prometheus call /actuator/prometheus to scrap metrics, this call will generate a trace/span so i want to prevent generating a traceid/spanid for this path. – Rejoin

You use zipkin/sleuth I guess. Try with spring.sleuth.web.skip-pattern=/actuator/prometheus in your properties. I am not sure is that a correct regex. That should help. https://github.com – Middlebrow

I am not using sleuth, it's no more supported in spring-boot 3. Spring-boot 3 use micrometer – Rejoin