Memory Leak Issue with spring-cloud-starter-hystrix and spring-cloud-starter-archaius integration

Question

We are using spring-cloud-starter-hystrix with spring-cloud-starter-archaius where we are unable to stop the poolingconfigurationSource thread of archaius once the war is un-deployed. But spring-cloud-starter-archaius is working fine without hystrix and thread is stopped once war is un-deployed.

Duiker

"pollingConfigurationSource" thread is not killed automatically when application is un deployed from application server. This thread is created by archaius API. Checked thread dump in visualVM after starting application and undeploying application. At the time of un deployment , same thread is killed automatically and then immediately new thread is created with same name (pollingConfigurationSource). – Duiker

Answer 1

**Issue resolved permanently.**

**There are 2 approach :**
1) Create ContextListener in Servlet and in destroy method , copy below code.

2) If you are using Histrix + Spring Boot + Archaius then on main spring application java file , copy below code in method annonated with @PreDestory annotations.

    **Solution :**

    try {
    if (ConfigurationManager.getConfigInstance() instanceof DynamicConfiguration) {
    DynamicConfiguration config = (DynamicConfiguration) ConfigurationManager.getConfigInstance();
    config.stopLoading();
    } else if (ConfigurationManager.getConfigInstance() instanceof ConcurrentCompositeConfiguration) {
    ConcurrentCompositeConfiguration configInst = (ConcurrentCompositeConfiguration) ConfigurationManager
    .getConfigInstance();
    List<AbstractConfiguration> configs = configInst.getConfigurations();
    if (configs != null) {
    for (AbstractConfiguration config : configs) {
    if (config instanceof DynamicConfiguration) {
    ((DynamicConfiguration) config).stopLoading();
    break;
    }
    }
    }
    }
    } catch (Exception e) {
    e.printStackTrace();
    } 

Duiker

Answer 2

Try reseting Hystrix before the Spring Application shuts down

@EnableCircuitBreaker
@SpringBootApplication
public class Application {

  public static void main(String[] args) {
    SpringApplication.run(Application.class, args);
  }

  @PreDestroy
  public void cleanUp() {
    Hystrix.reset();
  }

}

Beastly

Thanks Davin for prompt response. – Duiker

I have tried same but its not working.. After undeploying app from server , its automatically created new thread again. see below thread dump for your information. – Duiker

Thread dump : "pollingConfigurationSource" #227 daemon prio=5 os_prio=0 tid=0x000000001cafd800 nid=0x218 waiting on condition [0x000000002dd0f000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) – Duiker

You should be able to remove spring-cloud-starter-archaius from your project as Archaius is a transitive dependency of Hystrix. Does the proposed solution above work without the extra dependency? – Beastly

@Beastly This not work for me. When i run project it give me log: Process finished with exit code 1 – Locative

@Beastly Please answer my question: #52811355 – Locative

Answer 3

both Davin and Ashish Patel are right: there are multiple leaks caused by Spring cloud.

The presence of a some threads named pollingConfigurationSource can be partially fixed by the solution proposed by Davin. You also need to make sure not to have any file named config.properties in your classpath because com.netflix.config.sources.URLConfigurationSource (look in the source for all the cases) will search for common patsh and start an exectutor thread. there are multiple path in the code that causes an executorservice on thread "pollingConfigurationSource" to be started (an not be always stopped). In my case removing "config.properties" solved this leak

The other leak I'm aware of is caused by Hystrix/RjJava. Instead of calling Histrix.reset call rx.schedulers.Schedulers.shutdown(); this will force threads "RxIoScheduler-" to exit.

Gleesome