Zuul timing out in long-ish requests

Question

I am using a front end Spring Cloud application (micro service) acting as a Zuul proxy (@EnableZuulProxy) to route requests from an external source to other internal micro services written using spring cloud (spring boot).
The Zuul server is straight out of the applications in the samples section

@SpringBootApplication
@Controller
@EnableZuulProxy
@EnableDiscoveryClient
public class ZuulServerApplication {
    public static void main(String[] args) {
        new SpringApplicationBuilder(ZuulServerApplication.class).web(true).run(args);
    }
}

I ran this set of services locally and it all seems to work fine but if I run it on a network with some load, or through a VPN, then I start to see Zuul forwarding errors, which I am seeing as client timeouts in the logs.

Is there any way to change the timeout on the Zuul forwards so that I can eliminate this issue from my immediate concerns? What accessible parameter settings are there for this?

Singley

Answer 1

The properties to set are: ribbon.ReadTimeout in general and <service>.ribbon.ReadTimeout for a specific service, in milliseconds. The Ribbon wiki has some examples. This javadoc has the property names.

Lorrielorrimer

Thanks that helps. Where would I find the default settings for all these parameters? – Singley

These are the configs: com.netflix.client.config.DefaultClientConfigImpl? – Singley

is it possible to set this property not only for ribbon if I'm not using it? – Latoshalatouche

Answer 2

In my case I had to change the following property:

zuul.host.socket-timeout-millis=30000

Deathless

this is the solution when not using ribbon – Passed

is it possible to appy this solution per host to avoid applying it for all endpoints? – Latoshalatouche

Can you please explain what this does? Is it a setting to increase the timeout between user logins, or to kill the page after 30 seconds? – Hadlock

Answer 3

I have experienced the same problem: in long requests, Zuul's hystrix command kept timing out after around a second in spite of setting ribbon.ReadTimeout=10000.

I solved it by disabling timeouts completely:

hystrix:
  command:
    default:
      execution:
        timeout:
          enabled: false

An alternative that also works is change Zuul's Hystrix isolation strategy to THREAD:

hystrix:
  command:
    default:
      execution:
        isolation:
          strategy: THREAD
          thread:
            timeoutInMilliseconds: 10000

Dorthydortmund

thank you very very very much:-) The last one worked for me! Has it some disadvantages? Oh my god.. no timeouts anymore.. thanks – Externalization

I'm not sure but I don't think there's a problem. Actually, we've used in in production without problems so far. You can read more about why SEMAPHORE is the default implementation here. #29966317 – Dorthydortmund

To use this isolated to a specific service: # Disable Hystrix timeout for a single service: hystrix.command.<serviceName>.execution.timeout.enabled: false # Increase the Hystrix timeout to 60s (per service) hystrix.command.<serviceName>.execution.isolation.thread.timeoutInMilliseconds: 60000 – Locally

Answer 4

This worked for me, I had to set connection and socket timeout in the application.yml:

zuul:
  host:
    connect-timeout-millis: 60000 # starting the connection 
    socket-timeout-millis: 60000  # monitor the continuous incoming data flow

Hyacinthie

This worked for me. Short and simple. Thanks bro. – Oilcan

Answer 5

I had to alter two timeouts to force zuul to stop timing out long-running requests. Even if hystrix timeouts are disabled ribbon will still timeout.

hystrix:
  command:
    default:
      execution:
        timeout:
          enabled: false
ribbon:
  ReadTimeout: 100000
  ConnectTimeout: 100000 

Haff

thanks for this, last time I logged in to stackoverflow was in 2016, I had to login to say thank you and upvote – Jeb

Answer 6

If Zuul uses service discovery, you need to configure these timeouts with the ribbon.ReadTimeout and ribbon.SocketTimeout Ribbon properties.

If you have configured Zuul routes by specifying URLs, you need to use zuul.host.connect-timeout-millis and zuul.host.socket-timeout-millis

by routes i mean

zuul:
  routes:
    dummy-service:
      path: /dummy/**

Diastrophism

Answer 7

I had a similar issue and I was trying to set timeout globally, and also sequence of setting timeout for Hystrix and Ribbon matters.

After spending plenty of time, I ended up with this solution. My service was taking upto 50 seconds because of huge volume of data.

Points to consider before changing default value for Timeout:

Hystrix time should be greater than combined time of Ribbon ReadTimeout and ConnectionTimeout.

Use for specific service only, which means don't set globally (which doesn't work).

I mean use this:

command:
   your-service-name:

instead of this:

command:
   default:

Working solution:

hystrix:
 command:
   your-service-name:
  execution:
    isolation:
      strategy: THREAD
      thread:
        timeoutInMilliseconds: 95000

your-service-name:
 ribbon:
  ConnectTimeout: 30000
  ReadTimeout: 60000
  MaxTotalHttpConnections: 500
  MaxConnectionsPerHost: 100

Reference

Almeria

What about timeout for specific route of service ? – Brazilein

Answer 8

Only these settings on application.yml worked for me:

ribbon:
    ReadTimeout: 90000
    ConnectTimeout: 90000

eureka:
    enabled: true

zuul:
    host:
        max-total-connections: 1000
        max-per-route-connections: 100
    semaphore:
        max-semaphores: 500

hystrix:
    command:
        default:
            execution:
                isolation:
                    thread:
                        timeoutInMilliseconds: 1000000

Hope it helps someone!

Abidjan