Okhttp3, http2 multiplexing POST requests high response time at peak load time
Asked Answered
A

1

6

My application will send approx 1000 POST requests/minute to my tomcat server(which is http/2 enabled) which will poll the given url and return back the html and response time, I want to achieve true http/2 multiplexing to reuse tcp connection between my application and tomcat server. my client uses okhttp and I can successfully establish connection and reuse it for a long time, but when the number of request count increases(something like, queued:50, running:50), the response time also increases(it spikes to 2000ms-15000ms or even worse, which would normally take 300ms-500ms). I can understand this is happening because of overloading of tcp connection with too many requests, So I have decided to open multiple tcp connections and allow it to distribute the request load across tcp connections. I force the client to open multiple connections using

Dispatcher dispatcher = new Dispatcher();
dispatcher.setMaxRequests(100);
dispatcher.setMaxRequestsPerHost(5);
ConnectionPool cp = new ConnectionPool(5, 5, TimeUnit.MINUTES);

With the help of wireshark, I can see 5 connections opened and also see 4 of the requests are closed immediately after the first handshake is succeeded, I don't know whether that's the expected behaviour of http/2 multiplexing.

If that's expected behaviour, how can I optimize it to reduce the response time of each request at peak load time?

Or is that possible to use multiple connections to distribute the load at peak load time?

My sample program as follows,

Security.insertProviderAt(Conscrypt.newProvider(), 1);
sslContext = SSLContext.getInstance("TLSv1.3");
sslContext.init(null, new TrustManager[] {
    new X509TrustManager() {
        @Override
        public void checkClientTrusted(java.security.cert.X509Certificate[] chain, String authType) throws CertificateException {
        }

        @Override
        public void checkServerTrusted(java.security.cert.X509Certificate[] chain, String authType) throws CertificateException {
        }

        @Override
        public java.security.cert.X509Certificate[] getAcceptedIssuers() {
            return new java.security.cert.X509Certificate[]{};
        }
    }       
}, new java.security.SecureRandom());
sslSocketFactory = sslContext.getSocketFactory();
Dispatcher dispatcher = new Dispatcher();
dispatcher.setMaxRequests(100);
dispatcher.setMaxRequestsPerHost(5);
ConnectionPool cp = new ConnectionPool(5, 1, TimeUnit.DAYS);

okHttpClient = new OkHttpClient().newBuilder()
.sslSocketFactory(sslSocketFactory, (X509TrustManager)trustManager[0])
.dispatcher(dispatcher)
.connectionPool(cp)
.hostnameVerifier(new HostnameVerifier() {
    @Override
    public boolean verify(String hostname, SSLSession session) {
        return true;
    }
})
.build();

We use conscryt to enable ALPN in jdk8 itself,

try {
    String url = "https://localhost:8081/myAent/getOutput?1000000004286_1520339351002_"+System.currentTimeMillis();
    String json = "<?xml version=\"1.0\" standalone=\"no\"?><UC mid=\"1000000005011\" pollNow=\"true\" use_ipv6=\"false\" rca=\"0\" rcaFN=\"P|TA|D|J|M\" pFtct=\"\" sFtct=\"\" issecondarydc=\"false\" ssDc=\"false\" nocache=\"1\" storeHtmlResp=\"true\" storeTroubleScrnSht=\"false\" moConfig=\"false\" xconlen=\"8265\"  noScreenshotRecheckForSSLErrors=\"true\" avgDnsTime=\"null\" isProxyRequired=\"false\" userroles=\"EVAL_USER\" uid=\"102030230293029021\" retryDelay=\"2\" retry=\"true\" idcLocUrl=\"http://localhost:8080/app/receivemultipartdata\" isRemoteAgent=\"true\" sendHeaders=\"false\" api=\"ab_345nnn4l4lj4lk23nl4k23342lk4l23j4\" ut=\""+(System.currentTimeMillis()+"")+"\" mt=\"URL\" dctimeout=\"70\" pollinterval=\"1440\" locid=\"48\" log=\"1\" currentstatus=\"1\" postUrl=\"https://example.com\"><Url acc=\"\" forced_ips=\"\" use_ipv6=\"false\" client_cert=\"\" mid=\"1000000005011\" sotimeout=\"60\" ua=\"\" ds=\"117.20.43.94\" ucc=\"\" md=\"false\" client_cert_pass=\"\" context=\"default\" unavail_alert=\"\" ssl_protocol=\"\" avail_alert=\"\" enabledns=\"false\" enableBouncyCastle=\"false\" cc=\"false\" a=\"https://www.example.com/tools.html\" upStatusCodes=\"\" regex_alert=\"\" probeproxy=\"false\" m=\"G\" keyword_case=\"0\" regex=\"\" rbc=\"\" t=\"30\" lc=\"English\"><PD></PD><CH hn=\"\" hd=\"_sep_\" hv=\"\"/><AI ps=\"\" un=\"\"/></Url></UC>";
    RequestBody body = RequestBody.create(MediaType.get("application/json; charset=utf-8"), json);
    Request request = new Request.Builder()
        .url(url)
        .post(body)
        .build();
    long nanoStartTime = System.nanoTime();
    okHttpClient.newCall(request).enqueue(new Callback() {
        @Override 
        public void onFailure(Call call, IOException e) {
            System.out.println("okhttp3:: Request failed"+ e);
        }

        @Override
        public void onResponse(Call call, okhttp3.Response responseObj) throws IOException {
            try (ResponseBody body = responseObj.body()) {
                long nanoEndTime = System.nanoTime();
                long nanoDiffTime = TimeUnit.NANOSECONDS.toMillis(nanoEndTime - nanoStartTime);
                System.out.println("okhttp3:: Succeded response ***"+body+"$$$");
                System.out.println("okhttp3:: Request Succeded protocol ***"+responseObj.protocol()+"$$$, time is "+nanoDiffTime);
            }
        }
    });

} catch (Exception e) {
    // TODO Auto-generated catch block
    e.printStackTrace();
}

How can I optimize okhttpclient to use multiple tcp sockets/connection for achieveing http/2 multiplexing to distribute the request load.

Client : Tomcat apache - 9.0.x, Jdk - 8, Http library - Okhttp3, Os - Ubuntu/Centos, Security Provider - Conscrypt(For supporting ALPN in jdk 8).

Server : Tomcat apache - 9.0.16, Jdk -10.0.1, Os - Ubuntu/Centos, OpenSSL - 1.1.1a for TLSv1.3 support

Ammoniacal answered 24/2, 2019 at 8:3 Comment(0)
I
6

What you are seeing is the result of connection coalescing in OkHttp. OkHttp doesn't know ahead of time whether HTTP/2 connections will be established (vs HTTP/1.1), so allows the setMaxRequestsPerHost value of connections to proceed. There is currently* no client side load balancing, so these are quickly coalesced to a single connection which is what you are seeing.

Currently you can achieve this via multiple clients, or careful management of completely distinct hosts and connections. n.b OkHttp will work against you here and optimise for the typical case where a single connection is better than multiple e.g. SSL certificate specifies overlapping Subject Alternative Names‎


* Follow this issue for support https://github.com/square/okhttp/issues/4530

Imprisonment answered 24/2, 2019 at 8:16 Comment(3)
Any other java client supports such feature? Like Apache httpcomponents5 or jetty?Ammoniacal
Also you can configure Tomcat’s maxConcurrentStreams and OkHttp will honor it. tomcat.apache.org/tomcat-8.5-doc/config/http2.htmlDrogue
The documentation explains, it will throw STREAM_REFUSED exception once the limit has reached, whether it allows client to create new connections after the configured limit is exceeded or just one connection with the configured limit and refuses to accept any more connections or request concurrently(which will create a delay in sending the request making the response time worse)??Ammoniacal

© 2022 - 2024 — McMap. All rights reserved.