CloseNowException: This stream is not writeable

Question

I'm really stumped. In my server logs I'm seeing:

org.apache.coyote.CloseNowException: Connection [215], Stream [95], This stream is not writable
        at org.apache.coyote.http2.Http2UpgradeHandler.reserveWindowSize(Http2UpgradeHandler.java:843) ~[tomcat-coyote.jar:9.0.30]
        at org.apache.coyote.http2.Stream$StreamOutputBuffer.flush(Stream.java:940) ~[tomcat-coyote.jar:9.0.30]
        at org.apache.coyote.http2.Stream$StreamOutputBuffer.doWrite(Stream.java:859) ~[tomcat-coyote.jar:9.0.30]
        at org.apache.coyote.http2.Http2OutputBuffer.doWrite(Http2OutputBuffer.java:59) ~[tomcat-coyote.jar:9.0.30]
        at org.apache.coyote.Response.doWrite(Response.java:601) ~[tomcat-coyote.jar:9.0.30]

It seems to happen when a user clicks too quickly, but that makes no sense, as Tomcat should be able to serve plenty of requests. This is on a server with a very light load, maybe 2 or 3 HTTP requests per second, on a very fast machine.

This is with Spring Boot 2 and Tomcat 9.0.30. It's really perplexing.

I did see a similar question on SO where someone got this using a web push, but we're not.

Here's how our HTTP/2 connector is configured:

<Connector port="443" protocol="org.apache.coyote.http11.Http11NioProtocol"
           maxThreads="150" SSLEnabled="true"
           keystoreFile="/etc/ssl/keys.p12"
           keystorePass="changeit"
           keyAlias="tomcat"
           sslProtocol="TLS"
           sslEnabledProtocols="TLSv1.3,TLSv1.2"
           connectionTimeout="20000"
           >
  <UpgradeProtocol className="org.apache.coyote.http2.Http2Protocol"
                   keepAliveTimeout="20000"
                   />
</Connector>

It's running on JDK 13.0.2 on Ubuntu server 18.04.

Any ideas on this? It's definitely something users notice and I have no clue about how to solve this.

Posit

Found out some more: this doesn't happen when I remove the Http2 upgrade. It does happen with http2 and when the user clicks too quickly on links, which is really strange. For now we are running without http2 but that's unfortunate. – Posit

Do you get the same error if you use Jetty instead of Tomcat? – Existential

If you are clicking links before the page is fully loaded, you are closing the streams Tomcat currently attempts to write to - especially if multiple resources are pushed. So theses Exceptions do not mean Tomcat is outperformed by a single user. – Recrement

I am experiencing a similar issue, did you manage to solve it? – Culbertson

@Culbertson No it keeps happening, no solution. It's really weird. It happens on completely static resources which it should just send. The stream closes, it sends zero bytes and I get the CloseNowException. I've since tried it on Ubuntu 2004, JDK 14, and Apache Tomcat 9.0.36 and other versions and it's all the same. Is anyone investigating this? It's completely unusable as a production web server. I have seen this on two different projects, both using Spring Boot, but one of them a very minimal project without much going on. – Posit

Answer 1

This problem actually didn't go away. I tried many many things. I tried this on several newer version of Tomcat in the 9.0 series. I upgraded to JDK 14, 14.0.1 and 14.0.2, all from Oracle (I never tried the OpenJDK release, can't imagine it would make any difference). I tried all kinds of TLS settings, including limiting it to only TLS1.3.

The problem continued.

I tried running a completely different Spring Boot application, one that had to make a large number of small (500 byte) JSON POSTs to the server. Very often these POST messages would fail. It happened regularly on some browsers but not others. There wasn't any clear pattern to this. These would be browsers running on Windows, Linux and Safari, and on mobile devices. Sometimes it worked sometimes not.

Very often the most likely to fail GETs would be getting small static objects, like a small CSS file, which should in fact be the EASIEST and fastest thing to load. As a side consequence, failing to load CSS totally breaks the site, so this is really bad.

So we have many different versions of Tomcat, many versions of JDK, many different variations of config parameters for http2 and TLS, and two completely different Spring Boot applications, and we get the same problem: sporadic failures of http2 sessions. The only thing in common here is that the problems all go away when I switched back to http1.1.

Http1.1 as reliable as it is has significantly worse performance than http2 and I wanted it to work.

Out of total desperation I thought, ok, the only thing I haven't changed is the server software, and I know that Jetty's http2 implementation is completely different from Tomcat's.

I switched to a basic Jetty configuration, enabled http2, got my app deployed, and tested it. All the http2 problems went away. It started behaving as expected, performing well, and had the performance benefits of http2. This switch fixed the problems on both the Spring Boot apps that had problems before. By the way these two Spring Boot apps were very different, one involving a database and Spring Integration and tons of features, the other much simpler, involving reading and writing a few small files.

I can only reach one conclusion here: something is wrong in Tomcat's http2 implementation and it has not had sufficient testing or real-world use and should not be used in production. This was a painful thing to learn as I've been using Tomcat as my go-to server for many years and I assumed Tomcat releases would be stable and production ready but this simply was not. If you are running into these "closed stream" errors where the client simply fails to load, and it's immediate, not after any timeout, and it happens with http2 but not http1, then try switching to Jetty and see if that makes a difference. If any Tomcat developers wants to get further into this, reply to this and I can set up our system as a demo.

I really came to this with a strong bias in favor of Tomcat, given that it's the oldest and most widely used Servlet container, but by this point I conclude that Jetty is going to be my go-to.

I hope this post helps others who may be encountering this problem because it really was not easy to figure out, and in the end I didn't ever figure out what was happening, other than, something is amiss in Tomcat's http2.

Posit

Ran into the same problem with the latest Spring Boot. Thanks for your post, saved me a lot of time. – Fleenor

@Posit the same issue with OpenJDK release, this doesn't happen when I remove the Http2 upgrade – Tracheitis

Answer 2

The solution is very simple. I have had the same problem.

There are other parameter to configure in in the following link there is a detailed list https://tomcat.apache.org/tomcat-8.5-doc/config/http2.html

In your case a simple solution is :

<UpgradeProtocol className="org.apache.coyote.http2.Http2Protocol"                     
                 overheadCountFactor="-1"
                 overheadDataThreshold="0"
                 overheadWindowUpdateThreshold="0"/>

overheadCountFactor: The factor to apply when counting overhead frames to determine if a connection has too high an overhead and should be closed. The overhead count starts at -10. The count is decreased for each data frame sent or received and each headers frame received.ecc ecc ...... if the overhead count exceeds zero, the connection is closed. A value of less than 1 disables this protection. In normal usage a value of 3 or more will close the connection before any streams can complete. If not specified, a default value of 1 will be used.

Marion

This could be part of the problem. I guess because HTTP/2 uses a single connection, a mechanism has been added to terminate requests that are using lots of resources. And our requests are pretty large. So, after this config change, I have less Exceptions but still some. And this is also experienced by others (forum post). Overall, this is not a complete solution in my case – Gabriella

This in combination with readTimeout parameter (see other answer) seems to solve my issues. The linked Tomcat documentation of the parameters explains them very well. After reading it, I consider it a (configurable) feature, not a bug. Also, I suggest to enable HTTP/2 related logging in tomcat/conf/logging.properties to tackle HTTP/2 related issues. – Gabriella

Answer 3

I've had the same problem. For me the solution was to set readTimeout="20000" for the upgrade protocol:

<UpgradeProtocol className="org.apache.coyote.http2.Http2Protocol"
        readTimeout="20000" />

No guarantee it works for you though ;-)

Ritualize

documentation of the parameter can be found here: tomcat.apache.org/tomcat-8.5-doc/config/http2.html – Gabriella

Answer 4

TL;DR

There is strong evidence that this might be a bug of older versions of tomcat. Try to upgrade your tomcat's version as this helped me fix a similar problem. Below I'll give a more detailed explanation of what was happening to me.

---

Background

This error started appearing when I was trying to make an application work with HTTP 2. After I enabled HTTP 2 in my tomcat's server.xml this error started appearing in the log when I visited the main homepage. Some information stopped showing due to this and an error message was being presented to the client. This is a Java 8 application using JSP, Servlet 2.5 and Tomcat 9.0.8.

Explanation of the error

I found out, after some googling, that this error basically means that the client closed the connection and now the server cannot write a response for that connection. By connection I mean a TCP connection. For me this error was triggering when I tried to write to my response.getOutputStream(). The response in this case is an HttpServletResponse object. The only way for the server to know that the client closed the connection is to actually try to write a response to it.

Usually this error is not a big deal. Since the client closed the connection it does not matter if the server cannot give a response, the application will probably be fine. But, for some reason, this was impacting my application. Also, I was not closing any connection, so there was no reason for this error to show. Why there is a connection being closed? What connection is that?

What was causing it

After some more debugging I finally understood what was happening. According to my browser's devtools network tab, I was doing just a single request, but when I debugged my servlet actually two requests were coming to the server. One of those requests was the actual request that the browser made, but the other one I had no idea. The second request was the one that, when my servlet tried to write a response to, the error was occurring. From the perspective of my application there was two threads (one for each request) doing the exact same thing and this was causing some concurrency problems. That is why some information were not appearing in the screen.

The solution

The only thing that helped me solve this problem was to actually upgrade my tomcat. I was using version 9.0.8. When I upgraded it to version 9.0.37, the most updated version as the time of this writing, the problem stopped. I couldn't replicate it anymore. Only one connection was coming to the server, as it should. I can only assume that this is a bug in 9.0.8 version of tomcat, and may be present in other versions since I don’t know when this was eventually fixed. The tomcat’s changelog page may give some hints.

Heckman

I'm pretty sure the behavior I saw continued in Tomcat 9.0.37. The solution was switching to Jetty, at least if you want to use http2. I'm sure this is some kind of bad bug in Tomcat's http2, only question is, is it fixed or not. – Posit

Just checked, it is definitely still a problem on 9.0.37. Based on what I have experienced, I can say, Tomcat is not usable in http2 in production settings. Jetty doesn't have this bug and performance is great. I'm saying this as someone who has used Tomcat since version 4.1! I've had to switch to Jetty in production due to this issue. – Posit

Answer 5

For me this property solved the issue using Spring Boot 2.1 (or higher)

server.http2.enabled=true

Bunny

Answer 6

I think I tried all of the suggestions mentioned in this thread, including upgrading Tomcat to 10.0.6 to no avail. What did fix my problem was bumping up the number of threads on the Connector. I originally had maxThreads="10" when I bumped the threads up to 30 my problem completely went away.

(The page that was causing me problems had 25 icons on it)

Chatman

I have maxThreads="150" and still get a similar issue. However, that more threads solved the issue for you could be an indication that there is some concurrency issue in Tomcat's implementation of HTTP/2. – Gabriella