java.io.IOException: Broken pipe

Question

We are currently migrating a legacy application to Jetty. And I have somehow an exception regarding a broken pipe.

• Java 6
• Jetty 8.1.8
• Spring 3.2.0

I am trying to migrate a Glassfish web application to Jetty. In our testing environment we are using a load balancer and everything is working fine. Our clients are working without any problem.

WARN  [2013-04-03 13:34:28,963] com.myapp.bbb.config.MvcDefaultConfig$1: Handler execution resulted in exception
! org.eclipse.jetty.io.EofException: null
! at org.eclipse.jetty.http.HttpGenerator.flushBuffer(HttpGenerator.java:914)
! at org.eclipse.jetty.http.HttpGenerator.complete(HttpGenerator.java:798)
! at org.eclipse.jetty.server.AbstractHttpConnection.completeResponse(AbstractHttpConnection.java:642)
! at org.eclipse.jetty.server.Response.complete(Response.java:1234)
! at org.eclipse.jetty.server.Response.sendError(Response.java:404)
! at org.eclipse.jetty.server.Response.sendError(Response.java:416)
! at org.springframework.web.servlet.DispatcherServlet.noHandlerFound(DispatcherServlet.java:1111)
! at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:898)
! at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:856)
! at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:915)
! at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:811)
! at javax.servlet.http.HttpServlet.service(HttpServlet.java:735)
! at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:796)
! at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
! at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:669)
! at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1336)
! at com.magnetdigital.maggy.dropwizard.head2get.Head2GetFilter.doFilter(Head2GetFilter.java:22)
! at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1307)
! at com.yammer.dropwizard.servlets.ThreadNameFilter.doFilter(ThreadNameFilter.java:29)
! at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1307)
! at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:453)
! at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:229)
! at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1072)
! at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:382)
! at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
! at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1006)
! at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
! at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
! at com.yammer.metrics.jetty.InstrumentedHandler.handle(InstrumentedHandler.java:200)
! at org.eclipse.jetty.server.handler.GzipHandler.handle(GzipHandler.java:275)
! at com.yammer.dropwizard.jetty.BiDiGzipHandler.handle(BiDiGzipHandler.java:123)
! at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
! at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
! at org.eclipse.jetty.server.Server.handle(Server.java:365)
! at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:485)
! at org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
! at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:926)
! at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:988)
! at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:635)
! at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
! at org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
! at org.eclipse.jetty.server.nio.BlockingChannelConnector$BlockingChannelEndPoint.run(BlockingChannelConnector.java:298)
! at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
! at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
! at java.lang.Thread.run(Thread.java:662)
Caused by: ! java.io.IOException: Broken pipe
! at sun.nio.ch.FileDispatcher.write0(Native Method)
! at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
! at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69)
! at sun.nio.ch.IOUtil.write(IOUtil.java:26)
! at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334)
! at org.eclipse.jetty.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:293)
! at org.eclipse.jetty.server.nio.BlockingChannelConnector$BlockingChannelEndPoint.flush(BlockingChannelConnector.java:253)
! at org.eclipse.jetty.http.HttpGenerator.flushBuffer(HttpGenerator.java:850)
!... 44 common frames omitted

When I check the stacktrace I have seen this exceptions are triggered by always a 404 request.

org.springframework.web.servlet.DispatcherServlet.noHandlerFound(DispatcherServlet.java:1111)

• Why am I having this exception?
• How can I reproduce this exception at my machine locally?

Dubitation

in my case it happended when i set popup to load from my backend with time out to auto close, and it closes before loading the popup – Hurleigh

Answer 1

The most common reason I've had for a "broken pipe" is that one machine (of a pair communicating via socket) has shut down its end of the socket before communication was complete. About half of those were because the program communicating on that socket had terminated.

If the program sending bytes sends them out and immediately shuts down the socket or terminates itself, it is possible for the socket to cease functioning before the bytes have been transmitted and read.

Try putting pauses anywhere you are shutting down the socket and before you allow the program to terminate to see if that helps.

FYI: "pipe" and "socket" are terms that get used interchangeably sometimes.

Murphy

I almost know request which is creating this exception but I could not be successful to be create a similar one. Do you have any idea to generate a broken pipe by curl or something else? – Dubitation

rcook is correct, looks like the client didn't bother completely consumimg the response before exiting...you can often get this when a user navigates away from a page before it has finished loading completely – Charland

I found my problem is that there is a timeout setting on the client side, the client side will close the connection when the waiting time exceeds. After that, the server side wants to flush the response to the client side but the stream is already closed. So the "broken pipe" issue happened. You can adjust the client side timeout setting to eliminate this scenario. – Matchmaker

So I have handled this exception to warn level. – Galloromance

Clearly the code responsible for sending the info has to be put outside the user's control, i.e., if the user can click a button and cause that thread to exit, then you are at the user's mercy. Without knowing your application, it is difficult to make any specific suggestions, but you could think about ensuring that the code responsible for completing the socket send is on a non-daemon (or is it "on a daemon"?) thread, i.e., one that doesn't exit automatically when the app terminates, so that it can finish sending. This assumes you have control over the sender, of course... – Murphy

In my case, the connection was getting terminated by load-balancer after specific time – Ziagos

I thought Java was async. I occasionally have this issue. I do a request to an API. Often everything just works, but sometimes instead of awaiting this response, my Java program jumps to the next webhook it receives, before completing the previous functions. How can I force my function to await the restexchanges response? – Java

I don't know what you mean by "async" in this context, nor am I sure what "this respone" you're referring to or what a "webhook" is. If you're calling a web api, then whether it completes its operation before returning a result to you is up to the people programming that API, and they may or may not guarantee it to work the way it currently works. If it might be asynchronous -- that is, return to you before completing its requested task -- then in order to wait for completion, you're going to need that API to provide you something you can wait on or get a completion value from or something . – Murphy

Answer 2

I agree with @arcy, the problem is on client side, on my case it was because of nginx, let me elaborate I am using nginx as the frontend (so I can distribute load, ssl, etc ...) and using proxy_pass http://127.0.0.1:8080 to forward the appropiate requests to tomcat.

There is a default value for the nginx variable proxy_read_timeout of 60s that should be enough, but on some peak moments my setup would error with the java.io.IOException: Broken pipe changing the value will help until the root cause (60s should be enough) can be fixed.

NOTE: I made a new answer so I could expand a bit more with my case (it was the only mention I found about this error on internet after looking quite a lot)

Weide

Answer 3

Basically, what is happening is that your user is either closing the browser tab, or is navigating away to a different page, before communication was complete. Your webserver (Jetty) generates this exception because it is unable to send the remaining bytes.

org.eclipse.jetty.io.EofException: null
! at org.eclipse.jetty.http.HttpGenerator.flushBuffer(HttpGenerator.java:914)
! at org.eclipse.jetty.http.HttpGenerator.complete(HttpGenerator.java:798)
! at org.eclipse.jetty.server.AbstractHttpConnection.completeResponse(AbstractHttpConnection.java:642)
! 

This is not an error on your application logic side. This is simply due to user behavior. There is nothing wrong in your code per se.

There are two things you may be able to do:

1. Ignore this specific exception so that you don't log it.
2. Make your code more efficient/packed so that it transmits less data. (Not always an option!)

Poff

In case that you want (like me) to find a way to ignore this specific type of IOException (and not ignore all the others), then this blog-post does a good job at showing how to do it in Spring MVC: mtyurt.net/post/… – Harmless

Answer 4

Error message suggests that the client has closed the connection while the server is still trying to write out a response.

Refer to this link for more details:

https://markhneedham.com/blog/2014/01/27/neo4j-org-eclipse-jetty-io-eofexception-caused-by-java-io-ioexception-broken-pipe/

Pittsburgh

Answer 5

increase the response.getBufferSize() get the buffer size and compare with the bytes you want to transfer !

Laverty

Hello , I am practicing this , and this is a single line answer for this question. Does answer from arcy provide any valuable inputs for you ? Do you know the point to alter it ? – Laverty

There is no bufferSize() method in the JDK, or any of the variant spellings, and 'compare [thebuffer size] with the bytes you want to transfer' is meaningless. Answer is nonsense. – Perfuse

@VimalkumarNatarajan consider to leave some code snippet easy to follow by the OP, for my the length of the answer doesn't say nothing about the answer, so it is fine as look it is helpful to the op, if that is not the case, so consider editing and eventually deleting this answer. Voting up for the goodwill – Catboat

Answer 6

You may have not set the output file.

Gabby

Answer 7

Maybe still someone will find this answer relevant: In my case this error was due to file upload limit being smaller than uploaded file size.

Henrietta

Where is the file upload limit set in your case? On the client or server side? – Concussion