java.io.EOFException: Premature EOF: no length prefix available in Spark on Hadoop

Question

I'm getting this weird exception. I'm using Spark 1.6.0 on Hadoop 2.6.4 and submitting Spark job on YARN cluster.

16/07/23 20:05:21 WARN hdfs.DFSClient: DFSOutputStream ResponseProcessor exception  for block BP-532134798-128.110.152.143-1469321545728:blk_1073741865_1041
java.io.EOFException: Premature EOF: no length prefix available
    at org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:2203)
    at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:176)
    at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:867)
16/07/23 20:49:09 ERROR server.TransportRequestHandler: Error sending result RpcResponse{requestId=4719626006875125240, body=NioManagedBuffer{buf=java.nio.HeapByteBuffer[pos=0 lim=81 cap=81]}} to ms0440.utah.cloudlab.us/128.110.152.175:58944; closing connection
java.nio.channels.ClosedChannelException

I was getting this error when running on Hadoop 2.6.0 and thought the exception might be kind of a bug like this but after even changing this to Hadoop 2.6.4 I'm getting the same error. There is not any memory problem, my cluster is good with HDFS and memory. I went through this and this but no luck.

Note: 1. I'm using Apache Hadoop and Spark not any CDH/HDP. 2. I'm able to copy data in HDFS and even able to execute another job on this cluster.

Salford

i get similar issues. Did you find a solution ? @Salford – Bordie

@Bordie not yet. – Salford

I think Hadoop version was the issue. No issues in 2.7.2, which I updated later; Also, check your firewall rules. – Salford

Answer 1

Check file permissions of dfs directory:

find /path/to/dfs -group root

In general, the user permission group is hdfs. Since I started HDFS service with root user, some dfs block file with root permissions was generated.

I solved the problem after change to right permissions:

sudo chown -R hdfs:hdfs /path/to/dfs

Singleness