Login/Register
hadoop Protocol message was too large. May be malicious. Use CodedInputStream.setSizeLimit() to increase the size limit
Asked Answered
hadoophadoop2
N

2

9

I am seen this in the logs of the data nodes. This is probably because I am copying 5 million files into HDFS:

java.lang.IllegalStateException: com.google.protobuf.InvalidProtocolBufferException: Protocol message was too large.  May be malicious.  Use CodedInputStream.setSizeLimit() to increase the size limit.
    at org.apache.hadoop.hdfs.protocol.BlockListAsLongs$BufferDecoder$1.next(BlockListAsLongs.java:332)
    at org.apache.hadoop.hdfs.protocol.BlockListAsLongs$BufferDecoder$1.next(BlockListAsLongs.java:310)
    at org.apache.hadoop.hdfs.protocol.BlockListAsLongs$BufferDecoder.getBlockListAsLongs(BlockListAsLongs.java:288)
    at org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB.blockReport(DatanodeProtocolClientSideTranslatorPB.java:190)
    at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.blockReport(BPServiceActor.java:507)
    at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:738)
    at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:874)
    at java.lang.Thread.run(Thread.java:745)
    Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol message was too large.  May be malicious.  Use CodedInputStream.setSizeLimit() to increase the size limit.
    at com.google.protobuf.InvalidProtocolBufferException.sizeLimitExceeded(InvalidProtocolBufferException.java:110)
    at com.google.protobuf.CodedInputStream.refillBuffer(CodedInputStream.java:755)
    at com.google.protobuf.CodedInputStream.readRawByte(CodedInputStream.java:769)
    at com.google.protobuf.CodedInputStream.readRawVarint64(CodedInputStream.java:462)
    at com.google.protobuf.CodedInputStream.readSInt64(CodedInputStream.java:363)
    at org.apache.hadoop.hdfs.protocol.BlockListAsLongs$BufferDecoder$1.next(BlockListAsLongs.java:326)
    ... 7 more

I am just using hadoop fs -put .... to copy the files to HDFS. Recently I started getting these kinds of messages on the client side:

15/06/30 15:00:58 INFO hdfs.DFSClient: Could not complete /pdf-nxml/file1.nxml._COPYING_ retrying...
15/06/30 15:01:05 INFO hdfs.DFSClient: Could not complete /pdf-nxml/2014-full/file2.nxml._COPYING_ retrying...

I get a msesage like the above approximately 3 times per minute, but the exceptions are more frequent on the data nodes.

How can I fix this?

EDIT
I had to restart hadoop and now it doesn't start up properly with these in each data node's log file:

2015-07-01 06:20:35,748 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Unsuccessfully sent block report 0x2ac82e1cf6e64,  containing 1 storage report(s), of which we sent 0. The reports had 6342936 total blocks and used 0 RPC(s). This took 542 msec to generate and 240 msecs for RPC and NN processing. Got back no commands.
    2015-07-01 06:20:35,748 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: Exception in BPOfferService for Block pool BP-1043486900-10.0.1.42-1434126972501 (Datanode Uuid d5dcf9a0-c82d-49d8-8162-af5910c3e3fe) service to cruncher02/10.0.1.42:8020
    java.lang.IllegalStateException: com.google.protobuf.InvalidProtocolBufferException: Protocol message was too large.  May be malicious.  Use CodedInputStream.setSizeLimit() to increase the size limit.
    at org.apache.hadoop.hdfs.protocol.BlockListAsLongs$BufferDecoder$1.next(BlockListAsLongs.java:332)
    at org.apache.hadoop.hdfs.protocol.BlockListAsLongs$BufferDecoder$1.next(BlockListAsLongs.java:310)
    at org.apache.hadoop.hdfs.protocol.BlockListAsLongs$BufferDecoder.getBlockListAsLongs(BlockListAsLongs.java:288)
    at org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB.blockReport(DatanodeProtocolClientSideTranslatorPB.java:190)
    at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.blockReport(BPServiceActor.java:507)
    at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:738)
    at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:874)
    at java.lang.Thread.run(Thread.java:745)
    Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol message was too large.  May be malicious.  Use CodedInputStream.setSizeLimit() to increase the size limit.
    at com.google.protobuf.InvalidProtocolBufferException.sizeLimitExceeded(InvalidProtocolBufferException.java:110)
    at com.google.protobuf.CodedInputStream.refillBuffer(CodedInputStream.java:755)
    at com.google.protobuf.CodedInputStream.readRawByte(CodedInputStream.java:769)
    at com.google.protobuf.CodedInputStream.readRawVarint64(CodedInputStream.java:462)
    at com.google.protobuf.CodedInputStream.readSInt64(CodedInputStream.java:363)
    at org.apache.hadoop.hdfs.protocol.BlockListAsLongs$BufferDecoder$1.next(BlockListAsLongs.java:326)
    ... 7 more
Neukam answered 30/6, 2015 at 14:3 Comment(4)
Did you "Use CodedInputStream.setSizeLimit() to increase the size limit."? I don't know what this is exactly, but it seems to solve the problem. If you did, let us know what happened. – Schuler
well, this is the hadoop logs, I don't have any code running. This actually occurs even during startup of the hadoop cluster. – Neukam
My hadoop 2.7.0 cluster was not starting. I had to recompile protobuf-2.5.0, changing com.google.protobuf.CodedInputStream#DEFAULT_SIZE_LIMIT to 64 << 24. Then I modified hdfs-site.xml to include <property> <name>ipc.maximum.data.length</name> <value>134217728</value> </property> and now it seems back up. I am not sure if I could avoid recompiling protobuf, is there a hadoop option for that? – Neukam
This is being discussed at issues.apache.org/jira/browse/HDFS-7482 – Oyler
S
1

The answer to this question was already provided in the comments:

My hadoop 2.7.0 cluster was not starting. I had to recompile protobuf-2.5.0, changing com.google.protobuf.CodedInputStream#DEFAULT_SIZE_LIMIT to 64 << 24. Then I modified hdfs-site.xml to include ipc.maximum.data.length 134217728 and now it seems back up.

Shammy answered 8/8, 2017 at 13:48 Comment(1)
Thank you. I set DEFAULT_SIZE_LIMIT to Integer.MAX_VALUE and it works now. You can download patched protobuf-java-2.5.0.jar here: mega.nz/#!SddX0BCC!hbmB9lypAk0Qok2GtYEzXN814m7TzKv5kvl4iBrV814 – Qualified
J
0

Please try the below 3 steps, it should work. Worked for me like a champ..

  • Please change the DEFAULT_SIZE LIMIT as below in the class CodedInputStream of protobuf-java-2.5.0.jar
    private static final int DEFAULT_SIZE_LIMIT = 64 << 20;  // 64MB

to

    private static final int DEFAULT_SIZE_LIMIT = 64 << 21;  // 128MB
  • Replace the newly built jar in the below paths $HADOOP_HOME
$HADOOP_HOME/share/hadoop/common/lib/protobuf-java-2.5.0.jar
$HADOOP_HOME/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar
$HADOOP_HOME/share/hadoop/httpfs/tomcat/webapps/webhdfs/WEB-INF/lib/protobuf-java-2.5.0.jar
$HADOOP_HOME/share/hadoop/kms/tomcat/webapps/kms/WEB-INF/lib/protobuf-java-2.5.0.jar
$HADOOP_HOME/share/hadoop/mapreduce/lib/protobuf-java-2.5.0.jar
$HADOOP_HOME/share/hadoop/tools/lib/protobuf-java-2.5.0.jar
$HADOOP_HOME/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar
  • Restart hadoop daemons ./start-all.sh
Joyajoyan answered 7/5, 2019 at 6:57 Comment(0)

© 2022 - 2024 — McMap. All rights reserved.