Behavior of the parameter "mapred.min.split.size" in HDFS

Question

The parameter "mapred.min.split.size" changes the size of the block in which the file was written earlier? Assuming a situation where I, when starting my JOB, pass the parameter "mapred.min.split.size" with a value of 134217728 (128MB). What is correct to say about what happens?

1 - Each MAP process the equivalent of 2 HDFS blocks (assuming each block 64MB);

2 - There will be a new division of my input file (previously included HDFS) to occupy blocks in HDFS 128M;

Cinder

Answer 1

The split size is calculated by the formula:-

max(mapred.min.split.size, min(mapred.max.split.size, dfs.block.size))

In your case it will be:-

split size=max(128,min(Long.MAX_VALUE(default),64))

So above inference:-

1. each map will process 2 hdfs blocks(assuming each block 64MB): True

2. There will be a new division of my input file (previously included HDFS) to occupy blocks in HDFS 128M: False

but making the minimum split size greater than the block size increases the split size, but at the cost of locality.

Koral

Note that the values are in bytes, not megabytes. So for 128MB you should specify mapreduce.input.fileinputformat.split.minsize=134217728 – Osmund

Answer 2

Assume that the minimum split size is defined 128mb and the minimum block size is defined 64mb.

NOTE: As each block will be replicated to 3 different datanodes by HDFS by default. Also each map task performs its operation on single block.

Hence, the 128mb split size will consider 2 blocks as a single block and create a single map task for it that will run on a single datanode. This happens at the cost of data-locality. By "cost of data-locality" I am talking about the block that is residing on the datanode on which the map task is not running. Which has to be fetched from that datanode and processed on the datanode on which the map task is running, resulting in lower performance.

However if we consider a file of size 128mb, with default block size of 64mb and a default minimum split size of 64mb, then in that case as normally happens two map tasks will be created for each 64mb of block.

Rikkiriksdag

Answer 3

I have tried with the formula provided where I have block size as 128 MB but results are not coming as expected. I came up with the below formula but it does not make any sense but it is giving me right result every time.

min(mapred.min.split.size, max(mapred.max.split.size, dfs.block.size))

Killing