How does HDFS with append works

Question

Let's assume one is using default block size (128 MB), and there is a file using 130 MB ; so using one full size block and one block with 2 MB. Then 20 MB needs to be appended to the file (total should be now of 150 MB). What happens?

Does HDFS actually resize the size of the last block from 2MB to 22MB? Or create a new block?

How does appending to a file in HDFS deal with conccurency? Is there risk of dataloss ?

Does HDFS create a third block put the 20+2 MB in it, and delete the block with 2MB. If yes, how does this work concurrently?

Caterer

Answer 1

According to the latest design document in the Jira issue mentioned before, we find the following answers to your question:

1. HDFS will append to the last block, not create a new block and copy the data from the old last block. This is not difficult because HDFS just uses a normal filesystem to write these block-files as normal files. Normal file systems have mechanisms for appending new data. Of course, if you fill up the last block, you will create a new block.
2. Only one single write or append to any file is allowed at the same time in HDFS, so there is no concurrency to handle. This is managed by the namenode. You need to close a file if you want someone else to begin writing to it.
3. If the last block in a file is not replicated, the append will fail. The append is written to a single replica, who pipelines it to the replicas, similar to a normal write. It seems to me like there is no extra risk of dataloss as compared to a normal write.

Manard

Can you append to a closed file ? – Caterer

if you call append on a file you are opening it. you can't call append on an open file. then once you call append and get an output stream you can start dumping your bytes onto the end of the file. – Manard

If I remember correctly when this feature was introduced, you would need to leave a newly created file opened to be able to "append" to it (aka not a real append). Are you saying that now HDFS allows to 1) create a file 2) Close it 3) Re-open it 4) Append data to it ? – Caterer

yup. because when you reopen it you ask the name node what the last block is, and then you can start writing to it. – Manard

Answer 2

Here is a very comprehensive design document about append and it contains concurrency issues.

Current HDFS docs gives a link to that document, so we can assume that it is the recent one. (Document date is 2009)

And the related issue.

Wertz

Answer 3

Hadoop Distributed File System supports appends to files, and in this case it should add the 20 MB to the 2nd block in your example (the one with 2 MB in it initially). That way you will end up with two blocks, one with 128 MB and one with 22 MB.

This is the reference to the append java docs for HDFS.

Ataractic

I understand that the path remains the same. But since blocks are write-once, I would imagine HDFS would create a third block put the 20+2 MB in it, and delete the block with 2MB. But how does this work concurrently? – Caterer