Life of distributed cache in Hadoop

Question

When files are transferred to nodes using the distributed cache mechanism in a Hadoop streaming job, does the system delete these files after a job is completed? If they are deleted, which i presume they are, is there a way to make the cache remain for multiple jobs? Does this work the same way on Amazon's Elastic Mapreduce?

Astrahan

Answer 1

I was digging around in the source code, and it looks like files are deleted by TrackerDistributedCacheManager about once a minute when their reference count drops to zero. The TaskRunner explicitly releases all its files at the end of a task. Maybe you should edit TaskRunner to not do this, and control the cache through more explicit means yourself?

Slaton

That's a huge help. I think there might be other ways to get files loaded onto the nodes which I will explore. Distributed cache was just the method I was familiar with. Thanks for the code ref, that's amazingly helpful. – Astrahan

Answer 2

I cross posted this question at the AWS forum and got a good recommendation to use hadoop fs -get to transfer files in a way that persists across jobs.

Astrahan