It's possible only install Hadoop HDFS?

Question

I'm new on Hadoop world, and I need install mesos with Hadoop HDFS to make a fault-tolerant distributed file system, but all installation references include necessary components for my scenario as for example: MapReduce.

Do you have any idea or references about this?

Momentarily

Answer 1

Absolutely possible. Don't think Hadoop as an installable program, it's just composed by a bunch of java processes running on different nodes inside a cluster.

If you use hadoop tar ball, you can just run NameNode and DataNodes processes if you only want HDFS.

If you use other hadoop distros (HDP for instance), I think HDFS and mapreduce come from different rpm packages, but it does harm to install both rpm packages. Again just run NameNode and DataNodes if you only need HDFS.

Bernardobernarr

ok, thks @zhutoulala...do you know some tutorial to make this? – Momentarily

michael-noll.com/tutorials/… is one of the most popular ones – Bernardobernarr

I have one instance of postgres and I need that their data file it's store on HDFS, it's possible? I'm not sure because, when I was reading your link and other docs, all show that the way of read/write on HDFS it's using hadoop command. The read/write operation with HDFS not it's transparent to the user? – Momentarily

well, this is a different question:) You can also use HDFS api or webhdfs web service to get/put data into HDFS. But if you ask is using HDFS for your database's storage, that's a totally different story – Bernardobernarr

ok, thanks @zhutoulala, yes I was suppose that it's possible the use of HDFS for my database storage, but now, after some time reading docs, I think that this idea it's wrong or not available at this moment. – Momentarily

@kikicarbonell based on this article link it looks as if using FDW (Foreign Data Wrappers) it would be possible to store data in HDFS and access it via postgres. Not sure of the effectiveness / limitations of this method, but looks interesting. – Matteson