"Wrong FS... expected: file:///" when trying to read file from HDFS in Java

Question

I am unable to read a file from HDFS using Java:

String hdfsUrl = "hdfs://<ip>:<port>";
Configuration configuration = new Configuration();
configuration.set("fs.defaultFS", hdfsUrl);
FileSystem fs = FileSystem.get(configuration);
Path filePath = new Path(hdfsUrl + "/projects/harmonizome/data/achilles/attribute_list_entries.txt.gz");
FSDataInputStream fsDataInputStream = fs.open(filePath);

SEVERE: Servlet.service() for servlet [edu.mssm.pharm.maayanlab.Harmonizome.api.DownloadAPI] in context with path [/Harmonizome] threw exception
java.lang.IllegalArgumentException: Wrong FS: hdfs://146.203.54.165:8020/projects/harmonizome/data/achilles/attribute_list_entries.txt.gz, expected: file:///
    at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:310)
    at org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:47)
    at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:357)
    at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245)
    at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:125)
    at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:283)
    at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:356)
    at edu.mssm.pharm.maayanlab.Harmonizome.api.DownloadAPI.readLines(DownloadAPI.java:37)
    at edu.mssm.pharm.maayanlab.Harmonizome.api.DownloadAPI.doGet(DownloadAPI.java:27)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:622)
...

I didn't setup our HDFS, so I don't know what I don't know. Any help is appreciated.

Articulation

Answer 1

Try this:

Configuration configuration = new Configuration();
FileSystem fs = FileSystem.get(new URI(<url:port>), configuration);
Path filePath = new Path(<path/to/file>);
FSDataInputStream fsDataInputStream = fs.open(filePath);
BufferedReader br = new BufferedReader(new InputStreamReader(fsDataInputStream));

Please refer to http://techidiocy.com/java-lang-illegalargumentexception-wrong-fs-expected-file/

A similar problem is addressed.

Edra

While this link may answer the question, it is better to include the essential parts of the answer here and provide the link for reference. Link-only answers can become invalid if the linked page changes. – Hovercraft

I agree with @scoa. I was able to resolve my problem using this link and have edited your answer with the working code. – Articulation

@Hovercraft thanks for suggestion. I will follow this in future. – Edra

I don't get it. What was the root cause? – Varro

Now the link is invalid & I can't know the root cause anymore 😅 – Misbecome

Archived version: web.archive.org/web/20150805043209/http://techidiocy.com/… – Annals

From link, for posterity: "Your Configuration object is not initialized properly because of that it is trying to find this file in your local file system where it doesn’t exists because file [you are accessing] exists on HDFS." – Annals

Answer 2

Also below Configuration works!

Configuration conf = new Configuration();
conf.addResource(new Path("/usr/hdp/2.6.3.0-235/hadoop/etc/hadoop/core-site.xml"));
FileSystem fs = FileSystem.get(conf);

Derive

Answer 3

I fixed this issue by changing the order of the class path. MapR uses their own libraries and somehow it was picking up the classes from another library.

Shiller