Typical start of working with HDFS using org.apache.hadoop.fs classes on scala is val conf = new Configuration() fs = FileSystem.get(conf) Do we need to call something like IOUtils.closeStream(...

I am a newbie in Spark SQL world. I am currently migrating my application's Ingestion code which includes ingesting data in stage,Raw and Application layer in HDFS and doing CDC(change data capture...

In Hadoop fs how to lookup the block size for a particular file? I was primarily interested in a command line, something like: hadoop fs ... hdfs://fs1.data/... But it looks like that does not ...

I have a shell script like below. This script prints the path of a file located in HDFS #!/bin/bash TIMESTAMP=`date "+%Y-%m-%d"` path=/user/$USER/logs/${TIMESTAMP}.fail_log path1=/user/$USER/logs...

User rok uploaded file and set the permission to 770. The file on HDFS looks like this: -rw-rw---- 3 rok hdfs filename1 I'm using ksc user to consume the data uploaded by rok user. So first, I'd...

I want to read file paths irrespective of whether they are HDFS or local. Currently, I pass the local paths with the prefix file:// and HDFS paths with the prefix hdfs:// and write some code as the...

Problem - I am running 1 query in AWS EMR. It is failing by throwing exception - java.io.FileNotFoundException: File s3://xxx/yyy/internal_test_automation/2016/09/17/17156/data/feed/commerce_feed...

I just started reading about Hadoop and came across the CAP Theorem. Can you please throw some light on which two components of CAP would be applicable to a HDFS system?

I am trying to understand how spark runs on YARN cluster/client. I have the following question in my mind. Is it necessary that spark is installed on all the nodes in yarn cluster? I think it sho...

What is the reason for having column families? Example: Scenario 1 : Table Row-Key ColumnFamily1 ColumnFamily2 ColumnFamily3 Scenario 2 : Table1 Row-Key Column1...ColumnN Table2 Row-Key Column1......

I am installing HDFS on my local Windows machine. The installayion guide I am following is https://github.com/MuhammadBilalYar/Hadoop-On-Window/wiki/Step-by-step-Hadoop-2.8.0-installation-on-Window...

I get from time to time the following errors in cloudera manager: This DataNode is not connected to one or more of its NameNode(s). and The Cloudera Manager agent got an unexpected response fr...

Is the following code for Mappers, reading a text file from HDFS right? And if it is: What happens if two mappers in different nodes try to open the file at almost the same time? Isn't there a ne...

When I setup the hadoop cluster, I read the namenode runs on 50070 and I set up accordingly and it's running fine. But in some books I have come across name node address : hdfs://localhost...

I just downloaded Hortonworks sandbox VM, inside it there are Hadoop with the version 2.7.1. I adding some files by using the hadoop fs -put /hw1/* /hw1 ...command. After it I am deleting the ad...

How to find Hadoop HDFS directory on my system? I need this to run following command - hadoop dfs -copyFromLocal <local-dir> <hdfs-dir> In this command I don't knon my hdfs-dir. No...

I have created one oozie workflow for hive script to load data in a table. My workflow.xml contains - <workflow-app xmlns="uri:oozie:workflow:0.4" name="Hive-Table-Insertion"> <start to...

I get the error Cannot create directory /home/hadoop/hadoopinfra/hdfs/namenode/current While trying to install hadoop on my local Mac. What could be the reason for this? Just for reference, I'...

I have 3 DataNodes and 1 NameNode on a machine inside LXC containers. The DataNode on the same node as the NameNode works fine but the other 2 don't i get: Initialization failed for Block pool BP...

I'm trying to connect to HDFS through Pyarrow, but it does not work because libhdfs library cannot be loaded. libhdfs.so is in $HADOOP_HOME/lib/native as well as in $ARROW_LIBHDFS_DIR. print(os.e...

This query returns in 10 seconds most of the times, but occasionally it need 40 seconds or more. There are two executer nodes in the swarm, and there is no remarkable difference between profiles of...

I am facing a problem with hive default partition (null partition) in hive. I will explain the situation briefly here.. I have a hive main table and data ingestion is happening to that table everyd...

I find that my Impala swarm performs not stable, normally it takes only a few seconds (less than 10s) to finish a query, but occasionally it will take more than 40s (and this situation will last fo...

Is there a simple command for hadoop that can change the name of a file (in the HDFS) from its old name to a new name?

I am trying to read parquet file and perform some operations on it and save the result as parquet on HDFS. I am doing it using Spark. While doing so I am getting following exception. java.io.EOFEx...