I was wondering if someone could help me understand how to use Hive's regexp_replace function to capture groups in the regex and use those groups in the replacement string. I have an example probl...

I know both is performed on a column in the table but how is each operation different.

I've been getting the following error in several cases: 2017-03-23 11:55:10,794 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report...

I have the following problem in Azure Databricks. Sometimes when I try to save a DataFrame as a managed table: SomeData_df.write.mode('overwrite').saveAsTable("SomeData") I get the following err...

After installing jdk9 I have been seeing this problem: $hive Java HotSpot(TM) 64-Bit Server VM warning: Ignoring option MaxPermSize; support was removed in 8.0 SLF4J: Class path contains multiple ...

I'm trying to fetch last modified timestamp of a table in Hive.

Trying set Hive use Spark as engine. After long configurations got, Timeout error. Application log file yarn logs -applicationId <app_id> ERROR yarn.ApplicationMaster: User class threw except...

I'm trying to interact with Iceberg tables stored on S3 via a deployed hive metadata store service. The purpose is to be able to push-pull large amounts of data stored as an Iceberg datalake (on S3...

Is there any reference as to what sets of versions are compatible between aws java sdk, hadoop, hadoop-aws bundle, hive, spark? For example, I know Spark is not compatible with hive versions above...

I'm trying to get a table located in hive (hortonworks) ,to collect some twitter data to implement on a machine learning project, using pyhive since pyhs2 is not supported by python3.6. Here's my ...

What is the difference between an INNER JOIN and LEFT SEMI JOIN? In the scenario below, why am I getting two different results? The INNER JOIN result set is a lot larger. Can someone explain? I...

Every day I have a hive job that compute some aggregations for each quarter of hour for two months of data. It's resulting in submitting something like 5760 jobs to Tez. The job failed with the fo...

I have Hadoop 2.7.1 and apache-hive-1.2.1 versions installed on ubuntu 14.0. Why this error is occurring ? Is any metastore installation required? When we typing hive command on terminal how the...

I have just extracted and setup spark 1.6.0 into environment that has a fresh install of hadoop 2.6.0 and hive 0.14. I have verified that hive, beeline and mapreduce works fine on examples. Howev...

I'm trying to create to following view: CREATE OR REPLACE VIEW view_events AS ( SELECT "rank"() OVER (PARTITION BY "tb1"."innerid" ORDER BY "tb1"."date" ASC) "r" , "tb2"."opcode" , "tb1"."inn...

I installed Hadoop, Hive, HBase, Sqoop and added them to the PATH. When I try to execute sqoop command, I'm getting this error: Error: Could not find or load main class org.apache.sqoop.Sqoop D...

I have configured my Hive as given on link: http://www.youtube.com/watch?v=Dqo1ahdBK_A, but I am getting the following error while creating a table in Hive. I am using hadoop-1.2.1 and hive-0.12.0....

I am working on a test in which I must find out the number of partitions of a table and check if it is right. If I use show partitions TableName I get all the partitions by name, but I wish to get ...

I am using hive 1.1 hive> select country from releases limit 1; OK ["us","ca","fr"] For now country is of type string in hive . How do I convert that into Array[String]? I tried the below...

I'm looking for a workaround or hive date functions that gives day of the week , Sunday - 1 Monday - 2 Tuesday - 3 Wednesday - 4 Thursday - 5 Friday - 6 Saturday - 7 Requirement in detail : I'm ...

From hive -h : --hiveconf <property=value> Use value for given property --hivevar <key=value> Variable subsitution to apply to hive commands. e.g. --hivevar A=B

I'm using Java-Spark (Spark 2.2.0). I'm trying to drop Hive partitions as follow: spark.sql(""ALTER TABLE backup DROP PARTITION (date < '20180910')" And got the following exception: org.a...

There are at least two different ways of creating a hive table backed with Avro data: Creating a table based on an Avro schema (in this example, stored in hdfs): CREATE TABLE users_from_avro_schem...

i am bit new to hadoop. As per my knowledge buckets are fixed no. of partitions in hive table and hive uses the no. of reducers same as the total no. of buckets defined while creating the table. So...

I am trying to read a kafka stream and save it to Hive as a table. The consumer code is : import org.apache.spark.sql.{DataFrame, Dataset, SaveMode, SparkSession} import org.apache.spark.sql.functi...