Hive installation issues: Hive metastore database is not initialized

Question

I tried to install hive on a raspberry pi 2. I installed Hive by uncompress zipped Hive package and configure $HADOOP_HOME and $HIVE_HOME manually under hduser user-group I created. When running hive, I got the following error message: hive

ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console.

Exception in thread "main" java.lang.RuntimeException: Hive metastore database is not initialized. Please use schematool (e.g. ./schematool -initSchema -dbType ...) to create the schema. If needed, don't forget to include the option to auto-create the underlying database in your JDBC connection string (e.g. ?createDatabaseIfNotExist=true for mysql)

So I ran the command suggested in the above error message: schematool -dbType derby -initSchema I got the error message:

Error: FUNCTION 'NUCLEUS_ASCII' already exists. (state=X0Y68,code=30000) org.apache.hadoop.hive.metastore.HiveMetaException: Schema initialization FAILED! Metastore state would be inconsistent !! * schemaTool failed *

It seems there aren't any helpful information when I try to google this error online. Any help or any explanation on how Hive works with Derby would be appreciated!

Valedictory

Answer 1

After installing hive, if the first thing you did was run hive, hive attempted to create/initialize the metastore_db, but apparently might not get it right. On that initial run, maybe you saw your error:

Exception in thread "main" java.lang.RuntimeException: Hive metastore database is not initialized. Please use schematool (e.g. ./schematool -initSchema -dbType ...) to create the schema. If needed, don't forget to include the option to auto-create the underlying database in your JDBC connection string (e.g. ?createDatabaseIfNotExist=true for mysql)

Running hive, even though it fails, creates a metastore_db directory in the directory from which you ran hive:

ubuntu15-laptop: ~ $>ls -l |grep meta
drwxrwxr-x 5 testuser testuser 4096 Apr 14 12:44 metastore_db

So when you then tried running

ubuntu15-laptop: ~ $>schematool -initSchema -dbType derby

The metastore already existed, but not in complete form.

Soooooo the answer is:

1. Before you run hive for the first time, run

   schematool -initSchema -dbType derby

2. If you already ran hive and then tried to initSchema and it's failing:

   mv metastore_db metastore_db.tmp

3. Re run

   schematool -initSchema -dbType derby

4. Run hive again

**Also of note: if you change directories, the metastore_db created above won't be found! I'm sure there's a good reason for this that I don't know yet because I'm literally trying to use hive for the first time today. Ahhh here's information on this: metastore_db created wherever I run Hive

Lactiferous

Note that you need to restart derby after deleting or moving metastore_db folder before rerunning schema tool. Otherwise, you will get the same error on rerun. – Snooty

Nice explanation. (Not just press that switch and magic will happen) – Spiky

O MY GOD, why is this not mentioned in official docs. Thank you! – Banking

@Rebecca were you able to find out how to set the metastore permanently so that we don't create new metastore_db folder everytime invoking from a new directory the hive script. – Oxonian

For those looking for answer to fix the metastore_db location, look at this answer https://mcmap.net/q/373893/-metastore_db-created-wherever-i-run-hive – Oxonian

Note that this problem happens with other databases as well. If you are using mysql, do: 'drop database metastore; create database metastore' instead of removing a Derby file. – Shopper

the "also of note" save my life: If I run hive in different directory, it sometimes works well(After I init schema), but another times, it does not work, because it will auto generate a metastore_db which is not properly configured. – Windbroken

At last I found the reason why inconsistency for run hive in different directory, I should use an absolute path: /an/absolute/path/metastore_db for derby instead of relative path: metastore_db – Windbroken

THANKYOU SIR Cole! – Dambro

Answer 2

I installed hive with HomeBrew(MacOS) at /usr/local/Cellar/hive and afer running schematool -dbType derby -initSchema I get the following error message:

Starting metastore schema initialization to 2.0.0 Initialization script hive-schema-2.0.0.derby.sql Error: FUNCTION 'NUCLEUS_ASCII' already exists. (state=X0Y68,code=30000) org.apache.hadoop.hive.metastore.HiveMetaException: Schema initialization FAILED! Metastore state would be inconsistent !!

However, I can't find either metastore_db or metastore_db.tmp folder under install path, so I tried:

1. find /usr/ -name hive-schema-2.0.0.derby.sql
2. vi /usr/local/Cellar/hive/2.0.1/libexec/scripts/metastore/upgrade/derby/hive-schema-2.0.0.derby.sql
3. comment the 'NUCLEUS_ASCII' function and 'NUCLEUS_MATCHES' function
4. rerun schematool -dbType derby -initSchema, then everything goes well!

hope this help someone.

Nutshell

this is a great answer. How about using it to my question over here: #43948430 – Hereunto

Answer 3

I encountered a similar issue, which I fixed by removing the derby database files rm -Rf $HIVE_HOME/metastore_db

Be aware, this will remove your schema completely!

After cleaning the old schema I could reinitialize with:

$HIVE_HOME/bin/schematool -initSchema -dbType derby

It might differ from where you call hive, try to go to your hive installation directory and run ./bin/hive

Aiaia

have you any other suggestions I tried this and get the same schema tool failed error as above – Executor

Answer 4

Try this one: schematool -dbType mysql -initSchema

Read the documentation of Hive SchemTool: https://cwiki.apache.org/confluence/display/Hive/Hive+Schema+Tool

Fader

Answer 5

If you use Derby, you can remove DB files and run the init process.

Or if problem persists, you can edit the SQL files Hive use. For example:

/usr/hdp/2.5.6.0-40/hive2/scripts/metastore/upgrade/mysql/ contains those for mysql.

Stoic

Answer 6

I faced a similar error saying Schema initialization FAILED! Metastore state would be inconsistent !! though my problem was different. I had HMS and Postgres on Docker and after restarting HMS tried to initialize the Metastore while there was already a Metastore in place. The solution was adding --env IS_RESUME="true" as mentioned here.

Version