overwrite hdfs directory Sqoop import

Question

Is it possible to overwrite HDFS directory automatically instead of overwriting it every time manually while Sqoop import? (Do we have any option like "--overwrite" like we have for hive import "--hive-overwrite")

Halfcock

Answer 1

Use --delete-target-dir

​It will delete <HDFS-target-dir> provided in command before writing data to this directory.

Amora

Hello, How we can deal this with sqoop with hcatalog integration. I know that we have a way to truncate the table before sqoop import. – Aerialist

Answer 2

Use this: --delete-target-dir

This will work for overwriting the hdfs directory using sqoop syntax:

$ sqoop import --connect jdbc:mysql://localhost/dbname --username username -P --table tablename --delete-target-dir --target-dir '/targetdirectorypath' -m 1

E.g:

$ sqoop import --connect jdbc:mysql://localhost/abc --username root -P --table empsqooptargetdel --delete-target-dir --target-dir '/tmp/sqooptargetdirdelete' -m 1

This command will refresh the corresponding hdfs directory or hive table data with updated/fresh data, every time this command is run.

Script