SQOOP SQLSERVER Failed to load driver " appropriate connection manager is not being set"

Question

I downloaded sqljdbc4.jar. I'm invoking sqoop like so from the folder (where the jar is stored):

sqoop list-tables --driver com.microsoft.jdbc.sqlserver.SQLServerDriver --connect jdbc:sqlserver://localhost:1433;user=me;password=myPassword; -libjars=./sqljdbc4.jar

I'm getting the following warning & error:

13/10/25 18:38:13 WARN sqoop.ConnFactory: Parameter --driver is set to an explicit driver however appropriate connection manager is not being set (via --connection-manager). Sqoop is going to fall back to org.apache.sqoop.manager.GenericJdbcManager. Please specify explicitly which connection manager should be used next time.

13/10/25 18:38:13 INFO manager.SqlManager: Using default fetchSize of 1000
13/10/25 18:38:13 ERROR sqoop.Sqoop: Got exception running Sqoop: java.lang.RuntimeException: Could not load db driver class: com.microsoft.jdbc.sqlserver.SQLServerDriver
java.lang.RuntimeException: Could not load db driver class: com.microsoft.jdbc.sqlserver.SQLServerDriver
    at org.apache.sqoop.manager.SqlManager.makeConnection(SqlManager.java:727)
    at org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:52)
    at org.apache.sqoop.manager.SqlManager.listTables(SqlManager.java:418)
    at org.apache.sqoop.tool.ListTablesTool.run(ListTablesTool.java:49)
    at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
    at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
    at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
    at org.apache.sqoop.Sqoop.main(Sqoop.java:238)

UPDATE

I changed the command line to reflect the comments below, I get the same error:

sqoop list-databases -libjars=<ABSOLUTE_PATH>/jars/sqljdbc4.jar --connect jdbc:sqlserver://localhost:1433;user=me;password=password

13/10/28 17:00:33 ERROR sqoop.Sqoop: Got exception running Sqoop: java.lang.RuntimeException: Could not load db driver class: com.microsoft.sqlserver.jdbc.SQLServerDriver
java.lang.RuntimeException: Could not load db driver class: com.microsoft.sqlserver.jdbc.SQLServerDriver
    at org.apache.sqoop.manager.SqlManager.makeConnection(SqlManager.java:727)
    at org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:52)
    at org.apache.sqoop.manager.CatalogQueryManager.listDatabases(CatalogQueryManager.java:57)
    at org.apache.sqoop.tool.ListDatabasesTool.run(ListDatabasesTool.java:49)
    at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
    at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
    at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
    at org.apache.sqoop.Sqoop.main(Sqoop.java:238)

When I look at the listing of sqljdbc4.jar, I do see the class in that path... Is it possible that libjars option isn't doing what I think it is supposed to do?

Plagiarize

Did you manage to fix this? – Kalmick

Answer 1

In vast majority of cases using parameter --driver is not required and even more will lead to an undesirable behaviour. I would strongly recommend dropping this argument entirely from your command line. Check out Connectors vs Drivers blog post for more details.

Also in addition you are specifying a nonexistent JDBC Driver class. The correct one is:

com.microsoft.sqlserver.jdbc.SQLServerDriver

You can see it in the official docs, whereas you are specifying

com.microsoft.jdbc.sqlserver.SQLServerDriver

Notice the different order of jdbc and sqlserver packages. This is one of the reasons why it's recommended to not use the --driver option at all.

Midweek

Thank you @Jarek Jarcec Cecho - i removed the driver setting, I'm still getting the same issue...I'll update my question to reflect the change. – Plagiarize

changing to this 'com.microsoft.sqlserver.jdbc.SQLServerDriver' solved my problem ...this was driving me nuts..thank you – Cleon

Answer 2

You need to put sqljdbc4.jar in $SQOOP_HOME/lib and also add sqoop-1.4.4.jar or whatever version you are using along with sqljdbc4.jar to $HADOOP_HOME/lib.

I'm using Hadoop-2.2.0, so i put it inside $HADOOP_HOME/share/hadoop/common/lib directory, and use the following command to do the import:

export HCAT_HOME=/home/Kuntal/BIG_DATA/hive-0.12.0/hcatalog

(sometimes HCatlog of Hive needs to be exported or set.)

./sqoop-import --connect "jdbc:sqlserver://IP\INSTANCE;port=1433;username=USERNAME;password=PASSWORD;database=DATABASE_NAME" --table TABLE_NAME --target-dir hdfs://localhost:50315/sqoop --m 1

Sometimes you have to specify the port, otherwise default works. Hope you find it useful.

Scheld

Answer 3

According to this sqoop documentation, generic options like -libjars must come before tool-specific options:

Generic Hadoop command-line arguments:
(must preceed any tool-specific arguments)
...
-libjars <comma separated list of jars> specify comma separated jar files to include in the classpath.

Sure

Thanks @Mark Rotteveel. I tried that but no luck. I'll update my question. – Plagiarize

Answer 4

Recently came across this same problem. Even though documentation says that it will pick up additional jar files. The problem is I believe propagated from Hadoop jar command line option. -libjars is not reliable option to set additional jar file path. Instead, choose HADOOP_CLASSPATH option to setup additional jar files.

In my case, I had multiple different versions of driver JAR, but using -libjars was not correctly picking up file for me.

To resolve this, I specified

export HADOOP_CLASSPATH=/$SQOOP_HOME/<path_to_driver>.jar

This makes sure that correct JAR file gets loaded.

Gerard