When using --negotiate with curl, is a keytab file required?

Question

The documentation describing how to connect to a kerberos secured endpoint shows the following:

curl -i --negotiate -u : "http://<HOST>:<PORT>/webhdfs/v1/<PATH>?op=..."

The -u flag has to be provided but is ignored by curl.

Does the --negotiate option cause curl to look for a keytab that was created beforehand with the kinit command, or will curl prompt for credentials?

If it looks for a keytab file, what filename will the command be looking for?

Dinahdinan

curl tries to get a valid ticket in the cache (cf. env variable KRB5CCNAME or by default FILE:/tmp/krb5cc_$(id -u) on Unix / API: on Windows) -- it does not care how the ticket was created. – Immixture

Caveat: the Windows versions of curl are compiled to support the Microsoft SSPI libraries, not the GSSAPI libraries that are required by Hadoop REST services. – Immixture

BTW a keytab is a file containing a hashed password, created by ktutil on Linux or by an Active Directory utility like ktpass.exe. It is used by kinit, not created. – Immixture

Even better: if you update your question so that it is more specific, I can make a more specific answer (e.g. do you want to SPNEGO each time, or just once to get the reusable Delegation Token? do you want to know why HttpFs uses a signed cookie instead of the actual Token like WebHDFS does? are you worried about the lack of consistency checks using WebHDFS vs. using the Java API? etc.) – Immixture

An extra hint at no additional cost: #37459573 – Immixture

Thanks Samsom. I've just started exploring this area, so I don't have the knowledge to make my question more specific yet. I will post more as my knowledge builds. – Dinahdinan

@SHC Were you able to properly resolve the issue? Otherwise I would add a qualified answer. – Depilatory

Thanks, do you know how can we do the same in a Java http client, ie. Authorize the proxy using a keytab file? – Andryc

Answer 1

Being a once-in-a-while-contributor to curl in that area. Here is what you need to know:

curl(1) itself knows nothing about Kerberos and will not interact neither with your credential cache nor your keytab file. It will delegate all calls to a GSS-API implementation which will do the magic for you. What magic depends on the library, Heimdal and MIT Kerberos.

Based on your question, I assume that you have little knowledge about Kerberos and want simply automate API calls to a REST endpoints secured by SPNEGO.

Here is what you need to do:

1. Have a Unix-like OS
2. Install at least MIT Kerberos 1.11
3. Install at least curl 7.38.0 against MIT Kerberos
4. Verify this with curl --version mentioning GSS-API and SPNEGO and with ldd linked against your MIT Kerberos version.
5. Create a client keytab for the service principal with ktutil or mskutil
6. Try to obtain a TGT with that client keytab by kinit -k -t <path-to-keytab> <principal-from-keytab>
7. Verify with klist that you have a ticket cache

Environment is now ready to go:

8. Export KRB5CCNAME=<some-non-default-path>
9. Export KRB5_CLIENT_KTNAME=<path-to-keytab>
10. Invoke curl --negotiate -u : <URL>

MIT Kerberos will detect that both environment variables are set, inspect them, automatically obtain a TGT with your keytab, request a service ticket and pass to curl. You are done.

Note: this will not work with Heimdal.

Depilatory

Can you detail how exactly in step-3 do we install curl against MIT Kerberos? – Adlar

@Adlar at best, your OS package manager will do this for you. At worst, you need to pass the pass the path of krb5-config to the configure script of curl. That's it. – Depilatory

What is KRB5CCNAME supposed to be set to? – Foreign

@Foreign See web.mit.edu/kerberos/krb5-1.12/doc/basic/ccache_def.html – Depilatory

That doesn't really answer my question. Does it need to point to an existing file or something? What, specifically do I set the KRB5CCNAME environment variable to? – Foreign

@Foreign A late answer, but you set it with path to kerberos cache file, e.g. export KRB5CCNAME=/tmp/krb5cc_1000. After using kinit, the cache is created automatically, you can find it using klist: Ticket cache: FILE:/tmp/krb5cc_1000 – Zygosis

+9000 for this post. This is a Top 5 G.O.A.T. for me. Before I read this post, I never knew that Linux/Cygwin /usr/bin/curl with GSS-API + SPNEGO was using MIT Kerberos lib for auth. Blew my mind. There is no hint from the curl error. Only: gss_init_sec_context() failed: SPNEGO cannot find mechanisms to negotiate. Another good tip: Enable MIT Kerberos debug logs: env KRB5_TRACE=/dev/stdout curl ... – Belligerent

Answer 2

1. Check curl version

   $ curl -V - It should support the feature "GSS-Negotiate"

2. Login using kinit

   $ kinit <user-id>

3. Use curl

   $ curl --negotiate -u : -b ~/cookiejar.txt -c ~/cookiejar.txt http://localhost:14000/webhdfs/v1/?op=liststatus

   "--negotiate" option enables SPNEGO

   "-u" option is required but ignored (the principle specified during kinit is used)

   "-b" & "-c" options are used to store and send http cookies.

Laforge

It should not support GSS-Negotiate because all these version are broken prior 7.38.0. It has to support SPNEGO. – Depilatory

This works fine for me, and is a lot less setup/trouble than the accepted answer. curl 7.35.0, no SPNEGO, only GSS-Negotiate. – Theodora

Are the cookie options necessary for a basic test of connectivity? If yes, why? – Mickiemickle

Answering myself: without cookiejar the http will redirect infinitely even though the Kerberos part was successful. – Mickiemickle

@Depilatory I, too, am using a pre-7.38 version with SPNEGO with no issue. curl 7.29.0 (x86_64-redhat-linux-gnu) libcurl/7.29.0 NSS/3.28.4 zlib/1.2.7 libidn/1.28 libssh2/1.4.3 Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp scp sftp smtp smtps telnet tftp Features: AsynchDNS GSS-Negotiate IDN IPv6 Largefile NTLM NTLM_WB SSL libz unix-sockets. GSS-Negotiate appears to be the ticket. – Folkmoot

I'm having the same problem but none of the answers seem to help. Would you mind having a look at #59899025 maybe you can spot a mistake :-) – Capwell

Answer 3

I am using version 7.54.0. The following command works fine for me. (please remember/pay attention there are double dashes before the negotiate option).

curl --negotiate -u: '<the_full_url>'

Laundromat