Livy REST API: GET requests work but POST requests fail with '401 Authentication required'

Question

I’ve written a Java client for parts of Livy’s REST API at https://github.com/apache/incubator-livy/blob/master/docs/rest-api.md. The client uses Spring’s RestTemplate.getForObject() and postForObject() to make GET and POST requests respectively. The Livy server is secured with Kerberos.

GET /sessions and GET /batches requests work fine: I get the expected responses from Livy. But both POST /sessions and POST /batches requests fail with:

org.springframework.web.client.HttpClientErrorException: 401 Authentication required

Does anyone know why the POST requests fail when the GET requests succeed? My code does nothing explicit with authentication.

I've tried authenticating as several different users via Kerberos but I always get this problem. Does Livy need extra configuration to allow POST requests from particular users (since POST requests effectively create interactive sessions or submit jobs to Spark)?

Coffey

Did you trace the HTTP headers sent/received during the SPNego "negotiation" (that always starts with a 401 challenge but should end either with 200 or 403)? Any visible difference in the sequence of events for GET and POST e.g. a redirection at some point? – Dulosis

There was a similar issue with WebHDFS, long ago, but because of a bug in Java 6 netlibrary... hopefully you are using something more recent! – Dulosis

Thanks for the tips! No, I'm using Java 8 (and Livy v0.3.0 BTW). I haven't managed to trace those HTTP headers but it looks like Livy requires extra security for POST requests compared to GET requests. Currently trying with a special version of Spring's RestTemplate to use with Kerberos: KerberosRestTemplate. Will post more when I get another chance in a few days... – Coffey

For debugging purposes you may try a curl command-line HTTP query (if the Kerberos TGT is available in the default cache) with curl --negotiate -u : --trace-ascii - <some-GET-URL> then -X POST --data 'blah blah' <some-POST-URL> – Dulosis

Answer 1

It turns out that whilst the regular org.springframework.web.client.RestTemplate class is sufficient for GET requests, you need to use org.springframework.security.kerberos.client.KerberosRestTemplate for POST requests. You may also need to add an extra header to POST requests if the Livy server has CSRF (cross-site request forgery) protection enabled as described here.

GET /batches example

RestTemplate restTemplate = new RestTemplate();
GetBatchesResponse response2 = restTemplate.getForObject("http://your_livy_server:8998" + "/batches", GetBatchesResponse.class);

where GetBatchesResponse is a simple POJO I've written that represents the response body to GET /batches.

POST /batches example

PostBatchesRequest postRequestBody = new PostBatchesRequest();
postRequestBody.setFile("/path/to/your/application"); // In HDFS

KerberosRestTemplate kerberosRestTemplate = new KerberosRestTemplate("path_to_your_key_tab_file", "your_user@your_realm");

// Add CSRF header if required:
HttpHeaders headers = new HttpHeaders();
headers.set("X-Requested-By", "your_user@your_realm");
headers.setContentType(MediaType.APPLICATION_JSON);

HttpEntity<PostBatchesRequest> postRequest = new HttpEntity<PostBatchesRequest>(postRequestBody, headers);

Batch batch = kerberosRestTemplate.postForObject("http://your_livy_server:8998" + "/batches", postRequest, Batch.class);

where PostBatchesRequest and Batch are POJOs I've written to represent the request body and response respectively.

Coffey

Credit is due to Hortonworks for helping me arrive at this solution! – Coffey

If possible can you paste the imports and dependencies too? I am having trouble to find them. I am also trying to post a request to Livy. – Maximamaximal

@Maximamaximal - You can't find the POJOs (such as PostBatchesRequest) because I wrote them. They're company code so I can't share them but all I did was write a Java client for Livy's REST API and used org.springframework.web.client.RestTemplate to make GET requests or org.springframework.security.kerberos.client.KerberosRestTemplate to make POST requests.. – Coffey