CloudKit Server-to-Server authentication

Question

Apple published a new method to authenticate against CloudKit, server-to-server. https://developer.apple.com/library/content/documentation/DataManagement/Conceptual/CloudKitWebServicesReference/SettingUpWebServices.html#//apple_ref/doc/uid/TP40015240-CH24-SW6

I tried to authenticate against CloudKit and this method. At first I generated the key pair and gave the public key to CloudKit, no problem so far.

I started to build the request header. According to the documentation it should look like this:

X-Apple-CloudKit-Request-KeyID: [keyID]  
X-Apple-CloudKit-Request-ISO8601Date: [date]  
X-Apple-CloudKit-Request-SignatureV1: [signature]

• [keyID], no problem. You can find this in the CloudKit dashboard.
• [Date], I think this should work: 2016-02-06T20:41:00Z
• [signature], here is the problem...

The documentation says:

The signature created in Step 1.

Step 1 says:

Concatenate the following parameters and separate them with colons.
[Current date]:[Request body]:[Web Service URL]

I asked myself "Why do I have to generate the key pair?".
But step 2 says:

Compute the ECDSA signature of this message with your private key.

Maybe they mean to sign the concatenated signature with the private key and put this into the header? Anyway I tried both...

My sample for this (unsigned) signature value looks like:

2016-02-06T20:41:00Z:YTdkNzAwYTllNjI1M2EyZTllNDNiZjVmYjg0MWFhMGRiMTE2MjI1NTYwNTA2YzQyODc4MjUwNTQ0YTE5YTg4Yw==:https://api.apple-cloudkit.com/database/1/[iCloud Container]/development/public/records/lookup  

The request body value is SHA256 hashed and after that base64 encoded. My question is, I should concatenate with a ":" but the url and the date also contains ":". Is it correct? (I also tried to URL-Encode the URL and delete the ":" in the date).
At next I signed this signature string with ECDSA, put it into the header and send it. But I always get 401 "Authentication failed" back. To sign it, I used the ecdsa python module, with following commands:

from ecdsa import SigningKey  
a = SigningKey.from_pem(open("path_to_pem_file").read())  
b = "[date]:[base64(request_body)]:/database/1/iCloud....."  
print a.sign(b).encode('hex')

Maybe the python module doesn't work correctly. But it can generate the right public key from the private key. So I hope the other functions also work.

Has anybody managed to authenticate against CloudKit with the server-to-server method? How does it work correctly?

Edit: Correct python version that works

from ecdsa import SigningKey
import ecdsa, base64, hashlib  

a = SigningKey.from_pem(open("path_to_pem_file").read())  
b = "[date]:[base64(sha256(request_body))]:/database/1/iCloud....."  
signature = a.sign(b, hashfunc=hashlib.sha256, sigencode=ecdsa.util.sigencode_der)  
signature = base64.b64encode(signature)
print signature #include this into the header

Peripteral

the apple documentation is totally confusing, I am basically stuck in the same spot as you. I'm using Ruby, but the results are the same. – Magner

Same here, I'm using bash. I always get 401 back unless I send an incorrect date, then I'll get a 500. – Farcical

I am doing the exact same thing as described in the edit. But I still keep on getting 401. The correct way to do base64(sha256(request_body)) is base64.b64encode(hashlib.sha256(request_body).hexdigest()), right? Also, why does the ISO date time string have a Z at the end? – Creep

@Magner did you solve this with ruby? – Sherlene

@Sherlene no I never did get it working, I ended up just running my own MongoDB server instead. – Magner

If you are just fetching data from CloudKit (e.g. Query, Lookup), it is a million times easier just appending your ckAPIToken at the end of your request url. – Orthodontist

@AmateurProgrammer Did you ever get it working? – Orthodontist

@Orthodontist Yes I did. When creating the signature, the URL to be used should not contain the leading https://api.apple-cloudkit.com. This worked for me. – Creep

@AmateurProgrammer I did exactly the same but still getting 401. Do you mind posting your working code sample? I have been stuck with this forever – Orthodontist

@Orthodontist Unfortunately I don't have access to that code right now. Will probably be able to get access in two weeks. Will post it then. – Creep

@AmateurProgrammer Okay, looking forward to it. Thanks – Orthodontist

Answer 1

The last part of the message

[Current date]:[Request body]:[Web Service URL]

must not include the domain (it must include any query parameters):

2016-02-06T20:41:00Z:YTdkNzAwYTllNjI1M2EyZTllNDNiZjVmYjg0MWFhMGRiMTE2MjI1NTYwNTA2YzQyODc4MjUwNTQ0YTE5YTg4Yw==:/database/1/[iCloud Container]/development/public/records/lookup

With newlines for better readability:

2016-02-06T20:41:00Z
:YTdkNzAwYTllNjI1M2EyZTllNDNiZjVmYjg0MWFhMGRiMTE2MjI1NTYwNTA2YzQyODc4MjUwNTQ0YTE5YTg4Yw==
:/database/1/[iCloud Container]/development/public/records/lookup

The following shows how to compute the header value in pseudocode

The exact API calls depend on the concrete language and crypto library you use.

//1. Date
//Example: 2016-02-07T18:58:24Z
//Pitfall: make sure to not include milliseconds
date = isoDateWithoutMilliseconds() 

//2. Payload
//Example (empty string base64 encoded; GET requests):
//47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU=
//Pitfall: make sure the output is base64 encoded (not hex)
payload = base64encode(sha256(body))  

//3. Path
//Example: /database/1/[containerIdentifier]/development/public/records/lookup
//Pitfall: Don't include the domain; do include any query parameter
path = stripDomainKeepQueryParams(url) 

//4. Message
//Join date, payload, and path with colons
message = date + ':' + payload + ':' + path

//5. Compute a signature for the message using your private key.
//This step looks very different for every language/crypto lib.
//Pitfall: make sure the output is base64 encoded.
//Hint: the key itself contains information about the signature algorithm 
//      (on NodeJS you can use the signature name 'RSA-SHA256' to compute a 
//      the correct ECDSA signature with an ECDSA key).
signature = base64encode(sign(message, key))

//6. Set headers
X-Apple-CloudKit-Request-KeyID = keyID 
X-Apple-CloudKit-Request-ISO8601Date = date  
X-Apple-CloudKit-Request-SignatureV1 = signature

//7. For POST requests, don't forget to actually send the unsigned request body
//   (not just the headers)

Disquisition

I pick this as an answer. Thanks for your help and the pseudo code. The other answers are also helpful. – Peripteral

Answer 2

Extracting Apple's cloudkit.js implementation and using the first call from the Apple sample code node-client-s2s/index.js you can construct the following:

You hash the request body request with sha256:

var crypto = require('crypto');
var bodyHasher = crypto.createHash('sha256');
bodyHasher.update(requestBody);
var hashedBody = bodyHasher.digest("base64");

The sign the [Current date]:[Request body]:[Web Service URL] payload with the private key provided in the config.

var c = crypto.createSign("RSA-SHA256");
c.update(rawPayload);
var requestSignature = c.sign(key, "base64");

Another note is the [Web Service URL] payload component must not include the domain but it does need any query parameters.

Make sure the date value is the same in X-Apple-CloudKit-Request-ISO8601Date as it is in the signature. (These details are not documented completely, but is observed by looking through the CloudKit.js implementation).

A more complete nodejs example looks like this:

(function() {

const https = require('https');
var fs = require('fs');
var crypto = require('crypto');

var key = fs.readFileSync(__dirname + '/eckey.pem', "utf8");
var authKeyID = 'auth-key-id';

// path of our request (domain not included)
var requestPath = "/database/1/iCloud.containerIdentifier/development/public/users/current";

// request body (GET request is blank)
var requestBody = '';

// date string without milliseconds
var requestDate = (new Date).toISOString().replace(/(\.\d\d\d)Z/, "Z");

var bodyHasher = crypto.createHash('sha256');
bodyHasher.update(requestBody);
var hashedBody = bodyHasher.digest("base64");

var rawPayload = requestDate + ":" + hashedBody + ":" + requestPath;

// sign payload
var c = crypto.createSign("sha256");
c.update(rawPayload);
var requestSignature = c.sign(key, "base64");

// put headers together
var headers = {
    'X-Apple-CloudKit-Request-KeyID': authKeyID,
    'X-Apple-CloudKit-Request-ISO8601Date': requestDate,
    'X-Apple-CloudKit-Request-SignatureV1': requestSignature
};

var options = {
    hostname: 'api.apple-cloudkit.com',
    port: 443,
    path: requestPath,
    method: 'GET',
    headers: headers
};

var req = https.request(options, (res) => {
   //... handle nodejs response
});

req.end();

})();

This also exists as a gist: https://gist.github.com/jessedc/a3161186b450317a9cb5

On the command line with openssl (Updated)

The first hashing can be done with this command:

openssl sha -sha256 -binary < body.txt | base64

To sign the second part of the request you need a more modern version of openSSL than what OSX 10.11 comes with and use the following command:

/usr/local/bin/openssl dgst -sha256WithRSAEncryption -binary -sign ck-server-key.pem raw_signature.txt | base64

Thanks to @maurice_vB below and on twitter for this info

Chammy

Are you getting "unable to load key" or similar? That's because you're enforcing the wrong encoding, change -sha256 to -ecdsa-with-SHA1 and you're good to go. Even though it says SHA1, the key will tell openssl to use ecdsa. – Farcical

When you actually sign the payload, use "RSA-SHA256" instead of "sha256", e.g: var c = crypto.createSign("RSA-SHA256"); The reason is that the key does contain information about the algorithm (ECDSA vs RSA) but not about the encoding (SHA1 vs SHA256). – Disquisition

@MaxGunther yes you're right. The Apple implementation uses "RSA-SHA256". Using "sha256" does seem to work though. – Chammy

Answer 3

I made an working code example in PHP: https://gist.github.com/Mauricevb/87c144cec514c5ce73bd (based on @Jessedc's JavaScript example)

By the way, make sure you set the date time in UTC timezone. My code didn't work because of this.

Tiler

It's so weird, I was able to run Apples sample script, but this PHP script fails to load my key. And if I change it to use ecdsa-with-sha1 instead of rsa256, I get the infamous AUTHENTICATION_FAILED. – Farcical

Make sure you update openssl on your computer. Then it will work without problems. I had the same problem on my Mac. You can also try the script on a server instead of your local pc. OSX specific: Mac comes with an outdated openssl version so you always need to update openssl in order to make this work. – Tiler

If I use the latest openssl version (homebrew) it really doesn't work with ecdsa-with-sha1. Your php script works at least. My python code also doesn't work. I am still investigating the issue. Thanks for your script ;) – Peripteral

I'm not sure if here is a good way to put it, but a weird thing is that I've found out that GET requests actually works, but not POST requests. Might be something with how the body is constructed, but I've printed literally evert part of the request in apples node-example, and while it works in javascript, I can't get it to work elsewhere. – Farcical

Have you tried my PHP code? That code sends POST requests that work. – Tiler

Yes, I get wrong public key type every time. And that key works, because it's working in Apples example. – Farcical

Make sure you update openssl on your system. To verify it works, run: echo "signature" | openssl dgst -sha256WithRSAEncryption -sign private.pem -binary | base64. You can also try web server that already has an updated openssl. – Tiler

My version of openssl says OpenSSL 1.0.2f 28 Jan 2016, but your snipped failed. Do I need to compile openssl with some specific options? – Farcical

Worked on the first try -- awesome! – Alvie

Answer 4

Distilled this from a project I'm working on in Node. Maybe you will find it useful. Replace the X-Apple-CloudKit-Request-KeyID and the container identifier in requestOptions.path to make it work.

The private key/ pem is generated with: openssl ecparam -name prime256v1 -genkey -noout -out eckey.pem and generate the public key to register at the CloudKit dashboard openssl ec -in eckey.pem -pubout.

var crypto = require("crypto"),
    https = require("https"),
    fs = require("fs")

var CloudKitRequest = function(payload) {
  this.payload = payload
  this.requestOptions = { // Used with `https.request`
    hostname: "api.apple-cloudkit.com",
    port: 443,
    path: '/database/1/iCloud.com.your.container/development/public/records/modify',
    method: 'POST',
    headers: { // We will add more headers in the sign methods
      "X-Apple-CloudKit-Request-KeyID": "your-ck-request-keyID"
    }
  }
}

To sign the request:

CloudKitRequest.prototype.sign = function(privateKey) {
  var dateString = new Date().toISOString().replace(/\.[0-9]+?Z/, "Z"), // NOTE: No milliseconds
      hash = crypto.createHash("sha256"),
      sign = crypto.createSign("RSA-SHA256")

  // Create the hash of the payload
  hash.update(this.payload, "utf8")
  var payloadSignature = hash.digest("base64")

  // Create the signature string to sign
  var signatureData = [
    dateString,
    payloadSignature,
    this.requestOptions.path
  ].join(":") // [Date]:[Request body]:[Web Service URL]

  // Construct the signature
  sign.update(signatureData)
  var signature = sign.sign(privateKey, "base64")

  // Update the request headers
  this.requestOptions.headers["X-Apple-CloudKit-Request-ISO8601Date"] = dateString
  this.requestOptions.headers["X-Apple-CloudKit-Request-SignatureV1"] = signature

  return signature // This might be useful to keep around
}

And now you can send the request:

CloudKitRequest.prototype.send = function(cb) {
  var request = https.request(this.requestOptions, function(response) {
    var responseBody = ""

    response.on("data", function(chunk) {
      responseBody += chunk.toString("utf8")
    })

    response.on("end", function() {
      cb(null, JSON.parse(responseBody))
    })
  })

  request.on("error", function(err) {
    cb(err, null)
  })

  request.end(this.payload)
}

So given the following:

var privateKey = fs.readFileSync("./eckey.pem"),
    creationPayload = JSON.stringify({
      "operations": [{
          "operationType" : "create",
          "record" : {
            "recordType" : "Post",
            "fields" : {
              "title" : { "value" : "A Post From The Server" }
          }
        }
      }]
    })

Using the request:

var creationRequest = new CloudKitRequest(creationPayload)
creationRequest.sign(privateKey)
creationRequest.send(function(err, response) {
  console.log("Created a new entry with error", err, "and respone", response)
})

For your copy pasting pleasure: https://gist.github.com/spllr/4bf3fadb7f6168f67698 (edited)

Holmic

Why is it that when using a value like this "Théâtre des Variétés" it fails with an authentication error? – Exalted

@DDD found the issue. The hash needs to be explicitly set input encoding to utf8. So using hash.update(this.payload, "utf8") will fix the problem. Will update the examples. Thanks for the catch. – Holmic

Answer 5

In case someone else is trying to do this via Ruby, there's a key method alias required to monkey patch the OpenSSL lib to work:

def signature_for_request(body_json, url, iso8601_date)
  body_sha_hash = Digest::SHA256.digest(body_json)

  payload_for_signature = [iso8601_date, Base64.strict_encode64(body_sha_hash), url].join(":")

  OpenSSL::PKey::EC.send(:alias_method, :private?, :private_key?)

  ec = OpenSSL::PKey::EC.new(CK_PEM_STRING)
  digest = OpenSSL::Digest::SHA256.new
  signature = ec.sign(digest, payload_for_signature)
  base64_signature = Base64.strict_encode64(signature)

  return base64_signature
end

Note that in the above example, url is the path excluding the domain component (starting with /database...) and CK_PEM_STRING is simply a File.read of the pem generated when setting up your private/public key pair.

The iso8601_date is most easily generated using:

Time.now.utc.iso8601

Of course, you want to store that in a variable to include in your final request. Construction of the final request can be done with the following pattern:

def perform_request(url, body, iso8601_date)

  signature = self.signature_for_request(body, url, iso8601_date)

  uri = URI.parse(CK_SERVICE_BASE + url)

  header = {
    "Content-Type" => "text/plain",
    "X-Apple-CloudKit-Request-KeyID" => CK_KEY_ID,
    "X-Apple-CloudKit-Request-ISO8601Date" => iso8601_date,
    "X-Apple-CloudKit-Request-SignatureV1" => signature
  }

  # Create the HTTP objects
  http = Net::HTTP.new(uri.host, uri.port)
  http.use_ssl = true
  request = Net::HTTP::Post.new(uri.request_uri, header)
  request.body = body

  # Send the request
  response = http.request(request)

  return response
end

Works like a charm now for me.

Reube

Thanks for this. Only problem is it's giving an error of "wrong public key type (OpenSSL::PKey::PKeyError)" on the line signature = ec.sign(digest, payload_for_signature). Any ideas? – Marlenmarlena

My suspicion is that you may have generated the wrong type of key, or perhaps the CK_PEM_STRING isn't being read in from the file properly, or was somehow altered if you copied and pasted it from the terminal? I'd suggest reading it in directly from the file. – Reube

Answer 6

I had the same problem and ended up writing a library that works with python-requests to interface with the CloudKit API in Python.

pip install requests-cloudkit

After it's installed, just import the authentication handler (CloudKitAuth) and use it directly with requests. It will transparently authenticate any request you make to the CloudKit API.

>>> import requests
>>> from requests_cloudkit import CloudKitAuth
>>> auth = CloudKitAuth(key_id=YOUR_KEY_ID, key_file_name=YOUR_PRIVATE_KEY_PATH)
>>> requests.get("https://api.apple-cloudkit.com/database/[version]/[container]/[environment]/public/zones/list", auth=auth)

The GitHub project is available at https://github.com/lionheart/requests-cloudkit if you'd like to contribute or report an issue.

Rustle