Error connecting to azure blob storage API from R

Question

I am attempting to work with Azure storage via the REST API in R. I'm using the package httr which overlays Curl.

Setup

You can use R-fiddle: http://www.r-fiddle.org/#/fiddle?id=vh8uqGmM

library(httr)
requestdate<-format(Sys.time(),"%a, %d %b %Y %H:%M:%S GMT")
url<-"https://preconstuff.blob.core.windows.net/pings?restype=container&comp=list"
sak<-"Q8HvUVJLBJK+wkrIEG6LlsfFo19iDjneTwJxX/KXSnUCtTjgyyhYnH/5azeqa1bluGD94EcPcSRyBy2W2A/fHQ=="
signaturestring<-paste0("GET",paste(rep("\n",12),collapse=""),
"x-ms-date:",requestdate,"
x-ms-version:2009-09-19
/preconstuff/pings
comp:list
restype:container")

headerstuff<-add_headers(Authorization=paste0("SharedKey preconstuff:",
                         RCurl::base64(digest::hmac(key=sak,
                         object=enc2utf8(signaturestring),
                         algo= "sha256"))),
                    `x-ms-date`=requestdate,
                    `x-ms-version`= "2009-09-19")

Trying to list blobs:

content(GET(url,config = headerstuff, verbose() ))

Error

Top level message

The MAC signature found in the HTTP request 'Q8HvUVJLBJK+wkrIEG6LlsfFo19iDjneTwJxX/KXSnUCtTjgyyhYnH/5azeqa1bluGD94EcPcSRyBy2W2A/fHQ==' is not the same as any computed signature.

Response content

[1] "<?xml version=\"1.0\" encoding=\"utf-8\"?><Error>
<Code>AuthenticationFailed</Code><Message>Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.\nRequestId:1ab26da5-0001-00dc-6ddb-15e35c000000\nTime:2015-03-26T17:51:42.7190620Z</Message>
<AuthenticationErrorDetail>The MAC signature found in the HTTP request 'NTM1ODZjMjhhZmMyZGM3NDM0YTFjZDgwNGE0ODVmMzVjNDhkNjBkNzk1ZjNkZjJjOTNlNjUxYTMwMjRhNzNlYw==' is not the same as any computed signature. Server used following string to sign: 
'GET\n\n\n\n\n\n\n\n\n\n\n\nx-ms-date:Thu, 26 Mar 2015 17:52:37 GMT\nx-ms-version:2009-09-19\n/preconstuff/pings\ncomp:list\nrestype:container'.
</AuthenticationErrorDetail></Error>"

Verbose output

-> GET /pings?restype=container&comp=list HTTP/1.1
-> User-Agent: curl/7.39.0 Rcurl/1.95.4.5 httr/0.6.1
-> Host: preconstuff.blob.core.windows.net
-> Accept-Encoding: gzip
-> Accept: application/json, text/xml, application/xml, */*
-> Authorization: SharedKey preconstuff:OTRhNTgzYmY3OTY3M2UzNjk3ODdjMzk3OWM3ZmU0OTA4MWU5NTE2OGYyZGU3YzRjNjQ1M2NkNzY0ZTcyZDRhYQ==
-> x-ms-date: Thu, 26 Mar 2015 17:56:27 GMT
-> x-ms-version: 2009-09-19
-> 
<- HTTP/1.1 403 Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
<- Content-Length: 719
<- Content-Type: application/xml
<- Server: Microsoft-HTTPAPI/2.0
<- x-ms-request-id: 3d47770c-0001-0085-2313-6d466f000000
<- Date: Thu, 26 Mar 2015 17:56:27 GMT
<- 

Resolving the error

Googling for this issue doesn't seem to yield a consistent cause, but it's likely due to bad formatting / request structure on my part. To that end I checked:

1. I've verified my key is correct (it's just c&p from portal)
2. I've made sure the date is correctly formatted
3. There was a recent documentDB SO that suggested it could be to be a clock skew issue and I do note that my x-ms-date is a second ahead of the Date in the response. I've tried sending a fixed value that was definitely in the past, but within the 15 mins tolerance. Didn't get a change in message.
4. Added encoding="Base64" in headerstuff further to an MSDN forum question but the same error message was returned
5. Further to @Serdar's answer, I incorporated the construction of a signature string (I've verified that this matches the one provided in the error message so it), then encoding in base64 a hmac-sha256 (using the secondary access key (sak) as the encryption key) version of the UTF8 converted signaturestring as the value to be used in the SharedKey authorisation.
6. Further to @Serdar's comment, the date used in the signature string and for the main request must be the same, so defined once and reused

Is there anything obviously wrong? Are there other things to check? Does the code work for others?

Emmeram

Answer 1

Looks like your problem is with the key. The string of the key you have provided is actually base64 encoded. You need to decode that to the raw vector before you use it to sign the request. For example:

url<-"https://preconstuff.blob.core.windows.net/pings?restype=container&comp=list"
sak<-"Q8HvUVJLBJK+wkrIEG6LlsfFo19iDjneTwJxX/KXSnUCtTjgyyhYnH/5azeqa1bluGD94EcPcSRyBy2W2A/fHQ=="

requestdate<-format(Sys.time(),"%a, %d %b %Y %H:%M:%S %Z", tz="GMT")
signaturestring<-paste0("GET",paste(rep("\n",12),collapse=""),
"x-ms-date:",requestdate,"
x-ms-version:2009-09-19
/preconstuff/pings
comp:list
restype:container")

headerstuff<-add_headers(Authorization=paste0("SharedKey preconstuff:",
                         RCurl::base64(digest::hmac(key=RCurl::base64Decode(sak, mode="raw"),
                         object=enc2utf8(signaturestring),
                         algo= "sha256", raw=TRUE))),
                    `x-ms-date`=requestdate,
                    `x-ms-version`= "2009-09-19")

content(GET(url,config = headerstuff, verbose() ))

There are no more authentication errors this way, though no blobs are listed. Perhaps that's a different issue.

Also, I changed the way the date/time was created to more "safely" change the local time to GMT.

Capitulation

You @Capitulation are a genius! Cheers. No blobs because I was first of all trying to GET before I PUT – Emmeram

It is also a working solution for connecting to DocumentDB collection (with some different headers). tx @Capitulation !! – Mucosa

Answer 2

It looks like you are using the key of your account directly in the Authorization header. To authenticate a request, you must sign the request with the key for the account that is making the request and pass that signature as part of the request. Please see Authentication for the Azure Storage Services for more information on how to construct the Authorization header.

Please also note that the service returns StringToSign in the error response. So, what your code should have done is to apply the following formula to StringToSign="GET\n\n\n\n\n\n\n\n\n\n\n\nx-ms-date:Wed, 25 Mar 2015 22:24:12 GMT\nx-ms-version:2014-02-14\n/preconstuff/pings\ncomp:list\nrestype:container" (without quotes):

Signature=Base64(HMAC-SHA256(AccountKey, UTF8(StringToSign)))

How the service calculates StringToSign is explained in detail in the link shared above.

Cytochemistry

Cheers for quick response, I'm still reading that section of the BOL and not getting it. Looking at the bit about constructing signature strings, I'm very confused (I'm a relative newb with APIs) and there isn't a URL the VERB is going against. Would you be able to make a rough stab at how you would see it working in terms of calls? I can probably run from there and refine your answer for you. – Emmeram

I just added some more info to the answer. I don't know R, so I don't have a sample code, but your VERB is basically GET. Please try comparing the documentation with what the service returned as StringToSign. Then you should sign that string with your account key. – Cytochemistry

I think I'm doing it as per instructions, but still not working :-/ Question updated with latest code with StringToSign incorporated – Emmeram

1) You are calling Sys.time() twice. The date you add to the StringToSign and the header must exactly match, as StringToSign should contain the exact values of those headers. 2) Why are you adding timeout to StringToSign even though you do not have it in your URL? – Cytochemistry

Answer 3

Worked trough the example code by MrFlick above and to get it to work I had to change a few things.

The date string has to be set in an English locale, for example:

lct <- Sys.getlocale("LC_TIME") 
Sys.setlocale("LC_TIME", "us")
requestdate <- format(Sys.time(),"%a, %d %b %Y %H:%M:%S %Z", tz="GMT")
Sys.setlocale("LC_TIME", lct)

The 'signaturestring' should be formated with \n between parameters:

 signaturestring <- paste0("GET", paste(rep("\n", 12), collapse=""),
"x-ms-date:", requestdate, 
"\nx-ms-version:2009-09-19\n/preconstuff/pings\ncomp:list\nrestype:container")

EDIT: Following procedure works for me. Based on Steph Locke example.

library(httr)
library(RCurl)

azureBlobCall <- function(url, verb, key, requestBody=NULL, headers=NULL, ifMatch="", md5="") { 
  urlcomponents <- httr::parse_url(url)
  account <- gsub(".blob.core.windows.net", "", urlcomponents$hostname, fixed = TRUE)
  container <- urlcomponents$path

  # get timestamp in us locale
  lct <- Sys.getlocale("LC_TIME"); Sys.setlocale("LC_TIME", "us")
  `x-ms-date` <- format(Sys.time(),"%a, %d %b %Y %H:%M:%S %Z", tz="GMT")
  Sys.setlocale("LC_TIME", lct)

  # if requestBody exist get content length in bytes and content type
  `Content-Length` <- ""; `Content-Type` <- ""
  if(!is.null(requestBody)) {
    if(class(requestBody) == "form_file") {
      `Content-Length` <- (file.info(requestBody$path))$size
      `Content-Type` <- requestBody$type 
    } else {
      requestBody <- enc2utf8(as.character(requestBody))
      `Content-Length` <- nchar(requestBody, "bytes")
      `Content-Type` <- "text/plain; charset=UTF-8" 
    }
  } 

  # combine timestamp and version headers with any input headers, order and create the CanonicalizedHeaders
  headers <- setNames(c(`x-ms-date`, "2015-04-05",  unlist(headers)), 
                      c("x-ms-date", "x-ms-version", unclass(names(unlist(headers)))))
  headers <- headers[order(names(headers))]
  CanonicalizedHeaders <- paste(names(headers), headers, sep=":", collapse = "\n")

  # create CanonicalizedResource headers and add any queries to it
  if(!is.null(urlcomponents$query)) {
    components <- setNames(unlist(urlcomponents$query), unclass(names(unlist(urlcomponents$query))))
    componentstring <- paste0("\n", paste(names(components[order(names(components))]),
                                          components[order(names(components))], sep=":", collapse = "\n"))
  } else componentstring <- ""
  CanonicalizedResource <- paste0("/",account,"/",container, componentstring)

  # create the authorizationtoken
  signaturestring <- paste0(verb, "\n\n\n", `Content-Length`, "\n", md5, "\n", `Content-Type`, "\n\n\n", 
                            ifMatch, "\n\n\n\n", CanonicalizedHeaders, "\n", CanonicalizedResource)

  requestspecificencodedkey <- RCurl::base64(
    digest::hmac(key=RCurl::base64Decode(key, mode="raw"),
                 object=enc2utf8(signaturestring),
                 algo= "sha256", raw=TRUE)
  )

  authorizationtoken <- paste0("SharedKey ", account, ":", requestspecificencodedkey)

  # make the call
  headers_final <- add_headers(Authorization=authorizationtoken, headers, `Content-Type` = `Content-Type`)
  call <- httr::VERB(verb=verb, url=url, config=headers_final, body=requestBody, verbose())

  print("signaturestring");print(signaturestring); 
  print(headers_final); print(call)
  return(content(call))
} 

## Tests. Replace 'key' and 'accountName' with yours
key <- "YowThr***********RDw=="

# Creates a container named 'test'
azureBlobCall("https://accountName.blob.core.windows.net/test?restype=container", "PUT", key)
# Creates a blob named 'blob' under container 'test' with the content of "Hej världen!"
azureBlobCall("https://accountName.blob.core.windows.net/test/blob", "PUT", key, 
              headers = c("x-ms-blob-type"="BlockBlob"), requestBody = "Hej världen!") #upload_file("blob.txt"))
# List all blob in the container 'test'
azureBlobCall("https://accountName.blob.core.windows.net/test?comp=list&restype=container", "GET", key)
# deletes the blobl named 'blob' 
azureBlobCall("https://accountName.blob.core.windows.net/test/blob", "DELETE", key)
# Creates a blob named 'blob' under container 'test' with and upload the file 'blob.txt'
azureBlobCall("https://accountName.blob.core.windows.net/test/blob", "PUT", key, 
              headers = c("x-ms-blob-type"="BlockBlob"), requestBody = upload_file("blob.txt"))
# deletes the container named 'test' 
azureBlobCall("https://accountName.blob.core.windows.net/test?restype=container", "DELETE", key)

Lancer