R httr POST of JSON returns status 400

Question

I'm working on a few functions to get data from StatBank Denmark, and their API. They have made a console to test JSON calls and I know the basic_request I parse to JSON in the function dst_get_data works, as I have tested it in the console.

I get a "status 400" error and an error message that says that I should "supply object when posting".

The code below should make reproducible example. It is the third function (dst_get_data) where I am stuck.

dst_meta <- function(table, ..., lang = "en"){
  require(jsonlite)
  require(httr)

  dkstat_url <- "http://api.statbank.dk/v1/tableinfo"

  params <- list("lang" = lang,
                 "table" = table,
                 "format" = "JSON")

  meta <- POST(url=dkstat_url, body=params, multipart=FALSE)
  meta <- jsonlite::fromJSON(txt=content(meta, as="text"),simplifyDataFrame=TRUE)
  #meta <- RJSONIO::fromJSON(content=content(meta),asText=TRUE, simplify=TRUE, simplifyWithNames=TRUE)

  return(meta)
}

dst_meta_parse <- function(meta, lang){

  basics_names <- c("id", "text", "description", 
                    "unit", "updated", "footnote")

  basics <- meta[names(meta) %in% basics_names]

  variables <- meta[["variables"]][,c("id", "text")]

  values <- meta[["variables"]][,"values"]
  names(values) <- variables$id

  if(lang == "en"){
    test <- grepl(pattern="Tid", names(values))
    if(sum(test) > 0){
      values$Tid$id <- sub(pattern="Q", replacement="K", x=values$Tid$id)
    }
  }

  ## Create basic_request for the data_post file
  basic_request <- vector(mode="list", length=length(variables$id))
  for(variable in 1:length(variables$id)){

    var_name <- variables$id[variable]

    if(var_name == "Tid"){
      basic_request[[variable]] <- list("code" = var_name,
                                       "values" = as.character(values[[var_name]]$id[length(values[[var_name]]$id)]))
    } else {
      basic_request[[variable]] <- list("code" = var_name,
                                       "values" = as.character(values[[var_name]]$id[1]))
    }
  }
  return(list("basics" = basics, "variables" = variables, "values" = values, "basic_request" = basic_request))
}

dst_get_data <- function(request, table,..., lang = "en", format = "CSV", value_presentation = "Default"){
  require(httr)
  require(jsonlite)

  dst_url <- "http://api.statbank.dk/v1/data"

  final_request <- list("table" = table,
                        "lang" = lang,
                        "format" = format,
                        "valuePresentation" = value_presentation,
                        "variables" = request)

  final_request <- jsonlite::toJSON(x=final_request, .escapeEscapes=TRUE, asIs=TRUE)
  print(validate(final_request))
  data <- POST(url=dst_url, body=final_request, multipart=FALSE)

  return(data)
}

test <- dst_meta(table="folk1")

test2 <- dst_meta_parse(meta = test, lang = "en")

test3 <- dst_get_data(request = test2$basic_request, table = "folk1", format="JSON")
#test3 <- dst_get_data(request = test2$basic_request, table = "folk1", format="JSON")

Bettinabettine

Do you have any plans to make this code public as a package? – Nosy

@Nosy Yes, I have thought about it. But I just started yesterday, so not in its current form. It's my first package, so initially just on github though (I just need to catch up on some of the intro material). – Bettinabettine

Cool. Let me know if you want help putting it together (I'm on Github as leeper). I think there could be interest in such a package. – Nosy

Answer 1

When I change dst_get_data to:

dst_get_data <- function(request, table, ..., lang = "en", 
                         format = "CSV", value_presentation = "Default") {

  require(httr)
  require(jsonlite)

  dst_url <- "http://api.statbank.dk/v1/data"

  final_request <- list("table" = table,
                        "lang" = lang,
                        "format" = format,
                        "valuePresentation" = value_presentation)

  data <- POST(url=dst_url, body=final_request, multipart=FALSE)

  return(data)

}

and, the call to it to:

test3 <- dst_get_data(request = test2$basic_request, 
                      table = "folk1", format="JSONSTAT")

I get:

Response [http://api.statbank.dk/v1/data]
  Status: 200
  Content-type: text/json
{"dataset":{"dimension":{"Tid":{"label":"time","category":{"index":{"2014K2":0},"label":{"2014K2":"2014Q2"}}},"id":["Tid"],"size":[1],"role":{"time":["Tid"]}},"label":"Population at the first day of the quarter by time","source":"Statistics Denmark","updated":"2014-05-17T04:10:00Z","value":[5634437],"status":["a"]}} 

I think it was both the need for JSONSTAT and the fact that POST will do the JSON conversion for you automatically.The reason you need JSONSTAT is due to the fact that the "formats" available when you select data from the popup (in the console) are:

<select id="format" name="format"><option value="PX">PX</option>
<option selected="selected" value="CSV">CSV</option>
<option value="XLSX">XLSX</option>
<option value="HTML">HTML</option>
<option value="JSONSTAT">JSONSTAT</option>
<option value="DSTML">DSTML</option>
<option value="PNG">PNG</option>
<option value="BULK">BULK</option>
<option value="AREMOS">AREMOS</option>
<option value="SDMXCOMPACT">SDMXCOMPACT</option>
<option value="SDMXGENERIC">SDMXGENERIC</option>
</select>

Plain ol' JSON is not one of the options.

Millennium

I wonder why my POST isn't working with JSON?! It's seems to me that @hadley is doing it here https://github.com/hadley/httr/blob/master/vignettes/api-packages.Rmd Thank's for the pointer! – Bettinabettine

I updated the answer with the reason for JSONSTAT. The auto-convert to JSON may be a recent addition (hence the difference from Hadley's examples). – Millennium

EDIT: deleted comment because I didn't take close look at the response. – Bettinabettine