R: Emulating a complex form with httr

Question

I am trying to get the results of that form with httr.

Having looked the form results, I tried the following:

library(httr)
library(stringr)

r = str_c("http://www.memoiredeshommes.sga.defense.gouv.fr/fr/arkotheque/",
          "client/mdh/base_morts_pour_la_france_premiere_guerre/index.php")

q = list(
  "action" = 1,
  "todo" = "rechercher",
  "le_id"  = "",
  "multisite" = "",
  "r_c_nom" = "mo",
  "r_c_nom_like" = 1,
  "r_c_prenom" = "",
  "r_c_prenom_like" = 1,
  "r_c_naissance_jour_mois_annee_jj_debut" = "",
  "r_c_naissance_jour_mois_annee_mm_debut" = "",
  "r_c_naissance_jour_mois_annee_yyyy_debut" = 1890,
  "r_c_naissance_jour_mois_annee_jj_fin" = "",
  "r_c_naissance_jour_mois_annee_mm_fin" = "",
  "r_c_naissance_jour_mois_annee_yyyy_fin" = "",
  "r_c_id_naissance_departement" = "",
  "hidden_c_id_naissance_departement" = "",
  "r_c_id_naissance_pays" = "",
  "hidden_c_id_naissance_pays" = "",
  "r_annot_c_id_grade" = "",
  "hidden_c_id_grade" = "",
  "r_annot_c_id_unite" = "",
  "hidden_c_id_unite" = "",
  "r_annot_c_id_recrutement_bureau" = "",
  "hidden_c_id_recrutement_bureau" = "",
  "r_annot_c_classe" = "",
  "r_annot_c_recrutement_matricule" = "",
  "r_annot_c_id_naissance_lieu" = "",
  "hidden_c_id_naissance_lieu" = "",
  "r_annot_c_deces_jour_mois_annee_jj_debut" = "",
  "r_annot_c_deces_jour_mois_annee_mm_debut" = "",
  "r_annot_c_deces_jour_mois_annee_yyyy_debut" = "",
  "r_annot_c_deces_jour_mois_annee_jj_fin" = "",
  "r_annot_c_deces_jour_mois_annee_mm_fin" = "",
  "r_annot_c_deces_jour_mois_annee_yyyy_fin" = "",
  "r_annot_c_id_deces_lieu" = "",
  "hidden_c_id_deces_lieu" = "",
  "r_annot_c_deces_lieu_complement" = "",
  "r_annot_c_deces_lieu_complement_like" = 1,
  "r_annot_c_id_deces_departement" = "",
  "hidden_c_id_deces_departement" = "",
  "r_annot_c_id_deces_pays" = "",
  "hidden_c_id_deces_pays" = "",
  "r_annot_c_id_transcription_etablissement_lieu" = "",
  "hidden_c_id_transcription_etablissement_lieu" = "",
  "r_annot_c_id_transcription_etablissement_departement" = "",
  "hidden_c_id_transcription_etablissement_departement" = "",
  "r_annot_c_id_transcription_etablissement_pays" = "",
  "hidden_c_id_transcription_etablissement_pays" = ""
)

t = GET(r, query = q, verbose())
writeLines(content(t, "text", encoding = "UTF-8"), "~/Desktop/test.html")

… which is not working at all (all I get is NA).

What am I doing wrong?

Crosslet

Answer 1

You could try it like this

library(rvest)
html_session(url) %>%
  rvest:::request_POST(url, body = q, encode = "form") %>%
  read_html  %>%
  html_table 
# [[1]]
#             Nom                     Prénom(s) Date de naissance                   Département/Pays de naissance Détail     Images Panier Lien Fiche annotée
# 1          MOAL                    Alain Marc        10-08-1890                                  29 - Finistère Détail Visualiser Panier  Ark           oui
# 2          MOAL                          Jean        22-12-1890                                  29 - Finistère Détail Visualiser Panier  Ark           oui
# 3          MOAL                  Joseph Marie        29-04-1890                                  29 - Finistère Détail Visualiser Panier  Ark           oui
# 4        MOALIC           Pierre Joseph Marie        05-04-1890                                  29 - Finistère Détail Visualiser Panier  Ark           oui
# ...

Elwina

Brilliant trick, thank you so much! May I ask (1) how did you come across the request_POST function in rvest, and (2) if there's a way to do the same thing with httr? I'm going to try to answer (2) by looking at the code of request_POST. Thanks again. – Crosslet

Okay, forget (2), the answer is obvious from rvest's code. – Crosslet

You're welcome. (1) Cheers to Floo0 - this one helped me once, too. :) – Elwina

@Elwina : How should this be handled today while rvest:::request_POST is deprecated? Thanks – Krutz

@Krutz maybe rvest::html_form_submit – Equerry