curl vs python "requests" when hitting APIs

Question

I am trying to hit the Bitbucket API for my account, and a successful attempt looks like:

curl --user screename:mypassword https://api.bitbucket.org/1.0/user/repositories

in the command line. In python, I try:

import requests
url = 'https://api.bitbucket.org/1.0/user/repositories'

then

r = requests.post(url, data={'username': myscreename, 'password':mypassword})

and

r = requests.post(url, data="myscreename:mypassword")

and

r = requests.post(url, data={"user": "myscreename:mypassword"})

all get 405 error. The API is https://confluence.atlassian.com/bitbucket/rest-apis-222724129.html.

I wonder:

1. What am I doing wrong in the requests version, they all look similar to my curl attempt

2. What is the difference between requesting with curl and python requests module? What general pattern can I recognize when reading an API with a curl example and then writing it in python?

Thank you

ANSWER:

it required the right headers

https://answers.atlassian.com/questions/18451025/answers/18451117?flashId=-982194107

UPDATE:

# ===============
# get user
# ===============
import requests
import json

# [BITBUCKET-BASE-URL], i.e.: https://bitbucket.org/
url = '[BITBUCKET-BASE-URL]/api/1.0/user/'
headers = {'Content-Type': 'application/json'}

# get user
# [USERNAME], i.e.: myuser
# [PASSWORD], i.e.: itspassword
r = requests.get(url, auth=('[USERNAME]', '[PASSWORD]'), headers=headers)
print(r.status_code)
print(r.text)
#print(r.content)

Stagestruck

--user in curl uses HTTP auth headers. Your data={"username"... solution includes them as post data. The two are not the same thing, and it's unlikely that Bitbucket looks for the ones in the post data. – Predict

Answer 1

Here's a way to do basic HTTP auth with Python's requests module:

requests.post('https://api.bitbucket.org/1.0/user/repositories', auth=('user', 'pass'))

With the other way you're passing the user/pass through the request's payload, which is not desired since HTTP basic auth has its own place in the HTTP protocol.

If you want to "see" what's happening under the hood with your request I recommend using httpbin:

>>> url = "http://httpbin.org/post"
>>> r = requests.post(url, data="myscreename:mypassword")
>>> print r.text
{
  "args": {}, 
  "data": "myscreename:mypassword", 
  "files": {}, 
  "form": {}, 
  "headers": {
    "Accept": "*/*", 
    "Accept-Encoding": "gzip, deflate", 
    "Content-Length": "22", 
    "Host": "httpbin.org", 
    "User-Agent": "python-requests/2.5.1 CPython/2.7.6 Darwin/14.3.0"
  }, 
  "json": null, 
  "origin": "16.7.5.3", 
  "url": "http://httpbin.org/post"
}

>>> r = requests.post(url, auth=("myscreename", "mypassword"))
>>> print r.text
{
  "args": {}, 
  "data": "", 
  "files": {}, 
  "form": {}, 
  "headers": {
    "Accept": "*/*", 
    "Accept-Encoding": "gzip, deflate", 
    "Authorization": "Basic bXlzY3JlZW5hbWU6bXlwYXNzd29yZA==", 
    "Content-Length": "0", 
    "Host": "httpbin.org", 
    "User-Agent": "python-requests/2.5.1 CPython/2.7.6 Darwin/14.3.0"
  }, 
  "json": null, 
  "origin": "16.7.5.3", 
  "url": "http://httpbin.org/post"
}

And with curl:

curl -X POST --user myscreename:mypassword http://httpbin.org/post
{
  "args": {}, 
  "data": "", 
  "files": {}, 
  "form": {}, 
  "headers": {
    "Accept": "*/*", 
    "Authorization": "Basic bXlzY3JlZW5hbWU6bXlwYXNzd29yZA==", 
    "Host": "httpbin.org", 
    "User-Agent": "curl/7.37.1"
  }, 
  "json": null, 
  "origin": "16.7.5.3", 
  "url": "http://httpbin.org/post"
}

Notice the resemblance between the last python example and the cURL one.

Now, getting right the API's format is another story, check out this link: https://answers.atlassian.com/questions/94245/can-i-create-a-bitbucket-repository-using-rest-api

The python way should be something like this:

requests.post('https://api.bitbucket.org/1.0/repositories', auth=('user', 'pass'), data = "name=repo_name")

Incommutable

I like that httpbin site, but I edited the question to show my recent attempts, they did not work – Stagestruck

I got 405 for that too – Stagestruck

@Stagestruck That's out of the scope of this question, since it is not curl/python related – Incommutable

question 1 was what is my requests attempt I tried based on their curl example doing wrong, and the point is to take a curl example and be able to get a 200 using python requests. none of these similar attempts are working in requests – Stagestruck

requests.post('https://api.bitbucket.org/1.0/repositories', auth=('user', 'pass'), data = "name=repo_name") is the attempt that got 405 – Stagestruck

No, that request used 'api.bitbucket.org/1.0/user/repositories' as an url, you need to remove "user" from it – Incommutable

using that url correctly, I now get several 400s instead of 405, trying the original way then toying around. that url will make a new repo, which I did using the forum poster's exact syntax with curl, but I'm trying to grab existing repo info, which uses the other URL – Stagestruck

your answer cleared up the difference, and the best way to get the exactly request was asking bitbucket directly. thanks! – Stagestruck

@Stagestruck Great! It'd be nice if you shared the right way to do it – Incommutable

Answer 2

With python3, you can use json={...} instead of data={...}, and it will set the header automatically to application/json:

import requests
url = 'https://api.bitbucket.org/1.0/user/repositories'

data = {
    'data1': 'asd',
    'data2': 'asd'
}
req = requests.post(url, auth=('user', 'password'), json = data)
data = req.json()
# data['index']

Abase