Dealing with Bad request

Question

I’m getting: 'HTTP/1.1 400 Bad Request\r\n' and I don’t get why. It looks like it authenticates and then there is a redirection and then it now doesn’t work. Why is this happening?

I had thought it was the header and that it was missing content type, but even adding that produced the same outcome..

headers = {
    'basic_auth': 'brofewfefwefewef:EKAXsWkdt5H6yJEmtexN',
    'Content-Type': 'application/json'
}
client = Client(ClientConfig(), headers=headers, refresh=True)


class FileDownloader(object):
    ...Line 152...
    def _get_http_pool(self, secure=True):
        if secure:
            _http = urllib3.PoolManager(cert_reqs=str('CERT_REQUIRED'),
                                        ca_certs=certifi.where())
        else:
            _http = urllib3.PoolManager()

        if self.headers:
            content_type = self.headers.get('Content-Type')
            if 'Content-Type' in self.headers:
                del self.headers['Content-Type']
            _headers = urllib3.util.make_headers(**self.headers)
            _http.headers.update(_headers)
            if content_type:
                _http.headers['content-type'] = content_type
        print(_http.headers)
        return _http

https://github.com/JMSwag/PyUpdater/blob/master/pyupdater/client/downloader.py Line 366, is where the download itself starts. This is perplexing to say the least.

Error:

DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.bitbucket.org
send: b'GET /2.0/repositories/Anexampleuser/repo/downloads/keys.gz HTTP/1.1\r\nHost: api.bitbucket.org\r\nAccept-Encoding: identity\r\nauthorization: Basic YnJvZmV3ZmVmd2VmZXdlZjpFS0FYc1drZHQ1SDZ5SkVtdGV4Tg==\r\n\r\n'
reply: 'HTTP/1.1 302 Found\r\n'
DEBUG:urllib3.connectionpool:https://api.bitbucket.org:443 "GET/2.0/repositories/Anexampleuser/repo/downloads/keys.gz HTTP/1.1" 302 0
DEBUG:urllib3.util.retry:Incremented Retry for (url='https://api.bitbucket.org/2.0/repositories/Anexampleuser/repo/downloads/keys.gz'): Retry(total=2, connect=None, read=None, redirect=None, status=None)
INFO:urllib3.poolmanager:Redirecting https://api.bitbucket.org/2.0/repositories/Anexampleuser/repo/downloads/keys.gz -> https://bbuseruploads.s3.amazonaws.com/a0e395b6-0c54-4efb-9074-57ec4190020b/downloads/3fc0be6d-ca69-42d3-9711-fbb5cfd2bc38/keys.gz?Signature=ZQxeUTvYC3Q%2Fo1aaS1CSuzyit0Q%3D&Expires=1515976464&AWSAccessKeyId=AKIAIQWXW6WLXMB5QZAQ&versionId=n.ymY11KRkq36Xozy25aChvfUT.YzTf5&response-content-disposition=attachment%3B%20filename%3D%22keys.gz%22
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): bbuseruploads.s3.amazonaws.com
header: Server header: Vary header: Content-Type header: X-OAuth-Scopes header: Strict-Transport-Security header: Date header: Location header: X-Served-By header: ETag header: X-Static-Version header: X-Content-Type-Options header: X-Accepted-OAuth-Scopes header: X-Credential-Type header: X-Render-Time header: Connection header: X-Request-Count header: X-Frame-Options header: X-Version header: Content-Length send: b'GET /a0e395b6-0c54-4efb-9074-57ec4190020b/downloads/3fc0be6d-ca69-42d3-9711-fbb5cfd2bc38/keys.gz?Signature=ZQxeUTvYC3Q%2Fo1aaS1CSuzyit0Q%3D&Expires=1515976464&AWSAccessKeyId=AKIAIQWXW6WLXMB5QZAQ&versionId=n.ymY11KRkq36Xozy25aChvfUT.YzTf5&response-content-disposition=attachment%3B%20filename%3D%22keys.gz%22 HTTP/1.1\r\nHost: bbuseruploads.s3.amazonaws.com\r\nAccept-Encoding: identity\r\nauthorization: Basic YnJvZmV3ZmVmd2VmZXdlZjpFS0FYc1drZHQ1SDZ5SkVtdGV4Tg==\r\n\r\n'
reply: 'HTTP/1.1 400 Bad Request\r\n'

Plunger

Never paste the real auth data in your questions. If I had malicious intentions, I could easily loop through your repos and delete them one by one. I edited the question, but still would advise you to change your password/API token right now. – Alodee

Definitely change your credentials ASAP - the original post is still visible in the edit history. – Probation

App password changed. Though their was read-only access so not a huge security concern. – Plunger

Likely json relate fix. Error 400 Urlib3 answers are not really existent. I imagine something like that might fix in theory, but yet to find something – Plunger

Answer 1

The BitBucket API returns 302 for the /downloads/ endpoint, so the Authorization header is carried out to the next request, while Amazon does not like that header, so it returns 400. A workaround is recreating the redirected request manually. For example: (error checking omitted)

import urllib3

http_pool = urllib3.PoolManager()
req = http_pool.urlopen(
    'GET', 'https://api.bitbucket.org/2.0/repositories/brofewfefwefewef/eee/downloads/keys.gz',
    redirect=False, headers={'Authorization': 'Basic YnJvZmV3ZmVmd2VmZXdlZjpFS0FYc1drZHQ1SDZ5SkVtdGV4Tg=='})
redirected_req = http_pool.urlopen('GET', req.headers['Location'], preload_content=False)
with open('keys.gz', 'wb') as f:
    f.write(redirected_req.read())

By the way, your access token is still usable.

Jorge

Thanks :). Could you provide an example so that it will update to main-win-1.4.zip successfully? I appreciate it – Plunger

I'm too lazy to set up an repo for testing pyupdater :) I bet you'll get more feedback from pyupdater authors. – Jorge

Fair enough. To be fair if you copy paste config file and main.py into any directory, it should tell you exact output. You already have my app pass and username ;) – Plunger

This downloads the files into current directory. If it could update current main.exe it would be solved :). The author of pyupdater is not very active sadly. – Plunger

Answer 2

You need to give them access to the repository by creating a team inside bitbucket. Then you can use git --export to download files

git archive --remote=ssh://[email protected]/<your-username>/<reponame>.git <branchname> <filename> --output output.tar

Of course, you need to have git authed eg. using an ssh key or similar

I don't think there is a way to make direct download links with auth in bitbucket, then you need to set that up outside bitbucket.

Tartrate

Hmm... from what I have gather you cannot download repositories with bitbucket. You have to use https and app password if you do not want your account stolen. – Plunger

Answer 3

As this is a 400, I would guess the request to Amazon’s S3 service doesn’t like and shouldn’t have the Authorization header, but requests passes it anyway when redirecting.

What you should do is using allow_redirects=False and then doing the redirect yourself, extracting the location from the response’s Location header.

Craigcraighead

Thanks for your reponse :) Are you able to provide working example? I cannot get it to update. I added redirect=False and tried to include Yen Chi Hsuan's solution – Plunger