Adding custom headers to all boto3 requests

Question

I need to add some custom headers to every boto3 request that is sent out. Is there a way to manage the connection itself to add these headers?

For boto2, connection.AWSAuthConnection has a method build_base_http_request which has been helpful. I've yet to find an analogous function within the boto3 documentation though.

Tubercle

Answer 1

This is pretty dated but we encountered the same issue, so I'm posting our solution.

I wanted to add custom headers to boto3 for specific requests. I found this: https://github.com/boto/boto3/issues/2251, and used the event system for adding the header

def _add_header(request, **kwargs):
    request.headers.add_header('x-trace-id', 'trace-trace')
    print(request.headers)  # for debug


some_client = boto3.client(service_name=SERVICE_NAME)
event_system = some_client.meta.events
event_system.register_first('before-sign.EVENT_NAME.*', _add_header)

You can try using a wildcard for all requests:

event_system.register_first('before-sign.*.*', _add_header)

*SERVICE_NAME- you can find all available services here: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/index.html

For more information about register a function to a specific event: https://boto3.amazonaws.com/v1/documentation/api/latest/guide/events.html

Fanchette

Great answer, thanks for sharing the solution – Witte

This works, there is no way to pass custom data to event handlers, currently we have to do it in a non-pythonic way using global variables/queues :( I have opened issue ticket with Boto3 developers for this exact case: github.com/boto/boto3/issues/2889 – Maggiemaggio

@Maggiemaggio I added an answer showing how to dynamically pass in a different custom headers with every boto3 request – Sawdust

Answer 2

Answer from @May Yaari is pretty awesome. To the concern raised by @arainchi:

This works, there is no way to pass custom data to event handlers, currently we have to do it in a non-pythonic way using global variables/queues :( I have opened issue ticket with Boto3 developers for this exact case

Actually, we could leverage the python functional programming property: returning a function inside a function to get around:

In the case we want to add a custom value custom_variable to the header, we could do

some_client = boto3.client(service_name=SERVICE_NAME)
event_system = some_client.meta.events
event_system.register_first('before-sign.EVENT_NAME.*', _register_callback(custom_variable))

def _register_callback(custom_variable):
    def _add_header(request, **kwargs):
        request.headers.add_header('header_name_you_want', custom_variable)
    return _add_header

Or a more pythonic way using lambda

some_client = boto3.client(service_name=SERVICE_NAME)
event_system = some_client.meta.events
event_system.register_first('before-sign.EVENT_NAME.*', lambda request, **kwargs: _add_header(request, custom_variable))

def _add_header(request, custom_variable):
    request.headers.add_header('header_name_you_want', custom_variable)

Narbada

Answer 3

Answer from @May Yaari works great to add a constant header to every request you send. However, sometimes you need to send different headers with every request that's being sent, for example to make use of Cloudflare R2 conditional puts. For such use cases you can do the following:

event_system = client.meta.events

# Moves the custom headers from the parameters to the request context
# This is done because later in the processing of the request, there is
# a parameter validation step, which doesn't allow for custom arguments.
def process_custom_arguments(params, context, **kwargs):
    if (custom_headers := params.pop("custom_headers", None)):
        context["custom_headers"] = custom_headers

# Here we extract the headers from the request context and actually set them
def add_custom_headers(params, context, **kwargs):
    if (custom_headers := context.get("custom_headers")):
        params["headers"].update(custom_headers)

event_system.register('before-parameter-build.s3.PutObject', process_custom_arguments)
event_system.register('before-call.s3.PutObject', add_custom_headers)

custom_headers = {'If-Match' : '"3858f62230ac3c9ff15f300c664312c63f"'}

client.put_object(Bucket="my_bucket", Key="my_key", Body="my_data", custom_headers=custom_headers)

This registers 2 handlers into the event system, one to move the custom headers from the request parameters into the request context, and another one to set it. This is done to circumvent the request parameter validation that boto3 performs.

Sawdust