Disable boto logging without modifying the boto files

Question

I am using the Boto library to talk to AWS. I want to disable logging. (Or redirect to /dev/null or other file). I can't find an obvious way to do this. I tried this, but that doesn't seem to help:

import boto
boto.set_file_logger('boto', 'logs/boto.log')

This says it is possible, http://developer.amazonwebservices.com/connect/thread.jspa?messageID=52727&#52727 but as far as I know the documentation doesn't tell how.

Spallation

Answer 1

You could try

import logging
logging.getLogger('boto').setLevel(logging.CRITICAL)

which will suppress all (other than CRITICAL) errors.

Boto uses logging configuration files (e.g. /etc/boto.cfg, ~/.boto) so see if you can configure it to your needs that way.

The set_file_logger call simply adds a user-defined file to the logging setup, so you can't use that to turn logging off.

Nurmi

Thanks for this. For some reason I can't get the boto config files to switch off logging. code.google.com/p/boto/issues/detail?id=476 – Waikiki

In django setting this is like this: LOGGING['loggers'].update({ 'boto': { 'level': 'CRITICAL', } }) – Smollett

Using boto3, I had to logging.getLogger('botocore').setLevel(logging.INFO) – Mudra

I used; logging.getLogger('boto3').setLevel(logging.CRITICAL) – Greensand

As @charneykaye and @Greensand mention above, because boto3 is actually two distinct modules, boto3 and botocore, you'll need to suppress both with two calls like this : logging.getLogger('botocore').setLevel(logging.CRITICAL) and logging.getLogger('boto3').setLevel(logging.CRITICAL)) – Griseldagriseldis

Here are all boto3 loggers: boto3, botocore, nose. – Auramine

@DmitryVerhoturov - not sure why you classify nose as a boto logger - nose is an independent project which just happens to be used for testing of boto, right? – Nurmi

@VinaySajip, mindless copypaste, sorry. Just checked logging.Logger.manager.loggerDict.keys() with only boto3 imported, bcdocs, boto3, botocore, ndg are root level loggers which are used. – Auramine

Answer 2

I move the boto3 answer from the comments (namely charneykaye and gene_wood) to a proper answer:

import logging

logger = logging.getLogger()
logger.addHandler(logging.StreamHandler()) # Writes to console
logger.setLevel(logging.DEBUG)
logging.getLogger('boto3').setLevel(logging.CRITICAL)
logging.getLogger('botocore').setLevel(logging.CRITICAL)
logging.getLogger('s3transfer').setLevel(logging.CRITICAL)
logging.getLogger('urllib3').setLevel(logging.CRITICAL)

import boto3

s3 = boto3.resource('s3')

for bucket in s3.buckets.all():
    print(bucket.name)

To get all the loggers follow the response from leobarcellos:

import logging
loggers_dict = logging.Logger.manager.loggerDict

Ovid

Most logging i'm seeing from "boto" is really from s3transfer as noted in @Iony's list. (boto 1.9.28 and s3transfer 0.1.13). Note that s3transfer doesn't load until boto3 is used (ie doesn't load when boto3 is loaded) so one should import s3transfer before disabling s3transfer loggers – Input

Working on an app and botocore killed the logging I was looking to remove. – Hollands

I don't understand why this works. If the level is DEBUG and all the boto3 logs are CRITICAL, why these are not appearing in the logs anymore? – Hacker

DEBUG is the default and the boto3 log level is changed in the way that only errors critical and higher are presented, as the default boto3 messages are lower - I assume info or debug - they are no longer shown. Did that help? – Ovid

Answer 3

This is the only solution, which works for me as of today (2020/01/31):

for name in ['boto', 'urllib3', 's3transfer', 'boto3', 'botocore', 'nose']:
    logging.getLogger(name).setLevel(logging.CRITICAL)
logger = logging.getLogger(__name__)

The solution with

boto3.set_stream_logger('', logging.CRITICAL)

was killing my whole non-boto logs. It manipulates the root logger of the standard logging from python.

Try it out for yourself:

import logging
import boto3
import sys
logger = logging.getLogger(__name__)
boto3.set_stream_logger('', logging.CRITICAL)
logging.basicConfig(level=logging.DEBUG, stream=sys.stdout,
                    format='%(asctime)s - %(levelname)s - %(message)s')


if __name__ == '__main__':
    s3_client = boto3.client('s3')
    response = s3_client.list_buckets()
    logger.info(f'bucket list: {response}')

Regardless of where the init of the logger happens, it won't bring up the output. Remove the line of the boto3.set_stream_logger('', logging.CRITICAL) and the non-boto3 logs will re-appear again! Consequently the only working solution is NOT to use the approach with boto3.set_stream_logger() and apply it as I suggested.

Evelyn

Answer 4

Better yet, disable propagate for boto:

import boto
boto.set_file_logger('boto', 'logs/boto.log')
logging.getLogger('boto').propagate = False

Transduction

I'm not sure whether this is a new-ish feature, but this should be the accepted answer nowadays. – Enrage

I had to use logging.getLogger("botocore").propagate = False. – Brueghel

Why is this "better"? – Narcisanarcissism

If I recall correctly (and I wrote this 5+ years ago), by setting propagate to False it prevents it from being handled by other loggers. The second line in my example says "for the 'boto' logger, direct messages to logs/boto.log" and the third line says "...and then stop, don't pass it on to any other log handler" – Transduction

Answer 5

This answer is for those who're using logging.config.dictConfig.

It is recommended to disable DEBUG and INFO messages from all external packages, not limited to botocore and boto3:

LOGGING_CONFIG = { # Add your preexisting logging config here.
    "loggers": { # Add your preexisting loggers here.
        "": {"level": "WARNING", "handlers": ["console"], "propagate": False},  # Root logger.
     }

Alternatively, to disable debug messages from botocore and boto3 but not from all external packages:

LOGGING_CONFIG = { # Add your preexisting config here too.
    "loggers": { # Add your preexisting loggers here too.
        "botocore": {"level": "WARNING", "handlers": ["console"], "propagate": False},
        "boto3": {"level": "WARNING", "handlers": ["console"], "propagate": False},
     }

Assuming your logging configuration dict is named LOGGING, run this next:

logging.config.dictConfig(LOGGING)

The above must be run before boto3 is imported, irrespective of whether it is imported directly or indirectly! It won't entirely work if it's run after boto3 is already imported. You can choose to replace "WARNING" above with "INFO" or "ERROR" or "CRITICAL".

Eagleeyed

Probably the cleanest option, considering the logging config will just work for all packages relying on the logging module. – Campeche

Answer 6

Beware this solution will disable also non-boto logs. See mchlfchr's answer instead.

---

For me none of the posted solutions worked unfortunately. Probably due to meanwhile changes in boto itself.

But sometimes a look into the manual does help..

import logging
import boto3
boto3.set_stream_logger('', logging.CRITICAL)

Reverie

if you are using that, then all the other relevant logs (!= boto*) are disappearing as well – Evelyn

why do they disappear? – Reverie

see my extended answer: https://mcmap.net/q/235179/-disable-boto-logging-without-modifying-the-boto-files – Evelyn

I still don't understand why python's root logger is touched by that. But your example shows. Good job, man :) upvote. – Reverie

Answer 7

There is another annoying property of the logging module in python: if you import logging again (for instance because you import a function from a file where logging is imported) AFTER setting the loglevels for your code, setting the levels might not have any effect. I don't know the exact details of why that is, but I only managed to set correct log levels for imported libraries after implementing something like below, and ALWAYS using a logger created by that function. The file where this logger creation function is defined, is the only file in my code base where logging is imported.

import logging

def get_logger_by_name(logger_name: str, log_filepath: str = "/training.log") -> logging.Logger:
    """
    Function that reloads logging module to store logs into a file and creates logger
    :param logger_name: Name of the logger that is returned
    :param log_filepath: filepath to log_file to store all logs
    :return: logger object
    """
    reload(
        logging
    )  # we need to import logging again to configure log file. https://mcmap.net/q/246016/-python-logging-file-is-not-working-when-using-logging-basicconfig
    logging.basicConfig(
            level=logging.DEBUG,
            format="%(asctime)s %(levelname)-8s %(name)-10s %(message)s",
            datefmt="%Y-%m-%d %H:%M:%S",
            handlers=[logging.FileHandler(log_filepath), logging.StreamHandler()],
        )
    logger = logging.getLogger(logger_name)
    logger.setLevel(logging.DEBUG)

    # clean messy log output by setting log level to WARNING for unimportant loggers
    logging.getLogger("botocore").setLevel(logging.WARNING)
    logging.getLogger("boto3").setLevel(logging.WARNING)
    logging.getLogger("boto").setLevel(logging.WARNING)
    logging.getLogger("s3transfer").setLevel(logging.WARNING)
    return logger

Rightness

Answer 8

If you're dealing with boto3, it can be done by setting a general logging level in basicConfig. In the example below we try to check the existence of a file in a bucket on S3 service:

import boto3
import botocore
import logging

logging.basicConfig(..., level=logging.INFO) # ***** HERE IS THE SOLUTION *****

s3 = boto3.resource('s3', ...) # credetials
my_bucket = s3.Bucket('bucket_name')

logging.info('INFO TEST')
logging.error('ERROR TEST')
logging.warning('WARNING TEST')

try:
    my_bucket.Object('test/test.txt').load() # it will send some https requests and DEBUG them in logging
except botocore.exceptions.ClientError as e:
    if e.response['Error']['Code'] == "404":
        print('The object does not exist.')
else:
    print('The object does exist.')

Singleaction

Answer 9

If you're looking to disable boto and/or other loggers because logs generated by your code are bloated with debug logs of other modules, you can avoid using root logger.

Instead of importing with logging.getLogger(), import with logging.getLogger('module_name').

Referred from a comment to an issue on official boto3 github repo.

Multicolored