Using alembic.config.main redirects log output

Question

I have a script that performs database operations alongside an alembic API call to upgrade a newly created database to head. I am having an issue with a python logger instance where logs are written to a file using a module level logger.

Then the script invokes alembic.config.main(argv=alembic_args) to run a migration. However, every log statement after the alembic call, using the original logger instance, isn't written to the expected log file.

Here is an example script that reproduces the behavior.

#!/usr/bin/env python3

import logging
import os

import alembic.config

from .utilities import get_migration_dir

logging.basicConfig(filename='test.log',
                    level=logging.DEBUG)

CUR_DIR = os.path.dirname(__file__)
LOG = logging.getLogger('so_log')

LOG.info('Some stuff')
LOG.info('More stuff')

alembic_config = (
    '--raiseerr',
    'upgrade', 'head'
)

os.chdir(get_migration_dir())

alembic.config.main(argv=alembic_config)

os.chdir(CUR_DIR)

LOG.debug('logging after alembic call.')
LOG.debug('more logging after alembic call.')
print('Code still running after alembic')

Log file output

INFO:so_log:Some stuff
INFO:so_log:More stuff

stdout

INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
print statement before alembic
Code still running after alembic

It seems as though the logger instance, LOG, is losing context or being directed elsewhere after calling the alembic API.

Also, I've tried running the alembic call in a separate thread which yielded the same result. What I expect to happen should be that log statements continue to write to the specified file after using alembic for migrations but that is not happening. And further, it's actually breaking the LOG instance for any code that's called afterward; Unless I'm just missing something here.

Duenas

Answer 1

This is because alembic sets up logging using fileConfig from alembic.ini, you can see it in your env.py script:

# Interpret the config file for Python logging.
# This line sets up loggers basically.
fileConfig(config.config_file_name)

This effectively overrides your original logger config.

To avoid this, you can simply remove this line from env.py, however this will result in no logs being produced when running alembic from console.

A more robust option is to run alembic commands via alembic.command instead of alembic.config.main. This way you can override alembic config at runtime:

from alembic.config import Config
import alembic.command

config = Config('alembic.ini')
config.attributes['configure_logger'] = False

alembic.command.upgrade(config, 'head')

Then in env.py:

if config.attributes.get('configure_logger', True):
    fileConfig(config.config_file_name)

Missal

Ah, I see. I ended up running the alembic command in a separate process at runtime. Your solution is much better. Thank you. – Duenas

This is a great solution. I'd like to also recommend that we use the Config instantiation to set our configure_logger value instead of config.attributes. ie. config = Config('alembic.ini', attributes={'configure_logger': False}) – Carven

Answer 2

I just learned that fileConfig takes a keyword argument, disable_existing_loggers, which defaults to True. Simply adding disable_existing_loggers = False to the call to fileConfig in env.py e.g:

fileConfig(config.config_file_name, disable_existing_loggers=False)

appears to allow both logging configurations to work without interfering with each other (which may be preferred rather than having to choose one over the other, in some cases)

Herbal

This SHOULD be the answer. – Rotarian

Disable existing logger flag worked for me in FASTAPI based application. However I started seeing each message being logged twice. I had to use this flag in my FastApi logger : propogate = False . – Dissipation

Answer 3

In case anyone comes across this issue here is what worked for me:

• Updating the fileConfig call in env.py: fileConfig(config.config_file_name, disable_existing_loggers = False)

• Updating the level for [logger_root] in alembic.ini to a lower level, since it is set by default to WARN

Geoff