Mask out sensitive information in python log

Question

Consider the following code

try:
    r = requests.get('https://sensitive:[email protected]/')
    r.raise_for_status()
except requests.HTTPError:
    logging.exception("Failed to what.ever")

Here, if the endpoint returns non-successful http status code, the following will be logged

Traceback (most recent call last):
  File "a.py", line 5, in <module>
    r.raise_for_status()
  File "venv/lib/python3.5/site-packages/requests/models.py", line 928, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://sensitive:[email protected]/

The problem is that the password is logged. I could create a logging filter to filter out this line completely. However, it would be more convenient if the password was just masked out somehow. As no string is passed to logging.exception filtering on the app side is tricky. Where in the logging framwork can I transform a log record?

Ty

Is that url a magic value or did you create the url from some variables? – Lieabed

It's a magic actually, but if you've got suggestions either way I'm all ears. 😊 – Ty

Related: Hiding Sensitive Data from Logs with Python – Hypnosis

Answer 1

Apparently, this is done with a Formatter. Example below

import logging
import re


class SensitiveFormatter(logging.Formatter):
    """Formatter that removes sensitive information in urls."""
    @staticmethod
    def _filter(s):
        return re.sub(r':\/\/(.*?)\@', r'://', s)

    def format(self, record):
        original = logging.Formatter.format(self, record)
        return self._filter(original)

Use like so

import logging
import requests

from sensitive_formatter import SensitiveFormatter

LOG_FORMAT = \
    '%(asctime)s [%(threadName)-16s] %(filename)27s:%(lineno)-4d %(levelname)7s| %(message)s'
logging.basicConfig(level=logging.DEBUG)
log = logging.getLogger(__name__)

# Don't actually configure your logging like this, just to showcase
# the above answer. :)
for handler in logging.root.handlers:
   handler.setFormatter(SensitiveFormatter(LOG_FORMAT))

log.warning('https://not:[email protected]/basic-auth/expected-user/expected-pass')
try:
    r = requests.get('https://not:[email protected]/basic-auth/expected-user/expected-pass')
    r.raise_for_status()
except requests.exceptions.RequestException as e:
    log.exception('boom!')

The user/password will be masked out. See example log below

$ python log_example.py 
2018-05-18 11:59:22,703 [MainThread      ]                      log.py:14   WARNING| https://httpbin.org/basic-auth/user/secret
2018-05-18 11:59:22,747 [MainThread      ]           connectionpool.py:824    DEBUG| Starting new HTTPS connection (1): httpbin.org
2018-05-18 11:59:23,908 [MainThread      ]           connectionpool.py:396    DEBUG| https://httpbin.org:443 "DELETE /basic-auth/user/secret HTTP/1.1" 405 178
2018-05-18 11:59:23,913 [MainThread      ]                      log.py:19     ERROR| boom!
Traceback (most recent call last):
  File "log.py", line 17, in <module>
    r.raise_for_status()
  File "/Users/vidstige/src/so/venv/lib/python3.6/site-packages/requests/models.py", line 935, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 405 Client Error: METHOD NOT ALLOWED for url: https://httpbin.org/basic-auth/user/secret

Ty

Can you show what the error log looks like with the password masked out? – Hypnosis

@StevenVascellaro absolutely – Ty

Can SensitiveFormatter also be instantiated without argument? – Reed

Yes, it can. Then the default value of '%(message)s will be used. – Ty