Rails: Filter sensitive data in JSON parameter from logs

Question

I am running Rails 3 and trying to filter sensitive information out of our logs which are JSON blobs that are passed as post parameters. For example, user creation might take a post param called user with a string value that is a JSON object. One of the keys in the JSON object is password and we want to filter this out of our logs. The best way I found to do this was to add a block to our filter_params, like so:

keys_to_filter = ['password', 'password_confirmation']
config.filter_parameters << lambda do |k,v|
  if v.is_a? String
    keys_to_filter.each do |key|
      # Match "key":"<filter_out>", or "key":"<filter_out>"}, allowing for whitespace
      v.sub!(/("\s*#{key}\s*")\s*:\s*"[^,\}]*"\s*([,\}])/, "\\1:\"[FILTERED]\"\\2")
    end
  end
end

This adds a block to the filter_params, which causes an error which is described in another question: Rails: ParameterFilter::compiled_filter tries to dup symbol

It appears that it is not safe to pass a block to filter_parameters, so I'm wondering if there is another way to solve this problem.

Gertiegertrud

have you considered encrypting the password, then it would be safe from people simply looking at the logs – Kaffiyeh

@paul-kaplan: How does that make the logs more secure? It would obscure human-readable passwords, true. But, now the server is expecting an encrypted password, but that's just another string. If I wanted to, I could take those encrypted strings from the logs, put them in a post to the server, and log into the account. I agree that it is slightly better, and that the risk of exploitation is small, but it doesn't really solve the problem. – Gertiegertrud

I'm having the same problem. Seems the JSON being spit to my log is heavily escaped, probably causing filter_parameters to fail in finding password. Parameters: {"{\"user\":{\"first_name\":\"Barry\",\"last_name\":\"Hess\",\"email\":\"[email protected]\",\"password\":\"notfilterediswear\"}}"} – Erlandson

My problem was improper Content-Type (or complete lack of it) in API calls meaning Rails didn't format properly to the log. Proper Content-Type, proper parameter filtering. – Erlandson

Answer 1

There is an example of passing a block to filter_parameters in the Rails test suite:

config.filter_parameters += [ :foo, 'bar', lambda { |key, value|
  value = value.reverse if key =~ /baz/
 }]

Fleisher

Answer 2

Why is @Nesbitt's answer downvoted (down below)? Sure, it references code in a test suite, but that test suite is just testing a feature documented in ActionDispatch::Http::FilterParameters:

If a block is given, each key and value of the params hash and all subhashes is passed to it, the value or key can be replaced using String#replace or similar method.

See comments in the API docs for Rails 3.x here.

I needed to do the same thing: transform a field in a JSON blob that was sent with an HTTP POST to a Rails 3 app. In my case I just wanted to include a hashed digest of a field, so in config/application.rb:

config.filter_parameters += [:password, lambda {|key, value|
  if key.to_s == 'my_key'
    value.replace(calculate_my_hash(value))
  end
}]

Despot

Answer 3

I'm developing a Rails 3 app, and using Backbone.js. I'm passing JSON objects when I create or update a record. I overrode my Backbone model's toJSON function and hardcoded a password param just to test your issue. In my case, config.filter_parameters is just [:password], and it filters the password param correctly in the logs. At first I tested this in update which submits a PUT request. I then tested this in create with a POST request just to check if there's any special bug when submitting a POST. Password is still filtered correctly. Am I missing something?

It looks like config.filter_parameters works correctly without needing to pass a block.

Geophilous

Answer 4

I have dealt with the same issue in our application. I ended up blocking the entire JSON string from logging where there was secure data, and then adding explicit loggers for any information I wanted logged.

Dodecahedron