Rails converts empty arrays into nils in params of the request

Question

I have a Backbone model in my app which is not a typical flat object, it's a large nested object and we store the nested parts in TEXT columns in a MySQL database.

I wanted to handle the JSON encoding/decoding in Rails API so that from outside it looks like you can POST/GET this one large nested JSON object even if parts of it are stored as stringified JSON text.

However, I ran into an issue where Rails magically converts empty arrays to nil values. For example, if I POST this:

{
  name: "foo",
  surname: "bar",
  nested_json: {
    complicated: []
  }
}

My Rails controller sees this:

{
  :name => "foo",
  :surname => "bar",
  :nested_json => {
    :complicated => nil
  }
}

And so my JSON data has been altered..

Has anyone run into this issue before? Why would Rails be modifying my POST data?

UPDATE

Here is where they do it:

https://github.com/rails/rails/blob/master/actionpack/lib/action_dispatch/http/request.rb#L288

And here is ~why they do it:

https://github.com/rails/rails/pull/8862

So now the question is, how to best deal with this in my nested JSON API situation?

Balcke

I found where it's doing this deep_munge github.com/rails/rails/blob/master/actionpack/lib/…. Still not sure why it's doing it. – Balcke

Links to master/actionpack are no longer pointing at the correct line in question. Link to a tag or commit. – Woodpecker

Answer 1

After much searching, I discovered that you starting in Rails 4.1 you can skip the deep_munge "feature" completely using

config.action_dispatch.perform_deep_munge = false

I could not find any documentation, but you can view the introduction of this option here: https://github.com/rails/rails/commit/e8572cf2f94872d81e7145da31d55c6e1b074247

There is a possible security risk in doing so, documented here: https://groups.google.com/forum/#!topic/rubyonrails-security/t1WFuuQyavI

Baryram

Do you know how can I disable it if I'm using an older rails version? – Swaddle

This didn't seem to work for me when using post in an rspec test. – Havildar

Answer 2

Looks like this is a known, recently introduced issue: https://github.com/rails/rails/issues/8832

If you know where the empty array will be you could always params[:...][:...] ||= [] in a before filter.

Alternatively you could modify your BackBone model's to JSON method, explicitly stringifying the nested_json value using JSON.stringify() before it gets posted and manually parsing it back out using JSON.parse in a before_filter.

Ugly, but it'll work.

Eurus

That's how it originally worked, I was parsing/stringifying in Backbone. However I thought it's nice not to have partially stringified JSON when posting - it seems cool to just POST one large JSON as if I'm talking to a .. document based store API. – Balcke

Opps.. enter submits this textarea. For now, I just went with patching ActionDispatch::Request.deep_munge as discussed in github.com/rails/rails/pull/8862 until it rolls out in Rails stable, which it might as they merged the PR 2 days ago. – Balcke

Answer 3

You can re-parse the parameters on your own, like this:

class ApiController
  before_filter :fix_json_params    # Rails 4 or earlier
  # before_action :fix_json_params  # Rails 5

  [...]

  protected

  def fix_json_params
    if request.content_type == "application/json"
      @reparsed_params = JSON.parse(request.body.string).with_indifferent_access
    end
  end

  private

  def params
    @reparsed_params || super
  end
end

This works by looking for requests with a JSON content-type, re-parsing the request body, and then intercepting the params method to return the re-parsed parameters if they exist.

Biondo

I've found this gist working with Rails 3.2.13. – Quaker

@Biondo wouldn't it be better to merge the reparsed params with the original params? (e.g. url request parameters would be lost) – Devereux

Word of caution. Redefining params would break down for params-specific methods, like params.permit() – Tripoli

Answer 4

I ran into similar issue.

Fixed it by sending empty string as part of the array.

So ideally your params should like

{
  name: "foo",
  surname: "bar",
  nested_json: {
    complicated: [""]
  }
}

So instead of sending empty array I always pass ("") in my request to bypass the deep munging process.

Blew

Answer 5

Here's (I believe) a reasonable solution that does not involve re-parsing the raw request body. This might not work if your client is POSTing form data but in my case I'm POSTing JSON.

in application_controller.rb:

  # replace nil child params with empty list so updates occur correctly
  def fix_empty_child_params resource, attrs
    attrs.each do |attr|
      params[resource][attr] = [] if params[resource].include? attr and params[resource][attr].nil?
    end
  end

Then in your controller....

before_action :fix_empty_child_params, only: [:update]

def fix_empty_child_params
  super :user, [:child_ids, :foobar_ids]
end

I ran into this and in my situation, if a POSTed resource contains either child_ids: [] or child_ids: nil I want that update to mean "remove all children." If the client intends not to update the child_ids list then it should not be sent in the POST body, in which case params[:resource].include? attr will be false and the request params will be unaltered.

Woodpecker

@aceofspades I think you're right! Although if a client sent a PATCH to remove all relations to another resource, like {user: {child_ids:[]}} would that work without this solution? – Woodpecker

Otherwise, the above could be updated to return immediately if the request method is PATCH. Shame it's not easy to determine if the param was sent an nullified or if it was not sent by the client in case of a PATCH. – Woodpecker

Answer 6

I ran into a similar issue and found out that passing an array with an empty string would be processed correctly by Rails, as mentioned above. If you encounter this while submitting a form, you might want to include an empty hidden field that matches the array param :

<input type="hidden" name="model[attribute_ids][]"/>

When the actual param is empty the controller will always see an array with an empty string, thus keeping the submission stateless.

Bathhouse

This helped me with a multi-select element that was using hidden fields for submission of selected values, but none if there was no selection. The technique reminds me of the checkbox hack that is build into the check_box_tag form helper. – Currajong