REXML :: RuntimeError (entity expansion has grown too large)

Question

After upgrading to Ruby-1.9.3-p392 today, REXML throws a Runtime Error when attempting to retrieve an XML response over a certain size - everything works fine and no error is thrown when receiving under 25 XML records, but once a certain XML response length threshold is reached, I get this error:

Error occurred while parsing request parameters.
Contents:

RuntimeError (entity expansion has grown too large):
  /.rvm/rubies/ruby-1.9.3-p392/lib/ruby/1.9.1/rexml/text.rb:387:in `block in unnormalize'

I realize this was changed in the most recent Ruby version: http://www.ruby-lang.org/en/news/2013/02/22/rexml-dos-2013-02-22/

As a quick fix, I've changed the size of REXML::Document.entity_expansion_text_limit to a larger number and the error goes away.

Is there a less risky solution?

Mccarver

My version of p392 says it doesn't even recognize that setting. Causes my deployment to fail. – Tusker

Answer 1

This issue is generated when you send too much content as XML response.

To fix this issue : You need to restrict the data(< 10k) in the individual node (Instead of sending the whole data, show truncated data and provide a seperate link to view full content)

The error is being raised from the below file : ruby-2.1.2/lib/ruby/2.1.0/rexml/text.rb

# Unescapes all possible entities
def Text::unnormalize( string, doctype=nil, filter=nil, illegal=nil )
  sum = 0
  string.gsub( /\r\n?/, "\n" ).gsub( REFERENCE ) {
    s = Text.expand($&, doctype, filter)
    if sum + s.bytesize > Security.entity_expansion_text_limit
      raise "entity expansion has grown too large"
    else
      sum += s.bytesize
    end
    s
  }
end

The limit ruby-2.1.2/lib/ruby/2.1.0/rexml/text.rb defaults to 10240 which means 10k data per node.

REXML already defaults to only allow 10000 entity substitutions per document, so the maximum amount of text that can be generated by entity substitution will be around 98 megabytes. (Refer https://www.ruby-lang.org/en/news/2013/02/22/rexml-dos-2013-02-22/ )

Gillespie

Answer 2

That sounds like a LOT of XML. Do you really need to get all of it? Maybe you can just request certain fields from the remote server? One option might be to try another XML parser (Nokogiri for example). Another option to maybe use something other than XML as a transport (JSON? Binary?).

Herlindaherm

I'm pulling item inventory for retail stores through Quickbooks Web Connector and Point of Sale... all of these stores have over 25 items in stock, thus the large XML pulls. – Mccarver