Slower while generating the XML from the bunch of model object

Question

class GenericFormatter < Formatter
 attr_accessor :tag_name,:objects

 def generate_xml
   builder = Nokogiri::XML::Builder.new do |xml|
   xml.send(tag_name.pluralize) {
   objects.each do |obj|
        xml.send(tag_name.singularize){

            self.generate_obj_row obj,xml
        }                
    end
    }
   end
   builder.to_xml
 end


def initialize tag_name,objects
  self.tag_name = tag_name
  self.objects = objects
end


def generate_obj_row obj,xml
   obj.attributes.except("updated_at").map do |key,value|
     xml.send(key, value)
   end
   xml.updated_at obj.updated_at.try(:strftime,"%m/%d/%Y %H:%M:%S") if obj.attributes.key?('updated_at')
end
 end 

In the above code, I have implemented the formatter where I have used the nokogiri XML Builder to generate the XML by manipulating the objects passing out inside the code.It's generated the faster XML when the data is not too large if data is larger like more than 10,000 records then It's slow down the XML to generate and takes at least 50-60 seconds.

Problem: Is there any way to generate the XML faster, I have tried XML Builders on view as well but did n't work.How can I generate the XML Faster? Should the solution be an application on rails 3 and suggestions to optimized above code?

Tung

Is the issue simply that storing a huge list of objects is using up all your RAM? You can refactor the code to handle the objects in batches. – Benedikt

Also, if you run this as an asynchronous task then performance becomes less of an issue. – Benedikt

No it's not about storing the huge list it's all about manipulating the object to xml.If we divide it into batches so it will not affect on performance – Tung

Are the objects rails models? If they are, do they have associations? Do the associations have associations? If so, how deep do you want your XML to be? Have all the associations fetched from database? Are there any potential infinite recursion? – Permission

Aetherus Zhou yes they are rails models but there is only one model data having no associated data.But huge data – Tung

@TusharPal My interpretation of your problem description was that the task does not scale linearly. Is that correct? I.e. It takes, say, 2 seconds to generate a report for 3000 records, but 100 seconds to generate a report for 10,000 records - something like that? Or is the problem a more basic performance issue, e.g. it takes 1 second for 100 records, 10 seconds for 1000 records, 100 seconds for 10,000 records, ... ? – Benedikt

Tom second one is my problem – Tung

@TusharPal, the second one (1 second for 100 records, 10 seconds for 1000 records, 100 seconds for 10,000 records) isn't a performance problem, it's a design problem. Even if you find an XML generator that's twice as fast, you will still have a problem (even though you will experience the issue only later)... Consider creating a design where the XML is limited to a small number of objects. A single object is probably ideal. – Holmun

@Holmun That's why i am sharing above code to you all If it's problem with design then how I can increase it by above code.I need a proper solution where I can generate XML Faster because for now Active Record takes 500 ms and XML Generators takes 34-35 seconds. – Tung

Answer 1

Your main problem is processing everything in one go instead of splitting your data into batches. It all requires a lot of memory, first to build all those ActiveRecord models and then to build memory representation of the whole xml document. Meta-programming is also quite expensive (I mean those send methods).

Take a look at this code:

class XmlGenerator
  attr_accessor :tag_name, :ar_relation

  def initialize(tag_name, ar_relation)
    @ar_relation = ar_relation
    @tag_name = tag_name
  end

  def generate_xml
    singular_tag_name = tag_name.singularize
    plural_tag_name = tag_name.pluralize

    xml = ""
    xml << "<#{plural_tag_name}>"

    ar_relation.find_in_batches(batch_size: 1000) do |batch|
      batch.each do |obj|
        xml << "<#{singular_tag_name}>"

        obj.attributes.except("updated_at").each do |key, value|
          xml << "<#{key}>#{value}</#{key}>"
        end

        if obj.attributes.key?("updated_at")
          xml << "<updated_at>#{obj.updated_at.strftime('%m/%d/%Y %H:%M:%S')}</updated_at>"
        end

        xml << "</#{singular_tag_name}>"
      end
    end

    xml << "</#{tag_name.pluralize}>"
    xml
  end
end

# example usage
XmlGenerator.new("user", User.where("age < 21")).generate_xml

Major improvements are:

• fetching data from database in batches, you need to pass ActiveRecord collection instead of array of ActiveRecord models
• generating xml by constructing strings, this has a risk of producing invalid xml, but it is much faster than using builder

I tested it on over 60k records. It took around 40 seconds to generate such xml document.

There is much more that can be done to improve this even further, but it all depends on your application.

Here are some ideas:

• do not use ActiveRecord to fetch data, instead use lighter library or plain database driver
• fetch only data that you need
• tweak batch size
• write generated xml directly to a file (if that is your use case) to save memory

Commonly

Hey Michal so basically your are generating the the string as XML so How i can be able to get it in plain XML format because i think if I will call the method to_xml so it will also take more time on string to generate over the string. – Tung

I don't know how you plan to use this class. If some code is calling to_xml then just change the name of generate_xml to to_xml. – Redletter

Thanks michal Little Performance is increased by applying this so it can be easily integrated. good to points which you have given for it. – Tung

Answer 2

The Nokogiri gem has a nice interface for creating XML from scratch, Nokogiri is a wrapper around libxml2.

Gemfile gem 'nokogiri' To generate xml simple use the Nokogiri XML Builder like this

xml = Nokogiri::XML::Builder.new { |xml| 
    xml.body do
        xml.test1 "some string"
        xml.test2 890
        xml.test3 do
            xml.test3_1 "some string"
        end
        xml.test4 "with attributes", :attribute => "some attribute"
        xml.closing
    end
}.to_xml

output

<?xml version="1.0"?>
<body>
  <test1>some string</test1>
  <test2>890</test2>
  <test3>
    <test3_1>some string</test3_1>
  </test3>
  <test4 attribute="some attribute">with attributes</test4>
  <closing/>
</body>

Demo: http://www.jakobbeyer.de/xml-with-nokogiri

Hickox

Hey Mayur, Thanks for Demo of Nokogiri. If you see above my code it's also in the Nokogiri Builder so what's new thing are you expecting from my side. – Tung