performance is very bad to construct boost property tree from json file?

Question

I am using boost property tree to load/dump json file. However, the performance is very bad.

For example, I have a json file whose size is 1.8M. The boost C++ program spends 3 seconds to load the json file and construct the property tree. If I use python to load the json file, it only need 0.1 second. And python will also construct everything as object as well.

The C++ program is like:

int main(int argc, char **argv){
        std::fstream fin;
        fin.open(argv[1], std::fstream::in);
        if (!fin.is_open()){
            ASSERT(false);
        }

        boost::property_tree::ptree pt;
        try{
            read_json(fin, pt);
        }catch(ptree_error & e) {
            ASSERT(false);
        }
        fin.close();

    return 0;
}

The python script which is doing same thing is like:

#!/usr/bin//python

import sys
import json

fp = open(sys.argv[1],"r")
objs = json.load(fp)

I tried the lastest boost (1.54). It's still very slow on doing this.

Appreciate for any advice.

If there is no solution, do you know any other C++ library to load/dump json?

Lenticular

How did you compile your C++ program? Did you turn on optimization? – Florafloral

It's compiled with optimizations on. – Lenticular

have you profiled to see where the C++ program spends most of its time? Even something as simple as strace might show some potential bottlenecks. – Proportionate

Yes, I actually have a profile generated by collect. But I didn't understand it much yet. dl.dropboxusercontent.com/u/623030/boost_json.collect.incl.txt – Lenticular

Would you mind to provide the file you are trying to parse (or a similar one)? – Florafloral

Answer 1

It doesn't matter much what's really in the JSON file. I tried multiple JSON files with different conent. Boost is just slow.

Now I already switched to jansson which is much better - fast and nice API to use.

Lenticular

It is probably std::string that is causing your problem rather than boost. – Erotogenic

Answer 2

We had significant performance problems with boost::property_tree and JSON. Our approach was to stop using std::string and use an in-house string class with a custom allocator, and hash tables for not reallocating the same string twice. This improved performance and memory usage by at least a few orders of magnitude for large JSON files.

Our JSON files were large enough that the std::string allocation consumed all available address space on a 32-bit machine. This approach let us run with headroom.

Erotogenic

Answer 3

I found that there is a huge difference between Release Build vs Debug Build performance numbers from VS for Property Tree. on my specific hardware a parsing through a 1 MB JSON File using read_json was taking 8 sec in Debug build , but only 0.7 sec in release version.

Caricature

Great tip! I have the same results here, release performs just fine, saved me from going off to find another solution. – Salty