What's the most efficient way of converting a 10 MB JSON response into an NSDictionary?

Question

Our app must display a big chunk of data with minimal remote http requests, so we have added an endpoint to our backend that provides all the necessary data as a single json response. This results in ~1.5MB (compressed) or roughly 8 MBs of uncompressed json-formatted text.

Not much of a problem, it downloads in 10 - 30 seconds and we're using ASIHTTPRequest to write the whole response to disk.

Now comes the fun part - after reading the uncompressed file into a memory mapped string, we use stig's json-framework to convert it into an NSDictionary. This has worked very well for the rest of our app and the typical 2 KB json response for the rest of our API endpoints. However, deserializing these 8 MBs of data takes from a couple of seconds (simulator) to minutes (3G and 2nd gen iPod Touch).

I'm researching the best approach to read in all this data.

I would love to use binary plists served straight from the backend, but we are using Java and I haven't found a proper library that fits our requirements, and with such a tight deadline, writing our own might not be the best idea.

If it helps in any way, the json string we are parsing is mostly an array of X items, like so:

{
    "items": [ { "key1": "value1", "key2": "value2" },
               { "key1": "value1", "key2": "value2" },
               { "key1": "value1", "key2": "value2" },
               { "key1": "value1", "key2": "value2" },
               { "key1": "value1", "key2": "value2" },
               { "key1": "value1", "key2": "value2" }
             ]
}

What is the most efficient method to read in this 8 MB json formatted string into a NSDictionary in memory?

Addieaddiego

Can you split it into smaller files and read each one on demand? Or do you need all 8 MB at once? – Snowshoe

Gotta go with: egads. How can you display 8MB of text on such a small screen? I'll go with there's likely another approach that'll be more efficient. – Caruthers

It's not displayed all at once. It's an interactive TV programming guide, divided into one hour blocks with 900 channels each. So it's ~7*24 sequential table views, each with >900 programs. The user might only need the first 4 hours or so at the moment, but a requirement is that he must be able to fetch any of the other days/times without pause. Initially I would download only 4 hours of data at a time, but the code that kept track of which were available in Core Data and which needed to be downloaded yet, was too complex to maintain, so we opted to download all of the data in one pass. – Baily

Answer 1

Take a look at JSONKit. It's very fast, and if your JSON has a lot of keys and values that repeat themselves (as is common), its "recently instantiated object cache" will dramatically cut down on the amount of memory used for the "final" NSDictionary.

Arria

Very cool. Hadn't seen this before. I would love to hear how fast this is with the 8MB of data. – Cline

+1 I'd forgotten about this project! thanks for the reminder. :) – Pang

Ran some benchmarks with JSONKit and got a 200% improvement. Great suggestion. – Baily

Modified Sam Soffes's benchmark tool to run with my json string. Reduced the iterations to 10 for this run, will leave it running later at the initial 100. JSONKit came up ahead: hram.us/3VPL – Baily

This was with just one run through, which made it look like JSONKit was 5x faster: hram.us/3VDc – Baily

Is there a quick tutorial to get started using JSONKit? – Carducci

Answer 2

While JSONKit is faster than YAJL, YAJL supports parsing of the stream. You should be able to wire it up with ASIHTTPRequest to parse the JSON while it's downloading it. Check out the section labeled "Document style parsing as data becomes available" here:

http://gabriel.github.com/yajl-objc/

Rudich