JSONKit benchmarks

Question

I pulled the code from http://www.bonto.ch/blog/2011/12/08/json-libraries-for-ios-comparison-updated/ and tested it on my local machine.

I was surprised by the results and NSJSonSerialization gave much better performance than JSONKit.

Has NSJSonSerialization really overtaken JSONKit in terms of speed or did I do something wrong?

twitter_timeline.json

repeat.json

random.json

Bearer

Nice to see these results, thanks! I used JSONKit before but moved to NSJSONSerialization because it was fast and saves me another external library. But heh, what could you have done wrong with basically comparing two method calls? – Clasping

Are you running these on the simulator or on a device? – Huldahuldah

These were run on the iPhone 6.1 simulator. – Bearer

can anyone back this up? this says otherwise but is from 2011 (github.com/johnezang/…) ... – Thracian

You should ensure that you have disabled assertions in the build settings, which make JSONKit much slower. – Withhold

Answer 1

You have not done anything wrong.

Things have changed in iOS 6. Apple have greatly improved the performance of NSJSONSerialization. When using iOS 6 and up, it is the fastest.

JSONKit has also not been updated in a year. It is falling behind.

Pirbhai

+1: "JSONKit has also not been updated in a year." Yeah, it appears JSONKit repo is dead... :/ – Aldridge

Answer 2

Interestingly, there are still people out there who care about JSON parser performance ;)

About three years ago, I developed my own JSON parser with the goal to make it fast and with a low memory foot-print. Since then, I was interested how it compares to others, especially NSJSONSerialization and JSONKit. I do have a number of bench marks which I run occasionally, to check how new compilers and OSs did improve the parsers. I also forked Bonto's bench, fixed a few errors, added a few pathological JSON files and added my own parser as well.

Well, I can confirm that during the last years, the performance of NSJSONSerialization has been improved greatly. It's now comparable to JSONKit for many work loads. If I remember right, a certain test took 690ms for NSJSONSerialization iOS 5.0, while it now performs at 520ms on iOS 6.1. JSONKit took about 524ms - and now with the same sources - it takes about 500ms, the latter solely due to better compilers.

Since I was curious, I gathered some results, compiled with Xcode (soon available) and a few screenshots taken from my forked Bonto's bench mark.

The test includes my own parser library (JPJson) in the test with name "JPJsonData" (the newest version which is not yet checked in into GitHub), in a fair and comparable test. I ensured, JSONKit has been compiled with all optimizations enabled, and disabled assertions, so that it is as fast as possible.

The test has been run on this hardware, an iPad 2 (the second variant), with these specs:

System Name: iPhone OS System Version: 6.1.3 Model: iPad Localized Model: iPad Platform: iPad2,2 Hardware Model: K94AP Platform Type: 14 Platform String: iPad 2G CPU Frequency: 475.113 MHz BUS Frequency: 475.113 MHz Total Memory: 502 MByte User Memory: 419 MByte

---

Some other test file, pretty much average JSON:

---

JSON with mostly numbers:

---

JSON with mostly random Unicode strings.

These JSON Strings do contain "Unicode noncharacters" and "Unicode NULL" characters. JSONKit does not allow those characters - thus is fails. JPJson won't allow them by default, but it has options which when set, allow those characters in JSON Strings:

---

Update

I ran the benchmark for an iPhone 5s on iOS 7, for both 32-bit and 64-bit.

JSONKit failed to compile for 64-bit.

System Name: iPhone OS
System Version: 7.0.4
Model: iPhone
Localized Model: iPhone
Platform: iPhone6,2
Hardware Model: N53AP
Platform Type: 18
Platform String: Unknown iPhone
CPU Frequency: 0 MHz
BUS Frequency: 0 MHz
Total Memory: 999 MByte
User Memory: 858 MByte

All timings in milli seconds.

-----------------------------------
twitter_timeline.json

| Library             |  32-bit |  64-bit |
|---------------------|--------:|--------:|
| JSONKitData         |   1.68  |       *)|
| JPJsonData          |   1.16  |    1.29 |
| NSJSONSerialization |   2.44  |    2.21 |
| SBJsonData          |   6.05  |    6.69 |



-----------------------------------
appache_builds.json

| Library             |  32-bit |  64-bit |
|---------------------|--------:|--------:|
| JSONKitData         |    2.80 |      *) |
| JPJsonData          |    3.21 |    3.52 |
| NSJSONSerialization |    3.23 |    4.17 |
| SBJsonData          |   10.41 |   11.75 |


-----------------------------------
mesh.json

| Library             |  32-bit |  64-bit |
|---------------------|--------:|--------:|
| JSONKitData         |   87.64 |      *) |
| JPJsonData          |   25.46 |   20.34 |
| NSJSONSerialization |  144.25 |   34.25 |
| SBJsonData          |  105.60 |  107.70 |



-----------------------------------
sample.json

| Library             |  32-bit |  64-bit |
|---------------------|--------:|--------:|
| JSONKitData         |  failed |      *) |
| JPJsonData          |    5.97 |    6.39 |
| NSJSONSerialization |   18.94 |   13.66 |
| SBJsonData          |  failed |   88.19 |


-----------------------------------
random.json

| Library             |  32-bit |  64-bit |
|---------------------|--------:|--------:|
| JSONKitData         |   23.01 |      *) |
| JPJsonData          |   17.69 |   20.62 |
| NSJSONSerialization |   28.83 |   28.60 |
| SBJsonData          |   82.93 |   82.74 |

*) Failed to compile

Withhold

@PsychoDad Yes, faster than light -- no, that simply means that JSON Kit encountered an error. ;) The details: JSONKit does not allow Unicode NULL characters, and Unicode non characters. Usually, they don't occur in reasonable JSON, but strictly they are allowed. The "sample.json" contains randomly created JSON, thus it occasionally happens that Unicode non characters have been created. – Withhold

Have you tested on iPhone 5/5S or iPad 4 / iPad Air? – Betterment

The type of device is given in the answer. Didn't test on most recent hardware, but the relative numbers should be about the same. – Withhold

@croyneaus4u I've updated the benchmark results for an iPhone 5s running on iOS 7. – Withhold

Answer 3

I run the tests on a iPhone 4 (ios7) and iPhone 4S (ios 6.1) and NSJSONSerialization is faster in both cases.

Angelenaangeleno

In order to make fair comparison, you need to ensure that you compile with correct build settings which disable NSAssert macros: In "Preprocessor Macros" define NDEBUG and NS_BLOCK_ASSERTIONS. Furthermore, you need to compare "JSONKitData" test, since otherwise, you feed JSONKit a "NSString" which needs to be converted to NSData first. – Withhold