What performance can we expect from kdb+ 32bit

Question

Kx announced their free kdb+ (32bit) time series database.

What performance in terms of reads/writes per second can we typically expect from this database?

I do understand this is a complex question as the answer depends on a setup, number of nodes, etc. I am hoping someone will be able to provide us some numbers as well as use cases.

Rolanda

Did you look here: kx.com/benchmarks.php – Dethrone

Thanks, I am not sure which of these benchmarks apply to the open source version of kdb+ 32bit. – Rolanda

@Rolanda free =/= open source – Birthmark

An interesting question would be how the performance of the free 32-bit version compares to the paid-for 64-bit version. – Yttria

Answer 1

After more investigation I found sample performance benchmarks on the kx.com website:

On an AMD Opteron box with 4 GB of RAM, we get

0.672 million inserts per second (single insert)
6.944 million inserts per second (bulk insert 10)
20.408 million inserts per second (bulk insert 100)
24.39 million inserts per second (bulk insert 1000)
25 million inserts per second (bulk insert 10000)

On an AMD Turion64 laptop with 0.5 GB of RAM,

0.928 million inserts per second (single insert)
8.065 million inserts per second (bulk insert 10)
16.129 million inserts per second (bulk insert 100)
16.129 million inserts per second (bulk insert 1000)
16.129 million inserts per second (bulk insert 10000)

Running the same benchmarks on my machine (32GB RAM, Xeon E5 3.2GHz, SSD):

1.972 million inserts per second (single insert)
30.303 million inserts per second (bulk insert 10)
142.857 million inserts per second (bulk insert 100)
250 million inserts per second (bulk insert 1000)
200 million inserts per second (bulk insert 10000)

Rolanda

Take note this is the speed of in-memory operations. – Rolanda

Answer 2

The 32bit limits addressable memory. So the size of in-memory database is constrained. Common practice is to keep the current day (RDB) in memory.

Pish

So up to 4GB data can be loaded to 32bit kdb into the memory. Probably less as some memory is needed for performing operations on that data. – Rolanda

@Rolanda In fact I read the other day on kx's website a recommendation to provision 4 times as much memory as the largest expected data size. – Yttria

Answer 3

The STAC M3 benchmarks have this nicely covered too. Aside from that, I wouldn't expect major differences in 32bit v 64bit binaries. I believe the memory allocation strategy is the same in both.

As Datageek has already shown in his post, the performance for kx is second to none. Sure the language will be new for most developers who are used to OOP+SQL but this is a massive step forward for kx IMO.

Hyehyena