Tokyo Cabinet - Memory Tuning

Question

I have more than 100 million key-value pairs (one key can have multiple values). I am using Tokyo Cabinet's BDB (B+Tree DB) whose key, value is 32 bit byte-array.

Is it possible in Tokyo Cabinet to set how much memory (lower and upper limit) it can use ? Actually, I have 8GB memory, but it seems it can't take advantage of large memory.

Another point is that, I want to know that how to set,

tokyoCabinet.tune() tokyoCabinet.optimize() tokyoCabinet.setxmsiz() tokyoCabinet.setcache() these parameters thus I can get maximal tuning for my case. Thanks in advance. Any idea is also helpful to me.

Hargett

You may want to check out Kyoto Cabinet, the sequel to Tokyo Cabinet. The creators strongly suggest moving to Kyoto Cabinet - it doesn't look like Tokyo Cabinet is being maintained. – Gaudy

From the Tokyo Cabinet website, "BTW, do you know Kyoto Cabinet? Actually, it is more powerful and convenient library than Tokyo Cabinet. At this distance of time, Kyoto Cabinet surpasses Tokyo Cabinet in every aspects. I strongly recommend you to use Kyoto Cabinet." – Sensitive

Answer 1

tokyoCabinet.setcache() - set the cache size, usually is half or more filesize

tokyoCabinet.tune() - set the hash bucket file and BTree index parameters before open

tokyoCabinet.optimize() - the set new parameters of the hash bucket file and BTree index when file already opened, if the numbers of record was increasing more than set in the tune( bnum), You can increase by optimize(), for example.

tokyoCabinet.setxmsiz() set the size of the extra mapped memory, as using mmap, usually is filesize

100 million key-value pairs - bnum > 100M * 2 value is 32 bit byte-array -> recsize = 32bit = 8 byte -> apow = 3 ( = 2^3 )

If you have 1024 elements of Array: the recsize, set apow = 13

lmemb specifies the number of members in each leaf page. If it is not more than 0, the default value is specified. The default value is 128.

nmemb specifies the number of members in each non-leaf page. If it is not more than 0, the default value is specified. The default value is 256. fpow specifies the maximum number of elements of the free block pool by power of 2.

The official documentation

Griseous