KDB/Q memory consumption

Question

I have a KDB/Q databse which has around ~2M records per day consuming about ~2G of memory. At end of day it runs some reporting stuff doing joins between the tables and outputting result into files on disk. During the computation the memory usage grows to ~15G. My problem is that once this operation finishes the memory is never released back and until the DB is restarted it consumes all the 15G of memory.

I would like to tell KDB to unload some tables from memory (not drop them though) but I don't want to restart the DB since some other apps are still connecting to it.

Is there a way to tell KDB to unload something from memory?

EDIT:

If anyone finds it interesting I suggest to have a look on .Q.gc[] for KDB 2.5+, looks promising.

Pence

Answer 1

Here is the sum up of my research:

• KDB prior to ver. 2.5 allocates 64MB memory chunks on need and never release them. It is able to reuse them though.
• recent KDB versions allow .Q.gc[] call which is on-request call to garbage collector (KDB uses ref. counting btw.)
• this is especially useful when you invoke some memory intensive computation which allocates lot of memory (in my case it was ~20gB) and you want to release the memory after the computation finishes.
• you can always consider putting the memory intensive script into a separate Q process so the memory will be released once the script finishes

Pence

Answer 2

This may be obvious, but in addition to checking the garbage collection modes for your version of q, do make sure that you've actually gotten rid of the in-memory data that's using the memory. If you're ok with getting rid of the whole table (e.g. this is a temp table involved in the computation), just delete it from the root namespace

delete table from`.

if not, you can delete all its rows

delete from`table

Woman

Answer 3

For anyone trying this in future the easiest way would have been to:

1. Start a new KDB process.
2. From that process query to select out the smallest limited subsets of data needed.
3. Perform any joins / calculations / writing to file from that process. (allowing the original to carry on processing requests)
4. Close the process, freeing up all memory.

As mentioned by the above posters newer versions of KDB free up memory better but not perfect.

There's a good article on our company website that details KDB+ Memory Management: http://timestored.com/kdbGuides/memoryManagement

Urethra

Answer 4

http://code.kx.com/q4m3/12_Workspace_Organization/#125-expunging-from-a-context

I used a few different commands. As long as your table is stored on disk before you expunge it you should be OK.

This is the session before creating the table.

q).Q.w[]
used| 290192
heap| 67108864
peak| 67108864
wmax| 0
mmap| 0
mphy| 8589934592
syms| 629
symw| 20704

This command creates the table and then saves it to disk.

q)t:([]10000?"ab"; 10000?5)
q)save `t
`:t

The table is still in memory

q).Q.w[]
used| 437808
heap| 67108864
peak| 67108864
wmax| 0
mmap| 0
mphy| 8589934592
syms| 629
symw| 20704

Let's expunge the variable from memory and garbage collect.

q)delete t from `.
`.
q).Q.gc[]
0

Now the memory `used has been reduced to an amount similar to the start of the session.

q).Q.w[]
used| 290208
heap| 67108864
peak| 67108864
wmax| 0
mmap| 0
mphy| 8589934592
syms| 630
symw| 20730
q)\v
`symbol$()

Relativity