Sidekiq worker is leaking memory

Question

Using the sidekiq gem - I have sidekiq worker that runs a process (git-tf clone of big repository) using IO.popen and tracks the stdout to check the progress of the clone.

When I am running the worker, I see that sidekiq memory is getting larger over the time until I get kernel OOM and the process get killed. the subprocess (java process) is taking only 5% of the total memory.

How I can debug/check the memory leak I have in my code? and does the sidekiq memory is the total of my workers memory with the popen process?
And does anyone have any idea how to fix it?

EDIT
This is the code of my worker - https://gist.github.com/yosy/5227250

EDIT 2
I ran the code without sidekiq, and I have no memory leaks.. this is something strange with sidekiq and big repositories in tfs

Divot

Answer 1

I didn't find the cause for the memory leak in sidekiq, but I found a away to get a way from sidekiq.

I have modified git-tf to have server command that accepts command from redis queue, it removes lot of complexity from my code.

The modified version of git-tf is here: https://github.com/yosy/gittf

I will add documentation later about the sever command when I will fix some bugs.

Divot