Git gc using excessive memory, unable to complete

Question

Final update and fix: The solution here turned out to be a combination of two things: using Windows Git rather than Cygwin Git as Graham Borland suggested, and the Git config settings pack.threads = 1 and gc.aggressiveWindow = 150.

I have a large local Git repository, a git svn clone of an SVN repository with about 40,000 commits. I'm trying to run git gc over this repository, and getting nowhere:

$ git gc --auto
Auto packing the repository for optimum performance. You may also
run "git gc" manually. See "git help gc" for more information.
Counting objects: 25966, done.
Compressing objects: 100% (25249/25249), done.
fatal: Out of memory, malloc failed (tried to allocate 426523986 bytes)
error: failed to run repack

I'm running Git 1.7.5.1 inside Cygwin on a 64-bit dual-core Win7 machine with 4GB RAM. The .git directory is currently a little over 6.1GB.

I've tried running git gc --aggressive, to see if the more complete system is able to fix it, but no luck: I get a similar message to the above, with the same size attempted malloc, but a considerably higher object count (508,485 counted, 493,506 compressed).

I've also tried—as suggested by Google—assorted twiddles to the [pack] part of my .gitconfig file; the most complete being from another StackOverflow question. My .gitconfig now has the following relevant lines, but setting these appears to have made no difference:

[pack]
        windowMemory = 16m
        threads = 1
        window = 1
        depth = 1
        deltaCacheSize = 1

Any suggestions on how I can get git to gc my repository?

Edit: Mark Longair suggested some more .gitconfig file changes. Which I made, new lines below. But the changes made no difference whatsoever.

[core]
        packedGitWindowSize = 1m
        packedGitLimit = 256m
[pack]
        packSizeLimit = 128m

Edit 2: Michael Krelin suggested increasing the swap/page file size (WinXP instructions here, and it's similar for Win7). I tried that, but it made no difference, and indeed I only increased the maximum size available, and it looks as if Windows never tried to increase the size of the page file it was using.

I'm now looking at whether this was caused by a memory limit within or imposed upon Cygwin. To check "imposed upon", I'm trying running Cygwin with administrator privileges. To check "within" (which looks more likely), I'm having a play with Cygwin's maximum memory settings.

Edit 3: Much though I may prefer using Cygwin, it turns out the Windows Git client deals with the memory issue just fine. Seems I'll be falling back to that every so often when my repository needs a tidy.

Wedlock

Answer 1

You might have more luck running a native Windows client such as msysGit, rather than trying to do it inside Cygwin.

Plumper

My experience of Windows Git clients has generally been that they handle memory management even worse—I've been able to get Cygwin Git to handle a ~15,000 commit SVN repository that the Windows Git client I tried barfed on. Still, worth a shot, I guess! – Wedlock

Whee! Much as I may not like using the Windows Git client normally, that worked just fine. Thank you! – Wedlock

Answer 2

I had the same problem, tried the solutions mentioned so far without success. But my problems with git gc began after I added big image files to repo, so I created .gitattributes file and turned off delta compression for those big files:

*.tga -delta
*.psd -delta

It worked.

Stationary

ZOMG it worked! I've been having trouble with large files in my git repositories on a shared host which kills processes that take up too much memory. THIS SOLVED A PROBLEM I'VE HAD FOR YEARS :DDDD – Spiculate

After a lot of trial and error, sometimes getting it working and then after a few commit have the repo break again, this brought the solution (so far). I think this answer should be higher up. – Sugared

Answer 3

Some other config options that you might want to try restricting to lower than default values include:

• pack.packSizeLimit
• core.packedGitWindowSize
• core.packedGitLimit

... all of which are documented in the git config documentation. It's particularly worth checking in each case what units are understood, which I've made mistakes with in the past.

Bartie

There is no core.deltaCacheSize. I suspect you're referring to pack.deltaCacheSize, which I've already been adjusting. I'll try the rest of those out now, though, thanks! – Wedlock

@me_and: oops - I've removed that one from my answer – Bartie

No luck. I'll update the question with the new config I used, but the short version is that none of your suggested changes seemed to make any difference to the amount of memory git gc tried to allocate. – Wedlock

Answer 4

The only thing that help to avoid this error on shared Linux hosting was to add

  [pack]
    packSizeLimit = 64m
    threads = 1

to

.gitconfig

Most important was "threads = 1"

Medellin

Answer 5

Maybe temporarily adding a swap file bigger than life and going for a few cups of coffee elsewhere will help?

Avernus

If I don't get anywhere today, I'll reboot my computer with a massive page file this evening and leave it going overnight… – Wedlock

Oh, I didn't realize you were on windows — I assumed adding swap file is a matter of a few commands in shell. Well, I hope it's still possible on windows, but you know better how to do that ;-) – Avernus

No luck. In fact, I increased the available page file space, but Windows didn't decide to use it, which implies to me that there may be some Cygwin memory limit issue. I'm investigating that now… – Wedlock

Maybe cygwin implements ulimit? (I have no idea) – Avernus

There does indeed seem to be something similar: there's a Windows registry setting for Cygwin's maximum memory usage. More experimenting is occurring as I type! – Wedlock