git out of memory on checkout

Question

I have cloned a large repo and got an error (after several attempts)

Clone succeeded, but checkout failed

When trying to fix this with

git checkout -f HEAD

an error comes back

Fatal: Out of memory, realloc failed2

I've already set some memory limits higher because the cloning also ran into problems by setting

git config pack.WindowMemory 256m && git config pack.packSizelimit 256m

Based on advice below from Punit Vara (below) I've also edited the .git/config to:

[core]
packedGitLimit = 128m
  packedGitWindowSize = 128m

[pack]
  deltaCacheSize = 128m
  packSizeLimit = 128m
  windowMemory = 128m

And I've tried changing these values to: 128m, 256m, 512m, 1024m. This didn't work for me. I still get the same error that seems to appear at 41%.

Anyone has experience with this or any idea where this is going wrong and/or what can be done to solve this? Thanks.

Hachmin

#10293403 – Premillennial

Thanks, but that didn't solve it. I tried different values i.e 256m 512m and 128m but I still get the same error.The error does seem to be always at the same point, 41%. – Hachmin

I guess it's time to git fsck. I hope that this repository doesn't contain unique data. – Forbear

Git fsck: 100% directories, 100% objects. – Hachmin

Since you are not packing but just reading the pack.* config values should not matter. Which version of git it was and have you tried the latest one? Sounds like something worth reporting to authors – Prakrit

How large is the repo? I've heard it's pretty well known that git doesn't work well past a certain repo size, which is why common practice to limit what goes into the repo (no binaries, use git lfs or rsync, etc.). – Raber

Answer 1

I encountered the same annoying problem after my server was updated to 64Bit architecture. The OS memory limit for git was at 600m.

core.preloadIndex = false

finally did the trick for me. It defaults to true since git version 2.1

Wove

Answer 2

Try this:

git gc --auto --prune=today --aggressive 
git repack 
git config --global http.postbuffer 524288000 
git config --global pack.windowMemory 256m

Found on git push Out of memory, malloc failed.

Barger

Thanks for the post. Tried this just now. Didn't work. – Hachmin

Answer 3

I got this error message: "Fatal: Out of memory, realloc failed2 " when trying to use git add --all. The reason is that I attempted to add a very large csv file (>1.6 GB). Git/Github does not allow to upload such a large file. As a solution you can put the file on gitignore or move the file to another directory not connected to git.

Alameda

Answer 4

Set the following parameters to fix this issue:

core.packedGitWindowSize

Number of bytes of a pack file to map into memory in a single mapping operation. Larger window sizes may allow your system to process a smaller number of large pack files more quickly. Smaller window sizes will negatively affect performance due to increased calls to the operating system’s memory manager, but may improve performance when accessing a large number of large pack files.

Default is 1 MiB if NO_MMAP was set at compile time, otherwise 32 MiB on 32 bit platforms and 1 GiB on 64 bit platforms.
This should be reasonable for all users/operating systems. You probably do not need to adjust this value.

core.packedGitLimit

Maximum number of bytes to map simultaneously into memory from pack files.
If Git needs to access more than this many bytes at once to complete an operation it will unmap existing regions to reclaim virtual address space within the process.

Default is 256 MiB on 32 bit platforms and 8 GiB on 64 bit platforms.
This should be reasonable for all users/operating systems, except on the largest projects.
You probably do not need to adjust this value.

---

But sometimes you do need to set those values

Here is a complete list of config values which i can think of and can be relevant to your case :

[http]
    postbuffer = 524288000
[pack]
    threads = 1
    deltaCacheSize = 1024m
    packSizeLimit = 1024m
    windowMemory = 1024m
[core]
    packedGitLimit = 1024m
    packedGitWindowSize = 1024m

Antimony