Git on Windows, "Out of memory - malloc failed"

Question

Have run into a problem with repository and tried almost every possible config setting found out there eg. pack.WindowMemory etc etc

I believe someone has checked in a large file to remote repository and now each time I try and pull or push to it, GIT tries to pack it and runs out of memory:

Auto packing the repository for optimum performance. You may also
run "git gc" manually. See "git help gc" for more information.
Counting objects: 6279, done.
Compressing objects: 100% (6147/6147), done.
fatal: Out of memory, malloc failed (tried to allocate 1549040327 bytes)
error: failed to run repack

Have tried git gc & git repack with various options but keeps returning same error.

Almost given up and about to just create a new repo but thought I'd ask around first :)

Twocolor

Maybe related. Have you managed to solve yours, tho? – Spectroscope

In our case, we solved it with my link on a 7 gb repo. My suggestion, if you can't go with GIT, is trying Hg or Perforce. – Spectroscope

I think it's a bug in MsGit rather than an issue with Git itself. Don't think it warrants moving away from Git altogether either, especially not to Mercurial! – Twocolor

if so, can't u get an ubuntu pen drive just to try it? – Spectroscope

an update, I've made a new repository with new files, which has 3gb and nothing from the link solved in my case here... Only thing that did help was git config --global pack.threads 1 cited below. That worked. But anyway... Have you tried the ubuntu pen drive? – Spectroscope

possible duplicate of Git gc using excessive memory, unable to complete – Renfro

Answer 1

I found a solution Here that worked for me.

In .git/config file (client and/or server) I added the following:

[core]
  packedGitLimit = 128m
  packedGitWindowSize = 128m

[pack]
  deltaCacheSize = 128m
  packSizeLimit = 128m
  windowMemory = 128m

Head

It looks like msysgit does not use the correct default values for 32 bit if run on an 64 bit machine. However it should, as the client is 32 bit. These values worked for me: [core] packedGitLimit = 256m packedGitWindowSize = 32m [pack] deltaCacheSize = 256m packSizeLimit = 1g windowMemory = 256m – Kinney

I had similar problem and in my case disabling the delta compression for zips and a custom binary compressed file worked. In .git/info/attributes, I've added "*.zip binary -delta". – Rothstein

128m seems a bit low. is that per-thread? if so it does make a lot more sense, as if 32-bit git were to allocate more than 4096m across all threads, of course it would crash – Overbold

Helped on Linux too, for git lfs, when average file size is ~100 mb, commit size is ~8gb. – Koehler

Answer 2

For reference (you might already seen it), the msysgit case dealing with that issue is the ticket 292.

It suggests several workarounds:

• Disable delta compression globally. For this you have to set pack.window to 0. Of course this will make the repository much larger on disc.
• Disable delta compression for some files. Check the delta flag on the manpage to gitattributes.
• git config --global pack.threads 1
• git config --global pack.windowMemory 256m (you already tried that one, but also illustrated in "Error when pulling warning: suboptimal pack - out of memory")
• other settings are mentioned in "git push fatal: unable to create thread: Resource temporarily unavailable" and "Git pull fails with bad pack header error" in case this is pack-related.
• sm4 adds in the comments:

To disable the delta compression for certain files, in .git/info/attributes, add:

*.zip binary -delta

From Gitattributes man page:

Delta compression will not be attempted for blobs for paths with the attribute delta set to false.

---

Maybe a simpler workaround would be to somehow reset the history before that large file commit, and redo the other commits from there.

Gunnery

Unfortunately already tried most of these without any luck, thank you though +1ed – Twocolor

git config --global pack.windowMemory 256m worked for me – Holomorphic

To disable the delta compression for certain files, in .git/info/attributes, add "*.zip binary -delta" – Rothstein

Answer 3

EDIT:  Since git-v2.5.0 _^(Aug/2015), git-for-windows (formerly MSysGit)
      provides 64-bits versions as noticed by Pan.student.
      In this answer I was advising to install Cygwin 64-bits (providing 64-bits Git version).

---

I got a similar Out of memory, malloc failed issue using MSysGit when reaching the 4GB barrier:

> git --version
git version 1.8.3.msysgit.0

> file path/Git/cmd/git
path/Git/cmd/git: PE32 executable for MS Windows (console) Intel 80386 32-bit

> time git clone --bare -v ssh://linuxhost/path/repo.git
Cloning into bare repository 'repo.git'...
remote: Counting objects: 1664490, done.
remote: Compressing objects: 100% (384843/384843), done.
remote: Total 1664490 (delta 1029586), reused 1664490 (delta 1029586)
Receiving objects: 100% (1664490/1664490), 550.96 MiB | 1.55 MiB/s, done.
Resolving deltas: 100% (1029586/1029586), done.
fatal: Out of memory, malloc failed (tried to allocate 4691583 bytes)
fatal: remote did not send all necessary objects

real    13m8.901s
user    0m0.000s
sys     0m0.015s

Finally git 64 bits from Cygwin fix it:

> git --version
git version 1.7.9

> file /usr/bin/git
/usr/bin/git: PE32+ executable (console) x86-64 (stripped to external PDB), for MS Windows

> time git clone --bare -v ssh://linuxhost/path/repo.git
Cloning into bare repository 'repo.git'...
remote: Counting objects: 1664490, done.
remote: Compressing objects: 100% (384843/384843), done.
remote: Total 1664490 (delta 1029586), reused 1664490 (delta 1029586)
Receiving objects: 100% (1664490/1664490), 550.96 MiB | 9.19 MiB/s, done.
Resolving deltas: 100% (1029586/1029586), done.

real    13m9.451s
user    3m2.488s
sys     3m53.234s

FYI on linuxhost 64 bits:

repo.git> git config -l
[email protected]
core.repositoryformatversion=0
core.filemode=true
core.bare=true

repo.git> git --version
git version 1.8.3.4

repo.git> uname -a
Linux linuxhost 2.6.32-279.19.1.el6.x86_64 #1 SMP Sat Nov 24 14:35:28 EST 2012 x86_64 x86_64 x86_64 GNU/Linux

If my answer does not fix your issue, you may also check these pages:

• git clone out of memory even with 5.6GB RAM free and 50 GB hard disk
• Git clone fails with out of memory error - “fatal: out of memory, malloc failed (tried to allocate 905574791 bytes) / fatal: index-pack failed”
• git-clone memory allocation error
• MSysGit issues tracker

Anselmi

I have used 64bit git from git-scm.com to solve out of memory error when checking out branch on large repositary – Photima

installing the 64bit version solved the “Out of memory - malloc failed” error when I was doing a rebase in a repository with "large" "*.lib" files (20-100mb+ each one). – Betake

Answer 4

Some of the options suggested in the selected answer seem to be only partially relevant to the issue or not necessary at all.

From looking at https://git-scm.com/docs/git-config, it appears that just setting the following option is sufficient (set only for the project here):

git config pack.windowMemory 512m

From the manual:

pack.windowMemory

The maximum size of memory that is consumed by each thread in git-pack-objects[1] for pack window memory when no limit is given on the command line. The value can be suffixed with "k", "m", or "g". When left unconfigured (or set explicitly to 0), there will be no limit.

With this, I never went over the specified 512m per thread, actually used RAM was about half of that most of the time. Of course, the amount chosen here is user-specific depending on the available RAM and number of threads.

Waldemar

Answer 5

I came across this error message on Ubuntu. In my case a large committed file was causing issues. I realized the issue was due to a file that I hadn't noticed initially.

Identify Large Files:

To identify large files in the Git history, you can use the following command:

git rev-list --objects --all | git cat-file --batch-check='%(objecttype) %(objectname) %(objectsize) %(rest)' | sed -n 's/^blob //p' | sort --numeric-sort --key=2 | cut -c 1-12,41- | $(command -v gnumfmt || echo numfmt) --field=2 --to=iec-i --suffix=B --padding=7 --round=nearest

This command lists the objects in the Git history, sorts them by size, and displays the file sizes in a human-readable format.

Source: How to find/identify large commits in Git history

Remove Large File:

Once identified, you can remove the large file from the Git history using the following command:

git filter-branch --force --index-filter 'git rm --cached --ignore-unmatch path/to/my/file.jpg' --prune-empty --tag-name-filter cat -- --all

This command filters the branch history and removes the specified file.

Source: Removing large files from Git history

Update .gitignore:

Afterwards, make sure to either delete the file or add it to .gitignore:

echo 'path/to/my/*' >> .gitignore

Remember to add, commit, and push the changes to reflect them in your remote repository.

Hope this will help.

Labor

Answer 6

In my case the config changes above didn't help. What helped was a simple server reboot (in my case sudo shutdown -r now). Looks like something was eating lots of memory on the server. Hope this helps someone, too.

Glum

Answer 7

This worked for me, but I had to set the options via the command line using:

git --global core\pack [param] value

Lepore