fatal: protocol error: bad line length 819264.11 MiB/s

Question

I forgot to push my changes to a project I have been working on incrementally for months. This has resulted in an extremely large .git directory (~11 GB). The rest of the files in the repo combine for much less than that. I would like to push all the commits to github, if not at one time then incrementally. Thus, I am trying to push the commits one by one in order stay under the limits for each push.

My first commit's (~5.5 GB) push is failing and giving me the error fatal: protocol error: bad line length 819264.11 MiB/s.

I find somewhat similar messages when googling, but they all refer to a particular character error. I cannot seem to understand what this error I am getting is.

I am using Git 2.22.0 and macOS 10.15.4; this was pushed over HTTPS although I tried ssh and it ran into other problems, so I switched back. I had also run into another problem right before this. The steps I took before this are in more detail there.

Terminal Output with SSH:

client_loop: send disconnect: Broken pipeB | 1.39 MiB/s   
fatal: the remote end hung up unexpectedly
fatal: the remote end hung up unexpectedly

Terminal Output with HTTPS:

Enumerating objects: 667, done.
Counting objects: 100% (667/667), done.
Delta compression using up to 8 threads
Compressing objects: 100% (661/661), done.
fatal: protocol error: bad line length 819264.11 MiB/s   
error: failed to push some refs to 'https://github.com/USER/REPO.git'

Joanajoane

Why is your first commit 5.5 GB in size? What are you trying to push? Do you have generated files in your repository? – Fauteuil

Also, are you pushing over SSH or HTTPS? What version of Git are you using? What OS? Can you include more of the push output in your question as a code block? – Fauteuil

@Fauteuil no, I had just started the repo a little late into development and committed the entire block without breaking it up. I could probably break it up, but it doesn't seem to be a problem. I also added the relevant details to the question. – Joanajoane

@matt I'm unsure why you think this would be one file. It is not. – Joanajoane

No, I don't think it's any file. That was just a response to the way you worded the question, so I may have been misled. The related posts that I can find say it has nothing to do with your file length, but rather with something about remote communication. – Blume

@Blume Sorry, I am new to SO and was encouraged to keep questions small. This seemed to be out of context. Added the output for SSH, although there is a chance I misconfigured SSH. – Joanajoane

Have you done the usual git config http.postBuffer fix? – Blume

Let us continue this discussion in chat. – Joanajoane

By the way, another thing I would try is pushing the commits one at a time, and if necessary breaking up the large first commit into multiple commits. This would not substantially damage the integrity of your history, and it would be worth it just to be able to get the stuff uploaded. – Blume

try this configs: git config --global http.postBuffer 500M git config --global http.maxRequestBuffer 100M git config --global core.compression 0 Specifically the last one worked for me – Lindon

Answer 1

The actual message is:

fatal: protocol error: bad line length 8192

which merely has been written over an ongoing update that keeps printing the transfer rate. The last printed rate ended with ...61.11 MiB/s.

It's not clear what is causing the bad packet. I'd suggest switching to ssh, which is more likely to have solve-able problems. However, you could set the packet tracing option:

GIT_PACKET_TRACING=1 git push ...

so that you get packet-by-packet tracing, and then someone (you, probably) could spend a lot of time figuring out why they're sending a longer line than your Git is willing to receive.

Note that GitHub will not accept a file larger than 100 MB. See Repository size limits for GitHub.com.

Elevator

Switching to SSH solved a similar issue that I faced. Thanks! – Incurve

Thanks for the repository size limit on github. It would have been nice if the error were more indicative of the cause of the problem. – Parkinson

Answer 2

What kind of files do you have? If you have several large files, you can also check whether git lfs solves the problem. To do this, you must first install git lfs.

1. On Mac, for example, run brew install git-lfs.
2. Then git lfs install.
3. Specify which large files you want to track: with git lfs track "*.safetensors".
4. Add the created gitatributes file to the index: git add .gitattributes
5. Commit .gitattributes-File: git commit -m "Add gitattributes for git lfs"
6. Migrate the already commited files with git lfs migrate import --include="*.safetensors"

Here is a tutorial for migrating Files to git lfs: https://github.com/git-lfs/git-lfs/wiki/Tutorial#migrating-existing-repository-data-to-lfs

Moonscape