Error when deploying with git (dokku) - timeout / broken pipe

Question

I use dokku with digital ocean for a while now without any problem

I have a problem now when deploying to dokku leveraging the following command:

git remote add dokku dokku@some-ip:myapp
git push dokku develop:master

I have the following problem:

Counting objects: 528, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (509/509), done.
packet_write_wait: Connection to some-ip port 22: Broken pipe
fatal: The remote end hung up unexpectedly
error: failed to push some refs to 'dokku@some-ip:myapp'

I try

• set ServerAliveInterval for the ssh client
• git config http.postBuffer 209715200 / git config ssh.postBuffer 209715200

but it doesn't work. I think it's because the size of the content to push is a bit high...

Thanks for your help!

Update

After set the postBuffer option to the value 209715200, I got the following error:

Counting objects: 528, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (353/353), done.
Timeout, server some-ip not responding.38.00 KiB/s   
fatal: The remote end hung up unexpectedly
error: failed to push some refs to 'dokku@some-ip:myapp'

The last progress hint I have before the upload hangs is:

Writing objects:  87% (466/531), 33.85 MiB | 120.00 KiB/s 

Mayda

Is sshd running at some-ip? Check connection to some-ip and to some-ip:22. – Mothball

@Mothball thanks very much for your answer! Yes I can connect to some-ip with ssh. When doing the git push, I saw some network traffic at the beginning, but this stops after an amount of time... – Mayda

What is an amount of time -- seconds, minutes or hours? – Ambry

@Ambry thanks very much for your answer! It's around 5 / 10 minutes. FYI I increased the postBuffer value and now got a timeout error... – Mayda

I guess that there is a config entry to change but I don't know which one and if it's an ssh or git / client or server... – Mayda

How much memory do you have? The docs mentioned insufficient memory may lead to git push errors: dokku.viewdocs.io/dokku/getting-started/advanced-installation/… – Stockholder

@Stockholder thanks very much for your answer! I have 1Go. I had this problem when I started to use DigitalOcean / Dokku... ;-) – Mayda

Answer 1

Firstly, this

    git config http.postBuffer 209715200 / git config ssh.postBuffer 209715200

Won't make any difference, as long as you are pushing over SSH(port 22):

packet_write_wait: Connection to some-ip port 22: Broken pipe

Your error has nothing to do with configuration, but with bad internet connection/low bandwidth somewhere between you and destination point, which results in big size packets being broke in tcp and lost.

It's extensively discussed here. Try:

1) switching to http (this will increase size limit for brake due to a bit smaller header overhead)

2) decreasing repo size (repack or gc --aggressive)

However, this will work, only if you are on the verge of getting it there. If your connection is really bad you can't physically transfer big( also because containing multiple headers) packets of data, then the only fix:

Increase your bandwidth (switch to LAN, change network etc). It may be a requirement both on client and receiving end.

Bursitis

Answer 2

This generally occurs when the server runs out of memory. You can either add more ram to your server or setup swap space. The follow script will create 2GB of swap space.

sudo install -o root -g root -m 0600 /dev/null /swapfile
dd if=/dev/zero of=/swapfile bs=1k count=2048k
mkswap /swapfile
swapon /swapfile
echo "/swapfile       swap    swap    auto      0       0" | sudo tee -a /etc/fstab
sudo sysctl -w vm.swappiness=10
echo vm.swappiness = 10 | sudo tee -a /etc/sysctl.conf

Also try Git Garbage Collection and Clearing Application cache.

Cringe