Recently I've started seeing a lot of timeouts when deploying to one of my EC2 servers. After some investigating I narrowed the problem down to
git ls-remote and
This works 100% of the time:
eval `ssh-agent -s` && ssh-add key.pem && git ls-remote -h [email protected]:repo
This hangs 90% of the time:
GIT_SSH=wrapper.sh git ls-remote -h [email protected]:repo
This happens only on some of the servers. I've used the same wrapper for many projects and never had problems with it. I just setup Capistrano with
forward_agent that uses a different wrapper and it's also failing.
p.s. The wrapper, for the sake of completeness:
/usr/bin/env ssh -o "StrictHostKeyChecking=no" -i "/path/to/key.pem" $1 $2
p.p.s. The original version of git was 1.9.1. Updating to 2.1.1 didn't help.
Adding -v to the wrapper script allowed me to find the culprit:
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
Seems like a bug: https://bugs.launchpad.net/ubuntu/+source/openssh/+bug/1254085
Specifying cipher with
-c 3des-cbc seems to solve the problem with custom wrapper but it doesn't solve my problem with Capistrano.
Answering my own question.
To fix a single command or script specify a cipher used by SSH:
ssh -c 3des-cbc ...
To fix ssh for good, specify which ciphers you want to use by editing
Could probably also be fixed by altering MTU, as per https://bugs.launchpad.net/ubuntu/+source/openssh/+bug/1254085 but I prefer to edit ssh config.
Although I initially fixed it by changing the cipher it did turn out the underlaying problem was the MTU. Even though the above solution fixes the problem with SSH many SSL connections are also affected by it. Below I present a solution that solved all of them once and for all.
Set the MTU to something lower. Default for Ethernet is 1500, but I kept experiencing problems until I lowered it to 1468
sudo ip link set dev eth0 mtu 1468
If the above solves the problem, add the following two lines to
/etc/dhcp/dhclient.conf to make sure reboot doesn't reset the setting:
default interface-mtu 1468; supercede interface-mtu 1468;