20th May 2020

SSH Connection Fix may not have solved the issue completely

It seems, as we continue to watch the infrastructure that the fix we reported earlier today to our SSH connection issues may not have completely cleared up the issue. Although the vast majority of users can connect, in some cases delays of up to 20 seconds or more before the connection is successful has been observed. The issue seems to still be the PAM subsystem which has trouble completing authentication. The delay seems directly tied to load on the host system and we now suspect that recent kernel tweaks implemented to speed up the disk I/O system may be at fault.

As we continue to venture down the kernel rabbit hole we ask our customers to be patient with us and we promise to resolve this as quickly as is possible. In the meantime, if you are experiencing delays but can connect, you can try to disable PAM authentication (should be OK to do on most stacks) by editing /etc/ssh/sshd_config and setting "UsePAM no" and then restarting your ssh daemon with systemctl restart sshd

If you cannot connect at all, be in touch with support and we will try to restore your connectivity while we work through the issue. We are hitting the night cycle here in Europe, so we will not be working on this full strenght till morning.

We hope to update you all with the good news of a permanent resolution tomorrow at the latest.