Abnormally high latency in Canada Friday 18th November 2022 18:19:48

We are seeing high latency in Canada and we are investigating the cause

The issue was tracked down to the processing cards on our firewalls being overloaded by abnormal traffic. We analyzed the cause and found it was misbehaving crypto workloads insinde our infrastructure spawning hundreds if not thousands of speedtest-cli instances. We shut down the offending servers and implemented auto-banning of these workloads on our infrastructure. We will look into limiting these traffic types on our firewalls to avoid this kind of instability in the future if it happens with other types of workloads.

It looks like up to 500ms of latency is being added by our firewalls at the moment. We are still investigating.