Some systems are experiencing issues

Stickied Incidents

3rd December 2022

Webdock Datacenter Montreal Short interruption in service this evening

It seems we had a short interruption in service in Canada this evening where we blinked for about 10 minutes as a network device crashed. Usually it should not take this long for the other members of our network cluster to take over, but something prevented the spare unit from taking over immediately. We are investigating the cause.

30th November 2022

Webdock Network Infrastructure Montreal Minor latency spiking in Canada this evening

We have a bit of latency spiking in Canada this evening. We see spikes every 5-10 seconds that go up to 500ms+ but we have no packet loss and all servers are reachable from all parts of the world. We are checking to see if these spikes are on our end or with our ISP and if they are on our end we will see what we can tune on our equipment to eliminate these.

  • These latency issues have resurfaced in Canada this morning. We are aware of the issue and are investigting with our ISP why this has come back.

  • Our team has tuned some things and we see fewer and smaller latency spikes on our end now. If you run an MTR test and see that your traffic is routed through Zayo, this also seems to have a negative impact on latency as the Zayo network has been having issues lately. Our ISP informs us they will be dropping Zayo soon, which we hope will solve these performance issues outside our network. Customers routed through Cogent should see no issues at this time.

  • Past Incidents

    3rd December 2022

    Webdock Datacenter Montreal Short interruption in service this evening

    It seems we had a short interruption in service in Canada this evening where we blinked for about 10 minutes as a network device crashed. Usually it should not take this long for the other members of our network cluster to take over, but something prevented the spare unit from taking over immediately. We are investigating the cause.

    30th November 2022

    Webdock Network Infrastructure Montreal Minor latency spiking in Canada this evening

    We have a bit of latency spiking in Canada this evening. We see spikes every 5-10 seconds that go up to 500ms+ but we have no packet loss and all servers are reachable from all parts of the world. We are checking to see if these spikes are on our end or with our ISP and if they are on our end we will see what we can tune on our equipment to eliminate these.

  • These latency issues have resurfaced in Canada this morning. We are aware of the issue and are investigting with our ISP why this has come back.

  • Our team has tuned some things and we see fewer and smaller latency spikes on our end now. If you run an MTR test and see that your traffic is routed through Zayo, this also seems to have a negative impact on latency as the Zayo network has been having issues lately. Our ISP informs us they will be dropping Zayo soon, which we hope will solve these performance issues outside our network. Customers routed through Cogent should see no issues at this time.

  • 25th November 2022

    Webdock Network Infrastructure Montreal Network issue in Canada

    In what is becoming a rather tiring last couple of days with sporadic downtime due to continued DOS attacks we are once again seeing an issue with our network where our equipment is being overloaded causing downtime in Canada. Rest assured our networking team is hard at work resolving the issue.

  • We are flagging the network as operational as it has run without issues over the weekend.

  • We are online but as we have marked this as resolved so many times today without it actually sticking, then this time we will keep the issue open here until we have seen at least 24 to 48 hours of clean operation of our network.

    We will of course update here if we have further outages.

  • As we had feared it was indeed famous last words and we have an issue with the network yet again in Canada. We apologize profusely and assure you that we are getting rather annoyed by this recurring situation we are in. This is not acceptable and we will hopefully find a permanent resolution this time.

  • Apparently this downtime was due to human error: Our networking team had forgotten to apply the new security configuration on our firewalls which we had worked so hard to develop over the last couple of days and which was proven to be working for Finland. As soon as the config was applied and the malicious sessions cleared we came back online.

    We sincerely apologize for this sporadic downtime these last couple of days and thank you for your patience. We are now reasonably confident that these network issues should be over. Let's hope these are not famous last words ;)

  • Webdock Datacenter Montreal Single server in Canada needs restart

    A host in Canada has encountered a kernel issue and in order to prevent a sudden crash or issues for our customers we will be upgrading the kernel and rebooting the system. The system is a Ryzen system so we expect downtime to be minimal as they are extremely quick at restarting.

  • The system was upgraded and fully rebooted in less than 2 minutes. We apologize for any inconvenience caused during this minor downtime.

  • Webdock Network Infrastructure Finland High latency and packet loss in Finland

    While Canada has been OK since last night, just now we are seeing high latency and some packet loss in Finland. We are investigating.

  • Our network team now believes all required configuration is in place and that these events we have seen today are over. We will continue to keep a watchful eye. The networking team reports we should now be much better protected and will not suffer to the same extent during any future DOS event like we saw today and yesterday in Canada. We would like to thank all our customers for their patience during these issues.

  • Higher than normal latency has kicked in throughout the day and may continue for some time yet. Partially due to intermittent testing by our network team where we are intentionally stressing our equipment, and partially due to misbehaving workloads surfacing from time to time. We will update here once we believe we are in a stable state again.

  • Latency is back to normal low levels now after a lot of work by our network team. We are not finished however and cannot rule out more disruption today. We will update here if anything like that happens, of course.

  • Our network team has tweaked a number of things and packet loss is almost non-existent at this time. However, we are still seeing relatively high latency up to 10X from normal adding up to 200ms at the last hop. Our team is working hard to correct this isue.

  • 24th November 2022

    Webdock Network Infrastructure Montreal Network issue in Canada

    We have a network outage in Canada. We are investigating the cause.

  • We are operational yet again and have been for about 12 minutes. We are unfortunately still not confident we have solved the core issue here, and we are worried we will have another outage. The team is hard at work trying to get our firewalls to behave the way we want them to, as what's happening is that they are essentially failing to mitigate the DOS attack, despite being plenty capable of doing so in theory.

  • And we have another outage unfortunately. As before we are working to mitigate best we can.

  • We are operational again, but we will obviously continue working on this and keep a close eye on all traffic for the time being. We hope the issue has now been solved.

  • Although our initial mitigation has worked in limiting the amount of open sessions we are allowing through our network, our equipment is being overwhelmed by the sheer amount of packets being thrown at us, which is at this time about half a billion packets per second. Our team is working to mitigate the attack of course. We hope we have a resolution soon.

  • Seems like we are not out of the woods yet. We have another network outage in Canada. Likely it is another attack. We are investigating.

  • We were indeed hit with a rather large DOS attack and our firewalls were overwhelmed before they could block the incoming packets. We are adjusting our thresholds in order to protect our network from this type of event in the future. We expect to be able to fully mitigate this type of event once our new configuration is live.

  • The issue has already been solved but we are still determining the root cause. We are uncertain at this point what happened but our network equipment became overloaded with hundreds of thousands of connections for almost 30 minutes. This may have been a DOS attack of some description but we are still looking into it.

  • Webdock Network Infrastructure Montreal Intermittent high latency reaching google.com possibly others

    We are seeing regular spikes in latency reaching google.com from our Canada location. We are investigating the cause and trying to determine if it affects both ipv4 and ipv6, whether it happens with a particular upstream provider of ours and whether other destinations are equally affected.

  • The issue has been resolved and high latency is no longer a problem. We apologize for any inconvenience caused.

  • Our network team confirms the issue affects both ipv4 and ipv6 and all destinations, not just google. The issue seems to start at our equipment but gets worse once it hits our ISP, so we are still working on determining the cause and where the issue is located exactly.