We have a single host which experienced a hardware fault after a Remote Hands engineer accidentally popped out the wrong disk drive while doing some maintenance. The system could not recover from this human error and started freezing up. The system was unable to send alerts about its condition so it took a while before external monitors spotted the issue as load avg. became very large. Once we diagnosed the issue we initiated the reboot. All customer vps servers on that machine should be online soon once the reboot and startup sequence completes.
The host has been rebooted, all VPS servers and sites are coming up again and everything is OK
12th June 2021
Webdock Network infrastructure FinlandMicro outages and network instability in Helsinki
We have seen a lot of micro outages in the last 12 hours - a few packages being dropped here and there, essentially "noise" on the network - due to an extended DOS campaign against several of our customers. Our network team has mitigated most of this activity and is still monitoring the situation.
We now believe, after this latest incident this past hour, that we have gotten things locked down and under control. We are implementing new measures to avoid this type of incident in the future. Thank you for your patience today.
We think we have found a good way of dealing with the DOS issues at last. We are not calling the incident resolved however as we need to closely monitor the network for the next few hours and see how things evolve.
This issue is on-going and you can expect more micro outages throughout today as we coordinate with our upstream network providers to shut down this DOS compaign completely. We hope to be back up to normal operational stability by the end of today. It should be noted that for the vast majority of our clients this has no consequence as websites are still accessible pretty much as they always have been. It's mostly uptime monitoring tools which will show small outages, depending on the frequency of checks. What we are seeing is few packages being dropped in small "blips" of outage when various buffers and networking hardware CPU reach very high load.