Webdock - Notice history

This was resolved some time ago, but we kept the issue open here as we were still troubleshooting. All systems are now operational as normal. The cause was some config changes made by our network team today in efforts to resolve minor issues, which unfortunately went wrong due to human error introduced to some config files.

This incident has been resolved. This was not a DOS attack but an internal malfunction in a monitoring tool. There was packet loss, but this has been resolved after we identified the issue.

All customers have been migrated away and everything is operational, except for two almalinux instances which are having problems booting as they have selinux active which is causing issues. Our engineers are working on these last two customer instances and they should hopefully be up and running soon. This incident is now resolved and we will be diagnosing the failed host over the next days to ascertain if it needs an RMA or parts can be exhanged before it rejoins our fleet. Thank you for your patience during the instability and repeated outages today.

The fix suggested by Dell support seems to have had the exact opposite effect on system stability: This time the system was up for only 10 minutes. This tells us that firstly the problem does indeed have something to do with the CPUs and their Bios-managed performance profiles on the system, and second that this is likely a hard hardware fault which we cannot work around.

This means we will be starting migrations of all customers away from this host shortly. You will receive emails notifying you of when migration starts and ends.

Once all customers are away from this system, it will be sent back to manufacturer for further diagnosis and replacement.

The suggested BIOS setting from Dell support has been implemented and all customers are up. If this does not resolve the issue, we will perform an evacuation of this system in the form of a migration of all customers. This in effect means you may see up to two more restarts / brief outages today. Hopefully it doesn't come to that and the issue is resolved now, but we have no way of guaranteeing a positive result.

Unfortunately the issue happened yet again. We did not have time to implement the fix as suggested by Dell support. We will perform a reboot of the affected system now, in order to apply the BIOS change. After which, as outlined earlier, if that doesn't have the desired effect of a stable system, we will proceed with migrations of all customers away from the affected system.

We have come up with a gameplan for how to deal with this particular system: We have found a BIOS setting which may prevent the issue from happening. If this host system spontaneously reboots again with that error, we will firstly try that fix. If we get a fourth reboot, it is clear that the system has some fault related to CPU, which we are unable to diagnose at this time. In this case we'd migrate all customers away from this host to our other healthier hosts, and at that point send in the system for an RMA. We hope it doesn't come to migrations - but rather you experience a last restart as your instance comes up on a good host, than unplanned reboots/downtime and arbitrary times. We will update here as the situation develops.

Unfortunately this system experienced another fault just and rebooted. We have identified a potential issue with one of the CPUs - we are still diagnosing. We apologize for the inconvenience.

This incident has been resolved. The host system decided to reboot. We are analyzing logs in order to determine the cause. All customer VPS instances are up.

The issue has been resolved. A strange sequence of events occurred at about 2AM this evening, which was ultimately caused by Cloudflare rate-limiting requests. This in turn caused our search indexer to break our search index. With a broken search indexer, all database listing filtering was failing causing exceptions to be thrown on many of our dashboard pages. Without going into too much detail, the resolution was simply to rerun the search indexer and we are now operational. We will be looking closely at how to prevent this from happening again.

This incident has been resolved and snapshot operations are confirmed to be working normally.