While working with the failed node just now, we noticed that a lot of network packets had started flowing in to our network causing high SIRQ - this is looking like a DOS attack and was probably the reason for the network kernel driver failure on the node that went down just now.
You may see degraded performance while we work through the issue with our networking team.
Webdock Datacenter FinlandHost node up and running
Host node has restarted and all VPS servers are running
The host node that went down had a fault in a kernel component that killed networking. Reinitializing that component is not possible without a restart, so the system has initiated a reboot now. We expect the system to come up in no more than 10-15minutes.
Webdock Datacenter FinlandSingle host node down
We have a host node down in Finland and are working through the issue.
24th October 2020
No incidents reported
23rd October 2020
Webdock Network infrastructure FinlandIPv6 Performance back to normal
IPv6 Performance is back to nominal in our Helsinki, Finland datacenter
22nd October 2020
Webdock Datacenter USAMigration has been completed
We are finally away from the bad host with all the clients that were on there. All IP ranges have been migrated and all looks good. The few customers who got a new IP address will receive an email from us shortly.
Webdock Datacenter USAUnit overheated again
As we were migrating data off the faulty unit, it overheated again and shut down. We are therefore down on that node right now. We are waiting for it to be powered up again so we can resume the migration.
Webdock Datacenter USAMaintenance postponed 1 hour to 22.00 CEST
The planned maintenance has been postponed by 1 hour to 22.00 CEST and not 21.00 as reported earlier
The maintenance/migration in the US will happen at 21.00 CEST as planned
Webdock Datacenter USAStill on track for maintenance at 21.00 CEST
We are so far still on track for our planned maintenance at 21.00 CEST. There is one issue we are working through, but we hope to have that solved soon after which we will email all affected customers informing them of the final maintenance window time.
Webdock Datacenter USAMaintenance Notification for our US Datacenter
As we continue to observe instability on the node which had a thermal issue, we have decided to migrate everybody away from that node. As the operation requires us to reconfigure our networking and move some IP ranges around, we have decided it is safest to disable any new provisioning in our US location until the maintenance procedure has been completed. We expect the maintenance to happen around 21.00 to 22.00 CEST tonight - if all preparations go well. All affected customers will be notified by email today regarding the procedure.
You can expect availability in our US Datacenter to come back up once the procedure is complete.
21st October 2020
Webdock Datacenter USANode back online
Node is back online and healthy. New fans and thermal paste was added to the system and we believe the hardware issue to be resolved. All customer VPS servers are back online.
Webdock Datacenter USAIssue seems to be overheating
After checking the server our onsite team concluded there seems to be an issue with overheating. Our team is in the process of reapplying thermal paste and checking the fans. Unfortunately, we do not have an ETA but this should be completed as soon as possible.
Webdock Datacenter USATest / Maintenance status on US node
The affected host node is being checked for hardware faults, this includes a defect check on the disks and a memory test in order to look for bad RAM. These tests may take a little while to complete.
Webdock Datacenter USAKernel Panic / Host system down in the USA
One of our hosts in the USA is having problems again and has experienced a kernel panic, likely due to a hardware failure. We are beginning to suspect there is a systemic hardware issue on that host node. We will do some debugging and repairs now which will result in a small amount of downtime as we reboot the system.
If this incident occurs one more time, we will move all customers to a brand new host node in order to avoid more downtime moving forward. We apologize for the repetition of this incident.
20th October 2020
No incidents reported
19th October 2020
Webdock Network infrastructure FinlandImprovements to IPv6 in Finland underway
As we continue to work on improving our networking in Finland, we will be setting up new routers over the coming days. If we expect any outage to occur, all clients will be notified beforehand.
The momentary issue with Cloudflare has been resolved
Whatever CF did, they fixed the API 500 crash error quite quickly and everything is operational as seen from our end.
Cloudflare is having problems
Cloudflare is having issues at the moment which means that after creating your server, SSL generation will fail and you cannot reach it using the default Webdock alias. Your server will work just fine and the IP is reachable, and after a while once Cloudflare has propagated your DNS everything will work - you will then need to re-run the SSL Certbot tool. it is unclear how long the delay is at the moment.
18th October 2020
No incidents reported
17th October 2020
Webdock Network infrastructure FinlandSlight delays in IPv6 connectivity in Helsinki Finland
As we work to improve our filtering measures and network in general in order to make it more resilient against the type of blockage that happened a few days ago, some users may see IPv6 connectivity to be a bit slow to come up. This is due to some technical changes that are taking place in router tables and firewall filtering. Our network team expects this delay to be gone no later than monday or tuesday as we complete the on-going upgrades.
The incident has now been resolved and the affected host node is up and operational. The issue turned out not to be the network infrastructure at all as first suspected, but instead a failed hardware component on the node in question. No data loss was experienced whatsoever as it was not the storage subsystem which was affected in this case.
Webdock Network infrastructure USADC Staff is investigating
Datacenter staff is currently investigating the fault. Remote management interfaces are down as well, so this seems like either a localized routing fault or possibly a power failure in the rack.
Webdock Network infrastructure USANetwork issues
We seem to be having network issues in our US datacenter. One of our hosts is unreachable at the moment. It looks like something localized in the DC.