Affected Components:

Oct 9, 07:36:04 GMT+0
Resolved - The incident is resolved.

Oct 9, 07:31:15 GMT+0
Resolved - Instances on the host have trouble starting. The host needs a reboot. There will be downtime for 5 - 10 minutes. Sorry for the inconvenience..

Affected Components:

Oct 7, 12:55:02 GMT+0
Resolved - This is resolved. If you still see issues, please reach out to support..

Oct 7, 12:29:56 GMT+0
Resolved - We noticed issues within our infra. Dashboard actions like provisioning, rebooting and such are hanging. We are investigating this..

Affected Components:

Oct 1, 11:56:54 GMT+0
Resolved - The incident is resolved (kernel crash). Sorry for the inconvenience.

Oct 1, 11:52:46 GMT+0
Resolved - This incident has been resolved..

Duration: 14 days, 17 hours and 9 minutes

Affected Components:

Sep 28, 14:41:28 GMT+0
Completed - Maintenance has completed successfully..

Sep 28, 14:30:01 GMT+0
Completed - Maintenance is now in progress.

Sep 28, 08:36:52 GMT+0
Identified - We are upgrading the software on our backup cluster. This is a regular software upgrade. Snapshots and backups will be unavailable for about 1 hour between 16.30 and 17.30 CEST..

Affected Components:

Sep 27, 08:07:10 GMT+0
Resolved - This incident is resolved.

Sep 27, 07:57:57 GMT+0
Resolved - The host needs a reboot. There will be downtime for 5 - 10 minutes. Affected servers will see a notice in the dashboard. Sorry for the inconvnience..

Duration: 9 hours and 44 minutes

Affected Components: ,

Sep 22, 17:44:25 GMT+0
Completed - Maintenance has completed successfully..

Sep 22, 16:02:13 GMT+0
Identified - Maintenance is still in progress - we apologize for the earlier all-clear - we are almost there :).

Sep 22, 16:00:00 GMT+0
Completed - Maintenance has completed successfully.

Sep 22, 14:55:09 GMT+0
Identified - Extending by 1 more hour, but we believe that should do it. Thank you for your patience today..

Sep 22, 13:43:47 GMT+0
Identified - Maintenance is proceeding as it should, we have extended the window by 1 hour however in order for some slow running data migration jobs to complete. We will update here if the window is extended further, or when maintenance is complete..

Sep 22, 08:00:01 GMT+0
Identified - Maintenance is now in progress.

Sep 22, 07:52:37 GMT+0
Identified - We are upgrading all of our billing systems today. During the upgrade, we have to restrict any actions in our dash or API which may mutate billing information. This means, that during the maintenance window, customers cannot: Create new servers, Scale servers, Add Payment methods, Top up credits, manually pay invoices or transfer servers to other customers. All other functionality works as usual..

Affected Components:

Sep 17, 11:24:45 GMT+0
Resolved - This incident has been resolved..

Sep 17, 11:11:05 GMT+0
Resolved - This seems to be a kernel crash. Our NOC team is working on bringing up the host..

Duration: 21 minutes

Affected Components:

Sep 9, 08:43:31 GMT+0
Resolved - This incident has been resolved..

Sep 9, 08:22:18 GMT+0
Investigating - We are currently investigating this incident..

Duration: 10 minutes

Affected Components:

Sep 8, 11:02:43 GMT+0
Resolved - This incident has been resolved..

Sep 8, 10:53:03 GMT+0
Investigating - The host needs a reboot. Downtime of 5 - 10 minutes to be expected. Affected instances will see a message in the Webdock dash.

Duration: 21 minutes

Affected Components:

Sep 7, 14:50:40 GMT+0
Completed - Maintenance has completed successfully..

Sep 7, 14:30:01 GMT+0
Identified - Maintenance is now in progress.

Sep 7, 08:47:26 GMT+0
Identified - We are upgrading the software on our backup cluster. This is a regular software upgrade. Snapshots and backups will be unavailable for about 1 hour between 16.30 and 17.30 CEST..

Duration: 1 hour and 57 minutes

Affected Components:

Sep 7, 06:34:22 GMT+0
Resolved - This incident has been resolved. We truly apologize for the inconvenience this has caused. Thank you..

Sep 7, 06:10:38 GMT+0
Investigating - We are currently investigating this incident..

Sep 7, 05:59:11 GMT+0
Resolved - This incident has been resolved..

Sep 7, 04:37:35 GMT+0
Investigating - We are currently investigating this incident..

Duration: 5 minutes

Affected Components:

Sep 3, 10:02:00 GMT+0
Resolved - This incident has been resolved..

Sep 3, 09:56:46 GMT+0
Identified - The host is in trouble. Instant start operations hang. There will be downtime for 5 - 10 minutes. Sorry for the inconvenience..

Duration: 1 hour

Affected Components:

Sep 2, 14:00:00 GMT+0
Completed - Maintenance has completed successfully.

Sep 2, 13:00:01 GMT+0
Identified - Maintenance is now in progress.

Sep 1, 07:38:08 GMT+0
Identified - We'll be adding in more RAM sticks to two of our EPYC hosts tomorrow, September 2nd, 15.00 - 16.00 CEST (as we ran into issues today). There will be downtime during this period. Affected servers will see a blurb in the dashboard when the maintenance happens.

Duration: 7 minutes

Affected Components:

Sep 1, 08:02:21 GMT+0
Resolved - This incident has been resolved..

Sep 1, 07:55:16 GMT+0
Investigating - One of our legacy lxd-host needs a reboot. The host is frozen. Containers cannot start. Affected customers will see a blurb in the dashboard. Downtime will be about 5 - 10 minutes. Sorry for the inconvenience..

Duration: 35 minutes

Affected Components:

Sep 1, 07:34:31 GMT+0
Completed - We ran into some unexpected issues, so are aborting this maintenance and starting up the instances. We plan to do the maintenance tomorrow. So a new maintenance window will be opened soon. Sorry for the inconvenience..

Sep 1, 07:00:01 GMT+0
Identified - Maintenance is now in progress.

Aug 27, 10:59:19 GMT+0
Identified - We'll be adding in more RAM sticks to two of our EPYC hosts on September 1st (9.00 - 10.00 CEST). There will be downtime during this period. Affected servers will see a blurb in the dashboard when the maintenance happens.

Duration: 3 days, 23 hours and 30 minutes

Affected Components: ,

Aug 31, 07:16:50 GMT+0
Completed - This maintenance is cancelled as we discovered new issues over the weekend. We'll post a new incident here when we are ready to perform the upgrade..

Aug 27, 07:47:16 GMT+0
Identified - We are performing a major upgrade of our dashboard, monday 31st of August between 11.00 and 15.00 CEST. The dashboard and Webdock API will be completely inaccessible during this time period. We are performing critical upgrades of our entire billing infrastructure, and for that reason we need to keep the dash down in order to avoid any mutations to billing data while we perform the final data migrations required..

Duration: 6 minutes

Affected Components:

Aug 25, 06:41:02 GMT+0
Resolved - This incident has been resolved..

Aug 25, 06:34:42 GMT+0
Identified - The host machine has problems with sticking snapshot creation. Needs a reboot. VPS will be offline for 5 - 10 minutes. Affected server will see a blurb in the dashboard. Sorry for the inconvenience..

Duration: 8 minutes

Affected Components:

Aug 24, 10:47:06 GMT+0
Resolved - This incident has been resolved. We'll monitor and move way instances if need be. Sorry for the inconvenience.

Aug 24, 10:39:00 GMT+0
Identified - A single legacy lxd-based host is having issues, crashing down. We're looking into it. Possibly we'll move some of the servers away from the host to rebalance the load..

Duration: 12 minutes

Affected Components:

Aug 10, 14:45:11 GMT+0
Completed - Maintenance has completed successfully..

Aug 10, 14:33:17 GMT+0
Identified - Maintenance is now in progress..

Aug 10, 07:16:31 GMT+0
Identified - We are upgrading the software on our backup cluster again to prevent snapshot syncronization issues. Snapshots and backups will be unavailable for about 1 hour between 16.30 and 17.30 CEST..

Duration: 4 minutes

Affected Components:

Aug 11, 10:25:21 GMT+0
Resolved - This incident has been resolved. Sorry the inconvenience caused..

Aug 11, 10:21:16 GMT+0
Investigating - The system froze. Basic start/stop of containers (legacy) are hanging..

Duration: 15 minutes

Affected Components:

Aug 5, 15:58:08 GMT+0
Resolved - Following further investigation, we believe the most probable cause of the degraded connectivity affecting customers in Russia is a change in international transit and routing between Russian networks and Europe. Recent reporting indicates that Finland is in the process of discontinuing infrastructure that carries part of Russia's international Internet traffic: <https://thegaze.media/news/finland-to-dismantle-infrastructure-carrying-part-of-russias-internet-traffic> This is consistent with the routing changes we have observed, where traffic that would normally take a comparatively direct path between Russia and Northern Europe is instead being announced and carried through significantly longer transit paths, in some cases via Asia. Changes of this kind at the upstream carrier and BGP routing level can result in substantially increased round-trip times, less optimal peering paths, packet loss, and intermittent connectivity. These routes are selected outside of Webdock's network, and we are therefore unable to directly influence the path taken by traffic originating from affected Russian ISPs. Our own network, upstream connectivity, and infrastructure remain operational and we have not identified an issue within Webdock's network. As the underlying routing situation is external to our infrastructure, we are closing this incident. We will continue monitoring BGP paths and connectivity from Russian networks and will provide further updates if routing conditions change materially..

Aug 5, 15:42:48 GMT+0
Investigating - We are currently investigating degraded network connectivity and increased latency affecting customers connecting to our infrastructure from Russia. Our initial network observations indicate that traffic originating from Russia is no longer taking its usual routes toward European networks. Instead, a significant portion of the traffic appears to be routed through networks in Asia before reaching Europe. This considerably increases the physical network path and may result in higher latency, packet loss, unstable connections, or difficulty reaching services hosted with Webdock. At this time, the issue appears to be related to upstream international routing outside of Webdock's network and infrastructure. Our own network and services are operating normally. Our network team is continuing to investigate the affected routes and available alternatives. We will provide further updates as we learn more..

Duration: 18 minutes

Affected Components:

Aug 3, 15:51:08 GMT+0
Completed - Maintenance has completed successfully..

Aug 3, 15:33:01 GMT+0
Identified - Maintenance is now in progress..

Aug 3, 07:05:18 GMT+0
Identified - We are upgrading the software on our backup cluster. Snapshots and backups will be unavailable for about 1 hour between 17.30 and 18.30 CEST..

Duration: 17 hours and 32 minutes

Affected Components:

Aug 4, 06:16:08 GMT+0
Resolved - This incident has been resolved..

Aug 3, 12:44:29 GMT+0
Identified - We detected high packet loss towards Webdock.io. The NOC team are investigating this. Sorry for the inconvenience..

Duration: 1 hour and 41 minutes

Affected Components:

Jul 29, 18:01:46 GMT+0
Resolved - This incident has been resolved..

Jul 29, 16:20:24 GMT+0
Investigating - We are currently investigating this incident..

Duration: 4 hours

Affected Components: ,

Jul 28, 12:00:00 GMT+0
Completed - Maintenance has completed successfully.

Jul 28, 08:00:01 GMT+0
Identified - Maintenance is now in progress.

Jul 23, 12:25:43 GMT+0
Identified - We will be performing important security enhancements to our network on Tuesday the 28th betwen 10.00 and 14.00 CEST. The upgrade procedure includes a full kernel upgrade of all host systems. The procedure WILL result in brief network outage for ALL customers in Denmark. Typically we are looking at 5 seconds of interruption as systems are reloaded, but for some very busy systems or where networking hangs on some customer instance, the duration can be up to a few minutes. We apologize in advance for any inconvenience caused..

Duration: 5 hours and 4 minutes

Affected Components:

Jul 21, 11:03:33 GMT+0
Completed - Maintenance has completed successfully..

Jul 21, 09:34:14 GMT+0
Identified - Maintenance still in progress. We need more time to do the migrations. Hence increasing the maintenance duration by 2 more hours..

Jul 21, 06:00:01 GMT+0
Identified - Maintenance is now in progress.

Jul 20, 08:13:24 GMT+0
Identified - In order to balance load and stability we'll migrating Pico4 instances to other physical systems. Each server migration will be fairly quick (not more than 10 minutes, within 5 minutes in most cases). You will receive an email when the migration starts and also when the migration finishes. The migrations happen between a 4hr maintenace window, starting at 8 CEST, ending by 12 CEST..

Duration: 8 minutes

Affected Components:

Jul 18, 19:10:26 GMT+0
Resolved - This incident has been resolved..

Jul 18, 19:01:56 GMT+0
Investigating - We are currently investigating this incident..

Duration: 43 minutes

Affected Components:

Jul 15, 21:13:23 GMT+0
Resolved - This incident has been resolved..

Jul 15, 20:30:12 GMT+0
Investigating - We are currently investigating this incident. We are having issues with routing via IPV6\. The team is aware and looking into it. If you have any issues, you may turn off IPV6 via our dashboard..

Duration: 17 hours and 9 minutes

Affected Components:

Jul 8, 06:08:42 GMT+0
Completed - This maintenance has been cancelled..

Jul 7, 12:59:18 GMT+0
Identified - We'll be fusing the CPU so that IPMI can be used remotely. The maintenance starts at 7:30 CEST and should finish within 15 minutes, but we've opened a 1-hour incident window in case anything goes sideways. During this period, there will be downtime. The dashboard will show a blurb with the affected instances. .

Duration: 8 minutes

Affected Components:

Jul 6, 12:19:57 GMT+0
Resolved - This incident has been resolved..

Jul 6, 12:11:31 GMT+0
Identified - We need to reboot the host, there will be a dowtime of 5-10 minutes..

Duration: 5 minutes

Affected Components:

Jul 4, 12:22:53 GMT+0
Resolved - This incident has been resolved..

Jul 4, 12:17:28 GMT+0
Investigating - The host needs a reboot. There will be dowtime of 5-10 minutes.

Duration: 19 minutes

Affected Components:

Jun 29, 14:49:03 GMT+0
Completed - Maintenance has completed successfully..

Jun 29, 14:30:15 GMT+0
Identified - Maintenance is now in progress..

Jun 29, 09:03:59 GMT+0
Identified - We are upgrading the software on our backup cluster. Snapshots and backups will be unavailable for about 1 hour between 16.30 and 17.30 CEST..

Duration: 10 minutes

Affected Components:

Jun 26, 03:47:55 GMT+0
Resolved - This incident has been resolved..

Jun 26, 03:38:05 GMT+0
Investigating - We are currently investigating this incident..

Duration: 1 hour and 25 minutes

Affected Components:

Jun 17, 08:09:50 GMT+0
Completed - Maintenance has completed successfully..

Jun 17, 06:45:01 GMT+0
Identified - Maintenance is now in progress.

Jun 17, 06:36:53 GMT+0
Identified - We are upgrading the capabilities of our AI chat..

Duration: 21 minutes

Affected Components:

Jun 9, 06:33:43 GMT+0
Resolved - This incident has been resolved..

Jun 9, 06:13:13 GMT+0
Identified - Rebooting the host. The downtime will be 5-10 minutes. Apologies for inconvenience..

Duration: 1 day, 22 hours and 58 minutes

Affected Components:

Jun 10, 18:57:44 GMT+0
Resolved - This incident has been resolved..

Jun 8, 20:25:00 GMT+0
Identified - Hardware malfunction..

Jun 8, 20:00:00 GMT+0
Investigating - We are currently investigating this incident..

Duration: 43 minutes

Affected Components:

Jun 8, 16:13:16 GMT+0
Completed - Maintenance has completed successfully..

Jun 8, 15:30:01 GMT+0
Identified - Maintenance is now in progress.

Jun 8, 15:30:00 GMT+0
Identified - We are upgrading the software our backup cluster which will provide more logs if any specific backup fails. Snapshots and backups will be unavailable for about 1 hour between 17.30 and 18.30 CEST.

Duration: 1 hour

Affected Components:

Jun 1, 17:00:00 GMT+0
Completed - Maintenance has completed successfully.

Jun 1, 16:00:01 GMT+0
Identified - Maintenance is now in progress.

Jun 1, 16:00:00 GMT+0
Identified - We will be doing some light backup cluster maintenance tonight. Snapshots and backups will be unavailable for about 1 hour between 18.00 and 19.00 CEST.

Duration: 4 hours and 37 minutes

Affected Components:

May 30, 19:20:00 GMT+0
Resolved - This incident has been resolved..

May 30, 14:42:42 GMT+0
Identified - We noticed an issue with our AI chat and are working to fix it..

Duration: 6 minutes

Affected Components:

May 28, 10:13:06 GMT+0
Resolved - This incident has been resolved..

May 28, 10:07:08 GMT+0
Identified - The host needs a reboot. There will be downtime for few minutes. Apologies for the trouble caused..

Duration: 6 minutes

Affected Components:

May 20, 08:05:57 GMT+0
Completed - This maintenance has been cancelled due to short in staff today unfortunately. We'll post a new maintenance window when we are ready. Sorry for the inconvenience..

May 20, 08:00:01 GMT+0
Identified - Maintenance is now in progress.

May 20, 08:00:00 GMT+0
Identified - We will be performing important security enhancements to our network on Wednesday the 20th betwen 10.00 and 14.00 CEST. The upgrade procedure includes a full kernel upgrade of all host systems. The procedure WILL result in brief network outage for ALL customers in Denmark. Typically we are looking at 5 seconds of interruption as systems are reloaded, but for some very busy systems or where networking hangs on some customer instance, the duration can be up to a few minutes. We apologize in advance for any inconvenience caused..

Duration: 2 hours

Affected Components:

May 13, 17:00:00 GMT+0
Completed - Maintenance has completed successfully.

May 13, 15:00:01 GMT+0
Identified - Maintenance is now in progress.

May 13, 15:00:00 GMT+0
Identified - We are expanding our backup storage with another 200TB of storage. The storage backend will be unavailable for 1-2 hours..

Duration: 3 hours

Affected Components:

May 6, 09:30:00 GMT+0
Completed - Maintenance has completed successfully.

May 6, 06:30:01 GMT+0
Identified - Maintenance is now in progress.

May 6, 06:30:00 GMT+0
Identified - The danish utility service needs to do some electric cable splicing near our DC today. They will be cutting power to our facility for up to 3 hours between 8.30 and 11.30 CEST. We expect no impact to our operations or customers, as our emergency power systems should work nominally, just like they have in the recent power maintenance a couple of weeks ago. For this reason we expect no disruption of customer workloads..

Duration: 2 hours

Affected Components:

Apr 30, 06:00:00 GMT+0
Completed - Maintenance has completed successfully.

Apr 30, 04:00:01 GMT+0
Identified - Maintenance is now in progress.

Apr 30, 04:00:00 GMT+0
Identified - A single Ryzen system in our DK DC has a temperature anomaly. We believe one or more fans may have failed in the system and this needs to be investigated. We are reserving 2 hours for this maintenance on thursday morning at 06.00 CEST, just in case, but we expect this to take 45 minutes or less. Your server will go down during this time, and you have been notified by email if you are affected..

Duration: 2 hours

Affected Components:

Apr 27, 20:19:47 GMT+0
Completed - Maintenance has completed successfully. We have solved a number of minor issues, lastly it was Github login which no longer worked. We should be in a good state now. Enjoy and thanks for your patience with us this evening! :).

Apr 27, 20:00:00 GMT+0
Completed - Maintenance has completed successfully.

Apr 27, 18:48:49 GMT+0
Identified - We found a small issue in our API layer which was fixed a few minutes ago. The API is now fully operational again. We found a small issue in our automated jobs also, which we just fixed. We are continuing to monitor all systems and will update here once we feel we are in a perfect state..

Apr 27, 18:23:06 GMT+0
Identified - The maintenance was quicker than expected, due to us having extensively prepared and tested the procedure beforehand. We are pretty much done, but will be checking logs and other bits and pieces for a little while to make sure everything is running smoothly, before giving the all clear..

Apr 27, 18:00:01 GMT+0
Identified - Maintenance is now in progress.

Apr 27, 18:00:00 GMT+0
Identified - Monday April 27th at 20.00 CEST we will be performing major upgrades to the Webdock Dashboard at [app.webdock.io](http://app.webdock.io) \- the maintenance will take at most 2 hours, but we hope to complete it in 30 minutes or less. The upgrade requires us to take down the dashboard completely and in this time period you will be unable to use our dashboard to interact with your server, so please make sure you have completed any tasks like server snapshotting or restoring before this time. Your server(s) and all other services will be fully operational, it is only the main dashboard which will go down for maintenance. During the maintenance metrics collection will be paused so you will see a gap in data for utilization metrics for this period of time. No other impact is expected..

Duration: 10 hours and 20 minutes

Affected Components:

Apr 22, 15:19:35 GMT+0
Completed - Maintenance has completed successfully and without incident. Webdock is now one step closer to our next big expansion in our DK DC1..

Apr 22, 12:36:31 GMT+0
Identified - The first power outage happened this morning, a bit later than scheduled but without any impact to our services or customers as was expected. The second planned power outage will start within the next hour or so, and once that completes the maintenance is done. Since everything worked nominally this morning, we do not expect any issues with this second outage today. We will update here once the maintenance is completed..

Apr 22, 05:00:01 GMT+0
Identified - Maintenance is now in progress.

Apr 22, 05:00:00 GMT+0
Identified - We are planning for a scheduled maintenance of our power infrastructure on Wednesday April 22nd. In connection with this we will experience a power outage during two seperate windows, the first at 07:00 to 07:40 CEST and the second from 15:00 to 15:40 CEST. We have tested all of our emergency power equipment and we fully expect our UPS and Generator systems to take over during the planned power outage. We do not expect any disruption to our customers..

Duration: 1 hour and 23 minutes

Affected Components:

Mar 12, 20:23:08 GMT+0
Completed - Maintenance has completed successfully. All snapshots operations once again available..

Mar 12, 19:00:00 GMT+0
Identified - We are performing some needed maintenance in storage cluster DK DC1 now and for up to 2 hours. In this period, snapshot functionality will not be available..

Duration: 8 minutes

Affected Components:

Oct 11, 17:22:54 GMT+0
Completed - Maintenance has completed successfully. No impact to customer workloads was seen..

Oct 11, 17:15:01 GMT+0
Identified - Maintenance is now in progress.

Oct 11, 17:15:00 GMT+0
Identified - Our NOC team has identified and issue with a flapping link and high CRC errors rate for the past 2 days. We will attempt to fix the issue by changing optics..

Duration: 3 hours

Affected Components: , ,

Oct 2, 11:00:00 GMT+0
Completed - Maintenance has completed successfully.

Oct 2, 08:00:01 GMT+0
Identified - Maintenance is now in progress.

Oct 2, 08:00:00 GMT+0
Identified - Edit this maintenance has been bumped by +1 day from what was originally planned. We will be splitting our front-end website and docs from our dashboard systems tomorrow during the day, sometime between 10.00 and 13.00 CET. This should cause minimal disruption, but we do expect a couple of minutes of potential web server error messages as SSL Certificates are being generated and DNS is switching over..

Duration: 2 hours

Affected Components:

Sep 25, 08:00:00 GMT+0
Completed - Maintenance has completed successfully.

Sep 25, 06:00:01 GMT+0
Identified - Maintenance is now in progress.

Sep 25, 06:00:00 GMT+0
Identified - This thursday morning some electric works will be happening at the DK DC1\. We are getting new power supply added increasing our allowed power draw from the grid. The procedure means that mains power will need to be cut for about 5-10 minutes. This should be handled by our UPS and Generator systems - but just in case something unexpected happens we are announcing this 2 hour maintenance window..