Webdock - Backup cluster degraded – Incident details

Denmark: Storage Backend under maintenance

Backup cluster degraded

Monitoring
Degraded performance
Started 1 day ago

Affected

Denmark: Storage Backend

Degraded performance from 6:16 AM to 3:12 PM, Under maintenance from 3:12 PM to 12:00 AM

Updates
  • Monitoring
    Monitoring

    We implemented a temporary fix which seems to have elminated the context deadline exceeded errors we were seeing. The underlying issue is still present and pending investigation by the Incus team - what we did was just a "patch" which is fine for now. We will update once we know more.

  • Identified
    Identified

    After a major software upgrade of our backup cluster yesterday we are seeing frequent timeouts while doing backups. This is due to some regression in Incus, our hypervisor. We have the Incus team investigating this and we hope to have this solved within a day or two at most. In the interim, if you get an error trying to perform a snapshot you can retry and hopefully it goes through. Repeated failures however means that your snapshot is indeed present locally, it just isn't synched out to our storage backend, and it won't be synched until the underlying issue is solved. This issue seems to mostly affect new instances - "old" instances created before yesterday are not as affected.