Webdock - Network issue in DK DC – Incident details

All systems operational

Network issue in DK DC

Resolved
Major outage
Started 7 months agoLasted about 1 hour

Affected

Denmark: Network Infrastructure

Partial outage from 11:07 PM to 12:03 AM

Updates
  • Resolved
    UTC
    Resolved

    This incident has been resolved. We apologize for the inconvenience this evening.!

  • Monitoring
    UTC
    Monitoring
    We implemented a fix and are currently monitoring the result.
  • Identified
    UTC
    Identified

    NOC team is still working on this. We are unsure on this side why the fix is taking so long, especially given our pretty good DOS protection - this may be some sort of novel/interesting attack we haven't seen before. We will update once our NOC gives us information.

  • Monitoring
    UTC
    Monitoring

    Still seeing up to 90% packet loss from some regions, depending on the route the traffic takes to reach us. It's a single 10Gbit line which is currently overloaded. Our NOC team is looking at blocking the source of the DOS (which should be automatic, not sure why that is not happening) or shifting the traffic to our healthy 100Gbit lines. We will hopefully have this resolved soon.

  • Identified
    UTC
    Identified

    This is looking like a DOS event. We are investigating why auto-mitigation failed us.

  • Investigating
    UTC
    Investigating

    Something is happening with our network at this time. We are currently investigating this incident.