Webdock - Inability to reach Google services with a 403 Forbidden response in Finland – Incident details

All systems operational

Inability to reach Google services with a 403 Forbidden response in Finland

Resolved
Degraded performance
Started over 2 years agoLasted 2 months

Affected

Finland: Network Infrastructure

Degraded performance from 8:17 AM to 10:50 AM

Updates
  • Resolved
    Resolved

    We are calling this issue resolved in Finland for now as we are starting to see our nets become unblocked. The only remaining net which is still blocked as of today is:

    2a06:1301:4050:0200::/64

    But we expect this net to become unblocked soon as well. Reach out to support if you have any questions in this regard.

  • Update
    Update

    We have yet to get a response from Google on the specific issue, but we have observed today that one of our 3 blocked IPv6 nets has started working and is thus unblocked. The situation is as follows:

    Still Blocked by Google:
    2a06:1301:4050:45::/64
    2a06:1301:4050:0200::/64

    Currently Unblocked:
    2a06:1301:4050:0100::/64

    We will update here once we know more.

  • Update
    Update

    We have finally found a promising way forward to resolve this blocking issue. Apparently all we need to do is tell Google what the correct geolocation is for our IPv6 ranges. This is done through their ISP portal, which we have done already but we are still waiting on a response from there / for Google to take action. We are hoping this will clear up any day now.

  • Update
    Update

    Small update from here: Google claims they will give us a response / clarification on this issue no later than January 30th - we can't do much else but hope they resolve the issue by then or at least explain to us what is going on so we can remedy the situation.

  • Monitoring
    Monitoring

    In our Finland datacenter we have discovered an issue which started Friday December 9th. This issue affect all our IP ranges in Finland. The issue is that it seems Google, for some reason, has decided to block access to APIs and a wide range of services if calls are originating from our IP blocks. We are working with our ISP and have contacted Google in order to ascertain why this is.

    The issue is only present in Finland and a workaround for you, if viable, is to spin up your code/service that calls Google in our Canada location.

    We hope to have this resolved soon, but as we are dependent on Google Support investigating this, this may take some time to work through with them. We will update here as we go along.