Network outage GLB1

Incident Report for Statuspage Netrics Zürich AG

Resolved

After monitoring the affected systems for a week now, we are confident that the introduced measures will prevent further failures and are therefore closing this incident.
Posted Jul 01, 2019 - 10:01 CEST

Update

We have taken measures to mitigate the problem and are closely monitoring the systems.
Posted Jun 24, 2019 - 17:51 CEST

Update

We experienced another short outage between 16:41 and 16:47. WE are monitoring the situation closely.
Posted Jun 24, 2019 - 17:11 CEST

Monitoring

This outage was caused by human error, a configuration fault lead to the interruption. We sincerely apologize for the error and the inconvenience caused.
Posted Jun 24, 2019 - 11:46 CEST

Update

Services have been restored. Root cause analysis still in progress.
Posted Jun 24, 2019 - 11:03 CEST

Investigating

The same situation seems to have happend again. Engineers are investigating.
Posted Jun 24, 2019 - 10:38 CEST

Update

Our vendor is still analysing the fault and has escalated the case within their organization.
Posted Jun 20, 2019 - 15:11 CEST

Update

We are in the process of analysing the fault together with our vendor. We suspect a software bug in a switch as the root cause leading to a cascade of failures in the network.
Posted Jun 18, 2019 - 11:02 CEST

Monitoring

We experienced a short interruption (6 minutes) of our network services in the GLB1 datacenter between 08:38 and 08:44. We are investigating the root cause and will provide the information here.
Posted Jun 18, 2019 - 09:25 CEST
This incident affected: Backbone Network Service.