Connectivity issues

Incident Report for Riege Software

Postmortem

Statement regarding the network issue that occurred during planned maintenance of our network infrastructure on 2024-09-02.
As part of this maintenance, a security update has been applied to our Top-of-Rack switches, which required a reboot. After the reboot a wrong configuration was applied, leading to an upstream routing misbehavior. As a result, some of our services experienced an outage.
To address this issues, we have taken the following corrective actions:

The documentation has been reviewed and adjusted to ensure clarity and accuracy in the maintenance procedures.
Alerts have been added to our monitoring system to detect and address unintended changes.

Posted Sep 06, 2024 - 06:59 UTC

Resolved

This incident has been resolved.

Posted Sep 03, 2024 - 09:43 UTC

Monitoring

A fix has been implemented and we are monitoring the results.

Posted Sep 03, 2024 - 09:33 UTC

Identified

The issue has been identified and a fix is being implemented.

Posted Sep 03, 2024 - 09:28 UTC

Update

We are continuing to investigate this issue.

Posted Sep 03, 2024 - 08:41 UTC

Update

We are currently experiencing network issues and are working diligently to resolve them as quickly as possible. Login to Scope, as well as message communication to and from Scope and other services, may be affected.

Posted Sep 03, 2024 - 07:13 UTC

Update

We are continuing to investigate this issue.

Posted Sep 03, 2024 - 06:50 UTC

Investigating

Some customers report issues with Scope. You may experience problems connecting to Scope

We are actively investigating the cause of this issue.

Posted Sep 03, 2024 - 06:14 UTC

This incident affected: Scope.