Incident Report · March 8, 2026

Severity P0 — Resolved

Duration: 2 hours 14 minutes · Affected: All API Endpoints

A Message from Our CEO

Marcus Holt

Chief Executive Officer, NovaSaaS

On March 8, 2026, NovaSaaS experienced a significant service disruption that affected all API endpoints for 2 hours and 14 minutes. I want to personally apologize to every customer whose workflows, applications, and teams were impacted.

This outage fell well below the reliability standards we hold ourselves to and that you rightfully expect from us. We take full responsibility. Below, you will find a transparent account of what happened, why it happened, and the concrete steps we have already taken to ensure it does not happen again.

We are deeply grateful for your continued trust in NovaSaaS, and we will earn it back through action — not words alone.

Incident Summary

Incident ID

INC-2026-0308

Date

March 8, 2026

Total Downtime

2 hours, 14 minutes

Affected Services

REST API, GraphQL API, Webhook Delivery, SDK Authentication

Root Cause

Database failover misconfiguration introduced during a scheduled maintenance window

Root Cause Analysis

During a scheduled maintenance window on March 7 at 23:45 UTC, our infrastructure team deployed a configuration update to the primary database cluster as part of a routine capacity upgrade. A misconfiguration in the automated failover policy set the replica promotion timeout to 0 seconds — effectively disabling the grace period required for replica nodes to confirm data consistency before assuming primary responsibilities.

When a routine health check triggered a planned leader re-election at 09:12 UTC on March 8, the replica nodes entered a split-brain state. All inbound API requests were routed to a non-responsive primary, causing connection timeouts across every API surface. The misconfiguration was not caught by our pre-deployment validation suite because the relevant test case covered timeout values greater than zero but did not account for a zero-value edge case.

Timeline of Events (All Times UTC)

Mar 7 · 23:45

Maintenance window begins

Database configuration update deployed to production cluster. Automated checks pass. Team signs off.

Mar 8 · 09:12

Outage begins — API endpoints unreachable

Scheduled leader re-election triggers failover. Replica nodes enter split-brain state. API request timeouts spike to 100%.

Mar 8 · 09:15

Automated alerting fires — on-call team paged

PagerDuty escalation triggers. Senior SRE and database lead acknowledge within 3 minutes.

Mar 8 · 09:31

Root cause identified

Database team isolates the zero-value timeout misconfiguration in failover policy. Remediation plan drafted.

Mar 8 · 09:47

Status page updated — customers notified

Public status page updated with incident details. Email and in-app notifications sent to all affected accounts.

Mar 8 · 10:58

Configuration corrected — failover re-initiated

Corrected failover policy deployed. Replica nodes successfully elect a primary. Database cluster stabilises.

Mar 8 · 11:26

Full service restored — incident resolved

All API endpoints return to normal operation. Error rates drop to baseline. Status page updated to resolved.

Remediation Steps Taken

The following measures have already been implemented as of March 9, 2026.

Failover policy validation hardened

Pre-deployment validation suite updated to explicitly reject zero or negative timeout values across all database failover configurations. The edge case that bypassed checks has been closed.

Mandatory staging environment replay introduced

All database configuration changes must now complete a full failover simulation in our staging environment — including leader re-election under load — before being approved for production deployment.

Split-brain detection alert added

A dedicated monitoring rule now detects split-brain conditions within 60 seconds of onset and immediately pages the database on-call team, independent of general API error rate alerts.

Maintenance window change freeze policy updated

Database configuration changes are now prohibited within 4 hours of a scheduled leader re-election event. An automated calendar lock enforces this policy and blocks deployment pipelines accordingly.

Runbook and incident response drills scheduled

A revised database incident runbook has been published to our internal knowledge base. Quarterly failover drills have been added to the engineering calendar to ensure the team maintains hands-on familiarity with recovery procedures.

1 Month Service Credit Applied to Your Account

As a direct acknowledgment of the disruption caused on March 8, we have automatically applied a full one-month service credit to your NovaSaaS account. No action is required on your part.

The credit will appear on your next billing statement and will be applied before any charges are processed. If you have questions about your credit or billing, our support team is available 24/7.

Questions or Concerns?

We have published the full incident report, including all technical details and our ongoing monitoring improvements, on our status page. If you have questions, concerns, or would like to speak with a member of our team directly, please do not hesitate to reach out.

View Full Incident Report

Contact Support

Thank you for your continued partnership. We are committed to doing better.

Marcus Holt

Chief Executive Officer, NovaSaaS

marcus@novasaas.io

Twitter

GitHub

NovaSaaS, Inc. · 340 Pine Street, Suite 800 · San Francisco, CA 94104

You are receiving this message because you are a NovaSaaS customer affected by incident INC-2026-0308.

Manage notification preferences · Unsubscribe · Privacy Policy

Incident Post-Mortem