Foresight - Notice history

This incident has been resolved.

Our app and database infrastructure provider has implemented a fix and is now monitoring the incident. Foresight engineers determine that our services are now recovered and will continue to monitor.

The provider has identified the issue and is rolling out a mitigation. We are monitoring the situation and will apply any required updates on our side to restore normal service as quickly as possible.

We’ve mitigated the impact from AWS’s Availability Zone issue, and service has remained stable. We have not observed further node failures or customer impact, so we’re marking this incident as resolved. We’ll continue to monitor AWS’s recovery separately.

We’ve moved affected infrastructure out of the impacted AWS Availability Zone, and traffic should now be restored. We’re continuing to monitor for any residual impact while AWS’s recovery progresses.

AWS’s Availability Zone-level issue is continuing to cause impact to our infrastructure. Some customers may experience extended downtime or elevated error rates while affected workloads recover on healthy infrastructure. We’re actively working to restore service for impacted customers and will continue to provide updates as recovery progresses.

AWS has identified an Availability Zone-level issue affecting our infrastructure. Some customer workloads may continue to experience intermittent errors while affected jobs are reallocated to healthy hardware and databases fail over. We’re continuing to monitor recovery and will provide updates as AWS mitigation progresses.

This incident has been resolved.

A fix has been implemented and we are monitoring the results

This issue is considered resolved.

Impact: A small percentage of Convex deployments experienced sporadic availability for approximately 17 minutes, from 22:42 to 22:59 PM PT.

Root cause: one of our database clusters experienced a routine primary failover event. During the primary promotion process, the orchestration system failed to provision a replica that was ready to accept new database writes.

Out of an abundance of caution, Convex will only use database clusters that support semi-sync replication, where at least two machines persist the writes before considering the write successful.

Since the replicas were not ready, this cluster was not usable by Convex. Human operators were paged to intervene and fix the broken automation. We're working with our database partners to better understand what went wrong and what changes will prevent it in the future.

A fix has been implemented and we are monitoring the results.