Duration: 4 hours and 3 minutes

Affected Components: App & DB Services

May 22, 06:11:50 GMT+0
Identified - Our app and database infrastructure provider is experiencing an incident affecting some deployments. Some users may see temporary errors or delayed actions in the Foresight platform. We are monitoring the provider’s mitigation and will share updates as service recovers..

May 22, 06:55:16 GMT+0
Identified - The provider has identified the issue and is rolling out a mitigation. We are monitoring the situation and will apply any required updates on our side to restore normal service as quickly as possible..

May 22, 07:50:00 GMT+0
Monitoring - Our app and database infrastructure provider has implemented a fix and is now monitoring the incident. Foresight engineers determine that our services are now recovered and will continue to monitor..

May 22, 10:14:29 GMT+0
Resolved - This incident has been resolved..

Duration: 4 days, 13 hours and 4 minutes

May 8, 00:16:37 GMT+0
Investigating - We’re experiencing intermittent downtime from an upstream infrastructure provider. Some customer workloads may see brief error spikes while affected jobs are reallocated to healthy hardware and databases fail over. We’re monitoring recovery and will provide updates as mitigation progresses..

May 8, 00:29:51 GMT+0
Identified - AWS has identified an Availability Zone-level issue affecting our infrastructure. Some customer workloads may continue to experience intermittent errors while affected jobs are reallocated to healthy hardware and databases fail over. We’re continuing to monitor recovery and will provide updates as AWS mitigation progresses..

May 8, 00:54:51 GMT+0
Identified - AWS’s Availability Zone-level issue is continuing to cause impact to our infrastructure. Some customers may experience extended downtime or elevated error rates while affected workloads recover on healthy infrastructure. We’re actively working to restore service for impacted customers and will continue to provide updates as recovery progresses..

May 8, 01:51:00 GMT+0
Monitoring - We’ve moved affected infrastructure out of the impacted AWS Availability Zone, and traffic should now be restored. We’re continuing to monitor for any residual impact while AWS’s recovery progresses..

May 8, 02:05:41 GMT+0
Resolved - We’ve mitigated the impact from AWS’s Availability Zone issue, and service has remained stable. We have not observed further node failures or customer impact, so we’re marking this incident as resolved. We’ll continue to monitor AWS’s recovery separately..

Duration: 1 day, 17 hours and 2 minutes

May 6, 17:43:46 GMT+0
Investigating - Convex on-call engineers are investigated elevated latency for function calls on a small set of Convex deployments.

May 6, 18:04:13 GMT+0
Monitoring - A fix has been implemented and we are monitoring the results.

May 6, 18:24:48 GMT+0
Resolved - This incident has been resolved..

Duration: 1 day, 18 hours and 32 minutes

Apr 9, 05:42:27 GMT+0
Investigating - Seems related to a problematic database failover. Updates soon..

Apr 9, 05:59:27 GMT+0
Monitoring - A fix has been implemented and we are monitoring the results..

Apr 9, 06:24:59 GMT+0
Resolved - This issue is considered resolved. Impact: A small percentage of Convex deployments experienced sporadic availability for approximately 17 minutes, from 22:42 to 22:59 PM PT. Root cause: one of our database clusters experienced a routine primary failover event. During the primary promotion process, the orchestration system failed to provision a replica that was ready to accept new database writes. Out of an abundance of caution, Convex will only use database clusters that support semi-sync replication, where at least two machines persist the writes before considering the write successful. Since the replicas were not ready, this cluster was not usable by Convex. Human operators were paged to intervene and fix the broken automation. We're working with our database partners to better understand what went wrong and what changes will prevent it in the future..