Jul 31, 15:54 PDT Resolved - The issue with Observability Traces has been resolved. Missing trace data has been recovered. Jul 31, 14:58 PDT Monitoring - The issue with Observability Traces has been identified and a fix has been implemented. We have recovered most trace data that was missing from July 30th 5:16pm to July 31st 10:59am PT. We will resolve the incident once we fully complete data rec…
Jul 29, 11:15 PDT Resolved - This incident has been resolved. Jul 13, 11:08 PDT Update - All LaunchDarkly services are operational, though some customers need to take action to restore their connections to our Flag Delivery Network. Customers continue to successfully reestablish connections by completing the action guidance below and we continue to support customers working through the residual im…
Jul 28, 13:04 PDT Resolved - The issue with session replay and error ingest has been resolved. We've caught up on all backlogged data as of 1:01pm PT. Jul 28, 12:41 PDT Monitoring - The issue with session replay and error ingest has been identified and a fix has been implemented. We expect to catch up on all backlogged data ingest in ~15 minutes. No data has been lost. Jul 28, 12:24 PDT Identified…
Jul 26, 11:32 PDT Resolved - Between 11:32 and 11:48 PT, customers experienced increased latency and elevated error rates on API endpoints. Latency and error rates have returned to normal, and the incident has been resolved.
Jul 10, 11:19 PDT Resolved - All service availability has been fully restored.Customers using server-side SDKs might need to restart their applications to re-enable the SDKs' connection to LaunchDarkly. Customers using the Relay Proxy may also need to restart their Relay Proxy; in this case, the Relay Proxy should be restarted before restarting their applications that connect to it.To determine if…
Jul 10, 10:48 PDT Resolved - LaunchDarkly UI has recovered. We will continue to monitor Flag Delivery and event ingest here: https://status.launchdarkly.com/incidents/chywwz01ptb0. Jul 10, 10:43 PDT Update - We are continuing to work on a fix for this issue. Jul 10, 09:47 PDT Update - We are continuing to work on a fix for this issue. Jul 10, 09:26 PDT Identified - The issue has been identified an…
Jul 8, 12:41 PDT Resolved - This incident has been resolved. Jul 8, 11:51 PDT Monitoring - A fix has been implemented and we are monitoring the results. Jul 8, 11:12 PDT Identified - We have identified the issue and are working towards releasing a fix. Jul 8, 11:03 PDT Investigating - We are currently investigating an issue where experiments using frequentist analysis cannot be saved or started.
Jul 7, 08:10 PDT Resolved - Event ingestion is operating normally. Experimentation results, GuardedRollouts, and in-product data charts were delayed ~5 min between 7:13–7:53 AM PT. No data was lost. We apologize for any inconvenience.
Jul 6, 14:21 PDT Resolved - All error rates have normalized. Jul 6, 14:09 PDT Monitoring - We have rolled out a fix and error rates are returning to normal Jul 6, 14:08 PDT Update - We are continuing to investigate this issue. Jul 6, 15:30 PDT Investigating - At 13:21 Pacific Time, we experienced a high volume of error alerts affecting users' ability to interact with the LaunchDarkly application i…
Jul 6, 08:52 PDT Resolved - This incident has been resolved. Jul 6, 07:45 PDT Monitoring - A fix has been implemented and we are monitoring the results. Jul 6, 07:33 PDT Investigating - Some customers are not receiving account verification emails. Our team is actively working to mitigate this issue.
Jul 3, 09:06 PDT Resolved - This incident has been resolved. Jul 3, 08:13 PDT Monitoring - A fix has been implemented and we are monitoring the results. Jul 3, 07:29 PDT Investigating - We are investigating elevated error rates on our event data ingestion endpoints affecting production customers, between approximately 7:00 AM PDT and 7:15 AM PDT. During this window, a portion of feature flag, expe…
THIS IS A SCHEDULED EVENT Dec 31, 01:00 - 09:00 PST Aug 25, 11:43 PDT Scheduled - As part of our commitment to reliability and security, LaunchDarkly will retire all API versions except 20240415 on December 31, 2026.After this date, all requests made to retired versions will stop working. To avoid disruption, customers must update integrations and workflows to use version 20240415 before the retir…
Jun 17, 18:30 PDT Resolved - We're now seeing restored access in the affected regions. Jun 17, 18:23 PDT Update - We've identified the source of the problem for areas that were still seeing issues with the LaunchDarkly.com and docs and have a solution in place. We are now monitoring to ensure everything is resolved. Jun 17, 16:56 PDT Update - Incident seems to be resolved for the majority of users…
Jun 15, 10:38 PDT Resolved - The issue with Observability data ingest for OpenTelemetry data has been resolved. Impacted services have returned to normal operation. Jun 15, 10:33 PDT Monitoring - The issue with Observability data ingest for OpenTelemetry data has been identified and a fix has been implemented. We are continuing to monitor the performance of impacted services. We will continue to u…
Jun 13, 06:52 PDT Resolved - This incident has been resolved. Jun 12, 15:15 PDT Update - We have made major progress. We identified that a small percentage of event data was lost during two windows: June 9, 11:41 - 11:59 UTC and June 10, 23:55 UTC - June 11, 00:21 UTC. Some event-based features may show incomplete data while backfills and recovery continue. We are continuing to monitor and will pr…
Jun 11, 21:58 PDT Resolved - Monitoring has confirmed no further impact outside of the initially reported 7:30-7:37pm PT timeframe Jun 11, 21:23 PDT Monitoring - We identified a brief period of elevated errors affecting some LaunchDarkly application requests between 7:30-7:37pm PT. The issue has recovered, and we are continuing to monitor.
Jun 10, 09:00 PDT Completed - The scheduled maintenance has been completed. Jun 10, 08:00 PDT In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. May 27, 08:13 PDT Scheduled - We are removing 535 legacy individual IP addresses for stream.launchdarkly.com from our public IP list. The public IP list API will be updated when this maintenance window expi…
Jun 3, 13:26 PDT Resolved - The incident has been resolved and the flag dashboard is successfully loading. Jun 3, 13:06 PDT Monitoring - A fix has been implemented and were monitoring the results Jun 3, 12:54 PDT Identified - The issue has been identified and a fix is in progress. Jun 3, 12:31 PDT Investigating - Currently investigating an issue causing the flag dashboard to fail to load.
May 22, 12:41 PDT Resolved - We have confirmed Slack notifications are now coming through for segment approvals again. May 22, 12:33 PDT Monitoring - A fix has been implemented and we are monitoring notifications. May 22, 11:40 PDT Investigating - We are experiencing issues with sending Slack notifications for segment approvals. Email and in-app notifications still work as expected. We are working…
May 19, 07:54 PDT Resolved - This incident has been resolved. Event ingestion is operating normally and all Experimentation results and Guarded Rollouts are fully restored. No data was lost. We apologize for any inconvenience caused. Results were delayed from 6:58am PDT to 7:52am PDT May 19, 07:49 PDT Identified - We are currently experiencing delays in event ingestion, which is impacting Experime…
May 15, 18:13 PDT Resolved - All queued events have been processed and data ingestion is operating normally. No data was lost during this incident. May 15, 17:14 PDT Identified - We have identified the issue and a fix is in progress. The backlog is actively decreasing. No data has been lost — all queued events will be processed. May 15, 17:03 PDT Investigating - We are currently experiencing a del…
May 5, 21:26 PDT Resolved - We've implemented a fix and have mitigated impact as of 9:25pm PT. Streaming SDKs should be able to connect successfully at this time. May 5, 21:00 PDT Identified - We're identified an infrastructure issue causing elevated error rates and are working on a fix. May 5, 20:58 PDT Investigating - We are investigating elevated error rates for streaming initialization in the…
May 4, 13:25 PDT Resolved - All experiment and multi-armed bandit result data is caught up. No data was lost. May 4, 13:19 PDT Monitoring - We have implemented a fix and are catching up on backlogged data. We estimate that we will be fully caught up in about 1 hour. May 4, 12:50 PDT Identified - We have identified the issue and are continuing our work to resolve it. May 4, 12:47 PDT Investigating…
May 4, 09:23 PDT Resolved - This incident has been resolved. May 4, 08:47 PDT Monitoring - We've implemented a fix and have mitigated impact as of 8:35am PT. Streaming SDKs should be able to connect successfully at this time. May 4, 07:30 PDT Update - We're continuing to address the root cause of the elevated error rate. Server SDKs may require additional time to initialize while requests are retr…
May 2, 12:15 PDT Resolved - Experimentation results processing and data warehouse export was caught up as of 12:28 pm PT. May 2, 06:51 PDT Monitoring - We have implemented a fix and are catching up on backlogged experimentation results and data export. We estimate that we will be fully caught up in about 8 hours. May 2, 06:17 PDT Identified - We've identified the root cause of the experimentation…
Apr 30, 19:52 PDT Resolved - We've identified elevated error rates with server-side streaming initialization requests in the APAC region between 19:52 and 19:56 PT. Some customers may have experienced initialization timeouts despite SDK retries.
Apr 30, 17:09 PDT Resolved - We have validated a fix to resolve all remaining experiments with inaccurate sample ratio mismatches (SRMs) or empty experiment results. The vast majority of experiments and experimentation customers were not impacted. Apr 29, 17:04 PDT Monitoring - We've released a fix to prevent this issue in new experiments and iterations. We're working to resolve this issue for exp…
Apr 30, 04:58 PDT Resolved - We've caught up on the streaming data export backlog. Apr 30, 04:45 PDT Monitoring - We've identified latency in Pub/Sub export that were causing streaming data export delays that has since resolved. We're catching up on the export backlog now. Apr 30, 04:14 PDT Investigating - We are investigating a delay with streaming data export delivery. No data is lost, but custo…
Apr 29, 16:45 PDT Resolved - We've identified an infrastructure issue that caused some 503 responses in feature flagging streaming client initialization on the US LaunchDarkly instance between 16:45 and 17:05 PT. SDKs should automatically retry initialization in most cases.
Apr 28, 12:25 PDT Resolved - Observability alerts are functioning and delayed alerts have been redelivered. Apr 28, 12:10 PDT Monitoring - A fix has been deployed for Observability alerting and we are working to redeliver delayed notifications. Apr 28, 11:38 PDT Identified - We've identified the root cause delaying Observability alert notifications and are deploying a fix. All other notification t…