Nov 2, 19:40 PST Completed - The scheduled maintenance has been completed. Oct 21, 11:40 PDT In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Oct 21, 11:34 PDT Scheduled - We currently have no indication that LaunchDarkly-hosted services or customer data are impacted by the publicly-disclosed compromised intermediate CA certificates from eMudhra (…
Oct 26, 13:52 PDT Resolved - A scaling inflection point in our realtime event warehouse led to a delay for some event ingestion. The engineering team has remediated the problem and all events should be caught up.Some customers may have seen delays in experiments and in guarded release processing.This is now resolved and the team will carefully monitor.
Oct 15, 08:47 PDT Resolved - Between 15:11 and 15:32 UTC today, we observed elevated 5xx errors affecting requests across our services. Approximately 0.7% of requests failed during this period (99.3% succeeded). This incident has been resolved.
Oct 3, 02:06 PDT Resolved - This incident has been resolved. Oct 3, 01:51 PDT Monitoring - The backlog of delayed changes has been fully processed as of 1:45 AM PT on October 3. All changes that were delayed during the incident have been applied. No data was lost. We are continuing to monitor. Oct 3, 00:29 PDT Identified - Some automatic changes in LaunchDarkly are being applied later than expecte…
Oct 3, 02:05 PDT Resolved - This incident has been resolved. Oct 2, 18:26 PDT Monitoring - Monitoring. We've applied a fix and error rates have returned to normal, including custom role listing, search, and assignment. We're continuing to monitor while we complete recovery work. Oct 2, 10:29 PDT Update - Identified. We've identified the cause of the elevated error rates and are working on a fix. C…
Aug 26, 13:00 PDT Resolved - This incident has been resolved. Aug 26, 12:45 PDT Monitoring - Server-side streaming requests experienced elevated error rates between approximately 12:00 p.m. and 12:12 p.m. PDT, and some SDKs establishing new streaming connections may have been unable to connect. Polling request error rates remained below 1% throughout the incident.No customer data was lost or compr…
Aug 24, 06:36 PDT Resolved - Email delivery has been restored as of 2:32 PM UTC. Sign-up verification, password reset, teammate invitation, and notification emails are sending normally again.We are re-sending sign-up verification and password reset emails that failed to deliver during the incident window (August 22, 11:48 PM CDT through today). If you attempted to sign up or reset a password durin…
Aug 17, 09:03 PDT Resolved - This incident has been resolved. Aug 17, 08:29 PDT Monitoring - We've identified the root cause, our CDN provider had an issue on their end which is fixed now. Monitoring for a while to make sure things stabilise. Aug 17, 03:36 PDT Investigating - We're experiencing a service degradation which can result in Flag updates via the Polling API to be delayed.
Aug 14, 09:47 PDT Resolved - The issue is being marked as resolved. We're seeing improved latencies and our systems have recovered. Aug 14, 09:05 PDT Monitoring - We investigated elevated error rates reported for mobile SDK flag delivery in US East. We've confirmed that flag delivery itself was not impacted. No customer action is required and we are monitoring the situation. Aug 14, 07:47 PDT Inve…
Aug 7, 13:15 PDT Resolved - SDKs restored to operational Aug 7, 13:12 PDT Update - We are continuing to investigate this issue. Aug 7, 12:56 PDT Update - We released several mitigations and we're noticing that error rates are dropping. We'll continue to monitor the situation. Aug 7, 12:29 PDT Investigating - We are investigating elevated HTTP 500 errors affecting some server-side SDKstreaming conn…
Jul 31, 15:54 PDT Resolved - The issue with Observability Traces has been resolved. Missing trace data has been recovered. Jul 31, 14:58 PDT Monitoring - The issue with Observability Traces has been identified and a fix has been implemented. We have recovered most trace data that was missing from July 30th 5:16pm to July 31st 10:59am PT. We will resolve the incident once we fully complete data rec…
Jul 29, 11:15 PDT Resolved - This incident has been resolved. Jul 13, 11:08 PDT Update - All LaunchDarkly services are operational, though some customers need to take action to restore their connections to our Flag Delivery Network. Customers continue to successfully reestablish connections by completing the action guidance below and we continue to support customers working through the residual im…
Jul 28, 13:04 PDT Resolved - The issue with session replay and error ingest has been resolved. We've caught up on all backlogged data as of 1:01pm PT. Jul 28, 12:41 PDT Monitoring - The issue with session replay and error ingest has been identified and a fix has been implemented. We expect to catch up on all backlogged data ingest in ~15 minutes. No data has been lost. Jul 28, 12:24 PDT Identified…
Jul 26, 11:32 PDT Resolved - Between 11:32 and 11:48 PT, customers experienced increased latency and elevated error rates on API endpoints. Latency and error rates have returned to normal, and the incident has been resolved.
Jul 10, 11:19 PDT Resolved - All service availability has been fully restored.Customers using server-side SDKs might need to restart their applications to re-enable the SDKs' connection to LaunchDarkly. Customers using the Relay Proxy may also need to restart their Relay Proxy; in this case, the Relay Proxy should be restarted before restarting their applications that connect to it.To determine if…
Jul 10, 10:48 PDT Resolved - LaunchDarkly UI has recovered. We will continue to monitor Flag Delivery and event ingest here: https://status.launchdarkly.com/incidents/chywwz01ptb0. Jul 10, 10:43 PDT Update - We are continuing to work on a fix for this issue. Jul 10, 09:47 PDT Update - We are continuing to work on a fix for this issue. Jul 10, 09:26 PDT Identified - The issue has been identified an…
Jul 8, 12:41 PDT Resolved - This incident has been resolved. Jul 8, 11:51 PDT Monitoring - A fix has been implemented and we are monitoring the results. Jul 8, 11:12 PDT Identified - We have identified the issue and are working towards releasing a fix. Jul 8, 11:03 PDT Investigating - We are currently investigating an issue where experiments using frequentist analysis cannot be saved or started.
Jul 7, 08:10 PDT Resolved - Event ingestion is operating normally. Experimentation results, GuardedRollouts, and in-product data charts were delayed ~5 min between 7:13–7:53 AM PT. No data was lost. We apologize for any inconvenience.
Jul 6, 14:21 PDT Resolved - All error rates have normalized. Jul 6, 14:09 PDT Monitoring - We have rolled out a fix and error rates are returning to normal Jul 6, 14:08 PDT Update - We are continuing to investigate this issue. Jul 6, 15:30 PDT Investigating - At 13:21 Pacific Time, we experienced a high volume of error alerts affecting users' ability to interact with the LaunchDarkly application i…
Jul 6, 08:52 PDT Resolved - This incident has been resolved. Jul 6, 07:45 PDT Monitoring - A fix has been implemented and we are monitoring the results. Jul 6, 07:33 PDT Investigating - Some customers are not receiving account verification emails. Our team is actively working to mitigate this issue.
Jul 3, 09:06 PDT Resolved - This incident has been resolved. Jul 3, 08:13 PDT Monitoring - A fix has been implemented and we are monitoring the results. Jul 3, 07:29 PDT Investigating - We are investigating elevated error rates on our event data ingestion endpoints affecting production customers, between approximately 7:00 AM PDT and 7:15 AM PDT. During this window, a portion of feature flag, expe…
THIS IS A SCHEDULED EVENT Dec 31, 01:00 - 09:00 PST Aug 25, 11:43 PDT Scheduled - As part of our commitment to reliability and security, LaunchDarkly will retire all API versions except 20240415 on December 31, 2026.After this date, all requests made to retired versions will stop working. To avoid disruption, customers must update integrations and workflows to use version 20240415 before the retir…
Jun 17, 18:30 PDT Resolved - We're now seeing restored access in the affected regions. Jun 17, 18:23 PDT Update - We've identified the source of the problem for areas that were still seeing issues with the LaunchDarkly.com and docs and have a solution in place. We are now monitoring to ensure everything is resolved. Jun 17, 16:56 PDT Update - Incident seems to be resolved for the majority of users…
Jun 15, 10:38 PDT Resolved - The issue with Observability data ingest for OpenTelemetry data has been resolved. Impacted services have returned to normal operation. Jun 15, 10:33 PDT Monitoring - The issue with Observability data ingest for OpenTelemetry data has been identified and a fix has been implemented. We are continuing to monitor the performance of impacted services. We will continue to u…
Jun 13, 06:52 PDT Resolved - This incident has been resolved. Jun 12, 15:15 PDT Update - We have made major progress. We identified that a small percentage of event data was lost during two windows: June 9, 11:41 - 11:59 UTC and June 10, 23:55 UTC - June 11, 00:21 UTC. Some event-based features may show incomplete data while backfills and recovery continue. We are continuing to monitor and will pr…
Jun 11, 21:58 PDT Resolved - Monitoring has confirmed no further impact outside of the initially reported 7:30-7:37pm PT timeframe Jun 11, 21:23 PDT Monitoring - We identified a brief period of elevated errors affecting some LaunchDarkly application requests between 7:30-7:37pm PT. The issue has recovered, and we are continuing to monitor.
Jun 10, 09:00 PDT Completed - The scheduled maintenance has been completed. Jun 10, 08:00 PDT In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. May 27, 08:13 PDT Scheduled - We are removing 535 legacy individual IP addresses for stream.launchdarkly.com from our public IP list. The public IP list API will be updated when this maintenance window expi…
Jun 3, 13:26 PDT Resolved - The incident has been resolved and the flag dashboard is successfully loading. Jun 3, 13:06 PDT Monitoring - A fix has been implemented and were monitoring the results Jun 3, 12:54 PDT Identified - The issue has been identified and a fix is in progress. Jun 3, 12:31 PDT Investigating - Currently investigating an issue causing the flag dashboard to fail to load.
May 22, 12:41 PDT Resolved - We have confirmed Slack notifications are now coming through for segment approvals again. May 22, 12:33 PDT Monitoring - A fix has been implemented and we are monitoring notifications. May 22, 11:40 PDT Investigating - We are experiencing issues with sending Slack notifications for segment approvals. Email and in-app notifications still work as expected. We are working…
May 19, 07:54 PDT Resolved - This incident has been resolved. Event ingestion is operating normally and all Experimentation results and Guarded Rollouts are fully restored. No data was lost. We apologize for any inconvenience caused. Results were delayed from 6:58am PDT to 7:52am PDT May 19, 07:49 PDT Identified - We are currently experiencing delays in event ingestion, which is impacting Experime…