Jul 27, 18:27 UTC Resolved - The backlog has been cleared and ingestion is back to normal. Jul 27, 18:07 UTC Monitoring - We have resolved the incident and are monitoring our ingestion backlog. Jul 27, 16:56 UTC Identified - We have identified the issue and error and profile ingestion is back up in US. Expect delays in ingestion and alerting for errors and profiles. Jul 27, 16:23 UTC Investigating…
Jul 21, 14:14 UTC Resolved - We've burned through our spans backlog. Latency across span consumers are normal. Jul 21, 13:55 UTC Update - We're still burning through our spans backlog Jul 21, 13:04 UTC Monitoring - We've found and fixed the root cause and are burning through our spans backlog. Ingestion should be back to normal in around an hour. Jul 21, 12:20 UTC Update - We are still actively in…
Jul 20, 16:57 UTC Resolved - We have fully burned the backlog of EU error events ingested during the incident. Latency is normal across all event types. Jul 20, 16:09 UTC Update - We are still burning the backlog of EU region error events received during the incident. We are around halfway through the backlog. Jul 20, 14:50 UTC Update - The backlog on attachments in our EU region is fully burned,…
Jul 17, 06:57 UTC Resolved - The webhook backlog is now resolved. Jul 17, 05:49 UTC Update - The webhook backlog is processing. Jul 17, 04:50 UTC Update - The webhook backlog is processing. Jul 17, 03:49 UTC Update - The webhook backlog is processing. Estimated recovery is now 2-2.5 hours. Jul 17, 02:44 UTC Update - The webhook backlog is processing. Jul 17, 01:28 UTC Update - The webhook backlog…
Jul 14, 19:32 UTC Resolved - Feedback ingestion is back to realtime and the backlog has been processed. Jul 14, 18:16 UTC Update - We are continuing to monitor ingestion, and a fix is being deployed to unblock feedback ingestion. Jul 14, 17:40 UTC Monitoring - We are processing errors in realtime, and error backlogs have been processed. We're continuing to monitor the situation before restoring fe…
Jul 13, 14:25 UTC Resolved - Service has been restored. Jul 13, 12:41 UTC Monitoring - Service has been restored and we are actively monitoring the situation Jul 13, 12:24 UTC Identified - We identified the issue and restoring service. Jul 13, 11:51 UTC Investigating - We are investigating ingestion delays.
Jul 13, 02:56 UTC Resolved - Ingestion has caught up and alerts have been restored. Jul 13, 02:01 UTC Monitoring - We have implemented a fix and are monitoring the situation. We expect ingestion to catch up in the next hour, approximately. Jul 13, 01:46 UTC Identified - We have identified the problem and are working to restore ingestion Jul 13, 00:58 UTC Investigating - We are currently investigat…
Jul 11, 03:34 UTC Resolved - We have processed the backlog. Jul 11, 03:28 UTC Monitoring - We have processed the majority of our backlog. We are now recovering the tail of the backlog. Jul 11, 02:37 UTC Update - We have processed the majority of our backlog. We are now recovering an additional 25M error events ranging from 22:20 to 23:20 UTC, which should complete in the next 2 hours Jul 11, 01:46…
Jul 7, 14:36 UTC Resolved - Between July 6th 20:25 UTC and July 7th 12:25 UTC, a wrong configuration change was deployed causing over-sampling of logs, spans and metrics.The configuration change is now reverted and the issue is fixed. Jul 7, 08:49 UTC Investigating - We are currently investigating reports of inconsistent reported number of processed logs in Sentry for the last day
Jul 7, 00:55 UTC Resolved - The backlgo has been consumed and ingestion and alerting has returned to normal operation Jul 6, 23:55 UTC Update - We're continuing to burn down the EU spans, logs, and metrics ingestion backlog. Jul 6, 23:00 UTC Update - We are now processing the backlog in EU. Expect delays in ingestion and alerting for spans, logs, and metrics on the order of hours. We will keep thi…
Jul 6, 22:18 UTC Resolved - US ingestion is back to normal. EU ingestion is still delayed, but we are now recovering. We have created a new Statuspage incident for tracking EU, follow that for further updates: https://status.sentry.io/incidents/7tfjf5k3m0g3 Jul 6, 21:28 UTC Update - We're still burning the last of our backlog in US and expect to be back to normal the next hour.We're continuing to…
Jul 2, 03:56 UTC Resolved - All operations are back to normal, as of ~7:30PM PT Jul 2, 00:46 UTC Monitoring - The migration has been reversed in EU and US. Application functionality is restored but there is a 45 minute delay in ingestion in the US and no delay in EU Jul 2, 00:34 UTC Identified - We have identified a migration as the cause of the issue. We have reverted the migration in EU and are…
Jul 1, 10:00 UTC Resolved - Data is now processed live and the system are fully operational. Jul 1, 06:14 UTC Update - We continue to monitor the fix. Ingestion is around 45 minutes behind and improving. APIs and dashboards are functioning normally (outside of data waiting to be ingested). Jul 1, 05:22 UTC Update - We continue to monitor the situation Jul 1, 04:26 UTC Update - We continue to monit…
Jun 24, 20:48 UTC Resolved - The issue in our US region has been resolved, and ingestion latency is back to normal. Jun 24, 19:53 UTC Monitoring - We have implemented a fix for the problem and are monitoring the situation. We are burning our US region ingestion backlog for replays, spans, and performance issues and will update here as the situation progresses. Jun 24, 18:52 UTC Identified - We hav…
Jun 12, 11:30 UTC Resolved - From approximately 10:55 UTC to 11:10 UTC we experienced a loss of outcomes which power the "Stats" in the organization settings.The data loss is purely informational without impact on billing or ingested data.
Jun 11, 14:48 UTC Resolved - This issue has been resolved. Jun 11, 14:18 UTC Update - The issue involving some alerts not firing has been mitigated. We will continue to monitor the situation to ensure it does not reoccur. Jun 11, 13:23 UTC Update - We have identified an issue with one of our cloud providers that might have caused some alert notifications to not fire starting at around 4:00 AM GMT+…
Jun 10, 10:28 UTC Resolved - Our auth systems are back operational Jun 10, 10:17 UTC Update - We rolled back a configuration change we identified as the root cause of the issue. We'll keep monitoring our systems Jun 10, 10:15 UTC Monitoring - We deployed a fix and are monitoring the situation. Jun 10, 10:05 UTC Investigating - We are currently investigating the issue.
Jun 2, 07:45 UTC Resolved - Between 06:00-07:00 UTC, we saw increased numbers of API errors in the EU region. The largest impact was between 06:46-07:00. We've implemented a fix and we're no longer seeing elevated errors.
May 28, 14:28 UTC Resolved - The issue has been resolved May 28, 14:20 UTC Monitoring - The fix has been deployed and we are monitoring the situation May 28, 13:55 UTC Identified - We have identified the cause of the issue and are deploying a fix May 28, 13:40 UTC Investigating - We are currently investigating the issue
May 26, 08:28 UTC Resolved - Access to sentry.io is now fully operational May 26, 08:19 UTC Monitoring - We have implemented a fix and are monitoring. May 26, 08:11 UTC Investigating - We are currently investigating an issue causing 500 errors and preventing access to Sentry.io organizations in the DE region. The US region remains fully operational.
May 22, 08:23 UTC Resolved - Latency for sentry.io has returned to normal. May 22, 08:08 UTC Monitoring - We have implemented a fix and are monitoring. May 22, 07:42 UTC Investigating - We are currently investigating slow page loads on sentry.io.
May 19, 18:48 UTC Resolved - Ingestion for symbolicated issues is back to normal. May 19, 17:41 UTC Update - We are implementing a fix and monitoring the situation. Non-symbolicated issues are not experiencing delays. May 19, 15:58 UTC Investigating - We are experiencing ingestion delays for errors in the US region.
May 19, 14:18 UTC Resolved - Ingestion and alerting back to normal May 19, 14:05 UTC Monitoring - We've put in fixes and are monitoring the situation
May 18, 07:11 UTC Resolved - We have implemented a fix. Ingestion and alerting have returned to normal operation May 18, 05:35 UTC Update - We are continuing to investigate the issue. May 18, 04:27 UTC Update - We are still investigating the issue. May 18, 03:27 UTC Update - We are still investigating the issue. May 18, 02:29 UTC Update - We are still investigating the issue. May 18, 01:41 UTC Inv…
May 13, 17:56 UTC Resolved - Ingestion and alerting are back to steady state in the US region. May 13, 15:49 UTC Monitoring - Ingestion and alerting are back to normal. We continue to monitor the situation May 13, 14:38 UTC Investigating - We are experiencing ingestion and alerting delays in us for spans, transactions, and replays
May 12, 22:37 UTC Resolved - This incident has been resolved. May 12, 17:00 UTC Monitoring - We've applied a configuration change to mitigate the API/UI errors being experienced. May 12, 15:43 UTC Update - We are continuing to investigate increased UI and API timeouts and errors. May 12, 13:51 UTC Update - We are continuing to investigate increased UI and API timeouts and errors May 12, 12:28 UTC…
May 11, 22:11 UTC Resolved - Database performance has returned to normal May 11, 21:39 UTC Monitoring - We have implemented a fix and are monitoring the situation May 11, 21:08 UTC Update - We are working on restoring normal service. API and application performance should be improved, ingestion still has delays. May 11, 20:25 UTC Update - Our event database continues to have issues and we are acti…
May 6, 07:55 UTC Resolved - This incident has been resolved. May 6, 07:42 UTC Investigating - We are currently investigating the issue.
Apr 27, 20:59 UTC Resolved - This incident has been resolved. Apr 27, 17:48 UTC Monitoring - The system is operating as normal, we are monitoring the fix. Apr 27, 17:31 UTC Update - Ingestion is now back to normal, we are slowly increasing write traffic to reduce intermittent API failures Apr 27, 16:52 UTC Investigating - We are experiencing delays on our ingestion pipeline. Explorer views are in…
Apr 23, 02:17 UTC Resolved - All application features are now operating as normal. Apr 23, 02:12 UTC Monitoring - Ingestion for spans and replays is about 10 minutes behind and catching up. API and Dashboard performance has returned to normal. Apr 23, 02:06 UTC Investigating - We are currently investigating this issue.