Jul 11, 15:42 UTC Resolved - This incident has been resolved. Big thanks to Fastly for providing their CDN services and their fast response time! Jul 11, 15:06 UTC Monitoring - We have deployed countermeasures and our servers are largely back to normal again. We keep monitoring the situation. If you encounter any issues please let us know! Jul 11, 14:22 UTC Update - We are continuing to investigat…
Jun 14, 01:53 UTC Resolved - The primary source of the load appears to have been a scraper abusing our API. We have blocked the scraper and the replica database has returned to its normal heavily loaded state instead of being overloaded. Jun 14, 01:37 UTC Identified - The read-only replica database used for many of our common queries is under heavy load. We are evaluating options to deal with this…
May 26, 17:00 UTC Resolved - This incident has been resolved. May 23, 19:19 UTC Monitoring - Reverse dependencies remain disabled, but the rest of crates.io is functioning normally. May 23, 18:47 UTC Identified - The reverse dependencies pages are temporarily unavailable. Other crates.io functionality should continue to function normally.
May 20, 16:57 UTC Resolved - All endpoints are now operational and available again. May 20, 15:52 UTC Monitoring - Crates.io is currently under a suspected denial of service effecting the web frontend. The reverse dependency endpoint has been disabled. You will not be able to browse reverse dependencies on crates.io for the duration of the incident. All other operations, publishes and the index ar…
Apr 14, 23:44 UTC Resolved - The indexes are now back to normal operations. Apr 14, 21:08 UTC Monitoring - The background worker has been told to get itself together. The queue is now trending downwards, but git index updates are still running an hour behind. We are continuing to monitor, but we expect this to resolve over the next few hours. Apr 14, 20:48 UTC Investigating - The crates.io backgro…
Feb 12, 19:46 UTC Resolved - The background worker has caught up, and crate publishes are now updating normally in both indexes. Feb 12, 19:36 UTC Identified - The crates.io background worker is having a bad morning, and a backlog of Git index updates has accumulated. We have given the worker some coffee and it's catching up, but index updates for newly-published crates are running about an hour b…
Nov 11, 01:14 UTC Resolved - Everything continues to look stable, so we can consider this resolved. Nov 11, 01:04 UTC Monitoring - The 2025 version of "it's always DNS" might be "it's always AI scrapers pretending to be other things".Our database provider pointed out an expensive endpoint that was being hit just enough by a single IP to cause the incident, but not enough for our normal monitoring…
Nov 1, 06:38 UTC Resolved - All published crates have been synced to the Git index, and there is now no backlog. Nov 1, 06:01 UTC Update - 57 pending syncs. Nov 1, 05:01 UTC Update - Now at 116 pending syncs. Probably 1½ hours before we're caught up, give or take. Nov 1, 04:04 UTC Update - The queue burndown has slowed due to an unusual number of crate publishes in the last hour, but it is slowly…
Sep 2, 23:33 UTC Resolved - This incident has been resolved. Sep 2, 23:32 UTC Update - The backlog has cleared and all queues are nominal. Sep 2, 23:08 UTC Update - The sparse index has synchronized and is now up-to-date. CDN updates are still processing. Sep 2, 22:48 UTC Monitoring - A backlog of CDN invalidations is causing a backlog of approximately 350 crate backlog for sparse index updates an…
Aug 13, 01:55 UTC Resolved - The Git index has caught up and is now up to date. Aug 12, 22:21 UTC Monitoring - The Git index sync job backlog is slowly burning down, and is anticipated to catch up in a few hours. We will continue to monitor the situation. Aug 12, 21:47 UTC Identified - Due to a large number of crate versions being published in the last few hours, the legacy Git crate index is curr…
May 13, 07:44 UTC Resolved - Crates published since the background workers were restarted are now appearing in the index, and the backlog of background jobs has been processed successfully. May 13, 07:41 UTC Monitoring - Indexes are up to date, and we are monitoring the background workers to ensure they continue to pick up new jobs. May 13, 07:39 UTC Identified - Background workers have been resta…
Jun 5, 09:19 UTC Completed - Everything continues to work as intended so we will consider this migration successful. If you notice any issues, please contact us via https://github.com/rust-lang/crates.io/issues/new/choose. Jun 5, 08:12 UTC Verifying - The read-replica database available again and performance should be back to normal. The crates.io team will continue to monitor the situation for an…
May 20, 12:26 UTC Resolved - This incident has been resolved. May 20, 12:26 UTC Update - Database server upgrade is complete and crates.io should be running at full capacity again now. Hopefully even a bit faster than before due to increased memory on the new database servers. May 20, 12:14 UTC Monitoring - We noticed a small issue with our database indexes after the upgrade, resulting in several…
Mar 22, 13:25 UTC Resolved - The application is stable, marking the incident as resolved. We will continue investigating the root cause of this. Mar 22, 13:16 UTC Monitoring - The application is recovering, but we are still investigating trying to find the root cause. The incident seems to be unrelated to the one earlier today. Mar 22, 13:04 UTC Investigating - We are investigating the crates.io A…
Mar 22, 07:56 UTC Resolved - This incident has been resolved. Mar 22, 07:48 UTC Update - Since the last update, crates.io fully recovered and we are serving traffic as usual. We're continuing to monitor the situation and we'll re-enable the read-only replica again soon. Mar 22, 07:26 UTC Monitoring - We have identified a problem with our read-only database replica and routed traffic away from it.…
Mar 19, 00:29 UTC Completed - The scheduled maintenance has been completed. Mar 19, 00:00 UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Mar 18, 22:07 UTC Scheduled - After migrating to CDN driven crate downloads, the download counts on crates.io were undercounted for part of last week, with the 12th, 13th, and 14th being the days worst affe…
Mar 15, 12:11 UTC Resolved - Scheduled maintenance finished successfully. Mar 15, 12:00 UTC Update - Scheduled maintenance on our database is starting.We expect this to take less than 15 minutes to complete. During maintenance, crates.io will only be available in read-only mode: downloading crates and visiting the website will still work, but logging in, publishing crates, yanking crates, or chang…
Feb 15, 02:43 UTC Resolved - crates.io has returned to normal service with the processing of the download count backlog from earlier and the completion of the long running background jobs. Investigations will continue during normal hours for the crates.io team to ascertain what is causing elevated database load. Feb 15, 00:12 UTC Update - We are continuing to monitor for any further issues. Feb 15…
Feb 14, 19:25 UTC Resolved - crates.io continues to operate normally, and no further performance degradation or timeouts have been observed. Feb 14, 18:50 UTC Monitoring - Restarting the dynos has restored service, but we are still investigating the root cause of the issue, and continue to monitor crates.io performance. Feb 14, 18:45 UTC Investigating - Some crates.io endpoints are timing out at p…
Dec 27, 17:11 UTC Resolved - The incident has been resolved. Traffic and error rates are back to normal levels.The root cause has been identified and we are assessing solutions to avoid it from happening again in the future. Dec 27, 15:45 UTC Update - Download traffic is being treated and counted regularly again. We are continuing to monitor for any further issues. Dec 27, 14:37 UTC Monitoring - D…
Dec 21, 11:13 UTC Resolved - Scheduled maintenance finished successfully. Dec 21, 10:59 UTC Update - Scheduled maintenance on our database is starting.We expect this to take less than 15 minutes to complete. During maintenance crates.io will only be available in read-only mode: downloading crates and visiting the website will still work, but logging in, publishing crates, yanking crates or changin…
Nov 17, 01:52 UTC Completed - Deployment complete. Nov 17, 01:45 UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Nov 17, 01:42 UTC Scheduled - After a faulty crates deployment around 19:00 UTC (see https://status.crates.io/incidents/vth07cjjcr80 for more detail), we are going to retry the deployment. No downtime is expected, but given the ear…
Nov 16, 07:00 UTC Resolved - At 19:04 UTC a faulty deployment caused the crates.io API to not be reachable anymore. We immediately initiated a rollback to the previous deployment and the issue was resolved at 19:06 UTC. We are currently in the process of figuring out the root cause.
Oct 19, 06:45 UTC Resolved - This incident has been resolved. Oct 18, 19:06 UTC Monitoring - The traffic levels have returned to regular levels at this point. We will continue to monitor the situation. Oct 18, 19:03 UTC Investigating - We have been seeing an elevated level of failing requests due to response timeouts over the past couple of minutes. This appears to be caused by a significantly inc…
Sep 7, 01:10 UTC Completed - Maintenance was completed successfully. Sep 7, 00:30 UTC In progress - Maintenance is underway. Sep 6, 17:59 UTC Scheduled - The crates.io team will perform database maintenance on 2023-09-07 from 00:30 to 01:30 UTC to rebuild the version download charts that were affected by a crates.io bug from August 22nd to 29th.We expect this to take approximately 30 minutes to co…
Sep 5, 01:28 UTC Completed - Database maintenance is now complete. Sep 5, 00:57 UTC Update - Maintenance is going to run a little over due to an unexpected issue while rebuilding the per-version-per-day download count table, but there appears to be no user visible impact at this time. Next update around 01:20 UTC, assuming maintenance isn't complete by then. Sep 5, 00:01 UTC In progress - Maintena…