We are providing an update on the disruption affecting the Middle East (UAE) (me-central-1) Region. After a thorough assessment, we have determined that we are unable to restore access to the resources and data hosted exclusively in the mec1-az2 Availability Zone. We continue to work on recovering regional resources, as well as zonal resources hosted in the other affected Availability Zones (mec1-…
We are providing an update on the disruption affecting the Middle East (Bahrain) (me-south-1) Region. The damage to our infrastructure spanned multiple Availability Zones and exceeded what our regional and multi-AZ services are designed to withstand. After a thorough assessment, we have determined that we are unable to restore access to the resources and data hosted exclusively in this Region. Whe…
Between 2:12 PM and 3:18 PM PDT, we experienced increased API error rates affecting the STS AssumeRoleWithSAML and AssumeRoleWithWebIdentity APIs in the US-WEST-2 Region. The root cause was determined to be due to an issue with an STS subsystem responsible for communicating with external identity providers. Other AWS Services that rely on these identity federation protocols were also affected. At…
We continue to see recovery holding steady for the STS AssumeRoleWithSAML and AssumeRoleWithWebIdentityAPIs in the US-WEST-2 Region. Error rates are now back to pre-event levels, and continue actively monitoring to confirm full recovery. We will provide another update by 5:15 PM or earlier.
We are seeing early signs of recovery and continue to monitor for full recovery. We will provide another update at 4:15 PM, or sooner if we have additional information to share.
Starting at 2:12 PM PDT, we began experiencing increased API error rates for STS and Sign In when using SAML. Our engineering team was automatically engaged at 2:19 PM to begin investigating the root cause. There is no work around available at this time. We will provide another update by 3:30 PM PDT.
Between 4:58 PM and 6:34 PM PDT, we experienced increased delays affecting real-time metrics for Amazon Connect in the AP-NORTHEAST-1 Region, resulting in missing or stale data. During this time, customers may have experienced missing data within analytics reports, and may have observed issues if accessing realtime metrics within contact Flows, such as checking agent staffing. We identified the ro…
We are experiencing increased error rates affecting real-time metrics in the AP-NORTHEAST-1 Region. Customers may experience missing or delayed real-time metric data.
Between August 18 5:00 PM and August 19 11:00 AM PDT, we experienced elevated errors launching EC2 instances in a newly launched Availability Zone (euw2-az4) in the EU-WEST-2 Region. After the new Availability Zone launch, we began experiencing errors when using a default VPC. We discovered the root cause of the issue on August 19 at 9:00 AM and began deploying a change to resolve the issue at 9:3…
On August 18 we launched a new Availability Zone (euw2-az4) in the EU-WEST-2 Region. After the launch, we began experiencing errors launching EC2 instances in the new Availability Zone when a default subnet is not present. We can confirm that existing running instances and resources are not affected. Workflows that automatically get a list of Availability Zones in the Region via the DescribeAvaila…
We are investigating an issue that is impacting launching new EC2 instances and resources in a newly launched Availability Zone (euw2-az4) in the EU-WEST-2 Region. During this time, affected customers may experience issues when creating or modifying resources in the Region. Other AWS services may also be impacted. For immediate recovery, we recommend that customers use alternative Availability Zon…
Starting August 14 7:33 PM PDT, we experienced increased packet loss impacting AWS Direct Connect connectivity for customers with connections at the Equinix FR5 location in Frankfurt, DEU. Engineers were automatically engaged at 7:45 PM on August 14 and immediately began investigating mitigations. By 8:30 PM, we identified that network equipment at the FR5 location was impaired due to water ingres…
We are seeing significant recovery for most of the customer connections at this stage. While we are not yet fully recovered, restoration efforts are progressing as expected at the Equinix FR5 location in Frankfurt, DEU. Remediation of remaining infrastructure involves completion of hardware replacements and traffic validation, both of which are actively underway. We anticipate further customer-vis…
We continue to work on bringing affected hardware back online. Since our last update we have made progress that will not be visible to customers, but is required for recovery. We are working in parallel to bring all devices online as safely as possible. This work is expected to take several hours to complete and validate. For customers that require workarounds, we recommend that you consider faili…
We are seeing broad signs of recovery at the Equinix FR5 location in Frankfurt, DEU. We have restored connectivity for the majority of affected hardware and most of the connections are fully recovered and stable. There are a small number of customers that will remain affected until the remaining devices are fully restored. We will provide another update by 2:00 PM PDT, or sooner as remediation pro…
We continue to work on phased remediation incrementally at the Equinix FR5 location in Frankfurt, DEU. We are seeing early signs of recovery while we continue to fully remediate the issue. We are actively working to bring the remaining affected hardware back online and we will provide another update by 12:30 PM PDT, or sooner as remediation progresses.
We continue to make progress in our phased remediation at the Equinix FR5 location in Frankfurt, DEU. Network infrastructure and dependent systems continue to improve as we bring affected hardware back online. Some replacement hardware has been delivered and installation is proceeding as components arrive on-site. In parallel, we are shifting network traffic to allow restored devices to begin serv…
We continue to make progress in our phased remediation at the Equinix FR5 location in Frankfurt, DEU. Since our last update, some dependent network infrastructure has been restored. Remediation of remaining infrastructure is ongoing, with a portion of recovery dependent on the delivery of replacement hardware. Cooling has been fully restored, with environmental conditions stable within normal oper…
We have completed our assessment of the impacted network equipment at the Equinix FR5 location in Frankfurt, DEU and now have a clear understanding of the scope of impact. We are making progress toward restoring connectivity and will be taking a phased approach to remediation. Customers with connections exclusively at this location will continue to experience packet loss until remediation is compl…
The electrical isolation at Equinix FR5 in Frankfurt, DEU is now complete and our engineers have begun physically inspecting the impacted network equipment. We do not yet have a timeline for full resolution while we continue to assess the extent of impact to equipment. Direct Connect connections at this location remain impaired. Customers with connections exclusively at this location will continue…
We continue to work with our co-location partner to restore connectivity at the Equinix FR5 location in Frankfurt, DEU. Since our last update, we have made significant progress toward restoring safe access to the affected area. The electrical isolation procedure is now underway, with our teams on site in the electrical room executing the de-energization of the affected infrastructure. Once isolati…
We are actively engaged with co-location partner to restore connectivity at Equinix FR5 location in Frankfurt, DEU. Since our last update, we have made incremental progress to restore safe access to the affected area at the Equinix FR5 location in Frankfurt, DEU. In parallel, we have prioritized the order in which critical and high-priority racks will be restored, as part of the mitigation efforts…
Our co-location partner is continuing work to restore safe access to the affected area at the Equinix FR5 location in Frankfurt, DEU. Once safe access has been secured, our engineers will be able to assess the affected network devices. We continue to closely track progress and will share an update by 9:30 PM PDT, or sooner as new information becomes available.
Our co-location partner continues to work to resolve the underlying facility infrastructure issue at the Equinix FR5 location in Frankfurt, DEU. Access to the affected area is currently restricted due to safety concerns, which is impacting our ability to assess the physical condition of the network equipment and provide a more accurate recovery timeline. Based on current information, full recovery…
Engineers continue work to restore connectivity at Equinix FR5 location in Frankfurt, DEU. Our co-location partner is working to resolve the underlying facility infrastructure issue and while the improvements are not yet customer visible, we are making positive progress towards resolution. For customers who require immediate recovery, we recommend failing over to VPN, as outlined in our previous u…
AWS Direct Connect connectivity remains impaired for customers with connections at the Equinix FR5 location in Frankfurt, DEU. Customers with multi-site or redundant configurations across other locations continue to be unaffected. Engineers are actively working to restore connectivity, with efforts ongoing across multiple workstreams to resolve the underlying facility issue and bring the impacted…
We continue to work towards recovery of connectivity for AWS Direct Connect connections at Equinix FR5, Frankfurt, DEU. The root cause is related to a facility infrastructure issue at the location that is impacting network infrastructure. Customers with connections solely at this location will continue to experience packet loss or connectivity degradation. Customers with multi-site or redundant co…
We are actively working to restore connectivity through the Direct Connect location: Equinix FR5, Frankfurt, DEU. Customers whom only have connections at the Equinix FR5, Frankfurt, DEU location will continue to experience connectivity issues. For a workaround impacted customers who have the option available to failover to VPN are recommended to do so to achieve recovery. For customers using Direc…
Starting at 7:33 PM PDT, we began experiencing increased packet loss impacting AWS Direct Connect connectivity for some customers in the EU-CENTRAL-1 Region. While we have made progress, connections to the following Direct Connect location are still impaired: Equinix FR5, Frankfurt, DEU. Customers who have multi-site redundancy configured with their Direct Connect paths should not be observing imp…
We can confirm packet loss impacting Direct Connect connections in the EU-CENTRAL-1 Region. Engineers were automatically engaged and immediately began working to both identify the root cause, and identify multiple parallel paths to mitigate the issue. At this time, we are seeing early signs of recovery. We will provide another update in 60 minutes, or sooner if we have additional information to sh…