By 16:34 UTC, impacted resources and services were mitigated. By 10:15 UTC, Azure Storage engineers identified the configuration change as the source of the failed requests and began reverting it on select scale units.
Around 10:00 UTC, Azure Storage engineers were engaged and began investigating the cause and potential mitigation paths.
While this configuration change followed the usual Safe Deployment Practices (SDP) ( ), the change had adverse effects on a small set of scale units where the mechanism unexpectedly throttled some storage accounts in an attempt to bring the scale unit to a healthy state.Īround 08:14 UTC on 06 March 2023, automation detected an issue with Azure Key Vault requests. A new configuration was rolled out to improve the throttling algorithm.
In the event that a scale unit reaches such limits, the scale unit employs a safety protection mechanism to throttle accounts that are deemed to contribute to the overload, while also balancing load across other scale units.Ĭonfiguration settings are utilized to control how the throttling services monitor usage and apply throttling. This is our Preliminary PIR that we endeavor to publish within 3 days of incident mitigation, to share what we know so far.Īfter our internal retrospective is completed (generally within 14 days) we will publish a Final PIR with additional details/learnings.īetween 06:35 UTC and 17:55 UTC on 06 March 2023, a subset of customers using Azure Storage may have experienced greater than expected throttling when performing requests against Azure Storage resources located in the West Europe region.Īzure services dependent on Azure Storage, including Azure Key Vault, Data Factory V2, Azure Automation, Azure Bastion, Azure Synapse Analytics, Azure HDInsight, ExpressRoute Gateway, Azure Logic Apps, and Azure Arc Enabled Kubernetes may also have experienced intermittent failures and degraded performance due to this issue.Īzure Storage employs a throttling mechanism that ensures storage account usage remains within the published storage account limits ( ). This throttling mechanism is also used to protect a storage service scale unit from exceeding the overall resources available to the scale unit.
0 kommentar(er)
