Our Cloud Engineering team continues to migrate VMs off of the impacted host node, however with the mitigation steps put in place by our team there has been no further service impact. We will complete further troubleshooting once all VMs are off the node, but at this time all customer systems are operational and have been since 7:30 PM EST yesterday.
Posted Sep 29, 2021 - 19:06 EDT
Monitoring
At this time, all site monitoring is stable and has been stable since approximately 6:30 PM EST. We have received some reports of high latency for VMs still pending migration off the node, however these were last reported at 7:30 PM EST. We will continue work to evacuate the node overnight and will update this status with more information once troubleshooting is complete.
All systems should be operational at this time.
Posted Sep 28, 2021 - 20:00 EDT
Update
Our Cloud and Network Engineers have identified the cause of the intermittent host issues and have put a temporary fix in place. They will continue work to evacuate the node in order to ensure that customer VMs are not impacted. Additional troubleshooting will be completed once the node has been fully evacuated.
Posted Sep 28, 2021 - 18:30 EDT
Update
We are continuing to investigate this issue.
Posted Sep 28, 2021 - 17:30 EDT
Investigating
At 4:10 PM EST this afternoon our Cloud Engineers received notice of a node that stopped communicating with our internal monitoring. Upon investigation, it was discovered the system was experiencing intermittent network connection drops. Currently, our staff is working to migrate VMs off of this node while also isolating the cause of the intermittent drops.
Posted Sep 28, 2021 - 16:30 EDT
This incident affected: Regions (Dallas, Texas (USA-Central-1)).