Level3 Status Update ---- Root Cause: An offending flowspec announcement prevented BGP from establishing correctly, impacting client services.
Fix Action: The NOC deployed a configuration change to block the offending flowspec announcement, restoring stability to the BGP protocols.
Summary: On August 30, 2020 10:04 GMT, CenturyLink identified an issue to be affecting users across multiple markets. The IP Network Operations Center (NOC) was engaged, and initial research identified that an offending flowspec announcement prevented Border Gateway Protocol (BGP) from establishing across multiple elements throughout the CenturyLink Network. The IP NOC deployed a global configuration change to block the offending flowspec announcement, which allowed BGP to begin to correctly establish. As the change propagated through the network, the IP NOC observed all associated service affecting alarms clearing and services returning to a stable state.
This service impact has concluded.
Atlantic.Net Network Engineering will plan on reintegration of the Level3 carrier links this evening once they have validated that Level3 network is stable.
Posted Aug 30, 2020 - 12:47 EDT
Our BGP advertised routes are now starting to clear out of the Level3 network and other carrier connections are starting to withdraw the routes from their tables. Network connectivity is now starting to return.
We will continue to monitor the updates coming from Level3 --- At this time, Network Engineering will keep the Level3 circuit admin down state.
Posted Aug 30, 2020 - 11:06 EDT
Level3 has posted an update to this incident. ---- The IP NOC has isolated multiple Border Gateway Protocol (BGP) issues causing service impacts across multiple markets. Cooperative escalated investigation efforts with CenturyLink leadership are underway to isolate and troubleshoot the issue and expedite service restoral.
Posted Aug 30, 2020 - 09:23 EDT
Network Engineers have attempted to disconnect our Level3 carrier links --- however Level3 is still announcing our IP space via their network. Level3 has confirmed this to be a major network event within their network and are working to resolve the issue as quickly as possible.
We will post further updates as we have them available.
Posted Aug 30, 2020 - 08:33 EDT
This issue appears to be a global event within the Level3 network. We have attempted to switchover our carrier links to alternative carriers, however Level3 is still advertising our BGP peering routes -- not sure how, as we have disconnect our circuits to their network.
Posted Aug 30, 2020 - 07:39 EDT
We are aware of potential carrier transport issues in this region and are currently investigation the issue. We will update once we have more information.
Posted Aug 30, 2020 - 06:57 EDT
This incident affected: Regions (Orlando, Florida (USA - East-1)).