San Francisco, CA | Cloud Node Event

Incident Report for Atlantic.Net

Postmortem

Incident Summary

On April 16, 2026, at approximately 1:17 AM ET, the Cloud Host supporting one or more of your services experienced a critical shutdown due to a hardware failure. This resulted in an unplanned service interruption for the affected servers.

Our investigation began immediately at approximately 1:17 AM ET. After determining that the incident was caused by a critical hardware failure, our Engineering team began recovery efforts using available VM backups at approximately 4:00 AM ET.

All affected servers were restored from the most recently available backup, dated Wednesday, April 15, 2026, at 6:00 PM ET. Services were fully restored for all affected servers by approximately 6:00 AM ET, resulting in a total service interruption of approximately 5 hours and 43 minutes.

Actual downtime may vary by server, as some affected servers were restored earlier than others.

Root Cause

Our investigation determined that the interruption was caused by a hardware failure involving the storage drive backplane within the affected Cloud Host.

This failure caused the host to stop recognizing the installed storage drives, which resulted in the hypervisor crashing and the affected Cloud services becoming unavailable.

Immediate Actions Taken

To reduce the risk of further disruption, the affected host was removed from active service and marked so that no additional servers could be created on, or migrated to, that system.

To restore service as quickly as possible, our Engineering team restored the affected servers from the most recently available backups and brought them online on known-good Cloud Hosts.

Preventative Measures

Our Engineering team will fully service and evaluate the affected host before it is considered for any future production use.

No customer workloads will be returned to this system unless and until the evaluation is complete and the host is confirmed fit for continued production operation.

We apologize for the disruption this incident caused and appreciate your patience, understanding, and continued trust.

Posted Apr 28, 2026 - 03:46 EDT

Resolved

Our Cloud engineers have completed the restoration of all impacted servers to the date and time below. If your server was impacted, you will receive an email from our Support team with further information.

This incident has been resolved.
Posted Apr 16, 2026 - 06:13 EDT

Update

Our Cloud engineers are still working on restoring servers impacted by this incident. You may observe your services have been restored to the date and time outlined previously. Remaining impacted servers are expected to be restored shortly.
Posted Apr 16, 2026 - 05:33 EDT

Update

We are continuing to work on a fix for this issue.
Posted Apr 16, 2026 - 05:31 EDT

Update

Our Cloud engineers have determine that the Cloud node hardware is not able to be restored to a working state. As such, backups of individual servers affected by this incident will be restored to working nodes at the location. Backups from the following date and time will be used:

Wednesday, 4/15/2026, at 6:00PM Eastern Time

Our engineering team will continue to investigate the failure in an effort to perform data recovery for any affected VMs. If your services were affected by this incident, you will receive a separate email from our Support team with more information.
Posted Apr 16, 2026 - 04:01 EDT

Update

Our Cloud engineers are continuing to work diligently to return the Cloud node to full functionality. Further updates will be provided as soon as they are available. Thank you for your continued patience.
Posted Apr 16, 2026 - 02:20 EDT

Identified

A Cloud node in the San Francisco data center is currently experiencing unexpected issues which our Cloud Operations team is working to resolve. Virtual machines hosted on the associated node may be affected at this time. We will continue to provide updates as we work toward resolution. Please direct all questions to support@atlantic.net. Thank you for your patience.
Posted Apr 16, 2026 - 01:31 EDT
This incident affected: Services (Cloud Services - Server Provisioning, Cloud Services - Servers) and Regions (San Francisco, California (USA-West-1)).