AT&T’s massive outage on February 22, 2023, was far more extensive than initially reported. A new report from the Federal Communications Commission (FCC) reveals the full scale of the disruption, highlighting the impact on over 125 million devices and the blocking of over 92 million voice calls. The outage even affected 911 emergency services, preventing over 25,000 calls from reaching call centers. This inability to access emergency services is a major concern, as it highlights the critical need for reliable network infrastructure.
The FCC report indicates that AT&T prioritized restoring FirstNet, a dedicated network for first responders, before addressing service for residential and commercial users. While this approach appears reasonable, the FCC criticizes AT&T for multiple failures related to the network update, including insufficient testing. The report points to a lack of oversight and controls to ensure test processes were followed, or the processes themselves were inadequate. AT&T was also unprepared for the surge in traffic caused by devices attempting to reconnect to the network simultaneously. The company’s network infrastructure lacked the necessary robustness to manage this congestion.
The outage affected all 50 states, U.S. territories, including Washington D.C., Puerto Rico, and the U.S. Virgin Islands. The FCC reprimands AT&T for not adhering to industry best practices, which mandate thorough testing, review, and approval of all network changes before implementation. The outage lasted over 12 hours before AT&T fully restored service. This event echoes the recent Crowdstrike outage that disrupted Microsoft devices globally and was also attributed to an untested update with faulty code. It also resembles a similar outage experienced by Verizon in December 2022, which resulted in an FCC penalty and a consent decree.
While AT&T acknowledged responsibility for the outage and provided a $5 credit to affected customers, the FCC’s response is likely to be more severe. The Public Safety and Homeland Security Bureau has referred the matter to the FCC Enforcement Bureau to investigate potential violations of FCC rules. If found responsible, AT&T could face a substantial fine and be required to implement a compliance plan, similar to the $1.05 million fine imposed on Verizon for its December 2022 outage.
In response to the outage, AT&T implemented new technical controls within two days, including scanning the network for any elements lacking necessary controls and promptly addressing these deficiencies. The company is also conducting ongoing forensic work and implementing additional enhancements to bolster network robustness and resilience. Furthermore, AT&T has adopted new procedures to ensure peer reviews and confirm that maintenance work cannot proceed without completed peer reviews. These measures aim to prevent future outages.
However, the recurring issue of untested updates being pushed out across the industry suggests a broader problem. This incident serves as a stark reminder for companies to prioritize robust testing and rigorous oversight to prevent similar disruptions in the future.
