News

Microsoft: Azure Outage Caused by Ubuntu Update Issue

• By Kurt Mackie
• August 30, 2022

According to Microsoft, Azure customers who had upgraded Canonical Ubuntu on Azure virtual machines may be running into domain name system (DNS) problems.

The problem specifically affected customers "running Ubuntu 18.04 (bionic) VMs [that were] recently upgraded to systemd version 237-3ubuntu10.54," Microsoft's alert indicated. Microsoft also confirmed the problem in this Azure Support Twitter post.

Canonical issued this announcement yesterday describing a "confirmed bug" for Ubuntu Systemd upgrades, where there are problems with resolving DNS. The announcement didn't mention the Azure virtual machines problems, though.

Microsoft's alert included a table showing that multiple Azure services were affected -- such as Azure Container Apps, Azure Database for PostgreSQL Azure Kubernetes Service and Azure VMware Solution. Presumably, those services were just affected for the Ubuntu upgraders using Systemd (see chart snippet).

The problems started at about 6:00 in the morning UTC time on Aug. 30, according to the alert, or about 11:00 p.m. Pacific Daylight Time on Aug. 29, per a timeanddate.com reckoning. The outages apparently have lasted 12 hours- plus, so far.

At press time, Microsoft was investigating the issue, which was just described as affecting Ubuntu 18.04 (bionic) upgraders using systemd. Here's Microsoft's statement to that effect and its advice:

This bug and a potential fix have been highlighted on the Canonical / Ubuntu site, which we encourage impacted customers to read: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1988119
An additional potential workaround customers can consider is to reboot impacted VM instances so that they receive a fresh DHCP lease and new DNS resolver(s).
If you are running a VM with Ubuntu 18.04 image, and you are experiencing connectivity issues, we recommend you evaluate the above mitigation options.
If you are not experiencing impact on your Ubuntu 18.04 images, but you have unattended security updates enabled, we recommend you review this setting until the Ubuntu issue is mitigated.

Former Microsoft employee Kevin Beaumont described the DNS problems for Azure users in this Twitter post. He noted the odd circumstance that it's an Azure problem that's being blamed on Canonical.

"Azure are blaming Canonical (Ubuntu) but it only appears to impact Azure hosted VMs," Beaumont wrote.

The problem was also flagged by Dr. Nestori Syynimaa, a senior principal security researcher at Secureworks, in this Twitter post.

"These things happen all the time and will happen as long people are using technology," Syynimaa wrote in that post, adding "not on this scale that often though."