Virtual Machine: Disk Latency Regression — Impacting 25 Linux VMs

Thanks for the replies. We collected fresh Azure Monitor data for one affected VM. How should we interpret the absence of reported bandwidth burst-credit use alongside disk pressure and slower workloads? Does this evidence warrant a platform-side investigation?

Disk-intensive workloads became roughly 2–3 times slower around September 24–27, 2026. Our original monitoring comparison flagged approximately 25 production VMs, mostly in West Europe and one in East US 2. The example below concerns one West Europe VM.

Configuration

• Standard_D8as_v7 with a 512 GiB Standard SSD LRS OS disk, ReadOnly host caching, and no separate managed data disk.
• Ubuntu 24.04.5 LTS, currently running kernel 7.0.0-1017-azure.
• The managed OS disk is exposed through NVMe as /dev/nvme0n1, with ext4 root on /dev/nvme0n1p1. A loop filesystem is backed by a file on root. Root mount options include discard and commit=30.

Azure Monitor observations

We queried the same VM on November 3 using five-minute intervals. The baseline covers September 16–23, 2026, end exclusive; the recent period covers October 27 at 14:39:37 UTC through November 3 at 14:39:37 UTC.

MB/s means decimal MB/s. One recent burst-credit interval is missing and excluded. Azure Monitor reports a 100 MB/s target and 150 MB/s maximum burst bandwidth in both periods. It reports target/max-burst IOPS of 550/600, while the current disk resource reports 500 IOPS and 100 MB/s. What explains the difference in target IOPS?

Two five-minute intervals provide comparison points:

The November interval suggests disk-level pressure despite VM uncached bandwidth headroom. Write throughput alone was below the disk's 100 MB/s target; could reads or caching explain the near-100% disk bandwidth consumption?

We understand that 0% used credits does not mean credits were exhausted, and recent IOPS burst-credit use does not mean all bursting stopped. Production workloads differed between periods, so these observations do not prove failed burst delivery. We also understand that credit-based bursting is best effort.

Latency and guest evidence

Our earlier 100–200 ms figures came from Azure Monitor's OS Disk Latency metric, which Microsoft documents as unsupported for NVMe controllers. Those figures and the original latency-based fleet selection need qualification. The observations above do not rely on that latency metric.

On September 27 at 03:26:52 UTC, kernel 7.0.0-1014-azure reported dockerd blocked in D state for more than 122 seconds. The trace shows a filesystem writeback wait during OverlayFS unmount, but does not identify the cause. A containerd.io update occurred on September 26, and recent logs contain OverlayFS warnings about overlapping lower/upper/work directories, so a guest-side contribution remains possible. A mostly idle ten-second guest sample on November 3 did not reproduce the slowdown. Our original monitoring comparison flagged approximately 25 servers, although the latency metric used for that selection is unsupported on NVMe. Other servers were not flagged, and newly provisioned servers with comparable configurations have not shown the same workload slowdown.

Checks completed

Kernel rollback did not resolve the issue. Premium SSD upgrades helped partly but did not restore previous performance. Resource Health and Service Health showed no explanatory events. We have not performed a controlled resize or cache-mode change on this VM.

What does this evidence indicate about disk pressure and burst behavior? If another customer-side test is needed to distinguish possible causes, please explain what it would establish and which measurements to collect together.

I think the 0% burst reading needs checking. If it's OS Disk Used Burst IO Credits Percentage, that tracks credits used, not remaining. So 0% doesn't mean they're exhausted. OS Disk Latency (Preview) isn't available for NVMe either.

Which metric shows 0%, and how was the 100–200 ms measured? For one affected VM, compare disk IOPS, throughput and latency around Sept 24–27. Send that comparison, the VM/disk resource IDs and UTC timestamps to Azure support to check for platform-side issues.

Likely cause: because this started in a narrow time window across many VMs and multiple disk SKUs/regions, this is more likely a throughput/IO-path constraint at VM+disk combination level (or host/caching interaction) than a single-guest regression.

1. Validate effective limits at both layers (VM and disk), not disk alone:
   • VM-level IOPS/throughput caps can throttle all attached disks.
   • Disk SKU limits and bursting behavior must be checked together.
   Reference: https://learn.microsoft.com/en-us/azure/virtual-machines/disks-performance
2. Correlate per-VM metrics during the regression window:
   • OS disk Read/Write latency
   • VM uncached IOPS/throughput utilization
   • queue depth / await / %util in guest (iostat)
   If VM-level utilization is near cap while disk-level headroom exists, the bottleneck is likely VM-side.
3. Test one controlled mitigation branch:
   • Branch A: temporarily move one representative VM to a size with higher uncached disk throughput/IOPS.
   • Branch B: for one VM, test host caching mode change (ReadOnly → None) on OS disk in maintenance window.
   • Branch C: place workload on a dedicated data disk (or striped set) to isolate OS-disk bottleneck.
4. Because this affects ~25 production VMs across regions, open a support request for backend platform correlation and include:
   • impacted VM list + regions
   • exact onset window (Sept 24–27)
   • before/after latency metrics and burst-credit behavior
   • one VM where mitigation branch changed outcome
   Support process: https://learn.microsoft.com/en-us/azure/azure-portal/supportability/how-to-create-azure-support-request

Confirmed facts from your post:

• Cross-VM impact (~25 Linux VMs)
• Latency increased to ~100–200 ms
• Burst metrics changed behavior in the same period
• No conclusive cause after guest/kernel/SKU checks

Possible cause (not yet confirmed): VM-level disk bandwidth ceiling or host-path behavior change interacting with NVMe + ReadOnly cache pattern.

Validation checks:

• Latency returns near pre-regression baseline on test VM.
• VM uncached throughput utilization drops below sustained cap.
• Application response times recover proportionally.

Risk/side effect notes:

• VM resize and caching changes can require restart and may cause short downtime.
• Caching changes can alter consistency/performance characteristics; test on one VM first.

“If you find this answer helpful, please upvote it. If this resolves your issue, please mark it as the accepted answer so that it can help others in the community.”