"I/O timeout" error when you access Azure Container Registry

Summary

This article describes how to resolve an I/O timeout error that occurs when you try to access Azure Container Registry (ACR) to pull a container image or artifact.

Symptoms

You receive the following error message.

error pulling image configuration: download failed after attempts=6: dial tcp <storage-endpoint-ip-address>:443: i/o timeout

Cause 1: Port 443 is unavailable on the storage (data) endpoint

Azure allocates blob storage in Azure Storage accounts on behalf of each registry to manage data for container images and other artifacts. When a client accesses image layers in a container registry, it makes requests by using a storage account endpoint that the registry provides.

The domain name of the default storage (data) endpoint depends on whether you're using dedicated data endpoints, as shown in the following table.

To manually test the connectivity between your device and the storage (data) endpoint on port 443, run the telnet or NetCat command. If dedicated data endpoints are used, choose one of the following commands:

• Telnet command.

  telnet <container-registry-name>.<region-name>.data.azurecr.io 443
  

• NetCat command.

  nc -vz <container-registry-name>.<region-name>.data.azurecr.io 443
  

If dedicated data endpoints aren't used, choose one of the following commands:

• Telnet command.

  telnet <storage-account-name>.blob.core.windows.net 443
  

• NetCat command.

  nc -vz <storage-account-name>.blob.core.windows.net 443
  

Solution 1: Make port 443 available for communication between the device and the storage (data) endpoint

Ensure that there's networking connectivity between the device and the storage (data) endpoint over port 443.

If your device is part of a restricted network architecture, ensure that it has networking connectivity with the storage (data) endpoint over port 443. Consider checking potential existing firewalls, proxy servers, access control lists, internet service provider (ISP) restrictions, and so on.

Cause 2: Communication is blocked by an NSG that's associated with the NIC or subnet of a VM

If you're using an Azure virtual machine (VM) to pull from ACR, a network security group (NSG) might be blocking communication between the device and the sign-in server. The blocking NSG is associated with the network adapter or subnet of the Azure VM.

Run the telnet or NetCat command that's shown in Cause 1. If the command output shows that a time-out occurred, check whether the NSG configuration blocks the IP address of the storage account. To do this, follow these steps:

1. Go to Azure IP Ranges and Service Tags – Public Cloud, and then select Download to download the list of service tags as a JSON file.

2. Open the downloaded JSON file in a text editor, and then search for Storage.<region-name>. (Replace the <region-name> placeholder with the name of the region that contains your container registry, such as BrazilSoutheast.)

3. Choose an IP address that's allowable for use in that region. (You'll use this IP address in step 6 as the associated storage (data) endpoint IP address.)

4. In the Azure portal, search for and select Network Watcher.

5. In the Network Watcher menu pane, select Network diagnostic tools > NSG Diagnostics.

6. In the Network Watcher | NSG diagnostics page, complete the form by following the instructions in the following table.

   Field	Action
   Target resource type	Select Virtual machine in the list.
   Virtual machine	Enter or select the name of your Azure VM in the list.
   Protocol	Select TCP in the list.
   Direction	Select the Outbound option.
   Source type	Select IPv4 address/CIDR in the list.
   IPv4 address/CIDR	Enter the IP address of the Azure VM.
   Destination IP address	Enter the associated storage (data) endpoint IP address that you chose in step 3.
   Destination port	Enter 443.

7. Select Run NSG diagnostic.

8. In the Results box, check the value of the Traffic status field.

The Traffic status field can have a value of Allowed or Denied. The Denied status means that the NSG is blocking the traffic between the Azure VM and the associated storage (data) endpoint IP address that you chose. In this case, the name of the blocking NSG is also shown in the Results box. In the result table, locate the NSG name column for the row in which the corresponding Applied action column's value is Deny.

Solution 2: Modify the NSG configuration to allow connectivity between the VM and the associated storage (data) endpoint

Make changes at the NSG level so that connectivity is allowed on port 443 between the Azure VM and the associated storage (data) endpoint (all the IP address ranges or the associated service tag). Specifically, make sure that the following conditions are met.