Azure Private Endpoints Are Breaking DNS: Surviving the March 2026 Default Outbound Access Retirement

On March 31, 2026, Azure completed the retirement of default outbound internet access for new virtual networks. If you deployed a VM after that date without explicit outbound connectivity — NAT Gateway, Public IP, or Firewall — you had no internet access. Period.

That was the announcement. The reality, three months later, is messier. Organizations that rushed to deploy Private Endpoints to maintain connectivity to PaaS services are now dealing with a wave of DNS resolution failures that are subtle, hard to diagnose, and devastating when they hit production.

I have seen this pattern repeat across multiple client environments in Malaysia and Singapore. The root cause is always the same: recursive DNS forwarding loops between on-premises resolvers and Azure Private DNS Zones. Here is what is happening and how to fix it.

The Three Failure Modes

Failure Mode 1: Recursive DNS Forwarding Loops

The most common failure. When you enable Azure Private DNS Zones, Azure's internal DNS resolver (168.63.129.16) handles resolution for privatelink.database.windows.net and similar zones. But if your VM's DNS configuration points to an on-premises resolver that forwards queries to Azure DNS, and that resolver also tries to resolve private endpoints, you get a loop.

The query path looks like this:

VM → On-prem DNS (forwarder) → Azure DNS (168.63.129.16) → On-prem DNS (forwarder) → Azure DNS → ...

TTL expires. Resolution fails. Your application cannot connect to the SQL Server it has been using for years.

Symptoms: - Intermittent DNS timeouts - nslookup works from some VMs but not others - Connections to PaaS services fail from new subnets but work from existing ones

The fix: Stop forwarding all queries to Azure DNS. Instead, use conditional DNS forwarding with a suffix-based policy:

# On your on-premises DNS server (Windows DNS or BIND)
# Forward ONLY privatelink.* zones to Azure DNS

# Windows DNS example
Add-DnsServerForwarder -IPAddress "168.63.129.16" -PassThru
# Then create a zone delegation or conditional forwarder:
Add-DnsServerConditionalForwarderZone -Name "privatelink.database.windows.net" -ReplicationScope "Forest" -MasterServers "168.63.129.16"
Add-DnsServerConditionalForwarderZone -Name "privatelink.blob.core.windows.net" -ReplicationScope "Forest" -MasterServers "168.63.129.16"

# Terraform example for Azure DNS Private Resolver
resource "azurerm_private_dns_resolver" "main" {
  name                = "dns-resolver-main"
  resource_group_name = azurerm_resource_group.main.name
  location            = azurerm_resource_group.main.location
  virtual_network_id  = azurerm_virtual_network.main.id
}

resource "azurerm_private_dns_resolver_inbound_endpoint" "main" {
  name                    = "inbound-endpoint"
  private_dns_resolver_id = azurerm_private_dns_resolver.main.id
  location                = azurerm_resource_group.main.location

  ip_configurations {
    private_ip_allocation_method = "Dynamic"
    subnet_id                    = azurerm_subnet.dns_resolver.id
  }
}

Failure Mode 2: The WireServer Black Hole

The IP address 168.63.129.16 is Azure's WireServer — the internal metadata and DNS service. It is routable within a VNet but not across VPN tunnels or ExpressRoute circuits. When your on-premises DNS tries to forward queries to this IP, the packets are silently dropped.

This is particularly insidious because it does not cause an error — the query just times out after 5 seconds, and the retry behavior varies by DNS client implementation.

Why this matters for hybrid environments: If you have configured your on-premises DNS to forward Azure zone queries to 168.63.129.16, those queries will fail for any resolution originating outside Azure. This includes:

  • CI/CD pipelines running on-premises that need to connect to Azure SQL
  • Disaster recovery failover scenarios where on-premises systems need Azure PaaS access
  • Developer machines accessing staging databases

The fix: Deploy Azure DNS Private Resolver with both inbound and outbound endpoints. The inbound endpoint gives your on-premises DNS a routable IP (within the VNet) to forward queries to. The outbound endpoint lets Azure DNS forward non-Azure queries to your on-premises DNS.

# Create the resolver and endpoints
az network private-dns resolver create \
  --name dns-resolver-main \
  --resource-group rg-networking \
  --location malaysiaeast \
  --virtual-network id="/subscriptions/.../virtualNetworks/vnet-main"

# Inbound endpoint (routable from on-prem via VPN/ER)
az network private-dns resolver inbound-endpoint create \
  --name inbound-ep \
  --dns-resolver-name dns-resolver-main \
  --resource-group rg-networking \
  --ip-configurations '[{"private-ip-allocation-method":"Dynamic","subnet":"/subscriptions/.../subnets/dns-resolver"}]'

# Outbound endpoint (for conditional forwarding to on-prem)
az network private-dns resolver outbound-endpoint create \
  --name outbound-ep \
  --dns-resolver-name dns-resolver-main \
  --resource-group rg-networking \
  --subnet "/subscriptions/.../subnets/dns-resolver-outbound"

# Forwarding rule: forward *.internal.contoso.com to on-prem DNS
az network private-dns resolver forwarding-rule create \
  --name forward-to-onprem \
  --dns-forwarding-ruleset name=ruleset-main \
  --resource-group rg-networking \
  --domain-name "internal.contoso.com." \
  --forwarding-rules '[{"ip-address":"10.0.1.10","port":53}]' \
  --priority 100

Failure Mode 3: Subnet Exhaustion

This one is not a DNS failure per se, but it kills Private Endpoint deployments just as effectively. Azure reserves 5 IP addresses per subnet (first four plus the broadcast). A /27 subnet gives you 27 usable IPs, minus 5 reserved, leaving 22 addresses. If you also deploy an Azure DNS Private Resolver inbound endpoint in that subnet, that consumes one more, leaving 21.

Now deploy Private Endpoints. Each one needs its own IP. A typical enterprise environment with SQL Server, Storage Accounts, Key Vault, Service Bus, Cosmos DB, and a few App Services quickly exhausts a /27.

The math:

/27 subnet: 32 - 5 reserved = 27 usable
- DNS resolver inbound: 1 IP
- Available for Private Endpoints: 26

If you have more than 26 PaaS services, you need multiple subnets or a larger CIDR block.

The fix: Plan your Private Endpoint subnet sizing before deployment. Here is a capacity planning script:

#!/usr/bin/env python3
"""Private Endpoint capacity planner"""
import ipaddress

def plan_endpoints(cidr: str, resolver_count: int = 1, azure_reserved: int = 5):
    """Calculate available Private Endpoint slots in a subnet."""
    network = ipaddress.ip_network(cidr)
    total = network.num_addresses
    usable = total - azure_reserved
    available = usable - resolver_count

    print(f"Subnet: {cidr}")
    print(f"Total IPs: {total}")
    print(f"Azure reserved: {azure_reserved}")
    print(f"DNS resolver endpoints: {resolver_count}")
    print(f"Available for Private Endpoints: {available}")

    if available < 10:
        print("⚠️  WARNING: Consider a larger subnet (/26 or /25)")

    return available

# Example: planning for a /27 with 1 DNS resolver
plan_endpoints("10.0.10.0/27", resolver_count=1)

# For a larger deployment:
plan_endpoints("10.0.10.0/24", resolver_count=2)

Recommended subnet sizing by environment:

Environment Expected PEs Subnet Size Available PEs
Development 5-10 /27 21
Staging 10-20 /26 53
Production 20-50 /25 117
Enterprise hub 50-100 /24 247

The Correct Architecture

After diagnosing dozens of these failures, here is the DNS architecture that actually works for hybrid Azure environments with Private Endpoints:

On-premises VMs/Servers
    │
    ▼
On-premises DNS (Conditional Forwarder)
    │
    ├── *.internal.contoso.com → on-prem AD DNS (10.0.1.10)
    ├── privatelink.*.azure.com → Azure DNS Private Resolver inbound (10.0.10.4)
    └── everything else → public DNS (8.8.8.8)

Azure DNS Private Resolver
    │
    ├── Inbound endpoint: 10.0.10.4 (routable from on-prem via VPN/ER)
    └── Outbound endpoint → Azure DNS (168.63.129.16)

Azure VMs
    │
    ├── DNS servers: 10.0.10.4 (resolver inbound) + 168.63.129.16 (fallback)
    └── Private DNS Zones: linked to VNet

Key design principles:

  1. Never forward all queries to 168.63.129.16 from on-prem. It is not routable over VPN/ExpressRoute.
  2. Use conditional forwarding, not global forwarding. Only route privatelink.* zones to Azure.
  3. Deploy DNS Private Resolver for hybrid resolution. It provides a routable inbound IP and handles the split-brain DNS scenario cleanly.
  4. Size your Private Endpoint subnets generously. A /27 is fine for dev; production needs /25 or /24.
  5. Use Terraform or Bicep to manage Private Endpoints as code. Manual deployment at scale is error-prone and unrepeatable.

Terraform Auto-Generation for Bulk Private Endpoints

When migrating a large environment, you often need to create Private Endpoints for dozens of PaaS services. Here is a pattern that scales:

variable "private_endpoints" {
  type = map(object({
    service_id    = string
    subresource   = string
    dns_zone      = string
  }))
  default = {
    "sql-main" = {
      service_id  = "/subscriptions/.../Microsoft.Sql/servers/sql-main"
      subresource = "sqlServer"
      dns_zone    = "privatelink.database.windows.net"
    }
    "storage-main" = {
      service_id  = "/subscriptions/.../Microsoft.Storage/storageAccounts/storagemain"
      subresource = "blob"
      dns_zone    = "privatelink.blob.core.windows.net"
    }
    "keyvault-main" = {
      service_id  = "/subscriptions/.../Microsoft.KeyVault/vaults/kv-main"
      subresource = "vault"
      dns_zone    = "privatelink.vaultcore.azure.net"
    }
  }
}

resource "azurerm_private_endpoint" "this" {
  for_each            = var.private_endpoints
  name                = "pe-${each.key}"
  location            = azurerm_resource_group.main.location
  resource_group_name = azurerm_resource_group.main.name
  subnet_id           = azurerm_subnet.private_endpoints.id

  private_service_connection {
    name                           = "conn-${each.key}"
    private_connection_resource_id = each.value.service_id
    subresource_names              = [each.value.subresource]
    is_manual_connection           = false
  }

  private_dns_zone_group {
    name                 = "dns-group-${each.key}"
    private_dns_zone_ids = [azurerm_private_dns_zone.zones[each.value.dns_zone].id]
  }
}

This pattern lets you add new Private Endpoints by simply adding an entry to the private_endpoints map. Terraform handles the DNS zone linking, the service connection, and the subnet placement.

Auditing Your Current State

Before you start fixing, audit. Here is a script to discover all existing Private Endpoints and their subnet utilization across a subscription:

#!/bin/bash
# Audit Private Endpoints across a subscription
az account set --subscription "$SUBSCRIPTION_ID"

echo "=== Private Endpoints by Subnet ==="
az network private-endpoint list \
  --query "[].{name:name, subnet:ipConfigurations[0].subnet.id, group:privateLinkServiceConnections[0].privateLinkServiceId}" \
  --output table

echo ""
echo "=== Subnet Utilization ==="
az network vnet subnet list --vnet-name vnet-main --resource-group rg-main \
  --query "[?contains(name, 'private')].{name:name, addresses:addressPrefix, used:ipConfigurations | length(@)}" \
  --output table

Save the output and cross-reference it against your expected PaaS service count. If you are close to the limit, resize the subnet before it becomes a production incident.

What Surprised Me

The hardest part of this migration was not the Private Endpoints themselves — Azure makes that fairly straightforward. The hard part was the DNS. Specifically:

  1. The failure is delayed. You deploy Private Endpoints, everything looks fine, and then three days later a different application breaks because it happens to use a different DNS path.
  2. Debugging tools are limited. nslookup and dig show you the resolution chain, but they do not tell you where the loop is. You need packet captures or DNS query logging on your resolver to see the actual traffic pattern.
  3. Documentation assumes a greenfield. Microsoft's docs walk you through setting up Private Endpoints from scratch. They do not adequately address the scenario where you have 200 VMs with existing DNS configurations that all need updating.
  4. The 168.63.129.16 address confuses everyone. New team members consistently try to ping or telnet to it from on-prem. It is not a regular IP — it is a link-local service that only works within an Azure VNet.

Key Takeaways

  1. Azure's March 2026 default outbound retirement forces Private Endpoint adoption. If you have not migrated yet, you will, and DNS will be the hard part.
  2. Recursive DNS forwarding loops are the number one failure mode. Use conditional forwarding with suffix-based policies, not global forwarding to 168.63.129.16.
  3. Deploy Azure DNS Private Resolver for hybrid environments. It solves the routable-IP problem and the split-brain DNS scenario cleanly.
  4. Plan subnet sizing before deploying Private Endpoints. A /27 gives you ~21 slots; production environments with many PaaS services need /25 or /24.
  5. Automate Private Endpoint deployment with Terraform or Bicep. Manual management at scale is error-prone and does not survive team changes.

Resources