.NET & reliability · field note
Intermittent 502s on Azure App Service? Check for SNAT port exhaustion before you scale up
Intermittent 502s and SQL timeouts under load, while CPU and memory look fine, often mean the app has run out of outbound connections. Here is how to confirm SNAT port exhaustion and fix it in .NET.
A client's .NET app on Azure App Service started throwing 502s. Not all the time. A few minutes of failures under load, then everything looked fine again.
The logs also had SQL timeouts in them, so the database got the first suspicion. It wasn't the database. It was a few lines of code that created a new HttpClient inside a loop.
That's SNAT port exhaustion on Azure App Service, and it's one of the more frustrating production problems to chase, because nothing in the default metrics points at it. CPU and memory look normal. The app just can't open new outbound connections fast enough, and every symptom shows up somewhere else.
01 / Read the limit
What is actually running out?
Outbound connections from App Service to public addresses go through source network address translation (SNAT), and each instance starts with a preallocated pool of 128 SNAT ports. A port is used for every open connection to the same destination address and port. When the app closes a connection, Azure doesn't hand the port back straight away. It waits about four minutes.
That four-minute wait is the part people underestimate.
If your code opens a fresh connection to the same API about once every two seconds and never reuses it, you've used 120 ports in four minutes. That's the edge of the preallocated pool at modest traffic. Push the request rate up and new connections start queueing or failing. This is an illustration, not a client result.
Azure can allocate more ports on demand, but you don't control when, and the failures happen while you wait. Ports to different destinations don't compete in the same way, which is why this usually bites on the one endpoint the app calls most, like a payment API or a partner's REST service.
02 / Find the source
Why does a new HttpClient per call cause it?
Each HttpClient you create with new gets its own handler, and the handler owns the connection pool. Create a client per request and every request opens a new connection, with no pool to reuse. Here's the shape of what we found, simplified:
// Simplified version of the pattern. Don't do this.
foreach (var order in orders)
{
using var client = new HttpClient();
var response = await client.PostAsJsonAsync(endpoint, order);
// ...
} The using makes it look responsible. It doesn't help. Disposing the client closes the connection, and the closed connection still holds its SNAT port until Azure reclaims it. So the loop burns a port per iteration, and with enough orders in a batch you can empty the pool in a single run.
It also explains why the problem never showed up in testing. At test traffic, the four-minute window never fills. In production, with real batch sizes and a few users hitting the same path, it fills in minutes.
They were the misleading part. When outbound calls stall waiting for a port, requests back up behind them, and the database calls inside those same requests start hitting their timeouts too. Once the HTTP code was fixed, the SQL timeouts went away with the 502s. If you see both together, don't start by tuning the database.
03 / Check the evidence
How do you confirm it before changing anything?
Start in the portal. Open the app, go to Diagnose and solve problems, pick Availability and Performance, and open the SNAT Port Exhaustion detector. It shows SNAT port allocation per instance over time. Microsoft's guidance is to stay under 128. If you see spikes that line up with your 502s, you have your answer.
While you're there, search the same page for the TCP connections detector. It's a separate limit (total connections per worker, including local ones), and it tells you whether the app is opening too many connections in general or just too many to one place.
Two things the portal won't do for you:
- There's no SNAT metric you can alert on or autoscale on. You find out from the detector, usually after the incident.
- It won't tell you which code path is responsible. For that, search the solution for
new HttpClient(andnew HttpClientHandler(, then check which of those run per request or inside loops. Background jobs and WebJobs are common hiding places.
04 / Fix the code
The fix: let IHttpClientFactory own the connections
IHttpClientFactory fixes this by separating the HttpClient you use from the handler that holds the connections. The factory keeps a pool of handlers and hands each new client one that's already warm. You can create clients as often as you like and they share the same connections. Disposing a factory-made client doesn't dispose the handler, so the connection pool survives. The factory also recycles handlers every two minutes by default, so clients still pick up DNS changes.
For an ASP.NET Core app, a typed client is the cleanest way in. Register it once:
// Program.cs
builder.Services.AddHttpClient<OrdersApiClient>(client =>
{
client.BaseAddress = new Uri("https://api.example.com/");
client.Timeout = TimeSpan.FromSeconds(30);
}); Wrap the calls in a small class that takes HttpClient through its constructor:
public class OrdersApiClient(HttpClient http)
{
public Task<HttpResponseMessage> SendAsync(Order order, CancellationToken ct) =>
http.PostAsJsonAsync("orders", order, ct);
} And the loop becomes boring, which is what you want:
foreach (var order in orders)
{
await ordersApi.SendAsync(order, ct);
} Same loop, same traffic, but the connections get reused instead of thrown away.
One trap to watch. Typed clients are meant to be short-lived. If you inject one into a singleton service, the singleton keeps that client forever, the factory loses control of it, and it stops picking up DNS changes. In a singleton, inject IHttpClientFactory and create a named client when you need it, or configure the client with a SocketsHttpHandler and a PooledConnectionLifetime.
If you can't use dependency injection (an older console job, a library), one shared client works too. In .NET Core and .NET 5 onward, create a single static HttpClient with a SocketsHttpHandler and set PooledConnectionLifetime to a few minutes. That handles both port reuse and DNS. On .NET Framework, Microsoft recommends the factory.
Check your retries while you're in there. Aggressive retry loops make SNAT exhaustion worse, because every failed attempt can open another connection to the same endpoint. Use backoff, and cap the attempts.
My view: for anything running in ASP.NET Core, use typed clients through the factory. Reach for a static client only where DI genuinely isn't available. The code change is small, it reverts cleanly if something goes wrong, and it fixes the cause rather than adding capacity around it. If the failing path isn't obvious yet, that's the first job in a .NET and Azure stability review.
05 / Avoid the expensive detour
Why doesn't scaling up fix it?
Because the 128 preallocated SNAT ports are per instance, not per size. Moving to a bigger plan gives each worker more CPU and memory and a higher total TCP connection limit, but the SNAT starting pool is the same. If the code burns a port per request, a bigger worker burns them just as fast.
Scaling out does help, a bit. More instances means more ports in total and less traffic per instance. That's often how teams "fix" it during an incident, and it works well enough that nobody goes back. Then the bill carries extra instances for months to cover a code bug. If your App Service spend jumped after an incident and never came down, that's worth checking. It's one of the patterns in what I check first when an Azure bill goes up.
The same instinct shows up with VMs. A size that looks too big might be carrying a problem nobody has named yet, which is the point of Advisor's oversized VM that turns out to be load-bearing.
06 / Change the network
When do private endpoints or a NAT gateway make sense?
After the code is fixed, the network is the next layer, and it's worth doing for its own reasons.
Private endpoints for Azure services. If the app reaches Azure SQL, Storage or other Azure PaaS services over their public endpoints, that traffic goes through SNAT too. With regional virtual network integration plus private endpoints (or service endpoints), traffic to those services isn't subject to SNAT port limits at all. For the database side of the story, it's the better setup for the long run anyway.
A NAT gateway for external APIs. For third-party endpoints outside Azure, a NAT gateway on the integration subnet gives you 64,512 SNAT ports per public IP, up to 16 IPs, plus a fixed outbound IP you can give partners for allowlisting. There are conditions:
- The app needs regional virtual network integration, with all outbound internet traffic routed through it.
- Once a NAT gateway is in place, traffic to Azure Storage must use private endpoints or service endpoints.
- For App Service Environment, it's supported on v3 only.
- It's a billed resource, so it adds a line to the bill.
When I wouldn't start with it: as the first response to a code bug. A NAT gateway gives a leaking app enough room that the leak stops hurting, and the bad pattern spreads to the next service someone builds. Fix the client code, then decide whether you still need the gateway. Plenty of teams keep one afterwards just for the fixed IP, and that's a fine reason.
One caution on rollout. Routing all outbound traffic through the virtual network changes how every outbound call leaves the app, including calls you forgot it makes. Try it on a non-production copy first, or in a quiet window with a clear way back.
07 / Put it into practice
What to check this week
- For each busy App Service app, open the SNAT Port Exhaustion detector and look at the last week. Note anything near or above 128.
- Search your solutions for
new HttpClient(andnew HttpClientHandler(. Flag every one that runs per request or in a loop. - Move those calls to typed or named clients through
IHttpClientFactory. Deploy, then compare 5xx rates and failed dependency calls against the week before. - List the Azure services your apps reach over public endpoints. Those are your private endpoint candidates.
- Only then decide whether a NAT gateway is worth it, mainly for busy external APIs or when a partner needs a fixed IP.
- If you scaled out during an incident, check whether you still need those instances once the fix is live.
Primary references
Sources and further reading
- Microsoft: Troubleshoot intermittent outbound connection errors in Azure App Service — SNAT port allocation, symptoms, and the diagnostics detector
- Microsoft: Use SNAT for outbound connections — how SNAT ports are used and reused per destination
- Microsoft: Guidelines for using HttpClient — long-lived clients, PooledConnectionLifetime, and port exhaustion
- Microsoft: IHttpClientFactory with .NET — handler pooling, lifetimes, and the singleton trap
- Microsoft: Configure Azure NAT Gateway integration — prerequisites, routing, and SNAT capacity
- Microsoft: Manage connections in Azure Functions — reusing clients across function executions
Questions teams ask
Frequently asked questions
Does a bigger App Service plan give me more SNAT ports?
No. Each instance starts with the same 128 preallocated SNAT ports regardless of size. A bigger plan raises the total TCP connection limit per worker, which is a different limit. More instances give you more ports in total, but that is spending money around the problem.
Can I set an alert on SNAT port usage?
Not directly. SNAT port usage isn't exposed as a metric, so you can't alert or autoscale on it. Alert on what it causes instead: HTTP 5xx responses and failed outbound dependency calls. Then use the SNAT Port Exhaustion detector in Diagnose and solve problems to confirm.
Is one static HttpClient good enough?
In .NET Core and .NET 5 onward, yes, as long as you create it with a SocketsHttpHandler and set PooledConnectionLifetime. Without that setting, connections never refresh and the client can miss DNS changes.
Does this affect Azure Functions too?
Functions hosted on App Service hit the same outbound limits, and a function that creates a new client on every execution has exactly this problem. Create the client once and reuse it across executions. Microsoft's guide on managing connections in Functions covers the details.
Will private endpoints fix SQL timeouts?
They remove SNAT limits from the database traffic, which helps when the timeouts come from outbound connection pressure. They won't fix slow queries or an exhausted SQL connection pool. Check which one you have before you change the network.