AWS NLB Connection Teardown Quirks: Why Your Sockets Hang Without a FIN

SREs and DevOps engineers recently uncovered an unexpected operational quirk when dealing with AWS Network Load Balancers (NLB) handling TLS connections: under specific termination conditions, NLBs do not send a TCP FIN packet to gracefully close connection sockets.

The Anatomy of the Bug

When an application server or client relies on standard TCP state transitions, it expects either a FIN packet to initiate a graceful shutdown or an RST packet to force a reset. When AWS NLB drops or hangs up TLS sessions without sending a FIN, the client socket remains in an ESTABLISHED or CLOSE_WAIT state indefinitely until local OS kernel TCP keepalives eventually reap it.

This behavior can trigger cascading operational headaches:

• Connection Pool Exhaustion: App servers exhaust ephemeral ports or max connection limits waiting on hung sockets.
• Silent Cron Worker Failures: Background scripts or scheduled tasks relying on network calls hang forever, waiting for a response that will never close.
• Intermittent Latency Spikes: New traffic stalls while trying to reuse dead connections from client-side pools.

Mitigation & SRE Best Practices

1. Configure Aggressive TCP Keepalives: Ensure system kernel settings (tcp_keepalive_time, tcp_keepalive_probes) are tuned down from defaults to detect dead connections faster.
2. Set Client-Level Socket Timeouts: Never rely solely on network-level FIN packets; enforce strict application-layer read/write timeouts.
3. Proactive Monitoring with Rabbit SaaS:
   • Cron Rabbit: Prevent silent background job hangs caused by stuck TCP sockets. If a cron job hangs waiting for an NLB response and fails to ping its heartbeat endpoint, Cron Rabbit alerts your team immediately.
   • CloudStatusHQ: Aggregate and track upstream cloud provider behavior, service updates, and known infrastructure anomalies across AWS and other vendors in a single unified dashboard.