Completely solve the problem of too many SpringCloud TCP connections not being released!

I'm involved in a project using Spring Cloud Gateway, combined with Nacos, as a gateway to implement routing, forwarding, and load balancing. After the project ran for a while, I encountered the issue of too many unreleased TCP connections in the Spring Cloud Gateway service. To address this issue, I consulted relevant materials and summarized the basics of TCP connections, Http Keep-Alive, and TCP Keepalive. I also looked at the keepalive parameter configurations for Windows and Linux operating systems, and ultimately focused on configuring keepalive parameters for some common application services. Based on this, I adjusted the keepalive parameters for Spring Cloud Gateway in practice and verified their effectiveness. Other issues related to operating systems and Nginx and Tomcat settings require further verification in practice. I've documented this information here for further research and verification, and to provide a reference for future researchers. I hope readers will correct any omissions in this article. I'd be very grateful!

1. Introduction to TCP connection

To achieve reliable data transmission, TCP establishes a transport connection between application processes. It establishes a logical connection between two transmission users, allowing both communicating parties to confirm that the other party is their own transport connection endpoint.

1.1 Establishing a connection—three-way handshake

Before establishing a connection, the server first passively opens a well-known port and listens on it. When the client wants to establish a connection with the server, it initiates a request to actively open a port (usually a temporary port), and then enters the three-way handshake process.

picture

First handshake: When establishing a connection, the client sends a SYN packet (seq=x) to the server and enters the SYN_SEND state, waiting for the server to confirm;

Second handshake: The server receives the SYN packet and must confirm the client's SYN (ack=x+1). At the same time, it also sends a SYN packet (seq=y), that is, a SYN+ACK packet. At this time, the server enters the SYN_RECV state;

The third handshake: The client receives the SYN+ACK packet from the server and sends an acknowledgment packet ACK (ack=y+1) to the server. After this packet is sent, the client and server enter the ESTABLISHED state, completing the three-way handshake.

1.2 Release the connection - wave four times

picture

First wave: The client sends a FIN to close the data transmission from the client to the server, and the client enters the FIN_WAIT_1 state.
Second wave: After receiving FIN, the server sends an ACK to the client, confirming that the sequence number is the received sequence number + 1 (the same as SYN, one FIN occupies one sequence number), and the server enters the CLOSE_WAIT state.
The third wave: The server sends a FIN to close the data transmission from the server to the client, and the server enters the LAST_ACK state.
Fourth wave: After the client receives the FIN, the client enters the TIME_WAIT state, then sends an ACK to the server, confirming the sequence number is the received sequence number + 1, and the server enters the CLOSED state, completing four waves.

At this time, the TCP connection has not been released. The client must wait for 2MSL, the time set by the waiting timer, before entering the connection closing state.

illustrate:

2MSL stands for twice the MSL. The TCP TIME_WAIT state is also called the 2MSL wait state. When one end of a TCP connection initiates an active close, after sending the last ACK packet, that is, after the third handshake completes and the ACK packet for the fourth handshake is sent, it enters the TIME_WAIT state and must remain in this state for twice the MSL time. The main purpose of waiting for 2MSL is to prevent the other end from not receiving the last ACK packet. In this case, the other end will resend the FIN packet for the third handshake after a timeout. After receiving the resent FIN packet, the end that actively closed the connection can send another ACK packet. During the TIME_WAIT state, the ports on both ends cannot be used and must wait until the 2MSL time expires before they can be used again. During the 2MSL wait phase, any late segments are discarded. However, in actual applications, the SO_REUSEADDR option can be set to avoid waiting for the 2MSL time to expire before using the port.

If there are a large number of connections, three handshakes and four waves are required every time a connection is made or closed, which will obviously cause poor performance.

2. Introduction to KeepAlive and Keep-Alive

TCP's KeepAlive and HTTP's Keep-Alive are completely different concepts and should not be confused. In fact, HTTP's KeepAlive is written as Keep-Alive, which is also different from TCP's KeepAlive.

2.1 Http Keep-Alive

The HTTP protocol uses a "request-response" model. When using the normal mode (that is, non-Keep-Alive mode), the client and server must establish a new connection for each request/response, and disconnect immediately after completion. When using the Keep-Alive mode, the Keep-Alive function ensures that the connection between the client and the server remains valid. When subsequent requests to the server occur, the Keep-Alive function avoids establishing or re-establishing a connection.

In HTTP 1.0, Keep-Alive is disabled by default and can only be enabled by adding "Connection: Keep-Alive" to the HTTP header. In HTTP 1.1, Keep-Alive is enabled by default and can only be disabled by adding "Connection: close". Most browsers currently use the HTTP 1.1 protocol, which means they initiate Keep-Alive connection requests by default. Whether a full Keep-Alive connection can be completed depends on the server configuration.

Advantages and disadvantages of enabling Keep-Alive:

• Advantages: Keep-Alive mode is more efficient because it avoids the overhead of connection establishment and release.

• Disadvantages: Long-term TCP connections can easily lead to ineffective occupation of system resources and waste of system resources.

2.2 TCP KeepAlive

After a connection is established, if the client doesn't send data, or sends data only after a long period of inactivity, how can we determine if the other party is still online? Is it offline or simply not transmitting data? Should the connection be maintained? This is a consideration that needs to be taken into account in TCP protocol design. The TCP protocol solves this problem in a clever way. After a certain period of time, TCP automatically sends an empty packet (a detection packet) to the other party. If the other party responds to this packet, it indicates that the other party is still online and the connection can be maintained. If the other party does not return a packet and retries multiple times, the connection is considered lost and there is no need to maintain the connection.

TCP KeepAlive is a TCP connection preservation mechanism that detects the TCP connection status. The TCP KeepAlive preservation timer supports three system kernel configuration parameters:

net.ipv4.tcp_keepalive_intvl = 15  
net.ipv4.tcp_keepalive_probes = 5  
net.ipv4.tcp_keepalive_time = 18001.
2.
3.

KeepAlive is a TCP keepalive timer. After a TCP connection is established and idle (no data is exchanged between the two parties) for tcp_keepalive_time, the server will attempt to send a probe packet to the client to determine the status of the TCP connection (perhaps the client has crashed, the application has been forcibly closed, the host is unreachable, etc.). If no response (ack packet) is received from the other party (tcp_keepalive_intvl), the server will try to send a probe packet again until an ack is received. If no ack is received, the server will try again for tcp_keepalive_probes times, with the intervals set to 15s, 30s, 45s, 60s, or 75s respectively. If no ack is received after tcp_keepalive_probes, the TCP connection will be terminated. The default TCP connection idle timer is 2 hours, but 30 minutes is generally sufficient.

3. Operating system related keepalive parameter settings

3.1 Linux system

tcp_keepalive_time 7200 // How long does it take for no new message to be received since the last data transmission to be considered as the start of detection? The unit is seconds. The default value is 7200s (there is no need to check frequently, which wastes resources).
tcp_keepalive_intvl 75 // The interval at which heartbeat packets are sent after the detection starts, in seconds, the default is 75s.
tcp_keepalive_probes 9 // Send heartbeat packets several times and close the connection if the other party does not respond. The default is 9 times. You can modify the parameters in the following corresponding configuration files: /proc/sys/net/ipv4/tcp_keepalive_time /proc/sys/net/ipv4/tcp_keepalive_intvl /proc/sys/net/ipv4/tcp_keepalive_probes

3.2 Windows system

KeepAliveTime: The KeepAliveTime value controls how often the system attempts to verify that an idle connection is still alive. If the connection has no activity for a period of time, the system sends a keepalive signal, which it responds to if the network is up and the receiver is alive. If you need to be sensitive to lost receivers, meaning you need to detect lost receivers more quickly, consider lowering this value. If you have a high incidence of long periods of idle inactivity but a low incidence of lost receivers, you might want to increase this value to reduce overhead. By default, the system sends a keepalive message if an idle connection has no activity for 7200000 milliseconds (2 hours). A typical value of 1800000 milliseconds is recommended, so that lost connections are detected within 30 minutes. To do this: Navigate to the HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\TCPIP\Parameters registry subkey. Under the Parameters subkey, create or modify a REG_DWORD value named KeepAliveTime and set the value to an appropriate number of milliseconds.
KeepAliveIntervalThe KeepAliveInterval value indicates how often the system resends a keepalive signal if no response is received from the other party. If the number of consecutive keepalives without a response exceeds the value of TcpMaxDataRetransmissions (described below), the connection is abandoned. If network conditions are poor and longer response times are acceptable, consider increasing this value to reduce overhead. If you need to quickly verify that the recipient has been lost, consider decreasing this value or the TcpMaxDataRetransmissions value. By default, the system waits 1000 milliseconds (1 second) before resending a keepalive signal if no response is received. This can be modified to meet specific needs by navigating to the HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\TCPIP\Parameters registry subkey. Under the Parameters subkey, create or modify a REG_DWORD value named KeepAliveInterval and set the appropriate number of milliseconds.

4. Common server configuration Keepalive parameters

4.1 Nginx Keepalive Configuration

When using nginx as a reverse proxy, in order to support persistent connections, two things need to be done:

The connection from client to nginx is a long connection
The connection from nginx to the server is a long connection

4.1.1 The connection from the client to Nginx is a long connection

http {
    # 客户端连接的超时时间, 为 0 时禁用长连接。 tcp连接在传送完最后一个响应后，还需要hold住 keepalive_timeout秒后仍没有新的http请求，才开始关闭这个连接
    keepalive_timeout 120s;
    # 在一个长连接上可以服务的最大请求数目, 当达到最大请求数目且所有已有请求结束后, 连接被关闭, 默认为 100, 即每个连接的最大请求数
    keepalive_request 10000;
  }1.
2.
3.
4.
5.
6.

The keepalive_requests parameter actually means that after a keepalive connection is established, Nginx sets a counter for that connection, recording the number of client requests received and processed on the keepalive connection. If the maximum value for this parameter is reached, Nginx forcibly closes the keepalive connection, forcing the client to reestablish a new one. In most cases, when QPS (requests per second) is not very high, the default value of 100 is sufficient. However, for scenarios with high QPS (e.g., exceeding 10,000 QPS, or even reaching 30,000, 50,000, or even higher), the default value of 100 is too low. A simple calculation shows that at a QPS of 10,000, the client sends 10,000 requests per second (typically establishing multiple keepalive connections). Each connection can only process a maximum of 100 requests, meaning that, on average, 100 keepalive connections are closed by Nginx every second. This also means that to maintain QPS, the client must reestablish 100 connections per second. As a result, you will find a large number of TIME_WAIT socket connections (even if keep-alive is in effect between the client and nginx). Therefore, for scenarios with high QPS, it is very necessary to increase this parameter to avoid a large number of connections being generated and then abandoned, and to reduce TIME_WAIT.

4.1.2 Long Connection from Nginx to Server (Upstream)

To maintain a long connection between Nginx and the backend server (which Nginx calls upstream), the typical settings are as follows: (By default, Nginx uses short connections (HTTP 1.0) to access the backend. When a request comes in, Nginx opens a new port to establish a connection with the backend, and actively closes the connection after the backend completes the execution.)

http {
    upstream  BACKEND {
        server   192.168.0.1：8080  weight=1 max_fails=2 fail_timeout=30s;
        server   192.168.0.2：8080  weight=1 max_fails=2 fail_timeout=30s;
        keepalive 300;        // 这个很重要！
    }
server {
        listen 8080 default_server;
        server_name "";
        location /  {
            proxy_pass http://BACKEND;
            proxy_http_version 1.1;         // 这两个最好也设置
            proxy_set_header Connection "";
        }
    }
}1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.

The keepalive directive does not mean turning on or off the long connection switch; nor is it used to set the timeout; nor is it used to set the maximum number of connections in the long connection pool. Official explanation: The connections parameter sets the maximum number of idle keepalive connections to upstream servers connections. When this number is exceeded, the least recently used connections are closed. It should be particularly noted that the keepalive directive does not limit the total number of connections to upstream servers that an nginx worker process can open.

keepalive: This parameter is the maximum number of idle connections in the connection pool of nginx connection backend, for example: set to 300; if nginx creates a connection pool of 1000 connections to meet the QPS of the request, and only 500 requests come at this time, then 1000-500 = 500; then there will be 500 more idle connections, and 500 > 300; then nginx will disconnect 200 request connections according to this configuration; then at this time there will be only a connection pool of 800 connections, if 1000 requests come next time, then nginx will start creating connections again; all the configurations of this value should be configured carefully

4.1.3 Nginx has a large number of TIME_WAIT

1) There are two situations that can cause a large number of TIME_WAITs on the nginx side: 1. A relatively low keepalive_requests setting causes nginx to forcibly close the keepalive connection with the client if this value is exceeded under high concurrency. (Actively closing the connection causes TIME_WAIT in nginx.) 2. A relatively low keepalive setting (too few idle requests) causes nginx to frequently experience connection fluctuations under high concurrency (connections are closed if this value is exceeded), and it constantly closes and opens the keepalive connection with the backend server.

2) The situation that causes a large number of TIME_WAIT on the backend server: nginx does not open a long connection with the backend, that is, it does not set proxy_http_version 1.1; and proxy_set_header Connection ""; which causes the backend server to close the connection every time. Under high concurrency, a large number of TIME_WAIT on the server will appear.

4.2 Setting Keepalive in Tomcat

The browser adds "Connection: Keep-Alive" to the request header, telling the server, "I support persistent connections. If you do, please establish a persistent connection with me." If the server does support persistent connections, it adds "Connection: Keep-Alive" to the response header, telling the browser, "I do support it, so let's establish a persistent connection." The server can also use the "Keep-Alive: timeout=10, max=100" header to tell the browser, "I want 10 seconds to count as the timeout, and the maximum limit is 100 seconds."

Tomcat allows you to configure persistent connections. This is done by configuring the Connector node in the conf/server.xml file. This node controls the browser connection to Tomcat. Two properties are directly related to persistent connections: keepAliveTimeout, which specifies the number of milliseconds the Connector waits for another KeepAlive request before closing the connection. The default value is the same as connectionTimeout; and maxKeepAliveRequests, which specifies the maximum number of HTTP/1.0 KeepAlive and HTTP/1.1 KeepAlive/Pipeline requests. Setting this to 1 disables KeepAlive and Pipeline. Setting this to a value less than 0 sets an unlimited number of KeepAlive requests. In other words, persistent connections are enabled by default in Tomcat. To disable persistent connections, simply set maxKeepAliveRequests to 1.

Tomcat in the Connector element in server.xml:

• keepAliveTimeout: The unit is milliseconds, which indicates how long Tomcat will keep the connection open before the next request comes in. This means that if the client continues to make requests and the expiration time is not exceeded, the connection will be maintained.
• maxKeepAliveRequests: The maximum number of long connections (1 means disabled, -1 means unlimited, the default is 100. It is usually set between 100 and 200). This value indicates the maximum number of requests supported by the connection. Connections exceeding this number of requests will be closed (a Connection: close header will be returned to the client).

4.3 Netty Setting Keepalive

4.3.1 SO_KEEPALIVE

Socket parameter. Whether to enable the heartbeat keepalive mechanism, i.e., connection keepalive. When this function is enabled, TCP will actively detect the validity of idle connections.

This mechanism will be activated after the TCP sockets of both parties establish a connection (that is, both enter the ESTABLISHED state) and there is no data transmission in the upper layer for about two hours (the default heartbeat interval is 7200s, or 2 hours).

Default value: Netty turns off this feature by default, that is, the value is: false.

Code settings:

bootstrap.childOption(ChannelOption.SO_KEEPALIVE, true);1.

illustrate:

• If one side has closed or abnormally terminated the connection, but the other side is unaware of it, we call such a TCP connection half-open. TCP uses the KeepAlive timer to detect half-open connections.
• In high-concurrency network servers, sockets are often missed, resulting in a large number of connections in the CLOSE_WAIT state. This problem can be solved by setting the KEEPALIVE option.
• Set the SO_KEEPALIVE option to enable KEEPALIVE, and then set the keepalive start time, interval, number of times and other parameters through TCP_KEEPIDLE, TCP_KEEPINTVL and TCP_KEEPCNT.
• Of course, you can also achieve this by setting kernel parameters such as /proc/sys/net/ipv4/tcp_keepalive_time, tcp_keepalive_intvl and tcp_keepalive_probes, but this will affect all sockets.

4.3.2 SpringCloud Gateway setting Keepalive

@Configuration
public class NettyConfig {
    @Bean
    public NettyServerCustomizer nettyServerCustomizer() {
        return httpServer -> httpServer.tcpConfiguration(tcpServer -> {
            tcpServer= tcpServer.option(ChannelOption.SO_KEEPALIVE, true);
            tcpServer = tcpServer.doOnBind(serverBootstrap ->
                    BootstrapHandlers.updateConfiguration(serverBootstrap, "channelIdle", (connectionObserver, channel) -> {
                        ChannelPipeline pipeline = channel.pipeline();
                        pipeline.addLast(new ReadTimeoutHandler(5, TimeUnit.MINUTES));
                    }));
            return tcpServer;
        });
    }
}1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.