Retrieve TCP state details for an open connection without tcpdump?
I have an environment with a few hundred Linux (CentOS/RedHat) servers that all interact with each other through TCP connections. Above the TCP layer is an RPC protocol that has timeouts built into it such that the application will wait for about 10 seconds for a response after sending an RPC request before it considers the request to have timed out and the application takes alternate/corrective action to handle the timeout. I have been observing very intermittently that the RPC layer is logging a timeout, meaning the RPC layer on one end of the connection has sent an request to the other end but an RPC response has not yet ben received.
I am still trying to isolate whether the issue is at the TCP or RPC layer.
To investigate the TCP layer, I've been gathering samples of "netstat -pan" output on every server in the environment, every few seconds, over the course of a few days.
What I've noticed is that when I have a timeout for a request sent from server A to server B, the send queue size is non-zero (as shown by netstat -pan) for the TCP connection on server B. I am presuming that the RPC response has been partially or fully placed into the TCP send queue at that point but for some reason is not being transmitted.
I want to confirm my suspicion and dive further into the root cause. For this type of issue, my go-to is using tcpdump to capture the traffic from both ends of the connection and then analyze it with WireShark to understand the behavior. Unfortunately, given the number of servers involved, the intermittent nature of the issue, and the very high rate of traffic flow (well utilized 10GbE), it is not feasible to run tcpdump for an extended period of time across all machines in the environment.
As the send queue size is staying the same non-zero value for >10 seconds, what I'd like to do is a simple poll of the TCP state from all open connections on a machine every few seconds. Specifically, what I'd like to retrieve for each TCP connection is:
1. The last window size and scale factor advertised by the server
2. The highest acknowledgement number sent by the server for the connection along with any unacknowledged extents (as selective ack is enabled)
3. The next sequence number the server will use to transmit on the connection
4. Whether the TCP socket is corked (as the application uses TCP_CORK)
From this information, I should be able to identify if data has been transmitted but not acknowledged, if the window is exhausted and hence no more data can be sent, or if one of the servers is hitting an application layer bug where the TCP cork is in place and thus the RPC response is being blocked by some logical error in the application.
Unfortunately, I've looked around and could not find a *convenient* way to retrieve such information. I've checked through all the info I could find on the netstat and ss commands, and I've looked through a lot of /proc content but no luck.
I presume something like systemtap could be used, or a custom kernel could be built. But neither of those are particularly attractive options unless there is nothing else that can be done to retrieve the TCP state information for each open connection from a user space process.
Any ideas?
Categories
- All Categories
- 51 LFX Mentorship
- 104 LFX Mentorship: Linux Kernel
- 576 Linux Foundation IT Professional Programs
- 304 Cloud Engineer IT Professional Program
- 125 Advanced Cloud Engineer IT Professional Program
- 53 DevOps Engineer IT Professional Program
- 61 Cloud Native Developer IT Professional Program
- 5 Express Training Courses
- 5 Express Courses - Discussion Forum
- 2.1K Training Courses
- 19 LFC110 Class Forum
- 7 LFC131 Class Forum
- 27 LFD102 Class Forum
- 158 LFD103 Class Forum
- 21 LFD121 Class Forum
- 1 LFD137 Class Forum
- 61 LFD201 Class Forum
- 1 LFD210 Class Forum
- LFD210-CN Class Forum
- 1 LFD213 Class Forum - Discontinued
- 128 LFD232 Class Forum
- LFD237 Class Forum
- 23 LFD254 Class Forum
- 612 LFD259 Class Forum
- 105 LFD272 Class Forum
- 1 LFD272-JP クラス フォーラム
- 1 LFD273 Class Forum
- 2 LFS145 Class Forum
- 25 LFS200 Class Forum
- 739 LFS201 Class Forum
- 1 LFS201-JP クラス フォーラム
- 11 LFS203 Class Forum
- 75 LFS207 Class Forum
- 300 LFS211 Class Forum
- 54 LFS216 Class Forum
- 47 LFS241 Class Forum
- 41 LFS242 Class Forum
- 37 LFS243 Class Forum
- 11 LFS244 Class Forum
- 37 LFS250 Class Forum
- 1 LFS250-JP クラス フォーラム
- LFS251 Class Forum
- 141 LFS253 Class Forum
- LFS254 Class Forum
- 1.1K LFS258 Class Forum
- 10 LFS258-JP クラス フォーラム
- 93 LFS260 Class Forum
- 132 LFS261 Class Forum
- 33 LFS262 Class Forum
- 80 LFS263 Class Forum
- 15 LFS264 Class Forum
- 11 LFS266 Class Forum
- 18 LFS267 Class Forum
- 18 LFS268 Class Forum
- 23 LFS269 Class Forum
- 203 LFS272 Class Forum
- 1 LFS272-JP クラス フォーラム
- LFS274 Class Forum
- LFS281 Class Forum
- 236 LFW211 Class Forum
- 172 LFW212 Class Forum
- 7 SKF100 Class Forum
- SKF200 Class Forum
- 903 Hardware
- 219 Drivers
- 74 I/O Devices
- 44 Monitors
- 116 Multimedia
- 209 Networking
- 101 Printers & Scanners
- 85 Storage
- 763 Linux Distributions
- 88 Debian
- 66 Fedora
- 15 Linux Mint
- 13 Mageia
- 24 openSUSE
- 142 Red Hat Enterprise
- 33 Slackware
- 13 SUSE Enterprise
- 357 Ubuntu
- 479 Linux System Administration
- 41 Cloud Computing
- 70 Command Line/Scripting
- Github systems admin projects
- 95 Linux Security
- 78 Network Management
- 108 System Management
- 49 Web Management
- 68 Mobile Computing
- 23 Android
- 30 Development
- 1.2K New to Linux
- 1.1K Getting Started with Linux
- 538 Off Topic
- 131 Introductions
- 217 Small Talk
- 22 Study Material
- 826 Programming and Development
- 278 Kernel Development
- 514 Software Development
- 928 Software
- 260 Applications
- 184 Command Line
- 3 Compiling/Installing
- 76 Games
- 316 Installation
- 61 All In Program
- 61 All In Forum
Upcoming Training
-
August 20, 2018
Kubernetes Administration (LFS458)
-
August 20, 2018
Linux System Administration (LFS301)
-
August 27, 2018
Open Source Virtualization (LFS462)
-
August 27, 2018
Linux Kernel Debugging and Security (LFD440)