Troubleshooting_SQAnother one of my favorite tools for troubleshooting networks is TCPing, by Eli Fulkerson. This small, self-contained command line tool is to TCP what ping.exe is to ICMP, which is to say a totally awesome way to troubleshoot networks and more with minimal effort. If you’d like to learn how to evaluate firewall connectivity, check on service restarts, and determine if servers are up even when ICMP is being blocked, read on for more on TCPing.

Getting TCPing “installed”

For starters, you need to download TCPing from http://www.elifulkerson.com/projects/tcping.php. While you are there, grab TCProute, and we will cover that in an upcoming post. Once you download it, copy it to any directory that is in your path. It’s self-contained, so you don’t need to install anything, register any DLLs, or do anything else other than be able to execute it.

Getting started with TCPing

Once you have it somewhere in your path, call it without any parameters.

TCP 1

As you can see, there’s great help built right into the tool. At its simplest, you can simply enter TCPing server[enter] and TCPing will reach out to the server over TCP port 80 to see if anything is listening. You can specify an FQDN or an ip.addr.

TCP2

As you can see, a quick DNS lookup gives us the ip.addr of www.gfi.com, and TCPing tests TCP port 80 four times, giving us the round-trip time (RTT) for all four “pings” along with stats just like PING does; minimum, maximum, and average.

If you want to test for a different port, say TCP 443 to see if anything is listening like a webserver on HTTPS, just specify the port by appending the port number after the target, like this.

TCP 3

Think about how many different ways you could use this. I regularly use this to see if

  • A host is alive when ICMP is blocked
  • If a service is running
  • If the firewall is permitting connectivity to a service I know is running

But that is just the tip of the iceberg. There are other switches that TCPing includes that can help you test out so much more.

-t             a continuous ping allows you to see stats over time, and if the service drops anything

-n #        send the specified number of probes

-i #          wait the specified number of seconds between probes, such as for continuous testing without generating too much noise

-w #       wait the specified number of milliseconds for a response, which is useful over slow or very high latency networks

-d            include date and time on each line: think log files for ongoing monitoring, especially when combined with Splunk or other log analysis tool

-b #        beeps on up, down, or change, which we’ll discuss down below.

-r #         rechecks DNS for the FQDN, which is very cool when you are doing cutovers

Some examples of how to use TCPing

So I am sure everyone has run a continuous ping against a server to make sure it is up, to test for changes in response time, etc. You can see from above that there are several switches that you can use. What’s really cool is when you combine them. If I have to reboot a Windows server after patching or reconfiguration, and I want to be able to log back into it as soon as it is finished rebooting, I do this;

tcping –t –b 2 servername 3389

which will ping the server continuously on TCP port 3389, which is the RDP port. The continuous ping will let me know when the Terminal Services service is responding, which is when I can actually RDP back into the box. It might start to respond to ICMP pings several minutes before I can actually remote into it. The –b 2 is the really cool part. It will start to beep when the service responds, which means I can let the cmd prompt window go to the background, and I will hear when the server is ready for me to log on!

The –b switch is also really good for monitoring when you bounce services. You can beep on down, up, or change.

  • 1 for no response (down)
  • 2 for response (up)
  • 3 for change (if it was up and goes down, or down and then starts to respond)
  • 4 continuously, which is just likely to be annoying but someone may find it useful.

I have also used TCPing and Splunk as a poor man’s service monitor. If you output TCPing to a file, and set Splunk to tail the file, then you can ‘train’ Splunk to alert you when a response time goes above a certain value or a service fails to respond. Try this;

tcping –t –i 30 –d servername >> pathtofile.txt

This will probe the server on TCP port 80 (no port specified, so that is the default) once every 30 seconds, and write the output to a file that includes date and local time. So each line will have a date/time stamp, and whether the port is open or non-responsive, and the RTT. Have Splunk tail that file and alert you on down, or if RTT exceeds your desired value.

Playing with TCPing is the best way for you to come up with dozens of other ways to use it, so use the above to get you started and then see what else you come up with.

How TCPing works under the hood

TCPing is a very ‘polite’ network tool. It opens a TCP connection to the destination port with a SYN, and then waits for the SYN ACK back from the destination to determine if the service is up, and to measure RTT. When it receives the SYN ACK, it sends a RST ACK to close the half-open connection before it attempts to probe again. This ensures that you don’t inadvertently exhaust the server of resources by launching what is known as a SYN Flood attack, SYNing the server repeatedly until it runs out of memory or threads for processing legitimate requests.

TCPing is one of the cmd line tools that I copy onto every workstation and server I use. Seriously, it’s in my default image. I find it that useful. Take a look at it for yourself, and see how quickly it becomes invaluable to you too.

Whether you never heard of it before this post, or you’ve been using it for years, please leave a comment if you have a particular syntax mix you find really helpful, and share the knowledge with others.