Posted by: Mangesh_Linux_Administrator | July 22, 2010

Linux System Monitoring Tools (system admin should know)

# top – Process Activity Command

The top program provides a dynamic real-time view of a running system i.e. actual process activity. By default, it displays the most CPU-intensive tasks running on the server and updates the list every five seconds.

# vmstat – System Activity, Hardware and System Information

The command vmstat reports information about processes, memory, paging, block IO, traps, and cpu activity.
#vmstat 3

à Display Memory Utilization Slabinfo  :  vmstat –m

à Get Information About Active / Inactive Memory Pages : vmstat –a

# w – Find Out Who Is Logged on And What They Are Doing

w command displays information about the users currently on the machine, and their processes.

à w username

à w vivke

# uptime – Tell How Long The System Has Been Running

The uptime command can be used to see how long the server has been running. The current time, how long the system has been running, how many users are currently logged on, and the system load averages for the past 1, 5, and 15 minutes.


#ps – Displays The Processes

ps command will report a snapshot of the current processes. To select all processes use the -A or -e option:

# ps –A      [ps is just like top but provides more information.]

#Show Long Format Output  :  # ps -Al

Print All Process On The Server :  # ps –ax    and  # ps –aux

Display Only The Process IDs of Lighttpd :  #pgrep lighttpd

Find Out The Top 10 Memory Consuming Process

ps -auxf | sort -nr -k 4 | head -10

#free – Memory Usage

The command free displays the total amount of free and used physical and swap memory in the system, as well as the buffers used by the kernel.
# free

#iostat – Average CPU Load, Disk Activity

The command iostat report Central Processing Unit (CPU) statistics and input/output statistics for devices, partitions and network filesystems (NFS).

# iostat

#sar – Collect and Report System Activity

The sar command is used to collect, report, and save system activity information. To see network counter, enter:

# sar -n DEV | more # sar -n DEV -f /var/log/sa/sa24 | more

à To display the network counters from the 24th:   # sar -n DEV -f /var/log/sa/sa24 | more

You can also display real time usage using sar:  # sar 4 5

#mpstat – Multiprocessor Usage

The mpstat command displays activities for each available processor, processor 0 being the first one. mpstat -P ALL to display average CPU utilization per processor:

# mpstat -P ALL

#pmap – Process Memory Usage

The command pmap report memory map of a process. Use this command to find out causes of memory bottlenecks.

# pmap -d PID

To display process memory information for pid # 47394, enter:  # pmap -d 47394

#netstat and ss – Network Statistics

The command netstat displays network connections, routing tables, interface statistics, masquerade connections, and multicast memberships. ss command is used to dump socket statistics. It allows showing information similar to netstat. See the following resources about ss and netstat commands:

ss: Display Linux TCP / UDP Network and Socket Information

The ss command is used to dump socket statistics. It allows showing information similar to netstat command. It can display more TCP and state information than other tools. It is a new, incredibly useful and faster (as compare to netstat) tool for tracking TCP connections and sockets. SS can provide information about:

  • All TCP sockets.
  • All UDP sockets.
  • All established ssh / ftp / http / https connections.
  • All local processes connected to X server.
  • All the tcp sockets in state FIN-WAIT-1 and much more.

Most Linux distributions are shipped with ss and many monitoring tools. Being familiar with this tool helps enhance your understand of what’s going on in the system sockets and helps you find the possible causes of a performance problem.

Task: Display Sockets Summary

List currently established, closed, orphaned and waiting TCP sockets, enter:
# ss –s

Sample Output:

Total: 734 (kernel 904)
TCP:   1415 (estab 112, closed 1259, orphaned 11, synrecv 0, timewait 1258/0), ports 566

Transport Total     IP        IPv6
*                904       -         -
RAW           0         0         0
UDP            15        12        3
TCP            156       134       22
INET           171       146       25
FRAG         0         0         0

Task: Display All Open Network Ports

# ss -l
Sample Output:

ss -l
Recv-Q Send-Q                                                  Local Address:Port                                                      Peer Address:Port
0      0                                                                                                                  *:*
0      0                                                                                                                 *:*
0      0                                                                                                                 *:*
0      0                                                                   *:3306                                                                 *:*
0      0                                                                   *:http                                                                 *:*
0      0                                                                   *:4949                                                                 *:*
0      0                                                                   *:domain                                                               *:*
0      0                                                                   *:ssh                                                                  *:*
0      0                                                                   *:smtp                                                                 *:*
0      0                                                                                                                  *:*
0      0                                                                                                                  *:*
0      0                                                                      *:https                                                                *:*
0      0                                                                  :::34571                                                               :::*
0      0                                                                  :::34572                                                               :::*
0      0                                                                  :::34573                                                               :::*
0      0                                                                 ::1:rndc                                                                :::*

Type the following to see process named using open socket:
# ss -pl
Find out who is responsible for opening socket / port # 4949:
# ss -lp | grep 4949
Sample output:

0      0                            *:4949                          *:*        users:(("munin-node",3772,5))

munin-node (PID # 3772) is responsible for opening port # 4949. You can get more information about this process (like memory used, users, current working directory and so on) visiting /proc/3772 directory:
# cd /proc/3772
# ls -l

Task: Display All TCP Sockets

# ss -t –a

Task: Display All UDP Sockets

# ss -u –a

Task: Display All Established SMTP Connections

# ss -o state established '( dport = :smtp or sport = :smtp )'

Task: Display All Established HTTP Connections

# ss -o state established '( dport = :http or sport = :http )'

Task: Find All Local Processes Connected To X Server

# ss -x src /tmp/.X11-unix/*

Task: List All The Tcp Sockets in State FIN-WAIT-1

List all the TCP sockets in state -FIN-WAIT-1 for our httpd to network 202.54.1/24 and look at their timers:
# ss -o state fin-wait-1 '( sport = :http or sport = :https )' dst 202.54.1/24

Get Detailed Information About Particular IP address Connections Using netstat Command: –

etstat command and shell pipe feature can be used to dig out more information about particular IP address connection. You can find out total established connections, closing connection, SYN and FIN bits and much more. You can also display summary statistics for each protocol using netstat.

This is useful to find out if your server is under attack or not. You can also list abusive IP address using this method.
# netstat -nat | awk '{print $6}' | sort | uniq -c | sort -n

Dig out more information about a specific ip address:

# netstat -nat |grep {IP-address} | awk ‘{print $6}’ | sort | uniq -c | sort –n

Busy server can give out more information: #  netstat -nat |grep | awk ‘{print $6}’ | sort | uniq -c | sort -nsort –

Get List Of All Unique IP Address

1] To print list of all unique IP address connected to server, enter:

# netstat -nat | awk ‘{ print $5}’ | cut -d: -f1 | sed -e ‘/^$/d’ | uniq

2] To print total of all unique IP address, enter:

# netstat -nat | awk ‘{ print $5}’ | cut -d: -f1 | sed -e ‘/^$/d’ | uniq | wc –l

Find Out If Box is Under DoS Attack or Not

If you think your Linux box is under attack, print out a list of open connections on your box and sorts them by according to IP address, enter:

# netstat -atun | awk ‘{print $5}’ | cut -d: -f1 | sed -e ‘/^$/d’ |sort | uniq -c | sort -n


Q How do I track and monitor connection for eth1 public network interface under Redhat Enterprise Linux (RHEL) 5 server?

Ans : .You can use netstat command or tcptrack command. Both command can show established TCP connection and provides the ability to monitor the same.


Display Summary Statistics for Each Protocol

#  netstat –s

# netstat -s | less
# netstat -t -s | less
# netstat -u -s | less
# netstat -w -s | less
# netstat -s

Display Interface Table

You can easily display dropped and total transmitted packets with netstat for eth0:

# netstat –interfaces eth0


Read the following man pages for the details:

$ man netstat
$ man cut
$ man awk
$ man sed
$ man grep


#iptraf – Real-time Network Statistics

The iptraf command is interactive colorful IP LAN monitor. It is an ncurses-based IP LAN monitor that generates various network statistics including TCP info, UDP counts, ICMP and OSPF information, Ethernet load info, node stats, IP checksum errors, and others. It can provide the following info in easy to read format:

  • Network traffic statistics by TCP connection
  • IP traffic statistics by network interface
  • Network traffic statistics by protocol
  • Network traffic statistics by TCP/UDP port and by packet size
  • Network traffic statistics by Layer2 address


#tcpdump – Detailed Network Traffic Analysis

The tcpdump is simple command that dump traffic on a network. However, you need good understanding of TCP/IP protocol to utilize this tool. For.e.g to display traffic info about DNS, enter:

# tcpdump -i eth1 ‘udp port 53’

A] To display all IPv4 HTTP packets to and from port 80, i.e. print only packets that contain data, not, for example, SYN and FIN packets and ACK-only packets, enter:

# tcpdump ‘tcp port 80 and (((ip[2:2] – ((ip[0]&0xf)<<2)) – ((tcp[12]&0xf0)>>2)) != 0)’


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s


%d bloggers like this: