Posted by: Mangesh_Linux_Administrator | November 17, 2010

1 Server Monitoring


If the server load is high then do the following tricks:

1) Top
check for the server load and watch for process

2) free -m
it will display the memory status
# vmstat  : (virtual memory status)
# mymem   : (which services are consuming memory)
3)if  a perticular user is causing a high server laod then use “ps U username”: it will display the process that are running by the user .

4) Then you can take action against that user.

4) netstat -an |grep :80 |wc -l :
Show how many active connections there are to apache (httpd runs on port 80)

5) netstat -ntu | awk ‘{print $5}’ | cut -d: -f1 | sort | uniq -c | sort –n
it will display the no of connections form a certain ip

6)if there is MYSQL taking cpu usage then check for mysql status
# watch mysqladmin proc
# mysqladmin -u root processlist
check for the databases and suspend that user if any database is using high resources.

7)check /tmp if there is any process taking high resources : chown to root for that process and chmod 000 also if .sh process is running then chown it t0 root:root

8)if exim is taking high resources then check for exim logs : use this command :
replace :blackhole: : fail: — /etc/valiases/*

w : Current users

exim -bp | exiqsumm
ps aux | head -1;ps aux –no-headers| sort -rn +3 | head :

Use below mentioned command to get top memory consuming processes
ps aux | head -1;ps aux –no-headers | sort -rn +2
ps aux | head -1;ps aux –no-headers | sort -rn +2

———————————————————————————————————-

If Exim causes
exim –bpc
shows the total no of email in qmail

eximstats -nr -ne /var/log/exim_mainlog
Total mail server report

pidof exim
shows no of exim pids running

exim -bpr | grep frozen | wc -l
Shows no of frozen emails

exiqgrep -z -i | xargs exim –Mrm
exim -bp | exiqgrep -i | xargs exim -Mrm
it deletes the FROZEN mails from the server

tail -f /var/log/exim_mainlog | grep public_html
check for spamming if anybody is using php script for sending mail through public_html

tail -f /var/log/exim_mainlog | grep /tmp
Used for checking for who is spamming through the /tmp

tail -3000 /var/log/exim_mainlog |grep ‘rejected RCPT’ |awk ‘{print$4}’|awk -F\[ ‘{print $2} ‘|awk -F\] ‘{print $1} ‘|sort | uniq -c | sort -k 1 -nr | head -n 5
It will display the IP and no of tries done by the IP to send mail but rejected by the server.

netstat -plan|grep :25|awk {‘print $5’}|cut -d: -f 1|sort|uniq -c|sort -nk 1
shows the  connections from a certain ip to the   SMTP server

exim -bp | exiqsumm | more
It shows the domain name and the no of emails sent   by that domain

If  spamming from outside domain then you can block that domain or email id on the server
pico /etc/antivirus.exim
Add the following lines:
if $header_from: contains “name@domain.com”
then
seen finish
endif

eximstats -nr -ne /var/log/exim_mainlog

Catching spammer

exim -bp | exiqsumm | more

exim -bpr | grep “<*@*>” | awk ‘{print $4}’|grep -v “<>” | sort | uniq -c | sort -n
That will show you the maximum no of email currently in the mail queue have from or to the email address in the mail queue with exact figure.

exim -bpr | grep “<*@*>” | awk ‘{print $4}’|grep -v “<>” |awk -F “@” ‘{ print $2}’ | sort | uniq -c | sort -n
That will show you the maximum no of email currently in the mail queue have for the domain or from the domain with number.

Check if any php script is causing the mass mailing with
cd /var/spool/exim/input
egrep “X-PHP-Script” * -R
Just cat the ID that you get and you will be able to check which script is here causing problem for you.

To Remove particular email account email
exim -bpr |grep “ragnarockradio.org”|awk {‘print $3’}|xargs exim -Mrm

If Mysql causes :
mysqladmin -u root processlist
mysqladmin version
watch mysqladmin proc

If Apache causes :
netstat -ntu | grep :80 | awk ‘{print $5}’ | cut -d: -f1 | sort | uniq -c | sort –n
netstat -an |grep :80 |wc –l
netstat -n | grep :80 | wc -l;uptime ; netstat -n | wc –l
netstat –tupl
netstat -tupl

Other Commands

pidof php
shows the PIDs for php

history | netstat
shows no of connection details

lsof  -p pid
shows the details of pid

netstat -na |grep :80 |sort

Use below mentioned command to get top memory consuming processes”

ps aux | head -1;ps aux –no-headers| sort -rn +3 | head

Use below command to get top cpu consuming processes:

ps aux | head -1;ps aux –no-headers | sort -rn +2 |more

You can check if any backup is going on, run the following commands:
# ps aux | grep “pkg”
# ps aux | grep “gzip”
# ps aux | grep “backup”
If any backup process is going on, kill that process.

We can trace the user responsible for high web server resource usage by the folowing command
cat /etc/httpd/logs/access_log | grep mp3
cat  /etc/httpd/logs/access_log | grep rar
cat  /etc/httpd/logs/access_log | grep wav etc

cat /etc/httpd/logs/access_log | grep 408         can be used to check for DDOS attacks on the server.

cat  /etc/httpd/logs/access_log | grep rar

Port scanning :
root@ [/tmp]# nmap localhost

Starting Nmap 4.11 ( http://www.insecure.org/nmap/ ) at 2007-10-04 13:22 MYT
Interesting ports on localhost (127.0.0.1):
Not shown: 1664 closed ports
PORT     STATE SERVICE
1/tcp    open  tcpmux
21/tcp   open  ftp
25/tcp   open  smtp
53/tcp   open  domain
80/tcp   open  http
110/tcp  open  pop3
111/tcp  open  rpcbind
143/tcp  open  imap
443/tcp  open  https
465/tcp  open  smtps
631/tcp  open  ipp
783/tcp  open  spamassassin
953/tcp  open  rndc
993/tcp  open  imaps
995/tcp  open  pop3s
3306/tcp open  mysql

Nmap finished: 1 IP address (1 host up) scanned in 0.125 seconds
root@ [/tmp]#

netstat -anp |grep :3306

DDOS ATTACK :

ps -aux|grep HTTP|wc –l    : It will show you no of http connections to the server

netstat -lpn|grep :80 |awk ‘{print $5}’|sort    : It will helpful to check the no of connections from a certain ip

Check for the ips and block them with firewalls as apf/csf/iptables

For iptables : iptables -A INPUT -s <Source IP> -j DROP
For apf : apf –d ip_address
For CSF : csf -d ip_adrress

If load due to SPAMD:

Issue
The load issues on a cpanel 11 box (later versions too) because of spamd are quite common, sometimes they may take more than 90% of the CPU. Here are some ways to resolve it.
[edit] Solution
We can use the following steps to resolve the issue in a considerable manner.

1. Edit the /etc/mail/spamassassin/local.cf file

vi /etc/mail/spamassassin/local.cf

then add the following lines to it

a) use_bayes 0 (this will disable some spamassassin functionalities)
b) lock_method flock

(Uncomment these options, if they are commented)

2) Run scripts2/autofixer/spamd_dbm_fix

a) Log into WHM and use another URL as “http://your server IP:2086//scripts2/autofixer”
then type the script name “spamd_dbm_fix” and run

b) Or you can also do the same from server back end as follows.

# wget http://httpupdate.cpanel.net/autofixer/spamd_dbm_fix
# chmod 755 spamd_dbm_fix
# ./spamd_dbm_fix

3) Run the following lines of codes

# rm -f /home/*/.spamassassin/__db.bayes_toks.new
# rm -f /home/*/.spamassassin/*.lock

NOTE : If the load issue is related to a particular user then there is no need for running this,
you can simply enter into the ‘.spamassassin/’ directory for that user and delete the
files ‘__db.bayes_toks.new’ and *.lock.

4) Restart spam assassin (restart Exim)

5) :)Hope this will help you in such a situation.

: memory usage serverwide.
ps aux | awk ‘{print $4″\t”$11}’ | sort | uniq -c | awk ‘{print $2″ “$1” “$3}’ | sort –nr

[root@ ~]# mymem
Memory Usage
————
Current:        326.344 MB
Allocated:      492.012 MB
Guaranteed:     512 MB
Burstable:      768 MB

====================

5.1 1 spamd
0.7 1 spamd
0.6 1 /usr/bin/spamd
0.4 1 /usr/sbin/mysqld
0.2 1 cpsrvd
0.1 4 /etc/authlib/authProg
0.1 1 eximstats
0.1 1 cpdavd
0.1 1 cpbandwd
0.1 1 cpanellogd
0.1 1 cPhulkd
0.1 1 /usr/local/cpanel/bin/leechprotect
0.0 6 /usr/libexec/courier-authlib/authdaemond
0.0 4 /usr/sbin/courierlogger
0.0 4 -bash
0.0 3 sshd:
0.0 3 /usr/sbin/exim
0.0 3 /usr/lib/courier-imap/libexec/couriertcpd
0.0 2 sort
0.0 2 awk
0.0 11 /usr/local/apache/bin/httpd
0.0 1 xinetd
0.0 1 uniq
0.0 1 syslogd
0.0 1 pure-ftpd
0.0 1 ps
0.0 1 init
0.0 1 crond
0.0 1 chkservd
0.0 1 antirelayd
0.0 1 /usr/sbin/sshd
0.0 1 /usr/sbin/saslauthd
0.0 1 /usr/sbin/pure-authd
0.0 1 /usr/sbin/portsentry
0.0 1 /usr/sbin/named
0.0 1 /etc/authlib/authProg
0.0 1 /bin/sh
%MEM 1 COMMAND

====================================================

kill nobody process
ps aux | grep nobody | awk ‘{print $2}’ | xargs kill -9

To see only the memory resources occupied by each category of processes, such as Apache httpd, MySQL mysqld or Java, use the following command:

ps aux | awk ‘{print $4″\t”$11}’ | sort | uniq -c | awk ‘{print $2” “$1” “$3}’ | sort -nr
====================================================

ps -eo pid,%cpu,vsz,args,wchan

Shows every process, their pid, % of cpu, memory size, name, and what syscall they are currently executing. Nifty.

whereis locates source/binary and manuals sections for specified files. The supplied names are first stripped of leading pathname components and any (single) trailing
extension of the form .ext, for example, .c. Prefixes of s. resulting from use of source code control are also dealt with. whereis then attempts to locate the desired program in
a list of standard Linux places. For example find out where is ls command?
$ whereis ls

cd /var/spool/exim/input
egrep “X-PHP-Script” * -R

For netstat
netstat -nap | grep :110 | awk ‘{print $5}’ | cut -d”:” -f1 | sort | uniq -c | sort -nr | head
netstat -nap | grep :25 | awk ‘{print $5}’ | cut -d”:” -f1 | sort | uniq -c | sort -nr | head
netstat -n | grep :80 | wc -l;uptime ; netstat -n | wc –l
netstat -ntu | awk ‘{print $5}’ | cut -d: -f1 | sort | uniq -c | sort -n
netstat -nap | grep :80 | awk ‘{print $5}’ | cut -d”:” -f1 | sort | uniq -c | sort -nr | head

root@ [~]# tail -f /var/log/slow.log
# tail -500 /var/log/mysql-slow-queries.log

# How to find Top 10 memomry & cpu consuming processes
ps -eo pcpu,%mem,pid,user,args | sort -k 1 -r | head -10

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

Categories

%d bloggers like this: