Dedicated Servers | VPS Hosting » Linux server hosting

How to keep log of daily network traffic on Dedicated servers?

Thu, 01 Oct 2009 23:05:14 +0000

Here is how to keep log of daily network traffic on Dedicated Servers :

* It is a console-base network traffic monitor for Linux (perfect tool for remote Linux box hosted at ISP)
* It keeps a log of daily and monthly network traffic for the selected network interface(s).
* It collects all data from /proc file system it means normal user can monitor traffic (no need to run vnstat as a root user)
* Easy to setup & configure
* Ease of use

Step # 1: Install vnstat
Debian user can install vnstat using apt-get command

# apt-get install vnstat

Step # 2: Enable vnstat
Once installed you need to create a database with following command:

# vnstat -u -i eth0

Where,
u :forces a database update for interface or creates the database if it doesn’t exist
i eth0 : use to specify interface

Please note that it will start to collect data via cronjob

0-55/5 * * * * root /usr/bin/vnstat -u

You do not have to install cronjob yourself; it should be automatically configure by apt-get.

Step # 3 View statistics
Display default traffic statistics

$ vnstat

Display daily traffic statistics

$ vnstat -d

Display monthly traffic statistics:

$ vnstat -m

Display all time top10 traffic days:

$ vnstat -t

Try help option to get all query options:

$ vnstat –help

Optimization of MySQL Server

Wed, 27 Aug 2008 19:50:02 +0000

By following the below steps you can decrease the CPU load of a server,it decreases the page loading time of a website, and it also enhances the overall performance of the server. You can use some steps from the below compilation methods to optimize a web server,so that you can serve web pages faster and also decrease the overall CPU load on the box. Few best procedures a web host can do to optimize your web server

1) Basic Config File Editing

Before starting you must make a backup of your /etc/my.cnf file by simply typing copy “cp#” /etc/my.cnf /etc/my.cnf.back , and after that you can perform the following via SSH:

pico /etc/my.cnf
Now replace the current existing lines:

[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
skip-locking
skip-innodb
query_cache_limit=8M
query_cache_size=256M
query_cache_type=1
max_connections=500
max_user_connections=10
interactive_timeout=20
wait_timeout=20
connect_timeout=6
thread_cache_size=128
key_buffer=16M
join_buffer=1M
max_allowed_packet=16M
table_cache=1024
record_buffer=1M
sort_buffer_size=2M
read_buffer_size=2M
max_connect_errors=10
# Try number of CPU’s*2 for thread_concurrency
thread_concurrency=4
myisam_sort_buffer_size=64M
#log-bin
server-id=1

[mysql.server]
user=mysql
basedir=/var/lib

[safe_mysqld]
err-log=/var/log/mysqld.log
pid-file=/var/lib/mysql/mysql.pid
open_files_limit=8192

[mysqldump]
quick
max_allowed_packet=16M

[mysql]
no-auto-rehash
#safe-updates

[isamchk]
key_buffer=32M
sort_buffer=32M
read_buffer=16M
write_buffer=16M

[myisamchk]
key_buffer=32M
sort_buffer=32M
read_buffer=16M
write_buffer=16M

After that hit CTRL + X to exit and save the file

Once it is done Edit the httpd.conf:

pico /usr/local/apache/conf/httpd.conf (or wherever your httpd.conf is located)

Set “Timeout” value to “Timeout 300″
Change “KeepAlive on” to “KeepAlive off”
Set “MinSpareServers” to “MinSpareServers 8″
Set “MaxSpareServers” to “MaxSpareServers 13″
Set “MaxRequestsPerChild” to “MaxRequestsPerChild 50″
Set “HostnameLookups” to “HostnameLookups Off”

CTRL + X to exit and save the file

Restart Apache and MySQL with:
service httpd restart
service mysql restart OR service mysqld restart

People may need to restart the services with /etc/rc.d/init.d/httpd restart and /etc/rc.d/init.d/mysql restart

The above changes can be modified and configured according to your server’s needs .

2) Installing Mod_Perl

What is Mod_Perl?

mod_perl gives you a persistent Perl interpreter embedded in your web server. This lets you avoid the overhead of starting an external interpreter and avoids the penalty of Perl start-up time, giving you super-fast dynamic content.

YOu should Run these commands via SSH:

wget http://perl.apache.org/dist/mod_perl-1.0-current.tar.gz

tar zxvf tar zxvf mod_perl-1.0-current.tar.gz

cd mod_perl-1.29

perl Makefile.PL

If in case you face any errors about missing dependencies like CGI.pm, LWP::UserAgent & HTML::HeadParser which is required by mod_perl you may install them with:

perl -MCPAN -e shell
install HTML::HeadParser

If you had to install any dependencies you must re-run “perl Makefile.PL”. Back to setting up mod_perl, after perl Makefile.PL run:

make
make install

Now, you have just installed mod_perl. Restart Apache with:

service httpd restart
OR
/etc/rc.d/init.d/httpd restart

3) Install Turck MMCache for PHP

What is Turck MMCache for PHP?

Turck MMCache is a free open source PHP accelerator, optimizer, encoder and dynamic content cache for PHP. It increases performance of PHP scripts by caching them in compiled state, so that the overhead of compiling is almost completely eliminated. And it also uses some optimizations to speed up execution of PHP scripts. Turck MMCache typically reduces server load and increases the speed of your PHP code by 1-10 times.

Turck MMCache requires: apache 1.3, mod_php 4.1, autoconf, automake, libtool, m4. You should already have most of these on your server if not use the “apt-get install ” command to get them installed. I

To install Turck MMCache, perform the following commands via SSH:

wget http://aleron.dl.sourceforge.net/sou…e-2.4.6.tar.gz

export PHP_PREFIX=”/usr”

$PHP_PREFIX/bin/phpize

./configure –enable-mmcache=shared –with-php-config=$PHP_PREFIX/bin/php-config

(You must specify the real prefix where PHP is installed in the “export” command. It may be “/usr” “/usr/local”, or something else.)

make

make install

Turck MMCache can be installed both as Zend or PHP extension, so you need to edit your php.ini file (usually /etc/php.ini).
To install as Zend extension:

zend_extension=”/usr/lib/php4/mmcache.so”
mmcache.shm_size=”16″
mmcache.cache_dir=”/tmp/mmcache”
mmcache.enable=”1″
mmcache.optimizer=”1″
mmcache.check_mtime=”1″
mmcache.debug=”0″
mmcache.filter=”"
mmcache.shm_max=”0″
mmcache.shm_ttl=”0″
mmcache.shm_prune_period=”0″
mmcache.shm_only=”0″
mmcache.compress=”1″

To install as PHP extension:

extension=”mmcache.so”
mmcache.shm_size=”16″
mmcache.cache_dir=”/tmp/mmcache”
mmcache.enable=”1″
mmcache.optimizer=”1″
mmcache.check_mtime=”1″
mmcache.debug=”0″
mmcache.filter=”"
mmcache.shm_max=”0″
mmcache.shm_ttl=”0″
mmcache.shm_prune_period=”0″
mmcache.shm_only=”0″
mmcache.compress=”1″

You may need to copy the mmcache.so file to the directory specified to the above paths in the configuration entries.

Create the cache directory:
mkdir /tmp/mmcache
chmod 0777 /tmp/mmcache

Restart Apache with:

service httpd restart
OR
/etc/rc.d/init.d/httpd restart

4) Tuning sysctl.conf

What sysctl.conf?

The sysctl.conf of a server is something that is seldom optimized for performance. You can get a tremendous boost in throughput by adjusting these settings. This configuration has been written by Steve from Rack911. I have applied this configuration to servers ranging from Celeron 1.7Ghz to Dual Xeon 2.8Ghz servers, and on the whole, the load on each lowered after making the changes.

First make a backup of your old /etc/sysctl.conf file by running the following command, logged in as root:

cp /etc/sysctl.conf /etc/sysctl.conf.bak

Now enter:
pico /etc/sysctl.conf

and replace the contents of the file with the following:

# Kernel sysctl configuration file for Red Hat Linux
#
# For binary values, 0 is disabled, 1 is enabled. See sysctl(8) and
# sysctl.conf(5) for more details.

# Disables packet forwarding
net.ipv4.ip_forward=0

# Disables IP source routing
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.lo.accept_source_route = 0
net.ipv4.conf.eth0.accept_source_route = 0
net.ipv4.conf.default.accept_source_route = 0

# Enable IP spoofing protection, turn on source route verification
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.lo.rp_filter = 1
net.ipv4.conf.eth0.rp_filter = 1
net.ipv4.conf.default.rp_filter = 1

# Disable ICMP Redirect Acceptance
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.lo.accept_redirects = 0
net.ipv4.conf.eth0.accept_redirects = 0
net.ipv4.conf.default.accept_redirects = 0

# Enable Log Spoofed Packets, Source Routed Packets, Redirect Packets
net.ipv4.conf.all.log_martians = 0
net.ipv4.conf.lo.log_martians = 0
net.ipv4.conf.eth0.log_martians = 0

# Enable IP spoofing protection, turn on source route verification
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.lo.rp_filter = 1
net.ipv4.conf.eth0.rp_filter = 1
net.ipv4.conf.default.rp_filter = 1

# Disable ICMP Redirect Acceptance
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.lo.accept_redirects = 0
net.ipv4.conf.eth0.accept_redirects = 0
net.ipv4.conf.default.accept_redirects = 0

# Disables the magic-sysrq key
kernel.sysrq = 0

# Decrease the time default value for tcp_fin_timeout connection
net.ipv4.tcp_fin_timeout = 15

# Decrease the time default value for tcp_keepalive_time connection
net.ipv4.tcp_keepalive_time = 1800

# Turn off the tcp_window_scaling
net.ipv4.tcp_window_scaling = 0

# Turn off the tcp_sack
net.ipv4.tcp_sack = 0

# Turn off the tcp_timestamps
net.ipv4.tcp_timestamps = 0

# Enable TCP SYN Cookie Protection
net.ipv4.tcp_syncookies = 1

# Enable ignoring broadcasts request
net.ipv4.icmp_echo_ignore_broadcasts = 1

# Enable bad error message Protection
net.ipv4.icmp_ignore_bogus_error_responses = 1

# Log Spoofed Packets, Source Routed Packets, Redirect Packets
net.ipv4.conf.all.log_martians = 1

# Increases the size of the socket queue (effectively, q0).
net.ipv4.tcp_max_syn_backlog = 1024

# Increase the tcp-time-wait buckets pool size
net.ipv4.tcp_max_tw_buckets = 1440000

# Allowed local port range
net.ipv4.ip_local_port_range = 16384 65536

CTRL + X to exit and save the file

To make your changes take effect immediately, type this command:
/sbin/sysctl -p

You can easily watch your server load by entering “uptime” command via SSH.

Define RAID-4 and it’s advantages

Wed, 27 Aug 2008 19:22:45 +0000

Each entire block is written onto a data disk. Parity for same rank blocks is generated on Writes, recorded on the parity disk and checked on Reads.RAID Level 4 requires a minimum of 3 drives to implement.

1)Very high Read data transaction rate

2)Low ratio of ECC (Parity) disks to data disks means high efficiency

3)High aggregate Read transfer rate

Define RAID-3 and it’s advantages

Wed, 27 Aug 2008 19:21:57 +0000

The data block is subdivided and written on the data disks. Stripe parity is generated on Writes, recorded on the parity disk and checked on Reads.It requires a minimum of 3 drives to implement.

1)Very high Read data transfer rate

2)Very high Write data transfer rate

3)Disk failure has an insignificant impact on throughput

4)Low ratio of ECC (Parity) disks to data disks means high efficiency

Define RAID-2 and it’s advantages

Wed, 27 Aug 2008 19:19:53 +0000

Every single bit of data word is written to a data disk drive. Each data word has its Hamming Code ECC word recorded on the ECC disks. On Read, the ECC code verifies correct data or corrects single disk errors.

1)On the fly” data error correction

2)Very high data transfer rates possible

3)The higher the data transfer rate required, the better the ratio of data disks to ECC disks

4)Relatively simple controller design as compared to RAID levels 3,4 & 5

Define RAID-1 and it’s advantages

Wed, 27 Aug 2008 19:19:11 +0000

RAID-1 requires atleast 2 drives to implement.One writes or two Reads possible per mirrored pair

1)Doubled the Reading transaction rate of single disks, same Write transaction rate as single disks.

2)complete 100% redundancy of data means no rebuild is necessary in case of a disk failure, just a copy to the replacement disk.

3)Transfer rate per block is equal to that of a single disk

4) Under certain circumstances, RAID-1 can sustain multiple simultaneous drive failures.

5) Simplest RAID storage subsystem design.

Define RAID-0 and it’s advantages

Wed, 27 Aug 2008 01:02:04 +0000

RAID (Redundant array of independent disks) 0 implements a striped disk array, the data is broken down into different blocks and each block is written to a separate disk drive.

Advantages of RAID “0″ are as follows:-

1) I/O performance is greatly enhanced by spreading the I/O load among many channels and drives

2) Best performance is achieved when data is striped across multiple controllers with only one drive per controller

3) No parity calculation overhead is involved

4) Very simple design

5) It is very easy to implement.

Define::RAID and it uses

Wed, 27 Aug 2008 00:58:03 +0000

RAID stands for Redundant array of independent disks refers to a data storage scheme using multiple hard drives to share or replicate data among the drives.

1) No matter which RAID array you use backup regularly.

2) For performance always use hardware based RAID controller. If you are tight on budget go for software based RAID. For any production environment I recommend hardware based RAID controller.

3) To make your decision final regarding RAID consider following points:
* Fault tolerant
* Speed vs performance vs overhead
* Your application and required bandwidth
* Final Cost

Steps to Install AW-stats on your server

Mon, 25 Aug 2008 20:38:52 +0000

Step # 1
1) Visit AWStats project home page http://awstats.sourceforge.net/ to download source code or RPM file of awstats. I prefer to use RPM. At the time of writing this latest version of awstats was stable 6.2-1. Here is wget command to download file (make sure you replace appropriate version number in download):
# wget http://prdownloads.sourceforge.net/awstats/awstats-6.2-1.noarch.rpm

Step # 2
Now Install awstats the easiest way to install awstats is use rpm command:
# rpm -ivh awstats-6.2-1.noarch.rpm

Step # 3
Once AWstats installed, it is time to configure it. Fortunately it comes with nice configuration perl script:
1) Change current directory to /usr/local/awstats/tools where awstats configuration tools installed:
# cd /usr/local/awstats/tools

2) Make sure you know the domain’s access log file. Check apache configuration file /etc/httpd/conf/httpd.conf and look for access_log parameter. A quick grep will help you to find out your access log file:
# grep “CustomLog” etc/httpd/conf/httpd.conf

3) Run awstats_configure.pl script to configure awstats (this perl script will do little modification to your httpd.conf file. Here is typical session from above script [required input shown in BOLD + UNDERLINE text]):
# ./awstats_configure.pl
—–> Check and complete web server config file \’/etc/httpd/conf/httpd.conf\’
Add ‘Alias /awstatsclasses “/usr/local/awstats/wwwroot/classes/”‘
Add ‘Alias /awstatscss “/usr/local/awstats/wwwroot/css/”‘
Add ‘Alias /awstatsicons “/usr/local/awstats/wwwroot/icon/”‘
Add ‘ScriptAlias /awstats/ “/usr/local/awstats/wwwroot/cgi-bin/”‘
Add ” directive
AWStats directives added to Apache config file.

—–> Update model config file ‘/etc/awstats/awstats.model.conf’
File awstats.model.conf updated.

—–> Need to create a new config file ?
Do you want me to build a new AWStats config/profile
file (required if first install) [y/N] ? y

—–> Define config file name to create
What is the name of your web site or profile analysis ?
Example: www.mysite.com
Example: demo
Your web site, virtual server or profile name:
> www.cyberciti.biz

—–> Define config file path
In which directory do you plan to store your config file(s) ?
Default: /etc/awstats
Directory path to store config file(s) (Enter for default):

—–> Create config file ‘/etc/awstats/awstats.www.cyberciti.biz.conf’
Config file /etc/awstats/awstats.www.cyberciti.biz.conf created.
—–> Restart Web server with ‘/sbin/service httpd restart’
Stopping httpd: [ OK ]
Starting httpd: [ OK ]
—–> Add update process inside a scheduler

Sorry, configure.pl does not support automatic add to cron yet.
You can do it manually by adding the following command to your cron:
/usr/local/awstats/wwwroot/cgi-bin/awstats.pl -update -config=www.cyberciti.biz
Or if you have several config files and prefer having only one command:
/usr/local/awstats/tools/awstats_updateall.pl now
Press ENTER to continue…

A SIMPLE config file has been created: /etc/awstats/awstats.www.cyberciti.biz.conf
You should have a look inside to check and change manually main parameters.
You can then manually update your statistics for ‘www.cyberciti.biz’ with command:
> perl awstats.pl -update -config=www.cyberciti.biz
You can also read your statistics for ‘www.cyberciti.biz’ with URL:
> http://localhost/awstats/awstats.pl?config=www.cyberciti.biz

Press ENTER to finish…

In above session you need to input three different things.
(A) Answer question “Do you want me to build a new AWStats config/profile file (required if first install) [y/N] ?” with y
(B) Answer question “Your web site, virtual server or profile name:” with your domain name. For example www.cyberciti.biz.
(C) Answer question “Directory path to store config file(s) (Enter for default):” Press enter key to accept default /etc/awstats directory to store configuration for domain.

5) Open your configuration file created by previous script using text editor:
# vi /etc/awstats/awstats.www.cyberciti.biz.conf

6) Look for LogFile parameter and set value to your actual access_log file:
For example:
LogFile=”/home/httpd/vhosts/cyberciti.biz/statistics/logs/access_log”

7) Create directory to store some initial data:
# mkdir -p /var/lib/awstats

Create directory to store awstats html pages so that it can be viewed/analyzed from web browser(create appropriate directory according to your need):
# mkdir -p /home/httpd/vhosts/cyberciti.biz/httpdocs/tools/awstats

9) Build all sorts of webpage’s by running command:
(a) Change Directory:
# cd /usr/local/awstats/tools

(b) Run perl script to build stats:
# ./awstats_buildstaticpages.pl -config=www.cyberciti.biz -update -awstatsprog=/usr/local/awstats/wwwroot/cgi-bin/awstats.pl -dir=/home/httpd/vhosts/cyberciti.biz/httpdocs/tools/awstats/

Where
-config : Name of file for your domain stored in /etc/awstats or file which you modified step # 3 (5)
-update : Update statistics
-awstatsprog : Path to awstats.pl script to generate all statistics
-dir : Directory in which you need to store generated html statistics

10) It is time to test generated web pages aka stats. Fire FireFox (upgrade Internet explorer to FireFox!) and type url http://www.yourdomain.com/path/to/awstats/.

11) Tip you may find that you lost old month statistics with Awstats. To get your old stats you simply need to run following command:
/usr/local/awstats/tools/awstats_buildstaticpages.pl -config=www.cyberciti.biz -update -awstatsprog=/usr/local/awstats/wwwroot/cgi-bin/awstats.pl -dir=/home/httpd/vhosts/cyberciti.biz/httpdocs/tools/awstats/ -month=12 -year=2004

Please replace values of month and year according to your need.

Step # 4: Setup cron job
1) Edit existing contrab:
# crontab -e

2) Add entry to it (here I am generating stats every 10 minutes; generally, you should setup cron, according to your own requirements):
# 10 * * * * /usr/local/awstats/tools/awstats_buildstaticpages.pl -config=www.cyberciti.biz -update -awstatsprog=/usr/local/awstats/wwwroot/cgi-bin/awstats.pl -dir=/home/httpd/vhosts/cyberciti.biz/httpdocs/tools/awstats/ >/dev/null 2>&1
Note command must type in one line or spanned using \

3) Once cron job saved, you need to make sure that crond service running following command ensures that:
a) See if crond running or not:
# service crond status
crond (pid 1519) is running…

b) If it is not running then start it:
# service crond start

c) Make sure crond starts everytime linux comes up after reboot/shutdow:
# chkconfig crond on

Step # 5 Security
Well awstats does not open any port or open any major security risks. However, generally you should consider two important things:
1) Protect your AWstats html pages with password-protected directory. Use apache’s simple directory protection mechanism. This can easily accomplished with .htaccess file in your /path/to/awstats/ directory. You are going to use directory protection using apache’s .htaccess and .htpasswd :

(a) Create .htaccess file in /home/httpd/vhosts/cyberciti.biz/httpdocs/tools/awstats/ directory (add text as follows)
vi /home/httpd/vhosts/cyberciti.biz/httpdocs/tools/awstats/.htaccess
AuthName “AWStats restricted access”
AuthType Basic
AuthUserFile /home/httpd/vhosts/cyberciti.biz/private/.htpasswd
require user awstatsadmin

(b) Create user and password (-c assumes first time you are using .htpasswd file, otherwise skip this directive):
# htpasswd -c /home/httpd/vhosts/cyberciti.biz/private/.htpasswd awstatsadmin

For more information please read documents:
Authentication and access control
.htaccess other options/directives

2) Remember all logs rotated on regular basis including apache. Therefore, it is necessary to make sure that awstats does not misses your statistics. In logrotate httpd configuration file you need to add following 3 lines after /var/log/httpd/*log {:
(a) Open /etc/logrotate.d/httpd in vi text editor:
/etc/logrotate.d/httpd

(b) Add following following 3 lines after /var/log/httpd/*log {:
sharedscripts
prerotate
/usr/local/awstats/tools/awstats_buildstaticpages.pl -config=www.cyberciti.biz – update -awstatsprog=/usr/local/awstats/wwwroot/cgi-bin/awstats.pl -dir=/home/httpd/vhosts/cyberciti.biz/httpdocs/tools/awstats/ >/dev/null 2>&1
endscript
As a result, before log rotating take place our perl script will be execute to generate html stats.

Linux Password trick by using Linux second extended file system attributes

Mon, 25 Aug 2008 20:35:47 +0000

1) Login as root user

2) Run command:
# chattr +i /etc/shadow
Above command will write protect the /etc/shadow file.

3) Now lets test it with user Laurel. Login as laurel:
$passwd
Changing password for user laurel.
Changing password for laurel
(current) UNIX password: OLDPASSWED
New password: NEWPASSWD
Retype new password:NEWPASSWD
passwd: all authentication tokens updated successfully.

4) Now logout and try to login with new password. What you can’t? Yup use the old password you can login now.

5) Get the list of Linux second extended file system using the lsatter command (run as root user )
# lsattr /etc/shadow
—-i——– /etc/shadow

Note:Even root user is not allowed to change password. Before doing this root need to remove the attribute using command:
#chattr –i /etc/shadow
# lsattr /etc/shadow
————- /etc/shadow