Running a webserver with a lot quantity of virtual server is a double-edged sword. The more hosted site, the more income for us, but for every ten legitimate users, there’s at least one greed user that wants the whole site traffic. This is bad enough under normal circumstances, but it’s even worse when we’re on a co-hosted link and/or a shared server co-operation, because this will ruins the experience for the other clients AND the other servers on the same link.
Such was the situation on one of my server the other day, when I was recently made aware that one of the virtual servers was generating the vast majority of network traffic on a shared server on a shared network. I scanned quickly for runaway/zombie procs and found nothing, except an unusually large number of delay when opening one of the hosted site.
Well, I have to find a way how to limit that “greedy” user without too much modification made to the server itself because this is one is our main production server. So I started to google around for bandwitdh throttling measures that are spesific to web traffic, and I came across bw_mod by Ivan “Bruce” Barrera. It, along with a couple of built-in directives within apache, look to be the easiest solution to my problem, and here’s how I implemented them.
This mod_bw needs recompile the module using apxs2, so we need apache devel package to do that, install this using apt source from cdrom (to make sure I’ve got the same version libaprutil1 package along with my running apache package) :
komodo:/home/jfdesign# apt-get install libaprutil1-dev
Reading package lists… Done
Building dependency tree
Reading state information… Done
The following packages were automatically installed and are no longer required:
courier-ssl courier-base courier-authlib-userdb
Use ‘apt-get autoremove’ to remove them.
The following extra packages will be installed:
comerr-dev libapr1-dev libdb4.6-dev libexpat1-dev libkadm55 libkrb5-dev libldap2-dev libpcre3-dev libpcrecpp0 libpq-dev
libsqlite3-dev libssl-dev uuid-dev
Suggested packages:
doc-base db4.6-doc krb5-doc postgresql-doc-8.3 sqlite3-doc
The following NEW packages will be installed:
comerr-dev libapr1-dev libaprutil1-dev libdb4.6-dev libexpat1-dev libkadm55 libkrb5-dev libldap2-dev libpcre3-dev libpcrecpp0
libpq-dev libsqlite3-dev libssl-dev uuid-dev
0 upgraded, 14 newly installed, 0 to remove and 0 not upgraded.
Need to get 0B/6122kB of archives.
After this operation, 20.7MB of additional disk space will be used.
Do you want to continue [Y/n]? y
Selecting previously deselected package uuid-dev.
(Reading database … 115845 files and directories currently installed.)
Unpacking uuid-dev (from …/uuid-dev_1.2-1.41.3-1_i386.deb) …
Selecting previously deselected package libapr1-dev.
Unpacking libapr1-dev (from …/libapr1-dev_1.2.12-5+lenny1_i386.deb) …
Selecting previously deselected package libldap2-dev.
Unpacking libldap2-dev (from …/libldap2-dev_2.4.11-1_i386.deb) …
Selecting previously deselected package libexpat1-dev.
Unpacking libexpat1-dev (from …/libexpat1-dev_2.0.1-4_i386.deb) …
Selecting previously deselected package libdb4.6-dev.
Unpacking libdb4.6-dev (from …/libdb4.6-dev_4.6.21-11_i386.deb) …
Selecting previously deselected package libpcrecpp0.
Unpacking libpcrecpp0 (from …/libpcrecpp0_7.6-2.1_i386.deb) …
Selecting previously deselected package libpcre3-dev.
Unpacking libpcre3-dev (from …/libpcre3-dev_7.6-2.1_i386.deb) …
Selecting previously deselected package libsqlite3-dev.
Unpacking libsqlite3-dev (from …/libsqlite3-dev_3.5.9-6_i386.deb) …
Selecting previously deselected package libssl-dev.
Unpacking libssl-dev (from …/libssl-dev_0.9.8g-15+lenny3_i386.deb) …
Selecting previously deselected package libkadm55.
Unpacking libkadm55 (from …/libkadm55_1.6.dfsg.4~beta1-5lenny1_i386.deb) …
Selecting previously deselected package comerr-dev.
Unpacking comerr-dev (from …/comerr-dev_2.1-1.41.3-1_i386.deb) …
Selecting previously deselected package libkrb5-dev.
Unpacking libkrb5-dev (from …/libkrb5-dev_1.6.dfsg.4~beta1-5lenny1_i386.deb) …
Selecting previously deselected package libpq-dev.
Unpacking libpq-dev (from …/libpq-dev_8.3.7-0lenny1_i386.deb) …
Selecting previously deselected package libaprutil1-dev.
Unpacking libaprutil1-dev (from …/libaprutil1-dev_1.2.12+dfsg-8+lenny4_i386.deb) …
Processing triggers for man-db …
Setting up uuid-dev (1.2-1.41.3-1) …
Setting up libapr1-dev (1.2.12-5+lenny1) …
Setting up libldap2-dev (2.4.11-1) …
Setting up libexpat1-dev (2.0.1-4) …
Setting up libdb4.6-dev (4.6.21-11) …
Setting up libpcrecpp0 (7.6-2.1) …
Setting up libpcre3-dev (7.6-2.1) …
Setting up libsqlite3-dev (3.5.9-6) …
Setting up libssl-dev (0.9.8g-15+lenny3) …
Setting up libkadm55 (1.6.dfsg.4~beta1-5lenny1) …
Setting up comerr-dev (2.1-1.41.3-1) …
Setting up libkrb5-dev (1.6.dfsg.4~beta1-5lenny1) …
Setting up libpq-dev (8.3.7-0lenny1) …
Setting up libaprutil1-dev (1.2.12+dfsg-8+lenny4) …
komodo:/home/jfdesign#
Now insert this repo cause I don’t have apache devel on my cdrom :
deb http://ftp.us.debian.org/debian/ etch main contrib non-free
Reload and then installing apache-threaded devel package :
komodo:/home/jfdesign# apt-get install apache2-threaded-dev
Reading package lists… Done
Building dependency tree
Reading state information… Done
The following packages were automatically installed and are no longer required:
courier-ssl courier-base courier-authlib-userdb
Use ‘apt-get autoremove’ to remove them.
The following NEW packages will be installed:
apache2-threaded-dev
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 212kB of archives.
After this operation, 807kB of additional disk space will be used.
Get:1 http://ftp.us.debian.org lenny/main apache2-threaded-dev 2.2.9-10+lenny4 [212kB]
Fetched 212kB in 4s (44.1kB/s)
Selecting previously deselected package apache2-threaded-dev.
(Reading database … 117773 files and directories currently installed.)
Unpacking apache2-threaded-dev (from …/apache2-threaded-dev_2.2.9-10+lenny4_i386.deb) …
Processing triggers for man-db …
Setting up apache2-threaded-dev (2.2.9-10+lenny4) …
komodo:/home/jfdesign#
Now compile the mod_bw with apxs2 :
komodo:/home/jfdesign/mod_bw# apxs2 -i -a -c mod_bw.c
/usr/share/apr-1.0/build/libtool –silent –mode=compile –tag=disable-static i486-linux-gnu-gcc -prefer-pic -DLINUX=2 -D_GNU_SOURCE -D_LARGEFILE64_SOURCE -D_REENTRANT -I/usr/include/apr-1.0 -I/usr/include/mysql -I/usr/include/openssl -I/usr/include/postgresql -I/usr/include/xmltok -pthread -I/usr/include/apache2 -I/usr/include/apr-1.0 -I/usr/include/apr-1.0 -I/usr/include/postgresql -I/usr/include/mysql -c -o mod_bw.lo mod_bw.c && touch mod_bw.slo
/usr/share/apr-1.0/build/libtool –silent –mode=link –tag=disable-static i486-linux-gnu-gcc -o mod_bw.la -rpath /usr/lib/apache2/modules -module -avoid-version mod_bw.lo
/usr/share/apache2/build/instdso.sh SH_LIBTOOL=’/usr/share/apr-1.0/build/libtool’ mod_bw.la /usr/lib/apache2/modules
/usr/share/apr-1.0/build/libtool –mode=install cp mod_bw.la /usr/lib/apache2/modules/
cp .libs/mod_bw.so /usr/lib/apache2/modules/mod_bw.so
cp .libs/mod_bw.lai /usr/lib/apache2/modules/mod_bw.la
PATH=”$PATH:/sbin” ldconfig -n /usr/lib/apache2/modules
———————————————————————-
Libraries have been installed in:
/usr/lib/apache2/modulesIf you ever happen to want to link against installed libraries
in a given directory, LIBDIR, you must either use libtool, and
specify the full pathname of the library, or use the `-LLIBDIR’
flag during linking and do at least one of the following:
– add LIBDIR to the `LD_LIBRARY_PATH’ environment variable
during execution
– add LIBDIR to the `LD_RUN_PATH’ environment variable
during linking
– use the `-Wl,–rpath -Wl,LIBDIR’ linker flag
– have your system administrator add LIBDIR to `/etc/ld.so.conf’See any operating system documentation about shared libraries for
more information, such as the ld(1) and ld.so(8) manual pages.
———————————————————————-
chmod 644 /usr/lib/apache2/modules/mod_bw.so
komodo:/home/jfdesign/mod_bw#
Done !, easy isn’t it ? The results mod_bw.so module now on /usr/lib/apache2/modules/mod_bw.so, and it’s time to add some directive to the apache2.conf or httpd.conf to load the mod_bw module and adjust the parameter. Please make sure this line exist on the apache2.conf and refering to the right path :
LoadModule bw_module mod_bw.so
Within KLIXs, there are nothing to worries, I already compile them and you can install them easily using synaptic or download this and do rpm -Uvh from here.
Second note :
One of my server has already been configure well with ispcp under lenny 5.0.3, the problem is I cannot install apxs due to libaprutil1-dev missing and cannot installed by some dependencies issue with libldap2-dev. I cannot find the corresponding libldap2 that can be use to solve this on lenny, but then I found some workaround to achieve this. I have to step backward by using etch repo to track down the libldap2-dev to satisfied them.
[root@www jfdesign]# apt-get install libldap2
Reading package lists… Done
Building dependency tree
Reading state information… Done
The following extra packages will be installed:
libgnutls13 liblzo1 libopencdk8
Suggested packages:
gnutls-bin
The following NEW packages will be installed:
libgnutls13 libldap2 liblzo1 libopencdk8
0 upgraded, 4 newly installed, 0 to remove and 0 not upgraded.
Need to get 632kB of archives.
After this operation, 1610kB of additional disk space will be used.
Do you want to continue [Y/n]? y
Get:1 http://ftp.us.debian.org etch/main liblzo1 1.08-3 [53.9kB]
Get:2 http://ftp.us.debian.org etch/main libopencdk8 0.5.9-2 [102kB]
Get:3 http://ftp.us.debian.org etch/main libgnutls13 1.4.4-3+etch4 [315kB]
Get:4 http://ftp.us.debian.org etch/main libldap2 2.1.30-13.3 [162kB]
Fetched 632kB in 4s (133kB/s)
Selecting previously deselected package liblzo1.
(Reading database … 46970 files and directories currently installed.)
Unpacking liblzo1 (from …/liblzo1_1.08-3_amd64.deb) …
Selecting previously deselected package libopencdk8.
Unpacking libopencdk8 (from …/libopencdk8_0.5.9-2_amd64.deb) …
Selecting previously deselected package libgnutls13.
Unpacking libgnutls13 (from …/libgnutls13_1.4.4-3+etch4_amd64.deb) …
Selecting previously deselected package libldap2.
Unpacking libldap2 (from …/libldap2_2.1.30-13.3_amd64.deb) …
Replaced by files in installed package libldap-2.4-2 …
Processing triggers for man-db …
Setting up liblzo1 (1.08-3) …
Setting up libopencdk8 (0.5.9-2) …
Setting up libgnutls13 (1.4.4-3+etch4) …
Setting up libldap2 (2.1.30-13.3) …
[root@www jfdesign]#
then install the devel package :
[root@www jfdesign]# apt-get install libldap2-dev
Reading package lists… Done
Building dependency tree
Reading state information… Done
The following NEW packages will be installed:
libldap2-dev
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 251kB of archives.
After this operation, 885kB of additional disk space will be used.
Get:1 http://ftp.us.debian.org etch/main libldap2-dev 2.1.30-13.3 [251kB]
Fetched 251kB in 2s (96.3kB/s)
Selecting previously deselected package libldap2-dev.
(Reading database … 47015 files and directories currently installed.)
Unpacking libldap2-dev (from …/libldap2-dev_2.1.30-13.3_amd64.deb) …
Processing triggers for man-db …
Setting up libldap2-dev (2.1.30-13.3) …
[root@www jfdesign]#
[root@www jfdesign]# dpkg -l | grep ldap
ii libldap-2.4-2 2.4.11-1+lenny1 OpenLDAP libraries
ii libldap2 2.1.30-13.3 OpenLDAP libraries
ii libldap2-dev 2.1.30-13.3 OpenLDAP development libraries
ii php5-ldap 5.2.6.dfsg.1-1+lenny4 LDAP module for php5
ii proftpd-mod-ldap 1.3.1-17lenny4 versatile, virtual-hosting FTP daemon – LDAP
[root@www jfdesign]#
Now switch back to lenny repo package, and gladly installed libaprutil-dev with big smile :
[root@www jfdesign]# apt-get install libaprutil1-dev
Reading package lists… Done
Building dependency tree
Reading state information… Done
The following extra packages will be installed:
comerr-dev libapr1-dev libdb4.6-dev libexpat1-dev libkadm55 libkrb5-dev libmysqlclient15-dev libpcre3-dev libpcrecpp0 libpq-dev libsqlite3-dev
libssl-dev uuid-dev zlib1g-dev
Suggested packages:
doc-base db4.6-doc krb5-doc postgresql-doc-8.3 sqlite3-doc
The following NEW packages will be installed:
comerr-dev libapr1-dev libaprutil1-dev libdb4.6-dev libexpat1-dev libkadm55 libkrb5-dev libmysqlclient15-dev libpcre3-dev libpcrecpp0 libpq-dev
libsqlite3-dev libssl-dev uuid-dev zlib1g-dev
0 upgraded, 15 newly installed, 0 to remove and 1 not upgraded.
Need to get 13.4MB of archives.
After this operation, 47.7MB of additional disk space will be used.
Do you want to continue [Y/n]? y
Get:1 http://ftp.us.debian.org lenny/main uuid-dev 1.2-1.41.3-1 [59.4kB]
Get:2 http://ftp.us.debian.org lenny/main libapr1-dev 1.2.12-5+lenny1 [826kB]
Get:3 http://ftp.us.debian.org lenny/main libexpat1-dev 2.0.1-4+lenny3 [224kB]
Get:4 http://ftp.us.debian.org lenny/main libdb4.6-dev 4.6.21-11 [730kB]
Get:5 http://ftp.us.debian.org lenny/main libpcrecpp0 7.6-2.1 [94.2kB]
Get:6 http://ftp.us.debian.org lenny/main libpcre3-dev 7.6-2.1 [260kB]
Get:7 http://ftp.us.debian.org lenny/main libsqlite3-dev 3.5.9-6 [333kB]
Get:8 http://ftp.us.debian.org lenny/main zlib1g-dev 1:1.2.3.3.dfsg-12 [163kB]
Get:9 http://ftp.us.debian.org lenny/main libssl-dev 0.9.8g-15+lenny6 [2242kB]
Get:10 http://ftp.us.debian.org lenny/main libkadm55 1.6.dfsg.4~beta1-5lenny2 [170kB]
Get:11 http://ftp.us.debian.org lenny/main comerr-dev 2.1-1.41.3-1 [42.1kB]
Get:12 http://ftp.us.debian.org lenny/main libkrb5-dev 1.6.dfsg.4~beta1-5lenny2 [92.7kB]
Get:13 http://ftp.us.debian.org lenny/main libpq-dev 8.3.9-0lenny1 [453kB]
Get:14 http://ftp.us.debian.org lenny/main libmysqlclient15-dev 5.0.51a-24+lenny2+spu1 [7594kB]
Get:15 http://ftp.us.debian.org lenny/main libaprutil1-dev 1.2.12+dfsg-8+lenny4 [134kB]
Fetched 13.4MB in 1min4s (207kB/s)
Selecting previously deselected package uuid-dev.
(Reading database … 47171 files and directories currently installed.)
Unpacking uuid-dev (from …/uuid-dev_1.2-1.41.3-1_amd64.deb) …
Selecting previously deselected package libapr1-dev.
Unpacking libapr1-dev (from …/libapr1-dev_1.2.12-5+lenny1_amd64.deb) …
Selecting previously deselected package libexpat1-dev.
Unpacking libexpat1-dev (from …/libexpat1-dev_2.0.1-4+lenny3_amd64.deb) …
Selecting previously deselected package libdb4.6-dev.
Unpacking libdb4.6-dev (from …/libdb4.6-dev_4.6.21-11_amd64.deb) …
Selecting previously deselected package libpcrecpp0.
Unpacking libpcrecpp0 (from …/libpcrecpp0_7.6-2.1_amd64.deb) …
Selecting previously deselected package libpcre3-dev.
Unpacking libpcre3-dev (from …/libpcre3-dev_7.6-2.1_amd64.deb) …
Selecting previously deselected package libsqlite3-dev.
Unpacking libsqlite3-dev (from …/libsqlite3-dev_3.5.9-6_amd64.deb) …
Selecting previously deselected package zlib1g-dev.
Unpacking zlib1g-dev (from …/zlib1g-dev_1%3a1.2.3.3.dfsg-12_amd64.deb) …
Selecting previously deselected package libssl-dev.
Unpacking libssl-dev (from …/libssl-dev_0.9.8g-15+lenny6_amd64.deb) …
Selecting previously deselected package libkadm55.
Unpacking libkadm55 (from …/libkadm55_1.6.dfsg.4~beta1-5lenny2_amd64.deb) …
Selecting previously deselected package comerr-dev.
Unpacking comerr-dev (from …/comerr-dev_2.1-1.41.3-1_amd64.deb) …
Selecting previously deselected package libkrb5-dev.
Unpacking libkrb5-dev (from …/libkrb5-dev_1.6.dfsg.4~beta1-5lenny2_amd64.deb) …
Selecting previously deselected package libpq-dev.
Unpacking libpq-dev (from …/libpq-dev_8.3.9-0lenny1_amd64.deb) …
Selecting previously deselected package libmysqlclient15-dev.
Unpacking libmysqlclient15-dev (from …/libmysqlclient15-dev_5.0.51a-24+lenny2+spu1_amd64.deb) …
Selecting previously deselected package libaprutil1-dev.
Unpacking libaprutil1-dev (from …/libaprutil1-dev_1.2.12+dfsg-8+lenny4_amd64.deb) …
Processing triggers for man-db …
Setting up uuid-dev (1.2-1.41.3-1) …
Setting up libapr1-dev (1.2.12-5+lenny1) …
Setting up libexpat1-dev (2.0.1-4+lenny3) …
Setting up libdb4.6-dev (4.6.21-11) …
Setting up libpcrecpp0 (7.6-2.1) …
Setting up libpcre3-dev (7.6-2.1) …
Setting up libsqlite3-dev (3.5.9-6) …
Setting up zlib1g-dev (1:1.2.3.3.dfsg-12) …
Setting up libssl-dev (0.9.8g-15+lenny6) …
Setting up libkadm55 (1.6.dfsg.4~beta1-5lenny2) …
Setting up comerr-dev (2.1-1.41.3-1) …
Setting up libkrb5-dev (1.6.dfsg.4~beta1-5lenny2) …
Setting up libpq-dev (8.3.9-0lenny1) …
Setting up libmysqlclient15-dev (5.0.51a-24+lenny2+spu1) …
Setting up libaprutil1-dev (1.2.12+dfsg-8+lenny4) …
[root@www jfdesign]#
[root@www jfdesign]# apt-get install apache2-threaded-dev
Reading package lists… Done
Building dependency tree
Reading state information… Done
The following NEW packages will be installed:
apache2-threaded-dev
0 upgraded, 1 newly installed, 0 to remove and 1 not upgraded.
Need to get 211kB of archives.
After this operation, 807kB of additional disk space will be used.
Get:1 http://ftp.us.debian.org lenny/main apache2-threaded-dev 2.2.9-10+lenny6 [211kB]
Fetched 211kB in 2s (96.3kB/s)
Selecting previously deselected package apache2-threaded-dev.
(Reading database … 49016 files and directories currently installed.)
Unpacking apache2-threaded-dev (from …/apache2-threaded-dev_2.2.9-10+lenny6_amd64.deb) …
Processing triggers for man-db …
Setting up apache2-threaded-dev (2.2.9-10+lenny6) …
[root@www jfdesign]#
Now I have apxs2 on hand to compile the bandwidth limiter module.
Some slight modification needs to be patched to mod.bw.c before compilation if you face this error on restarting apache :
[root@www mod_bandwidth]# /etc/init.d/apache2 restart
Restarting web server: apache2We failed to correctly shutdown apache, so we’re now killing all running apache processes. This is almost certainly suboptimal, so please make sure your system is working as you’d expect now! (warning).
… waiting ….apache2: Syntax error on line 185 of /etc/apache2/apache2.conf: Syntax error on line 1 of /etc/apache2/mods-enabled/mod_bw.load: Cannot load /usr/lib/apache2/modules/mod_bw.so into server: /usr/lib/apache2/modules/mod_bw.so: undefined symbol: apr_atomic_cas
failed!
[root@www mod_bandwidth]#
And here is the change for mod_bw.c to bypass the compatibility problem :
/* Compatibility for ARP < 1 */
#if (APR_MAJOR_VERSION < 1)
#define apr_atomic_inc32 apr_atomic_inc
#define apr_atomic_dec32 apr_atomic_dec
#define apr_atomic_add32 apr_atomic_add
#define apr_atomic_cas32 apr_atomic_cas
#define apr_atomic_set32 apr_atomic_set
#endifto:
/* Compatibility for ARP < 1 */
/*
#if (APR_MAJOR_VERSION < 1)
#define apr_atomic_inc32 apr_atomic_inc
#define apr_atomic_dec32 apr_atomic_dec
#define apr_atomic_add32 apr_atomic_add
#define apr_atomic_cas32 apr_atomic_cas
#define apr_atomic_set32 apr_atomic_set
#endif
*/
Compile again the mod_bw.c and you should good to go.
Just in case the apxs2 doesn’t create you the right LoadModule at /etc/apache2/mods_available and the corresponding symlink on /etc/apache2/mods_enabled, then create yourself. Here is the mod_bw.load on mine :
LoadModule bw_module /usr/lib/apache2/modules/mod_bw.so
and don’t forget to create manually the mod_bw.load symlink on /etc/apache2/mods_enabled
Get it to work :
Most of us think the installation is the hardest part, no! … you wrong, make it work is the most confusing part. I don’t want to expose my setting due to some security problems, but here is some samples, it;s pretty straight forward, so I don’t see any difficulties for any webmaster that installed the virtual host hosting by them self. You should know where you put the Virtual Host setting do you ?
Configuration sample for Virtual Host :
Limit every user to a max of 10Kb/s on a vhost :
<Virtualhost *>
BandwidthModule On
ForceBandWidthModule On
Bandwidth all 10240
MinBandwidth all -1
Servername www.example.com
</Virtualhost>Limit al internal users (lan) to 1000 kb/s with a minimum of 50kb/s , and files greater than 500kb to 50kb/s.
<Virtualhost *>
BandwidthModule On
ForceBandWidthModule On
Bandwidth all 1024000
MinBandwidth all 50000
LargeFileLimit * 500 50000
Servername www.example.com
</Virtualhost>Limit avi and mpg extensions to 20kb/s.
<Virtualhost *>
BandwidthModule On
ForceBandWidthModule On
LargeFileLimit .avi 1 20000
LargeFileLimit .mpg 1 20000
Servername www.example.com
</Virtualhost>Using it the “right” way, with output filter by mime type (for text)
to 5kb/s:<Virtualhost *>
BandwidthModule On
AddOutputFilterByType MOD_BW text/html text/plain
Bandwidth all 5000
Servername www.example.com
</Virtualhost>
Now I can stop the small nagging sites hogging all the bandwidth of my server !!! I nailed you to the floor …
