HTTP Load Balancing

The following information is issued as a simple and basic tutorial of how to implement a load balancing system.

Load balancing is a technique to spread work between two or more computers, network links, CPUs, hard drives, or other resources, in order to get optimal resource utilization, throughput, or response time. – Wikipedia.

With load balancing we can get the power of a high availability cluster using cheap machines. Also it is easier to scale the systems horizontally and just add more servers as our needs increase.

Of course there exists hardware solutions specifically designed to take care of load balancing. Citrix and F5 being the most popular among datacenters but this are expensive and mostly targeted at the enterprise market, not the home user.

Fortunately Linux provides us with a powerful tool to develop load balancing: IPVS (Internet Protocol Virtual Server).

The procedure for the set up is fairly easy. First you need to define the machine that us gonna act as the main server and “father” of the cluster. Ideally this machine has to be one with post processing capabilities.

By default Linux comes with IPVS pre-installed, but in the case you don’t have it, just yum or apt-get it.

In this example we are going to set up a cluster with 4 machines:

Main cluster server: 10.50.51.52
cluster member A: 10.51.51.10
cluster member B: 10.51.51.20
cluster member C: 10.51.51.30
cluster member D: 10.51.51.40

Open a terminal on the machine that has installed IPVS (10.50.51.52) and create the virtual host and assing the child machines as follows:

$ ipvsadm -A -t 10.50.51.52:http -s rr -p5200
$ ipvsadm -a -t 10.50.51.52:http -r 10.51.51.10:http -m
$ ipvsadm -a -t 10.50.51.52:http -r 10.51.51.20:http -m
$ ipvsadm -a -t 10.50.51.52:http -r 10.51.51.30:http -m
$ ipvsadm -a -t 10.50.51.52:http -r 10.51.51.40:http -m

A little explanation of the used parameters:

t - Use TCP protocol pointing to the IP address of the server on the specified port.
s – Use allocation of TCP and UDP connections.
rr – use rr-type balancing Robin Robin (distributes the burden equally in all servers).
p – Specify the persistence of TCP connections. If empty, 300 is assigned by default.
A - Registers a new virtual service. This is the main server.

Thats it, we have a fully functional mini cluster on port 80 (http).

But what if someday, one of your machines burns out. Don’t let the panic attacks you.

To delete a “child” machine just issue your godlike root powers and type the following command:

$ ipvsadm -d -t 10.50.51.52:http

Or if you want to completely remove the whole clustering, then type the following command:

ipvsadm -D -t 10.50.51.52:http

The d deletes the specified server, and D deletes the cluster and all related servers it may have assigned.

Mounting NFS Resources

Last week I published an article about mounting remote filesystems in Linux using the SSHFS method.

Today I’m covering another way of implementing a network filesystem; This method is the classic one and maybe one of the most used out there.

NFS stands for Network File System and the main difference between SSHFS is that NFS by himself is a network protocol, while SSHFS makes use of our favorite terminal access method: SSH.
NFS does NOT send the information through a secure tunnel, hence it is only recommended(and I really mean ONLY) on those situations where you trust your network traffic or have already implemented another way of authenticating the server.

The first step is to check whether or not our server has NFS daemon already installed; NFS is an extension of the mount command. By default, most Linux distributions already include the mount libraries.

# mount.nfs -V

Now, lets start the daemon.

$ /etc/init.d/nfs start

Lets suppose we want the folder /usr/man/ from computer HOST_ONE, to be mounted on /mnt/backup/ on computer HOST_TWO.

What we have to do is connect to HOST_ONE and tell the system which folders are authorized to be shared. This is defined in the exports file:

$ vi /etc/exports/

Probably the file would be empty, so lets start filling it. The asterisk tells the system to grant r,w,x privileges. :

/usr/man    *

Alternally you can add specific privileges enclosed by parentheses.

/usr/man    *(rw)

After saving the changes, restart the FNS daemon:

$ /etc/init.d/nfs start

This is everything we have to do on HOST_ONE. Nos its time to actually mount the filesystem on the remote computer.

$ mount -t nfs 192.168.10.20:/usr/man /mnt/backup

Alternately you can add the IP address of the machine to the /etc/hosts file and use the host name instead of the IP:

$ mount -t nfs HOST_ONE:/usr/man /mnt/backup

x86_64 Kernel IOMMU Error on Intel Quad Core

For those of you that have tried installing a 64 bit kernel on a new intel box, the machine wont boot from media if the processor makes use of DMAR graphics engine.

When trying to boot Linux from the media (DVD/CD-ROM), upon loading the system files and standard drivers, the system hangs after the loading kernel screen.

Late quad core processors present this issue becoming a royal pain in the ass:

IOMMU: Setting identity map for device 0000:00:1a.1 [xdefd6000 - 0xdefd7000]

To solve this issue, you have to disable the graphics mapping Intel engine. On the initial GRUB installation screen lets edit the kernel parameters by pressing E; Then we just append the following phrase:

intel_iommu=off

Press B to boot and the system will start normally.

OK, but we don’t want to do this every time we power on the machine. There are two easy ways around this, the former being the recommended choice:

a) After the first successful booting, upgrade to the latest version of the kernel.
b) If you refuse to update your kernel, edit grub.conf to boot without loading DMAR.

Its up to you to decide the best way.

Apache Out Of Memory Error

For sure many people have had this kind of problem when deploying an application over Apache Web Server.

An Out of Memory (OOM) error occurs when the Apache instances start to increase their physical memory usage, overloading the server resources and forcing the system to start swapping RAM. When the swap memory depletes, the whole integrity and stability of the operating system is put at risk, then the kernel decides that is better to start killing processes in order to keep himself alive.

An example of a OOM error:
# vim /var/log/messages
July 12 13:11:19 localhost kernel: [ 8267.512850] Out of memory: kill process 7986 (apache) score 15147 or a child
July 12 13:11:19 localhost kernel: [ 8267.513008] Killed process 7986 (apache)

How the system knows which processes are safe to kill is calculated by a fast -but some times inefficient- algorithm but in general terms the parameters are the following:

1. Which processes are running as root?
2. How long a process has been running?
3. How many RAM/Swap is used by a process?
4. if (process == init) Don’t touch it!

Because in an OOM situation, the system has just a fraction of seconds to make the decision, the heuristics are not quite optimal and sometimes they will be inaccurate.

Apache processes use a ton of RAM. This becomes major threat when you realize that after each process has done its job, the bloated process sits and spoon-feeds data to the client, instead of moving on to bigger and better things.

The trick to get around this consists in fixing the Apache configuration file; The default file is the following:

# cd /etc/httpd/conf/
# vi httpd.conf

MaxClients
Max clients defines the maximum number of connections a child process can handle simultaneously. When a child process reach this limit, new connections start to queue; Of course this connections have to be saved somewhere on the system… ungracefully they’re stored on the RAM and then the swap memory.
If you use Apache to serve dynamic content, your simultaneous connections are severely limited. Exceed a certain number, and your system begins cannibalistic swapping.
By default this is set to 150. You could try increasing this value, just keep in mind that in a normal and standard installation of Apache you are limited to 256 MaxClients (I’m covering this topic on another post). If you set it to low, a message like the following will be printed in your error.log file:

[error] server reached MaxClients setting, consider raising the MaxClients setting.

MinSpareServers
When the Apache daemon (the father process) is started, a fixed number of child processes is spawned creating a pool of servers ready to serve the clients. By default this is set to 5; Creating new child processes is expensive for the system resources and the best practice is to always have servers ready for all the clients that may connect to the server. If you have a lot of traffic you should think in increasing this value.

MaxSpareServers
You can also define the max child processes your Apache server can spawn. As with the MaxClients option, when this value is reached new connections start to queue. A value between 20-50 should be fine in most situations.

MaxRequestsPerChild
Forcing your processes to die after awhile makes them start over with lower RAM usage. This can reduce total memory usage in many situations. This variable is very important on production systems. By default each Apache instance can serve an unlimited amount of clients/traffic. Child processes are never killed by the father process so, as time goes by, you can encounter with a process that has been running for, lets say, 28 days!.
If child processes never “recycle” themselves there is a high chance that the older ones may have used a lot of RAM they are not going to release until they die. Someday you will have to re-spawn this processes in order to keep the memory “clean”.
MaxRequestsPerChild defines the maximum connections a child process can serve. When the value is reached the child processes stops receiving new connections, awaits for the last current connection to be served and then it dies. Apache father process then re-spawns a new child and the cycle continues.
A value of 0 means unlimited (not recommended); In most cases values like 5000 would work fine but also values as low as 10 or 20 also may work for you. This is a game of catch-up, with your dynamic files constantly increasing total RAM usage, and restarting processes constantly reducing it. You have to experiment with this value.

There are more variables you can change and play with them, but those ones does have a little (if not null) relation with the OOM problem.

An OOM error is just a safety measure, NOT a system feature. The very existence of an OOM killer is a bug.

The goal of the sysadmin shall not be fix the OOM, but to get to the root of the problem and fix it. Poorly optimized queries and/or running the application on a piece of crap hardware, are common causes that may lead the server to an OOM.

SSHFS – Mounting Network Filesystems in Linux

Lets get a hold on the Secure Shell Filesystem Utilities (or just SSHFS to keep it more simple) using the awesome fuse libraries.

The things we need in order to use this method are the following:

• sshfs
• fuse-utils
• fuse-source
• kernel-headers
• kernel-devel

For the lazy people, all the these packages are available through the yum repository in Fedora (and apt-get in Debian).

$ yum install fuse-utils
$ yum install fuse-source
$ yum install kernel-header
$ yum install kernel-devel

Since yum takes care of all the versions compatibility and dependencies you wont have to configure anything.

Of course you can do the manual way by downloading and installing the packages by hand. For the kernel-devel and kernel-header you have to chose the one that matches your specific kernel version. Keep in mind that in most cases the kernel-devel package already includes the headers for your linux distribution:

Download fuse here
And fuse-ssh here
And the kernel packages here

Now that you have the needed files let’s start installing the kernel-devel.

As root do the following:

$ rpm -Uvf kernel-devel.rpm

Then install the fuse utils:

$ tar -zxvf fuse-2.7.4.tar.gz
$ cd /fuse-2.7.4
$ ./configure --with-kernel=/usr/source/kernel/[YOUR KERNEL]
$ make
$ make install

Now lets install sshfs:

$ tar -zxvf sshfs-fuse-2.1.tar.gz
$ cd sshfs-fuse-2.1
$ ./configure
$ make
$ make install

That’s it! You can check it by typing sshfs --help on a terminal window.

Using sshfs is actually like using a standard ssh connection command.

For example, suppose you want to mount the directory /home/ which is located on the remote computer 192.168.10.20, on the local directory /mnt/backup/:

$ sshfs root@192.168.10.20:/home /mnt/backup

To unmount it just issue the following command:

$ fusermount -u /mnt/backup