x86_64 Kernel IOMMU Error on Intel Quad Core

For those of you that have tried installing a 64 bit kernel on a new intel box, the machine wont boot from media if the processor makes use of DMAR graphics engine.

When trying to boot Linux from the media (DVD/CD-ROM), upon loading the system files and standard drivers, the system hangs after the loading kernel screen.

Late quad core processors present this issue becoming a royal pain in the ass:

IOMMU: Setting identity map for device 0000:00:1a.1 [xdefd6000 - 0xdefd7000]

To solve this issue, you have to disable the graphics mapping Intel engine. On the initial GRUB installation screen lets edit the kernel parameters by pressing E; Then we just append the following phrase:

intel_iommu=off

Press B to boot and the system will start normally.

OK, but we don’t want to do this every time we power on the machine. There are two easy ways around this, the former being the recommended choice:

a) After the first successful booting, upgrade to the latest version of the kernel.
b) If you refuse to update your kernel, edit grub.conf to boot without loading DMAR.

Its up to you to decide the best way.

Apache Out Of Memory Error

For sure many people have had this kind of problem when deploying an application over Apache Web Server.

An Out of Memory (OOM) error occurs when the Apache instances start to increase their physical memory usage, overloading the server resources and forcing the system to start swapping RAM. When the swap memory depletes, the whole integrity and stability of the operating system is put at risk, then the kernel decides that is better to start killing processes in order to keep himself alive.

An example of a OOM error:
# vim /var/log/messages
July 12 13:11:19 localhost kernel: [ 8267.512850] Out of memory: kill process 7986 (apache) score 15147 or a child
July 12 13:11:19 localhost kernel: [ 8267.513008] Killed process 7986 (apache)

How the system knows which processes are safe to kill is calculated by a fast -but some times inefficient- algorithm but in general terms the parameters are the following:

1. Which processes are running as root?
2. How long a process has been running?
3. How many RAM/Swap is used by a process?
4. if (process == init) Don’t touch it!

Because in an OOM situation, the system has just a fraction of seconds to make the decision, the heuristics are not quite optimal and sometimes they will be inaccurate.

Apache processes use a ton of RAM. This becomes major threat when you realize that after each process has done its job, the bloated process sits and spoon-feeds data to the client, instead of moving on to bigger and better things.

The trick to get around this consists in fixing the Apache configuration file; The default file is the following:

# cd /etc/httpd/conf/
# vi httpd.conf

MaxClients
Max clients defines the maximum number of connections a child process can handle simultaneously. When a child process reach this limit, new connections start to queue; Of course this connections have to be saved somewhere on the system… ungracefully they’re stored on the RAM and then the swap memory.
If you use Apache to serve dynamic content, your simultaneous connections are severely limited. Exceed a certain number, and your system begins cannibalistic swapping.
By default this is set to 150. You could try increasing this value, just keep in mind that in a normal and standard installation of Apache you are limited to 256 MaxClients (I’m covering this topic on another post). If you set it to low, a message like the following will be printed in your error.log file:

[error] server reached MaxClients setting, consider raising the MaxClients setting.

MinSpareServers
When the Apache daemon (the father process) is started, a fixed number of child processes is spawned creating a pool of servers ready to serve the clients. By default this is set to 5; Creating new child processes is expensive for the system resources and the best practice is to always have servers ready for all the clients that may connect to the server. If you have a lot of traffic you should think in increasing this value.

MaxSpareServers
You can also define the max child processes your Apache server can spawn. As with the MaxClients option, when this value is reached new connections start to queue. A value between 20-50 should be fine in most situations.

MaxRequestsPerChild
Forcing your processes to die after awhile makes them start over with lower RAM usage. This can reduce total memory usage in many situations. This variable is very important on production systems. By default each Apache instance can serve an unlimited amount of clients/traffic. Child processes are never killed by the father process so, as time goes by, you can encounter with a process that has been running for, lets say, 28 days!.
If child processes never “recycle” themselves there is a high chance that the older ones may have used a lot of RAM they are not going to release until they die. Someday you will have to re-spawn this processes in order to keep the memory “clean”.
MaxRequestsPerChild defines the maximum connections a child process can serve. When the value is reached the child processes stops receiving new connections, awaits for the last current connection to be served and then it dies. Apache father process then re-spawns a new child and the cycle continues.
A value of 0 means unlimited (not recommended); In most cases values like 5000 would work fine but also values as low as 10 or 20 also may work for you. This is a game of catch-up, with your dynamic files constantly increasing total RAM usage, and restarting processes constantly reducing it. You have to experiment with this value.

There are more variables you can change and play with them, but those ones does have a little (if not null) relation with the OOM problem.

An OOM error is just a safety measure, NOT a system feature. The very existence of an OOM killer is a bug.

The goal of the sysadmin shall not be fix the OOM, but to get to the root of the problem and fix it. Poorly optimized queries and/or running the application on a piece of crap hardware, are common causes that may lead the server to an OOM.

SSHFS – Mounting Network Filesystems in Linux

Lets get a hold on the Secure Shell Filesystem Utilities (or just SSHFS to keep it more simple) using the awesome fuse libraries.

The things we need in order to use this method are the following:

• sshfs
• fuse-utils
• fuse-source
• kernel-headers
• kernel-devel

For the lazy people, all the these packages are available through the yum repository in Fedora (and apt-get in Debian).

$ yum install fuse-utils
$ yum install fuse-source
$ yum install kernel-header
$ yum install kernel-devel

Since yum takes care of all the versions compatibility and dependencies you wont have to configure anything.

Of course you can do the manual way by downloading and installing the packages by hand. For the kernel-devel and kernel-header you have to chose the one that matches your specific kernel version. Keep in mind that in most cases the kernel-devel package already includes the headers for your linux distribution:

Download fuse here
And fuse-ssh here
And the kernel packages here

Now that you have the needed files let’s start installing the kernel-devel.

As root do the following:

$ rpm -Uvf kernel-devel.rpm

Then install the fuse utils:

$ tar -zxvf fuse-2.7.4.tar.gz
$ cd /fuse-2.7.4
$ ./configure --with-kernel=/usr/source/kernel/[YOUR KERNEL]
$ make
$ make install

Now lets install sshfs:

$ tar -zxvf sshfs-fuse-2.1.tar.gz
$ cd sshfs-fuse-2.1
$ ./configure
$ make
$ make install

That’s it! You can check it by typing sshfs --help on a terminal window.

Using sshfs is actually like using a standard ssh connection command.

For example, suppose you want to mount the directory /home/ which is located on the remote computer 192.168.10.20, on the local directory /mnt/backup/:

$ sshfs root@192.168.10.20:/home /mnt/backup

To unmount it just issue the following command:

$ fusermount -u /mnt/backup