• I’ve been using it for a long time; familiarity
• very simple/powerful plugin system
• tons of users, so lots of examples and plugins already available

Nagios version 3 is provided in the Ubuntu repositories, and is quite simple to install:

root@admin:~# apt-get install nagios3

The default config comes set up to monitor a set of services on localhost; I don’t really like the default Ubuntu/Debian setup of having one config file per host/service/etc, so on the master I’ve replaced the config file structure:

root@admin:/etc/nagios3/conf.d# cd /etc/nagios3/conf.d/
root@admin:/etc/nagios3/conf.d# ls
contacts.cfg  extinfo.cfg  groups.cfg  hosts.cfg  services.cfg  timeperiods.cfg

groups.conf contains the set of server types that I care about:

# A list of your web servers
define hostgroup {
hostgroup_name  http-servers
alias           HTTP servers
members         localhost
}

# A list of your mysql servers
define hostgroup {
hostgroup_name  mysql-servers
alias           MySQL servers
}

# A list of your VHosts
define hostgroup {
hostgroup_name  http-vhosts
alias           Virtual Host HTTP servers
}

Note that the “http-servers” can define “members” (localhost in this case), however in general I do not add members in this file but instead in the hosts.cfg:

define host {
host_name   anyhosting1
address     1.2.3.4
use         generic-host
hostgroups  http-servers
}

define host {
host_name   example.com
address     1.2.3.4
use         generic-host
hostgroups  http-vhosts
}

Note the “hostgroups” line; anyhosting1 is the physical server (this monitor is really checking on the reverse proxy), and example.com is a vhost (which is really proxying to a user running Apache for the “example.com” domain). These two checks make sure that the whole system is working and proxying correctly.

Finally, services.cfg brings it all together by defining which groups should run which services:

# check that web services are running
define service {
hostgroup_name                  http-servers
service_description             HTTP
check_command                   check_http
use                             generic-service
notification_interval           0 ; set > 0 if you want to be renotified
}
define service {
hostgroup_name                  http-vhosts
service_description             Virtual Host HTTP
check_command                   check_httpname
use                             generic-service
notification_interval           0 ; set > 0 if you want to be renotified
}

The Ubuntu nagios-plugins package (which by default is installed along with the nagios3 package) contains plugins that can intelligently check MySQL databases, disk space, load average, etc. By default these only work on the local machine, but these can be made to run on remote machines by installing the nagios-nrpe-server package. I will cover this further in a future blog post.

Computer crime is rampant and quite profitable. Websites have long been defaced for fun or viruses and worms released to cause disruptions.  Now there’s an incentive for criminals to avoid detection, to add your machines to a botnet for hire, or hide spam and ad links in your Wordpress blog.

This is why it’s super critical to us as at AnyHosting to focus on security, and follow best practices like secure backups, regular audits, intrusion detection, automatic updates, etc.

I actually helped to set up real-time web analysis with syslog-ng (using TCP) and a slightly hacked webalizer (it was ignoring multiple hits happening on the same second) from a FreeBSD/Apache web farm ~10 years ago, and have been looking into it again for my current logging needs.

His blog has consistently awesome posts (if you’re interested in systems administration), and as your doctor I highly suggest that you subscribe.

Our original goal was to provide websites for multiple users on one host, keeping them compartmentalized from each other, and to restrict break-ins. We chose to do reverse proxying using Apache, with separate Apache instances for each user, using mod_chroot.

Pros:

• user cannot see or read/write other users files, even on the same shared server, or use any system resources not explicitly provided in the chroot.
• mod_chroot is way less work than building a full chroot jail, and is potentially safer since you don’t have to provide system libraries or binaries, reducing your attack surface

Cons:

• multiple Apache instances consume more memory, and need special configuration and startup scripts
• it is possible to break out of a chroot. It is especially easy if the attacker can somehow get elevated to root (e.g. local security exploit)

Further investigation:

• apache-mpm-itk – This runs Apache child processes as a separate user per vhost. Cannot be combined with mod_chroot, unfortunately, and has some serious drawbacks (see the section on Quirks and Warnings).

• solaris containers, freebsd jail, linux vserver – These are real jails; less overhead than Virtual Machines, but provide more protection than a chroot. These tend to be significantly more complex to set up than something like mod_chroot, but are by design a lot harder to escape than a chroot.

There are some other interesting security-related modules such as mod_evasive that I am checking into further, which may mitigate some of the “cons” above.

Hopefully this will be useful to someone, however security is a process, not a product, and you should not rely solely on something like mod_chroot, but it can be a useful tool as part of an overall strategy that includes monitoring, intrusion detection, and regular security updates.

Ubuntu 8.10 Server

root@theo:~# apt-get install apache2 libapache2-mod-chroot

For each domain, create a virtual host config like /etc/apache2/sites-available/example.com:

<VirtualHost *:80>
ServerAdmin webmaster@example.com
ServerName example.com
ServerAlias www.example.com

ProxyPass / http://localhost:8080/
ProxyPassReverse / http://localhost:8080/

<Proxy *>
  Allow from all
</Proxy>

ErrorLog /var/www/example.com/logs/error.log

# Possible values include: debug, info, notice, warn, error, crit,
# alert, emerg.
LogLevel warn

CustomLog /var/www/example.com/logs/access.log combined

</VirtualHost>

This will be used by the Apache proxy, which is the normal system apache2 running as the default “www-data” user. This proxy handles name-based virtual hosts, and proxies the requests to a second process, running at localhost on port 8080.

Note that the logs are configured here and not in the user’s Apache process, for two reasons:

1. keep logs pristine in the event of a break-in on a user site (for example via a buggy or malicious PHP script)
2. single system-wide log file analysis process instead of per-user

Next, create a user account for each domain:

root@theo:~# mkdir /var/www/example.com
root@theo:~# cd /var/www/example.com/
root@theo:/var/www/example.com# useradd -d /var/www/example.com/htdocs example.com
root@theo:/var/www/example.com# mkdir htdocs logs conf
root@theo:/var/www/example.com# chown www-data:www-data logs
root@theo:/var/www/example.com# chown example.com:example.com htdocs/

Create the following in /var/www/example.com/conf/apache2.conf:

ServerRoot "/var/www/example.com/"
RequestHeader set Host example.com
LockFile /var/www/example.com/conf/accept.lock
PidFile /conf/apache2.pid
Timeout 300
KeepAlive On
MaxKeepAliveRequests 100
KeepAliveTimeout 15
User example.com
Group example.com
AccessFileName .htaccess
<Files ~ "^\.ht">
  Order allow,deny
  Deny from all
</Files>

DefaultType text/plain
HostnameLookups Off
ErrorLog /var/www/example.com/conf/error.log
LogLevel warn
LoadFile /lib/libnss_dns.so.2
LoadModule chroot_module /usr/lib/apache2/modules/mod_chroot.so
Include /etc/apache2/mods-enabled/*.load
Include /etc/apache2/mods-enabled/*.conf
Listen 8080
DocumentRoot /htdocs
ChrootDir /var/www/example.com/

Start up the user Apache process:

root@theo:~# apache2 -f /var/www/example.com/conf/apache2.conf -k start

Everything should now work, but you may notice some problems with PHP or CGI scripts. Generally this means that you will need to make parts of the system available in the chroot; however making copies takes up a lot of space and can quickly get out of date, which is a security risk.

One workaround is to provide needed directories using the “mount -o bind” option, which will remount an existing, mounted part of the file hierarchy somewhere else. For example, this will provide all of “/usr” inside the chroot:

root@theo:~# mkdir /usr /var/www/example.com/usr
root@theo:~# mount -o bind,ro /usr /var/www/example.com/usr

PHP sessions require /tmp to exist inside the chroot; there is no benefit to sharing this one, an empty directory is fine.

That’s pretty much it! Testing, getting startup right etc. is left as an exercise for the reader, since we’re venturing a bit out of the standard Ubuntu Apache setup. Entries in /etc/fstab and creating an init script to handle user Apache processes is probably the best way to go.

In part 3 we’ll cover take a more general look at the pros and cons to this setup, as well as possible future directions.

EDIT 2009-Oct-03 2:25 PM Pacific – create dir before cding into it; add read-only (ro) option to usr mount; override “host” header in user Apache process

EDIT 2009-Oct-05 4:41 PM Pacific – load resolver library and install caching proxy dns server, needed by popular wordpress anti-spam plugin Akismet

At AnyHosting we want to have multiple users on one shared server, but protect them from reading or writing each other’s files (whether accidental or intentional) and also limit damage caused by a break-in via a poorly configured PHP script or CGI, or even an Apache exploit.

The solution here is to have each customer run their own Apache process, and do name-based virtual hosting and forwarding via a reverse proxy. The proxy is currently Apache but we are also considering more scalable alternatives such as nginx. An important thing to note here is that the proxy could be on a separate machine and also combine load-balancing into the mix, so it provides a lot of flexibility.

The classic way to chroot Apache (or any server process) is to copy the server process and all of it’s dependencies into the chroot jail, which makes keeping the files up to date not just a depressing chore but also a serious security hazard. Also it’s a ton of work to get right, especially if you want to support server-side scripting like PHP, since it depends on lots of system files and libraries.

Enter mod_chroot. This Apache module runs in the user’s process, and does the chroot system call after opening all the files it needs. The only caveats I’ve found (besides the ones on the mod_chroot caveats page) are some files that the Ubuntu PHP install assumes it can reach (such as the MySQL server socket, timezone info, and random number generator). I will cover this in more detail in part 2.

If you’re looking for simple, secure and affordable web hosting, check it out!

I’m also working on a series of blog posts and articles about the new setup (reverse proxy, mod_chroot, on ubuntu server).

They seem to fall into three broad categories:

1. expensive proprietary packages
2. free open-source packages
3. custom software developed by hosting companies

The leader in #1 seems to be cPanel. I am not at all impressed with their demo, it looks very cluttered to me and I really want something simple, easy and aesthetically pleasing.

I have checked out all of the free/open-source web hosting control panels that Wikipedia lists, and I am also very displeased with the UI, and the code doesn’t look very easy to jump into to me. There seems to be a lot of custom code (I’d be happiest with something based on Django, but the PHP ones could at least use Cake). This makes me a little worried on the security aspect (XSS, SQL injection, etc.). Most of these code bases seem to be very old and not necessarily very active.

So is #3 the way to go? I’ve seen and heard about lots of great hosting control panels that only exist behind close doors, is this the big differentiator for modern hosting companies?