Since my last post on using Nginx to cache proxied content, they have added proper cache handling via their proxy_cache* directives. These are much more suitable for use, as they capture the HTTP response headers and also use more advanced Cache-Control checks.

To start, install the latest stable Nginx avaliable at http://wiki.nginx.org/NginxInstall.

Next edit your nginx.conf and add the proxy_cache_path directive to define a named cache storage. These are independant of servers and locations, and can be reused inside each later on.

...
http {
    ...
 
    proxy_cache_path /var/cache/nginx keys_zone=anonymous:10m;
 
    include vhosts/*.conf
}

Next create the directory for the cache:

mkdir -p /var/cache/nginx

Next define your server configuration, which can be done for example in conf/vhosts/example.com.conf if you defined the include above.

server {
    listen            80;
    servername        example.com;
 
    proxy_set_header  X-Real-IP  $remote_addr;
    proxy_set_header  Host       $host;
 
    location / {
        proxy_pass    http://localhost:8080/;
        proxy_cache   anonymous;
    }
 
    # don't cache admin folder, send all requests through the proxy
    location /admin {
        proxy_pass    http://localhost:8080/;
    }
 
    # handle static files directly. Set their expiry time to max, so they'll
    # always use the browser cache after first request
    location ~* (css|js|png|jpe?g|gif|ico)$ {
        root          /var/www/${host}/http;
        expires       max;
    }
}

As we don’t want the nginx worker processes to have root permissions when in use, add to the start of conf/nginx.conf:

user nginx
 
...

Then sort out the user and permissions:

useradd nginx
chown nginx:nginx /var/cache/nginx /usr/local/nginx/{fastcgi_temp,logs,proxy_temp}

To start nginx on bootup, add the following to the end of /etc/rc.local:

/usr/local/nginx/sbin/nginx

Then also run this command to start nginx now.

That is all that is needed, no patches this time. There are several more proxy_cache* directives avaliable that you can use to tweak its behaviour, see the proxy module documentation for more details.

mpm-itk is a fork of mpm-prefork (ironically in both process and project sense), which allows you to configure individual Apache vhosts to run as specified users and groups. This makes it extremely secure if used in a shared hosting environment.

I have provided a CentOS RPM for this in the Webtatic yum repository. This should work with your existing httpd installation, as it is installed as a separate mpm to be selected just as the worker or event mpms can.

To install, first add the Webtatic repository to your Yum configuration:

rpm -ivh http://repo.webtatic.com/yum/centos/5/`uname -i`/webtatic-release-5-0.noarch.rpm

Then install the package:

yum install --enablerepo=webtatic httpd-itk

Next, if you are currently running httpd, stop it, as the switch is done in the httpd control scripts.

service httpd stop

Then edit /etc/sysconfig/httpd and add the following line:

HTTPD=/usr/sbin/httpd.itk

Next you can add users and groups for your vhosts and configure httpd’s vhosts to use them:

...
<VirtualHost *:80>
    ServerName example.com
    DocumentRoot /path/to/web/root
 
    AssignUserId vhost-user vhost-group
</VirtualHost>
...

And then start up your Apache httpd:

service httpd start

The fun doesn’t stop there, however. You must configure websites to only be accessible to these linux users, and not to others. This can typically be done by setting the website’s root to be owned by the user, the group set to the vhost’s group, and turning off world read/write/execute, e.g.:

chown owner-user:vhost-group /path/to/webroot
chmod o-rwx /path/to/webroot
 
ls -ald /path/to/webroot
# drwxr-x--- 20 owner-user vhost-group 4096 Apr  5 19:38 /path/to/webroot

Now the only users able to access the webroot are the super user, owner-user, and any users in vhost-group. As the web server will be setuid’d to vhost-user/vhost-group it will be able to read files in the web root.

The reason I didn’t change the user of the directory to vhost-user is that it would give the httpd process write access, which you still should take care of, as if there are vulnerabilities in the user’s code, the files could be hacked.

VirtualHost security

mpm-prefork (and most of the other stable MPMs) runs under a single user and group. In a shared hosting environment using these processing models, every script that is run essentially has read access to every other vhost’s scripts, which is a frightening prospect as they can easily be compromised giving an attacker the full source to a website. Database passwords can be read, leading to additional private data being accessible.

There are methods to secure shared hosting, as detailed in my blog post “Techniques for creating a secure shared web server”. However they tend to decrease performance or don’t have total protection.

mpm-itk

mpm-itk is an alternative which runs the prefork model as root, giving it setuid capabilities. Each time a request is established, a preforked httpd process forks itself and setuid’s to the vhost’s defined user/group (or the default if unspecified). When the request is complete, the fork is terminated, allowing the parent to fork again on a later request.

The reason for this is that only the super user can setuid, and once done a process cannot go back to the super user’s context. Forking a process is extremely cheap on resources, so mpm-itk should not lose much in performance when compared to mpm-prefork.

Yes, there is a point where if there is a vulnerability an attacker might gain root permissions, specifically in between the established connection and the setuid, however as the project’s author mentions, this would only really happen if mod_ssl had a vulnerability.

A few days ago, Facebook released XHP, a PHP extension, which allows defining XML directly in PHP blocks, allowing you to “use PHP as a stricter templating engine”.

It seems a bit strange to be coding XML tags directly in PHP blocks, but it adds features such as automatic escaping, and the ability to manipuate the tags, and how they render.

I’ve compiled experimental RPMs, and put them in the Webtatic yum repository. These are compiled against PHP 5.3, which I also have in my repository.

I cannot guarantee the build of the exension is stable, as I had to add a few patches of my own to the source to get it to compile, so I don’t recommend you install it on a production site.

To install, first upgrade your PHP to 5.3, as detailed in my PHP 5.3 on CentOS post.

Then install the extension, and then you’re done:

yum install --enablerepo=webtatic php-xhp

To test if it is set up correctly, run the following:

php -r 'echo "XHP!\n"; exit; <a />;'

Currently XHP requires you to include a PHP file (xhp/init.php), which is installed in the include path, to define all the default elements e.g.:

require 'xhp/init.php';
 
$href = 'http://www.webtatic.com';
echo <a href={$href}>Webtatic.com</a>;

There are times when a Subversion working copy can mess up. This is usually due to human error, for example due to permissions problems or moving files or folders incorrectly

These can usually be easily recoverable, although at times it can seem there’s no solution. Here are a few examples and their solutions.

Incorrectly deleting files/folders without using svn rm

This will usually result in subversion complaining when you attempt to commit. The symptom of this is the svn status giving something like:

!       path/to/missing/file

To fix this, simply issue the svn rm command for the file, and Subversion will fix and mark it as deleted.

Incorrectly moving files/folders without using svn mv

Sometimes someone may forget to run the subversion move command, and move files/folders manually. If you attempt to add a file after this, you will lose the history of the file. If you attempt with a folder, it will give an error:

svn: warning: 'path/to/moved/folder' is already under version control

The symptom of this is an svn status of:

?       path/to/moved/folder
!       path/to/old/folder

The best solution for this is to simply move it back and run the svn mv command instead.

Adding a folder with permission problems

Sometimes either the group or permissions of a folder will be incorrect. If you add the folder, it will usually issue an error like:

svn: Can't create directory 'new/folder/.svn': Permission denied

Whats worse is that the working copy now registers the folder in a half-added state. The parent folder metadata says the folder is added, but the child folder is missing its .svn metadata. The symptom of this is an svn status of:

~       new/folder

The solution is to first fix the permissions of the folder, then move the folder to a temporary location, and revert the add:

# first fix the permissions
# ...
 
mv new/folder new/folder2
svn revert new/folder
 
mv new/folder2 new/folder
svn add new/folder

An .svn metadata transplant

There are sometimes some cases that can’t be explained. The best solution I’ve found for this is to do a .svn folder lobotomy and restore with fresh .svn folders. To do this:

# backup your project in case you run into trouble
cp -Rp /path/to/project /temporary/location
 
# strip out the old .svn folders
find /path/to/project -name .svn | xargs rm -rf
 
# check out a clean copy
svn co http://repo/location /temporary/location2
 
# move the .svn folders from the clean copy into the correct relative
# place in the broken copy
cd /temporary/location2
find . -name .svn -print0 | xargs -0 -I {} mv '{}' '/path/to/project/{}'
 
# remove the clean copy
rm -rf /temporary/location2

Centralised versioning systems are inherently authoritative, but when dealing with decentralised systems, either patches are made and applied to the maintainer’s repository, or one repository should be defined as the authoritative one.

If the authoritative repository requires commit access, it should be locked down as much as possible, requiring authentication, encryption, and push access without opening up raw file write access. If raw file write access is given, either intentionally or unintentionally, any user with access could corrupt or delete the repository.

Here are the protocols and methods used by Subversion(svn), Git and Mercurial(hg) along with their capabilities:

1. disabled by default
2. the login shell can be changed to a shell wrapper to allow only api commands e.g. git-shell
3. with mod_dav file provider
4. with a mod_auth_* module, and mod_ssl
5. achieved by file ownership and permissions

Git clone of a Subversion repository

An alternative, if you require git access, is to have a locked down Subversion repository cloned to a read-only git repository, with a svn post-commit hook running git svn fetch on the git repository.

Clone the Subversion repository via its publicly accessible url:

git svn clone -s https://example.com/svnrepo/ .

Create /path/to/svnrepo/hooks/post-commit with content:

#!/bin/sh
git --git-dir=/path/to/gitrepo/.git svn fetch

The suggested way (in “man git-svn”) of then cloning the git repository and initialising the svn metadata is a bit complicated, but you could instead rsync the git repository and use that instead, otherwise:

git init
git remote add origin url/to/gitrepo
git config --add remote.origin.fetch '+refs/remotes/*:refs/remotes/*'
git fetch
git checkout -b master FETCH_HEAD
git svn init -s https://example.com/svnrepo/
git svn rebase

Conclusion

The capabilities needed by a locked down repository (authentication, encryption, no raw file write access), mean that several methods aren’t viable, and only two with push capabilities (mod_dav_svn, and hgweb). The ssh methods can, as mentioned, be limited to api access only using wrappers, but require local users and those users can’t be used for regular activities, so if your users have ssh access as well, they’ll need two accounts

Public repositories can forgoe these capabilities, so any of these are suitable if anonymous read-only access is required.

I’ve been hearing good things about git lately, with many projects which used to use subversion converting to it, so I’ve decided to try it out myself. RPMForge only has version 1.5, so like the other software I’ve been using, I’ve converted Fedora 11′s rpm to CentOS and put it in the Webtatic repository.

Update 2010-02-14 – Updated Git 1.6.6 to 1.7.0

Update 2010-04-05 – Updated to 1.7.0.4, included all dependencies in the repository to remove any dependencies on other repositories

If you haven’t set up the Webtatic repository in yum, then add it in the command-line as following:

rpm -ivh http://repo.webtatic.com/yum/centos/5/`uname -i`/webtatic-release-5-1.noarch.rpm

Now install git:

yum install --enablerepo=webtatic git-all

The performance of a website is an important issue. Even fast responding dynamic pages can be hit with problems with sub-optimal static content such as high overhead on many HTTP requests and large javascript/css files. Tools like YSlow, and Google Page Speed help identify these problem areas.

Webtatic Optimizer is a tool that can be used to improve these areas, and can help get an almost perfect score.

Make fewer HTTP requests

JavaScript and Css files are naturally loaded individually. Having multiples of them being used across an entire site, e.g. including the JQuery library along with your own javascript, adds to the time the user has to wait for the files to download.

The Optimizer can concatenate these files and store in a new static file, so loading this instead will mean less requests, resulting in an “A” in this category. (images may decrease the score if you use many of them, but I will add in a future release the ability to easily create CSS sprites which rewrite the css file to accomidate the position of an image within its sprite).

Add Expires headers

Whilst you can force an expiry of X hours in apache, this means that changes to these files wont be reflected in the browser until the image has expired. However, browsers cache files based on their entire request url (including query string), so simply concatenating onto the referenced url a unique query string that changes when the contents are modified, is enough to be able to set a year’s expiry whilst changes are reflected immediately.

The Optimizer does this by rewriting css url() urls, calculating the last-modified date of the physical file, and appending this onto the url. There is also a php function helper to do this for static files included in dynamic pages. With this you can get an “A” in this section.

Compress components with gzip

Text-based http requests can be reduced in size significantly using gzip when the browser supports it as a content encoding. The Optimizer does this when creating the concatenated files, also creating a gzipped version at the same time.

This will give an “A” in YSlow.

Minify JavaScript and CSS

If a browser is not able to use the gzipped version of a file, optimized versions of the original JavaScript or CSS can be used instead. This is done at the same time as concatenating the files, using JsMin for JavaScript, and a simple CSS minifier, giving you an “A” for this category in YSlow.

Example

Using a configuration such as below, enables the above optimizations to be incorporated into the generated files:

[config]
version = 1.0.0
 
[javascript]
config.workingPath = http
defaults.output = site.js
files = "
  + javascript/jquery.min.js
  + javascript/my-javascript.js
"
 
[css]
config.workingPath = http
defaults.output = site.css
files = "
  + css/layout.css
  + css/global/**.css
"

The above configuration can passed to the optimizer command-line script:

optimizer-compile optimizer.ini

This will concatenate http/javascript/jquery.min.js and http/javascript/my-javascript.js, minimise it, and store it in http/site.js and a gzipped version in http/site.js.gz. It will also concatentate, minify, and rewrite url()’s for http/css/layout.css and any CSS files in http/css/global and sub-directories, and store it in http/site.css and a gzipped version in http/site.css.gz.

Advanced features

There are much more advanced features that can be configured, which you can find out more about in the documentation. The Webtatic Optimizer is a fully extensible library, using plugins and filters to achieve the above. You can enable/disable any features or can even write your own.

An example in use

Using each of the Webtatic Optimizer’s features above, I have managed to get the Fubra Passport, an authentication gateway and payment system for the sites Fubra Limited runs/is affiliated with, a score of “B”. It fails at getting an “A” as it doesn’t have a CDN, and cookies are passed for static content.

I will address the latter, along with making it easy to automatically rewrite local static file urls to a CDN, in a future release.

Recently I have been handling the security of some sensitive data. I had originally been encrypting/decrypting data with a symmetric-key system using mcrypt for PHP. This was due to the web frontend and the backend existing on the same server. However for security purposes I am now separating the frontend and backend onto different servers, so that there is no way the web accessible frontend, whether compromised or not, can get at the data it inserts into the database.

In order to do this, a asymmetric-key system is needed, such as public-key cryptography. Googling for examples of this in PHP, there doesn’t seem to be any results of this other than the php OpenSSL extension documentation, and systems that try to reinvent the wheel with their own implementations.

Using the PHP OpenSSL extension it is fairly easy to sort out a secure system for encrypting data with one key that only can be decrypted with another.

First, you need to generate your private and public keys. You can either do this yourself with the openssl command-line application:

# generate a 1024 bit rsa private key, ask for a passphrase to encrypt it and save to file
openssl genrsa -des3 -out /path/to/privatekey 1024
 
# generate the public key for the private key and save to file
openssl rsa -in /path/to/privatekey -pubout -out /path/to/publickey

or programatically using php-openssl:

// generate a 1024 bit rsa private key, returns a php resource, save to file
$privateKey = openssl_pkey_new(array(
	'private_key_bits' => 1024,
	'private_key_type' => OPENSSL_KEYTYPE_RSA,
));
openssl_pkey_export_to_file($privateKey, '/path/to/privatekey', $passphrase);
 
// get the public key $keyDetails['key'] from the private key;
$keyDetails = openssl_pkey_get_details($privateKey);
file_put_contents('/path/to/publickey', $keyDetails['key']);

Next, you can load the public key, and encrypt the data:

$pubKey = openssl_pkey_get_public('file:///path/to/publickey');
openssl_public_encrypt($sensitiveData, $encryptedData, $pubKey);
 
// store $encryptedData ...

When you need to get the sensitive data again, you can load the private key and decrypt:

// retrieve $encryptedData from storage ...
 
// load the private key and decrypt the encrypted data
$privateKey = openssl_pkey_get_private('file:///path/to/privatekey', $passphrase);
openssl_private_decrypt($encryptedData, $sensitiveData, $privateKey);

Alternatively you can use the private key to encrypt data, sign data or seal it against multiple other public keys, but that is beyond the scope of this article.

My previous articles on installing PHP on CentOS dealt with installing PHP 5.2.6. I have found this to have some bugs that kill the process without error information. One bug I found, which was on an x86_64 server, was that converting an object to a string did this.

So, I have compiled the latest PHP version, 5.2.10 5.3.0 5.3.3, and put it in my own repository for easy installation. I have compiled it for CentOS 5 i386 and x86_64, and provided the source RPMS in the repo, if anyone wants to compile it for another OS or architecture.

Update 2009-07-03 – I updated the version to PHP 5.3, which was released a few days before. This includes many new features such as closures, namespaces, and packaged scripts in phar files, which I’ll blog about soon. Check out PHP changelog for more details.
Update 2009-09-01 – Added a note about deprecated errors, and how to silence them. Also I have included a tip that might help those of you struggling to install.
Update 2010-03-03 – I’ve added both apc 3.1.3p1 beta (php-pecl-apc in yum) and eAccelerator 0.9.6 (php-eaccelerator in yum) RPMs to the repository, they are compiled for (and work on) php 5.3.x

I have also included the same php extensions I mentioned in my other article, php-mcrypt, php-mhash (in PHP 5.2), php-mssql and php-tidy

To install, first you must tell rpm to accept rpm’s signed by me, then add the yum repository information to yum:

rpm -ivh http://repo.webtatic.com/yum/centos/5/`uname -i`/webtatic-release-5-1.noarch.rpm

Now you can install php by doing:

yum --enablerepo=webtatic install php

Or update an existing installation of php, which will also update all of the other php modules installed:

yum --enablerepo=webtatic update php

If this does not work correctly, try disabling all other repositories while installing/updating, by adding the –disablerepo=* option. This will stop other dependencies from being installed, so you may want to install them first.

yum --disablerepo=* --enablerepo=webtatic update php

“Depsolving” problems

If you get depsolving problems when updating, you may have currently installed some extensions that have been removed, e.g. php-mhash, php-ncurses.

You will need to remove them before upgrading.

yum remove php-mhash php-ncurses

Deprecated Errors

Once you are running the new version, you may get “deprecated” errors in your error logs. This isn’t bad, it just means to tell you that some of the functions you are using are no longer prefered, and may be removed in a future major release. An example of this is the ereg functions. Preg functions are prefered over these, as they are much faster and more powerful, and in all cases do at least the same thing.

If upgrading the functions are not an option, and you would like to hide the deprecated errors from your error log, for example on a production server, just edit your /etc/php.ini file, find the line:

error_reporting  =  E_ALL

and replace to:

error_reporting  =  E_ALL & ~E_DEPRECATED

PHP 5.2.14

I am also maintaining a PHP 5.2.14 release, so should you prefer to install that (for reasons like incompatibilities or testing), you can force it to install that instead by doing:

yum --enablerepo=webtatic --exclude=php*5.3* update php

Or you can add to the /etc/yum.repos.d/webtatic.repo the line:

exclude=php*5.3*

Recently, I wrote a mod_perl module for using a database backend for basic and digest authentication in Apache, however I found it to be much slower than mod_auth_mysql. This would be due to using mod_perl and DBI. So I have written a patch for mod_auth_mysql which performs the same, which means its as fast.

The main reason why I chose to do this rather than use Webtatic::AuthDBI is because subversion checkouts were taking twice as long. A mod_perl authentication provider, even without performing authentication (just returning OK for any login details) seems to be the speed of the whole mod_auth_mysql without even establishing a mysql connection.

This patch seems to perform just as well as mod_auth_mysql.

Check it out here: /projects/mod_auth_mysql-auth/