Without transparant proxy (TPROXY), all request would appear to come from the load balancer's IP address, instead of the actual client (which can mess up your logging, scripts, ...).

This article is derived from the "How To Compile a Kernel - The CentOS way" pages at howtoforge.com, and the guides at LoadBalancer.org.

We will assume you have a running CentOS at this point (version 5.5 or later). First up, download the kernel source. Since the TPROXY patch isn't compatible with all Kernel versions, we're stuck with the 2.6.25 kernel. Please browse the Kernel Source Index, and pick the latest linux-2.6.25.* kernel work from. (if the version is no longer available for download, here's a local copy).

cd /usr/src
wget http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.25.20.tar.gz

Now, unpack the source code, and symlink the resulting directory to 'linux' (will make things easier later on).

tar xzf linux-2.6.25.20.tar.gz
ln -s linux-2.6.25.20 linux

Next step is to download the TPROXY patch. (if the version is no longer available for download, here's a local copy)

wget http://www.balabit.com/downloads/files/tproxy/tproxy-kernel-2.6.25-20080519-165031-1211208631.tar.bz2
tar xjf tproxy-kernel-2.6.25-20080519-165031-1211208631.tar.bz2

Now to apply the patch mentioned.

cd linux/
cat ../tproxy-2.6.25-20080519/00* | patch -p1 --dry-run
cat ../tproxy-2.6.25-20080519/00* | patch -p1

To start compiling the kernel, we'll first clean any leftovers that may exist.

make clean && make mrproper

And copy our current Kernel configuration to our local .config file. This will help in making sure our currently installed applications will continue working, and we don't modify the kernel too heavily.

cp /boot/config-`uname -r` ./.config

Make sure you have all the necessary developer-tools, to compile the kernel.

yum install make rpm-build gcc gcc-c++ ncurses-devel elfutils elfutils-libs libstdc++-devel

Let's pop up the kernel configuration menu, which will allow us to easily change configs.

make menuconfig

Since we saved our currently running Kernel's configuration in the .config file, we will choose the "Load an alternative Configuration File" option, and enter .config as the filename.

menuconfig: load config

Let's enable TPROXY support. Navigate in the menu to

• Networking
• > Networking support
• > Networking options
• > Network packet filtering framework (Netfilter)
  
• > Core Netfilter Configuration

and highlight:

Transparent proxying support (EXPERIMENTAL)
Netfilter Xtables support (required for ip_tables)
Netfilter Connection tracking
> Connection tracking flow accounting
> Connection mark tracking support
"TPROXY" target support (EXPERIMENTAL)
"socket" match support (EXPERIMENTAL)

Beware that you can check these options in 2 ways:

[*]: Built-in
[M]: Module

Try to select the options listed above as [M], so they are modules.

Then hit <ESC><ESC> a few times, to get back to the main menu, and navigate to General Setup > () Local version - append to kernel release. Add a custom suffix there, to identify this kernel. I've choosen "-tproxy" as suffix (so I know it has tproxy support). It's important you add a version number if you try to reinstall the kernel again afterwards, or you'll end up with "this package is already installed" messages.

Once you've applied the above changes, exit the menu and confirm you want to save your changes.

Now start compiling the source and create the RPM install file.

make rpm

This _will_ take a long time. If you're running this inside a virtual machine, consider adding more (virtual) CPU's to speed up this process. It's safe to assume this will run for at least 1 hour, probably more.

After it's been created, you will find the resulting Source RPM file in /usr/src/redhat/SRPMS/.

lb02.lab.mojah.be linux $ ls -al /usr/src/redhat/SRPMS/
total 62124
drwxr-xr-x 2 root root     4096 Aug 21 17:31 .
drwxr-xr-x 7 root root     4096 Aug 21 16:20 ..
-rw-r--r-- 1 root root 63536285 Aug 21 17:27 kernel-2.6.25.20tproxy-1.src.rpm

And the binary RPM file in /usr/src/redhat/RPMS/i386/ (or x86_64 if you're running 64 bit).

lb02.lab.mojah.be linux $ ls -al /usr/src/redhat/RPMS/i386/
total 120256
drwxr-xr-x 2 root root      4096 Aug 21 17:32 .
drwxr-xr-x 9 root root      4096 Aug 21 16:20 ..
-rw-r--r-- 1 root root 123002989 Aug 21 17:31 kernel-2.6.25.20tproxy-1.i386.rpm

Now it's time to install our custom kernel.

rpm -ivh --nodeps /usr/src/redhat/RPMS/i386/kernel-2.6.25.20tproxy-1.i386.rpm

And create the ramdisk for our system.

mkinitrd /boot/initrd-2.6.25.20-tproxy.img 2.6.25.20-tproxy

Let's see what files were made in the /boot partition. We'll need these filenames to edit the grub config later on.

lb02.lab.mojah.be linux $ ls -alh /boot/ | grep -i tproxy
-rw-r--r--  1 root root   73K Aug 21 17:27 config-2.6.25.20-tproxy
-rw-------  1 root root  3.2M Aug 21 17:33 initrd-2.6.25.20-tproxy.img
-rw-r--r--  1 root root 1015K Aug 21 17:27 System.map-2.6.25.20-tproxy
-rw-r--r--  1 root root  1.9M Aug 21 17:27 vmlinuz-2.6.25.20-tproxy

And edit the menu-file.

vi /boot/grub/menu.lst

And add the following snippet below the "hiddenmenu" line, and right above the first kernel declaration. This consists of copying an already existing boot-item, and modify the vmlinuz and initrd locations.

title CentOS-Tproxy (2.6.25.20-tproxy)
root (hd0,0)
kernel /vmlinuz-2.6.25.20-tproxy ro root=/dev/VolGroup00/LogVol00
initrd /initrd-2.6.25.20-tproxy.img

The /vmlinuz and /initrd should point to the filenames you discovered earlier. Please don't directly copy/paste the example above, but copy an entry from your file, and modify it (as to preserve the hard disk order and volume names).

Now reboot into your newly created kernel. Your boot screen would look a bit like this now, with a notice to the newly named kernel.

tproxy kernel

You can verify this once the server's booted up.

lb02.lab.mojah.be ~ $ uname -a
Linux lb02.lab.mojah.be 2.6.25.20-tproxy #1 SMP Sat Aug 21 17:22:41 CEST 2010 i686 i686 i386 GNU/Linux

Now we have our kernel with TPROXY support running, time to compile and patch our iptables to make use of it. To get started, download 1.4.0 iptables source. It's import you take the 1.4.0 version, newer versions won't work. (if the version is no longer available for download, here's a local copy).

cd /usr/src/
wget http://www.netfilter.org/projects/iptables/files/iptables-1.4.0.tar.bz2
tar xjf iptables-1.4.0.tar.bz2

Now also download the tproxy iptables patch. (if the version is no longer available for download, here's a local copy)

wget http://www.balabit.com/downloads/files/tproxy/tproxy-iptables-1.4.0-20080521-113954-1211362794.patch

Apply the tproxy path.

cd iptables-1.4.0/
cat ../tproxy-iptables-1.4*.patch | patch -p1
make && make install

Now you've installed both your kernel, and iptables, with the tproxy patch.

Downloads

iptables 1.4.0.tar.bz: full iptables source code
linux-2.6.25.20.tar.gz: full kernel 2.6.25.20 source code
tproxy-iptables-1.4.0-20080521.patch: tproxy patch for iptables
tproxy-kernel-2.6.25-20080519.tar.bz2: tproxy patch for 2.6.25 kernel

.config: the .config file I used to compile my kernel (with all necessary modules checked)
kernel-2.6.25.20tproxy.i386.rpm: rpm install file for a 32-bit (i386) kernel with tproxy support. Follow the "mkinitrd" steps above to install & use this kernel. The kernel is named "2.6.25.20-tproxyfourteen".

Troubleshooting: unknown match socket

If you've done the above steps, and still get the UNKNOWN match `socket' message in your iptables, you've probably skipped a kernel module required for this to work.

Troubleshooting: bad exit status during kernel compile

You could run into something similar to the following when compiling your kernel.

... [snip]
LD [M]  drivers/scsi/scsi_mod.o
LD      drivers/built-in.o
error: Bad exit status from /var/tmp/rpm-tmp.48540 (%build)

RPM build errors:
Bad exit status from /var/tmp/rpm-tmp.48540 (%build)
make[1]: *** [rpm] Error 1
make: *** [rpm] Error 2

That will prevent you kernel from successfully compiling. It usually means you selected a kernel option with a certain dependency that wasn't checked. So it's dependent on an option, that's not being compiled.

Tricky part here is to track down which one, and I'm afraid to say I don't know how. Also, it's beyond the scope of this document. If you want to retry the compilation again, make sure to run:

cp .config .config_backup
make clean && make mrproper

to reset your current attempt, clear created files and reset the config back to zero. Your "broken" config can then still be found in the .config_backup file. In my experience, it's better to just start all over ...

You could also consider deleting the generated files in /usr/src/redhat/BUILD/kernel-* as they are obsolete now.

In this example, I'm increasing a 3GB disk to a 10GB disk (so you can follow using the examples).

I would advise you to read the excellent documention on Logical Volume Management on tldp.org.

Just a small note beforehand; if your server supports hot adding new disks, you can just as easily add a new Hard Disk to your Virtual Machine. Doing so, would mean you can increase your LVM's size without having to reboot. If you increase the size of your currently attached disk (like the example below), you'll need to reboot your server at least once to re-read your partition table.

1) The "hardware" part, "physically" adding diskspace to your VM

Increasing the disk size can be done via the vSphere Client, by editing the settings of the VM (right click > Settings).

Edit settings

And increasing the privisioned disk space.

Increase disk size

If the "Provisioned Size" area (top right corner) is greyed out, consider turning off the VM first (if it does not allow "hot adding" of disks/sizes), and check if you have any snapshots made of that VM. You can not increase the disk size, as long as there are available snapshots.

Alternatively, you can also choose "Add..." to add new Hardware to your VM, with the desired extra space.

2) Partitioning the unalloced space

Once you've changed the disk's size, either boot up your VM again, or restart if it was still running. Linux needs to boot with the new disk, so it can see you've added (unallocated) disk space.

Once you've booted again, you can check if the extra space can be seen on the disk.

lb02.lab.mojah.be ~ $ fdisk -l

Disk /dev/sda: 10.7 GB, 10737418240 bytes
255 heads, 63 sectors/track, 1305 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1          13      104391   83  Linux
/dev/sda2              14         391     3036285   8e  Linux LVM

So the server can now see the 10GB hard disk. Let's create a partition, by start fdisk for the /dev/sda device.

lb02.lab.mojah.be ~ $ fdisk /dev/sda

The number of cylinders for this disk is set to 1305.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
(e.g., DOS FDISK, OS/2 FDISK)

Command (m for help): n

Now enter 'n', to create a new partition.

Command action
e   extended
p   primary partition (1-4)
p

Now choose "p" to create a new primary partition. Please note, your system can only have 4 primary partitions on this disk! If you've already reached this limit, create an extended partition.

Partition number (1-4): 3

Choose your partition number. Since I already had /dev/sda1 and /dev/sda2, the logical number would be 3.

First cylinder (392-1305, default 392): <enter>
Using default value 392
Last cylinder or +size or +sizeM or +sizeK (392-1305, default 1305): <enter>
Using default value 1305

Note; the cylinder values will vary on your system. It should be safe to just hint enter, as fdisk will give you a default value for the first and last cylinder (and for this, it will use the newly added diskspace).

Command (m for help): t
Partition number (1-4): 3
Hex code (type L to list codes): 8e
Changed system type of partition 3 to 8e (Linux LVM)

Now type t to change the partition type. When prompted, enter the number of the partition you've just created in the previous steps. When you're asked to enter the "Hex code", enter 8e, and confirm by hitting enter.

Command (m for help): w

Once you get back to the main command within fdisk, type w to write your partitions to the disk. You'll get a message about the kernel still using the old partition table, and to reboot to use the new table. Please obey kindly, and reboot the virtual machine.

After you've rebooted, you can see the newly created partition with fdisk.

lb02.lab.mojah.be ~ $ fdisk -l

Disk /dev/sda: 10.7 GB, 10737418240 bytes
255 heads, 63 sectors/track, 1305 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1          13      104391   83  Linux
/dev/sda2              14         391     3036285   8e  Linux LVM
/dev/sda3             392        1305     7341705   8e  Linux LVM

Now, create the physical volume as a basis for your LVM. Please replace /dev/sda3 with the newly created partition.

lb02.lab.mojah.be ~ $ pvcreate /dev/sda3
Physical volume "/dev/sda3" successfully created

Now find out how your Volume Group is called.

lb02.lab.mojah.be ~ $ vgdisplay
--- Volume group ---
VG Name               VolGroup00
...

Let's extend that Volume Group by adding the newly created physical volume to it.

lb02.lab.mojah.be ~ $ vgextend VolGroup00 /dev/sda3
Volume group "VolGroup00" successfully extended

With pvscan, we can see our newly added physical volume, and the usable space (7GB in this case).

lb02.lab.mojah.be ~ $ pvscan
PV /dev/sda2   VG VolGroup00   lvm2 [2.88 GB / 0    free]
PV /dev/sda3   VG VolGroup00   lvm2 [7.00 GB / 7.00 GB free]
Total: 2 [9.88 GB] / in use: 2 [9.88 GB] / in no VG: 0 [0   ]

Now we can extend Logical Volume (as opposed to the Physical Volume we added to the group earlier). The command is "lvextend -L+[SIZE] /dev/VolGroupxx".

lb02.lab.mojah.be ~ $ lvextend -L+7GB /dev/VolGroup00/LogVol00
Extending logical volume LogVol00 to 9.38 GB
Logical volume LogVol00 successfully resized

If you're running this on Ubuntu, use the following.

lb02.lab.mojah.be ~ $ lvextend -L+7GB /dev/mapper/vg-name

All that remains now, it to resize the file system to the volume group, so we can use the space. Replace the path to the correct /dev device if you're on ubuntu/debian like systems.

lb02.lab.mojah.be ~ $ resize2fs /dev/VolGroup00/LogVol00
resize2fs 1.39 (29-May-2006)
Filesystem at /dev/VolGroup00/LogVol00 is mounted on /; on-line resizing required
Performing an on-line resize of /dev/VolGroup00/LogVol00 to 2457600 (4k) blocks.
The filesystem on /dev/VolGroup00/LogVol00 is now 2457600 blocks long.

And we're good to go!

lb02.lab.mojah.be ~ $ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00 9.1G 1.8G  6.9G  21% /
/dev/sda1              99M   18M   77M  19% /boot
tmpfs                 125M     0  125M   0% /dev/shm

In order to inspire you to try the same (ahum), here are some screenshots of the developer build made at Thursday, August 26th 2010.

There are still some nasty bugs in the OS (especially in a VMware environment) which drop your screen/video from time to time to a black image, and there don't seem to be any apps available yet. So errr ...  it's only a quick-booting (8 seconds!) crashing "browser", but hey - it's cool, amiright?!

1) Get Ubuntu 10.04, 64-bit, and install it.

You'll need around 40GB of free space, and root privileges (through sudo) on your system.

2) Run the following to install the pre-requisites.

$ sudo apt-get install bison fakeroot flex g++ g++-multilib gperf   libapache2-mod-php5 libasound2-dev libbz2-dev libcairo2-dev   libdbus-glib-1-dev libgconf2-dev libgl1-mesa-dev libglu1-mesa-dev   libglib2.0-dev libgtk2.0-dev libjpeg62-dev libnspr4-dev libnss3-dev   libpam0g-dev libsqlite3-dev libxslt1-dev libxss-dev   mesa-common-dev msttcorefonts patch perl pkg-config python   python-old-doctools rpm subversion libcupsys2-dev libgnome-keyring-dev   libcurl4-gnutls-dev libcupsys2-dev

Or you can download the install-build-deps.sh and run it, to have it download automatically. It'll do some additional config as well, so I do prefer this method.

$ wget http://src.chromium.org/svn/trunk/src/build/install-build-deps.sh
$ sudo chmod +x install-build-deps.sh
$ sudo ./install-build-deps.sh

That'll probably take a while, it should download around 200MB worth of extra software.

3) Install the depot_tools (optional)

$ cd /usr/local/src/
$ svn co http://src.chromium.org/svn/trunk/tools/depot_tools
$ export PATH=`pwd`/depot_tools:"$PATH"

This will install the gclient binary. While it's part of the deprecated way of retrieving source code, it's still a valid tool to use.

4) Get git and repo

$ sudo apt-get install git-core subversion

The repo tool as its install instructions here:

$ wget http://android.git.kernel.org/repo -O /usr/bin/repo
$ sudo chmod a+x /usr/bin/repo

5) Get the Chromium OS source

Make sure you run this as a non-root user. Alle steps following this one, should be done as a normal user. This user needs to have sudo-powers.

$ cd /usr/local/src/
$ mkdir chromiumos
$ cd chromiumos/
$ repo init -u http://src.chromium.org/git/manifest -m minilayout.xml

Running the repo init will prompt you for your name and e-mail address, together with a question to enable color display for that user account.

Your Name [root]: Name
Your Email [root@server.(none)]: name@email.com
Your identity is: Name <name@mail.com>
is this correct [y/n]? y
Enable color display in this user account (y/n)? y

Afterwards, you can start the code sync by running:

$ repo sync

That will download a lot of source code. At the time of writing, it exists of ~93 projects varying in size, for a total of 2GB. So yes, this will take a while.

If you're getting "error: RPC failed; result=22, HTTP code = 502" error codes while it's syncing, have a look here for possible fixes.

6) Start building Chromium OS

This is fun part. I'll cover this briefly since it lacks some fine-grain explanations on the Project's Wiki: Building a Chromium-Based OS. Again, be sure to follow those steps as a non-root user, with normal privileges.

$ ln -s /usr/local/src/chromiumos/ ~/chromiumos
$ cd ~/chromiumos/src/scripts/
$ ./make_chroot

This will make the chroot-environment, in which Chromium will be built. On commodity hardware, this'll take anywhere between 10-30 minutes.

Now, enter the chroot.

$ cd ~/chromiumos/src/scripts/
$ ./enter_chroot.sh

And set up board for your system.

(cros-chroot) $ cd ~/trunk/src/scripts/
(cros-chroot) $ ./setup_board --board=x86-generic

Build the Chromium packages.

(cros-chroot) $ cd ~/trunk/src/scripts/
(cros-chroot) $ ./build_packages --board=x86-generic

And build the Chromium image for your board.

(cros-chroot) $ ./build_image --board=x86-generic

This will have generated a unique image for you.

(cros-chroot) $ ls -alh ~/trunk/src/build/images/x86-generic
total 16K
drwxr-xr-x 4 mattias root 4.0K Aug 26 23:22 .
drwxr-xr-x 3 mattias root 4.0K Aug 26 20:45 ..
drwxr-xr-x 2 mattias eng  4.0K Aug 26 23:22 0.8.69.2010_08_26_2313-a1
lrwxrwxrwx 1 mattias eng    25 Aug 26 23:22 latest -> 0.8.69.2010_08_26_2313-a1

Now, to easily test your image, you can convert it to a VMware VMDK file using the built-in convert script.

(cros-chroot) $ cd ~/trunk/src/scripts/
(cros-chroot) $ ./image_to_vm.sh --from=../build/images/x86-generic/latest/ --board=x86-generic --make_vmx --format vmware --vmdk chromium_os_x86-generic.vmdk --vmx chromium_os_x86-generic.vmx

This will try to generate the VMware image, but changes are you'll run into the following error.

./image_to_vm.sh: line 236: qemu-img: command not found

You can solve this by first installing the necessary "qemu-img" package.

(cros-chroot) $ sudo emerge app-emulation/qemu-softmmu

And re-generating your VMware image.

(cros-chroot) $ ./image_to_vm.sh --from=../build/images/x86-generic/latest/ --board=x86-generic --make_vmx --format vmware --vmdk chromium_os_x86-generic.vmdk --vmx chromium_os_x86-generic.vmx

And you'll find your VMware files in the following directory.

(cros-chroot) $ ls -alh ~/trunk/src/build/images/x86-generic/latest/
drwxr-xr-x 5 mattias eng  4.0K Aug 27 00:07 .
drwxr-xr-x 4 mattias root 4.0K Aug 26 23:22 ..
-rw-r--r-- 1 mattias eng   191 Aug 27 00:06 boot.config
-rw-r--r-- 1 mattias eng   834 Aug 26 23:13 boot.desc
-rw-r--r-- 1 mattias eng  675M Aug 27 00:07 chromium_os_x86-generic.vmdk
-rw-r--r-- 1 mattias eng   381 Aug 27 00:07 chromium_os_x86-generic.vmx

Now exit the chrooted environment, and copy the VMware files found in the following location to your local computer.

$ ls -alh ~/chromiumos/src/build/images/x86-generic/latest/

Fire up ye old VMware vCenter Converter, choose the "VMware Workstation or VMware Virtual Machine" as source, and browse to your .VMX file.

VMware Converter

Proceed with the wizard, crank up the assigned memory a bit (defaults to 32MB), and boot up your Virtual Machine.

Have fun!

This is usually how it's configured in the Security Policy Editor.

NetScreen-Remote: IP Subnet/Mask 0.0.0.0

If you want to keep having access to your local IP range(s), create a new connection under "My Connections" and add it as follows.

NetScreen-Remote: Local Network Exception(s)

Choose the following configuration settings.

• Connection Security: Non-secure
• ID Type: IP Subnet
• Subnet: 172.16.0.0
• Mask: 255.255.255.0
• Protocol: All
• Interface Name: Any

It goes without saying you you replace "172.16.0.0" with the IP range you want to have direct access to. Credits to David Geens for pointing it out!

So, let's fix that.

Open up /wp-includes/formatting.php and find the lines around 56 and 57 (this will vary depending on your WordPress version), which start with:

$static_characters = array_merge(array('---', ' -- ', '--', ' - ', ...
$static_replacements = array_merge(array('&#8212;', ' &#8212; ', '&#8211;', ' &#8211; ', ...

These lines effectively translate your double dashes (--) to the HTML character of a single dash. Remove the third element from every array there, so it looks something like this (please note the line continues, so just remove only the 3rd element from each array).

$static_characters = array_merge(array('---', ' -- ', ' - ', ...
$static_replacements = array_merge(array('&#8212;', ' &#8212; ', ' &#8211; ', ...

You can now happily write double dashes again!

# ./fix_wordpress.sh --without-silly-dashes --with-something-else

If you want to prevent the single & double quotes as being replaced automatically, open up your functions.php file in your /wp-includes/themes/<themename>/ folder. Create the file if doesn't exist. And add the following line of code all the way at the bottom.

# Custom: remove (single & double) quote replacements in posts
remove_filter('the_content', 'wptexturize');

This will remove the filter defined in WordPress, so it will no longer replace your single & double quotes.

root@server: # gclient sync
Syncing projects:  27% (26/93)
Initialized empty Git repository in /usr/local/src/chromiumos/chromiumos.git/src/third_party/chromiumos-overlay/.git/
error: RPC failed; result=22, HTTP code = 502

After which the gclient would hang. Here's a quick work-around to at least get the rest of the source code, while Google fixes this git-problem.

root@server: # gclient sync -j 2

That will continue the sync, but while running multiple commands in parallel. So while 1 command hangs on the checkout, the others will continue to checkout the rest of the (gigantic) source code. As of yet, there's a bug report for this, but no solution yet.

It should be noted that gclient is now deprecated, and you might consider switching over to "repo" as a Git overlay tool. This doesn't seem affected by this problem. More info is on "How to get the Chromium Source" wiki-article, which will let you install repo.

A general system error occured: internal server error: vmodl.fault.HostCommunication

You might be prompted to blame DNS resolution or Date/Time settings, but if you're running the previous version of vCenter Server, you won't be able to add the new ESX 4.1 host to your existing vCenter 4.0.

Solution, so far, is to either upgrade your vCenter Server installation, or downgrade / reinstall your ESX host. If you're going to upgrade your vCenter, read the vSphere Upgrade Guide. Also note that vCenter 4.1 is x64 only, there's no longer a 32-bit version.

Once you've upgraded vCenter, remove the host from the inventory, and re-add it. Choosing "connect" from the right-click dropdown menu on the host, likely won't solve it. The host needs to be removed first.

• Front End Tech Talk (1h:08m:40s): presentation about Javascript, optimization, performance, usability, ... Interesting to see how they used Bootloader (on-demand loading), Primer, Haste (packaging & dependency management), Javelin, BigPipe, Pagelets, ... to optimize their javascript. Bit of PHP info on their backend as well.
• Rethinking Servers & Datacenters (44m:46s): Datacenter optimization, infrastructure, ... Very good introduction in power usage/supply in datacenters, traditional cooling, PUE. Some good alternatives as well.
• Design Tech Talk (1h:02m:22s): quick iterations, small teams & much creativity. How to walk through design processes, get interactions, get comments, ... using tools like Pixelcloud,
• Typeahead Search Tech Talk (51m:13s): using intuitive search results as you type with Typeahead. Both front- and backend consequences for introducing it. Also read the "classic" Facebook Search, the Social Role that was integrated in displaying wanted results.

Want more? Lucky you, there's an entire Facebook Techtalk page for those movies!

This is because in MySQL 5.1 the data directory structure changed.

You can either alter your queries, so you retrieve from the database "#mysql50#dbname", or you can be sane and do the following query to upgrade the data directory structure. Note; this assumes you have shell access to the server with enough privileges! If you don't, please contact your server administrator.

mysql > ALTER DATABASE `#mysql50#dbname` UPGRADE DATA DIRECTORY NAME;

Note the backticks (`), that's not a single quote.

You might run into the following error when you first do so:

ERROR 1558 (HY000): Column count of mysql.proc is wrong. Expected 20, found 16. Created with MySQL 50077, now running 50148. Please use mysql_upgrade to fix this error.

To fix this, run the following on the server's command line.

srv # mysql_upgrade -h localhost -u root -p
# For Plesk environments, you can use
srv # mysql_upgrade -h localhost -u admin -p`cat /etc/psa/.psa.shadow `

This will update the necessary database entries to the latest version of MySQL. Chances are, this will even have fixed the '#mysql50#' database prefix for you. If it hasn't, return to the first step here.

More information can be found at the bottom of MySQL's Documentation on the ALTER DATABASE command.

Very fragible ... There was a very interesting video round-up by Discovery Channel, on how those cables are being deployed (in this case, for Global Crossing). It _was_ worth a watch, before they deleted it -_-.

I love being in IT. :-)

Installing phpsh

SSH into your server, and download, build & install the binaries. This assumes you've downloaded the .tar.gz version.

# wget http://github.com/facebook/phpsh/tarball/master
# tar xzf facebook-phpsh-1bc1c01.tar.gz
# cd facebook-phpsh-1bc1c01/
# python setup.py build
# python setup.py install
# phpsh

As simple as that.

How do I use it?

Even more simple. Once the install is done, execute phpsh on the CLI, and get started writing PHP code.

srv ~ # phpsh
Starting php
type 'h' or 'help' to see instructions & features
php>
php>
php> echo "Testing this new PHPSH";
Testing this new PHPSH
php>
php>
php> $dbconn = mysql_connect('localhost', 'user', 'pass');
PHP Warning:  mysql_connect(): Access denied for user 'user'@'localhost' (using password: YES) in /usr/lib/python2.4/site-packages/phpsh/phpsh.php(534) : eval()'d code on line 1
php>
php>
php> exit;

This can help to easily test or debug some PHP code. It has syntax highlighting, tab completion (!!), manual pages, ... Still not convinced? Have a look at some real-life examples for a PHP interface shell.  Seriously, awesome tool.

1. What is DNSSEC?
2. Enabling DNSSEC in your environment
3. Generating your keys: Key Signing Key (KSK) and Zone Signing Key (ZSK)
4. Adding the public keys to your zone
5. Signing the zone
6. Key rotation, zone maintenance
7. Summary
8. Links, Articles & Video

There are a few assumptions throughout this article, mostly related to directory structures as I like to keep things organized. Another method would be to create a directory per zone, and store all data within that directory.

1. I'm doing the signing for the dummy host "dns.org"
2. I have my zonefiles stored in /var/named/zones/
3. I have my Key Signing Keys stored in /var/named/KSK
4. I have my Zone Signing Keys stored in /var/named/ZSK
5. I have my Delegation Signer and Keyset stored in /var/named/SET

1) What is DNSSEC?

Here are a few links that very accurately explain DNSSEC.

• http://blog.techscrawl.com/2009/01/06/dnssec-101/
• http://en.wikipedia.org/wiki/DNSSec
• http://www.dnssec.net/
• http://www.icann.org/en/announcements/dnssec-qaa-09oct08-en.htm

2) Enabling DNSSEC in your environment

For my examples, I'll be signing the zone "dns.org", for which my nameserver has been configured to be authoritive. To enable DNSSEC, you'll need to add the following to your /etc/named.conf file. Note; this is only supported in Bind version 9.3 and upwards.

# named -v
BIND 9.3.6-P1-RedHat-9.3.6-4.P1.el5_4.2

So, edit the file /etc/named.conf and add the lines in bold to the options-section.

options {
...
// DNSSEC
dnssec-enable yes;
};

3) Generating your keys: Key Signing Key (KSK) and Zone Signing Key (ZSK)

Just a small clarification on what the KSK and ZSK actually are.
ZSK: Zone Signing Key - this key will sign your records in your zonefile.
KSK: Key Signing Key - this key will sign your Zone Signing Key. This is generally a key with greater key size.

In my set-up, I have all my zones in seperate files located in /var/named/zones/*. I'll be storing my keys in seperate folders, to keep the overview.

• /var/named/ZSK: all zone signing keys
• /var/named/KSK: all key signing keys

Note; ideally, you would want to store your keys on a offline machine, sign your zones there, and transfer them to your online nameserver. However, in set-ups where zones are reconfigured daily, this would cause too much overhead.

First, we'll generate the Zone Signing Key for dns.org:

# cd /var/named/ZSK
# dnssec-keygen -r /dev/urandom -a RSASHA1 -b 1024 -n ZONE dns.org

This will generate 2 files, using the same naming format: K<zone>.<id>.key (ie:  Kdns.org.+005+03486.key) and K<zone>.<id>.private (ie: Kdns.org.+005+03486.private). The first file holds the public key, and the .private file holds our private key.

Now, to generate the Key Signing Key for dns.org:

# cd /var/named/KSK
# dnssec-keygen -r /dev/urandom -a RSASHA1 -b 4096 -n ZONE -f KSK dns.org

This will also generate 2 files, using the same naming format as the ZSK.

Note; every time you generate a KSK or a ZSK, it will assign a new random ID to the filename.

4) Adding the public keys to your zone

After having generated the KSK and ZSK, we need to add the public key of each file to the zone. In my environment (using the folders described above), this would mean:

# cat /var/named/ZSK/Kdns.org.*.key >> /var/named/zones/dns.org
# cat /var/named/KSK/Kdns.org.*.key >> /var/named/zones/dns.org

This will add the appropriate public key to my zone. I've used the wildcard "Kdns.org.*.key" because we can't predict the ID. However, if you regenerate your keys, you'll have multiple files that would be added with that wildcard (as there are multiple public keys), so be cautious to remove the "old" public and private keys first.

If you take a look at your zonefile now, you'll see there are now 2 extra DNSKEY records which have been added.

dns.org. IN DNSKEY 256 3 5 "random chars"
dns.org. IN DNSKEY 257 3 5 "random chars"

The first DNSKEY, with the number "256", is the smaller Zone Signing Key, with the actual key appended at the back. The second DNSKEY with number "257" is the larger Key Signing Key with the actual key at the end.

After you've restarted your nameserver (to activate the newly updated zone) you can check if the records were added OK by trying to query for them:

# dig @localhost DNSKEY dns.org

The ANSWER SECTION should give you 2 DNSKEY's.

5) Signing the zone

To sign the zone, we need the command to point at the zonefile, the ZSK and KSK.

# cd /var/named/SET
# dnssec-signzone -o dns.org -k /var/named/KSK/Kdns.org.+005+48967.key /var/named/zones/dns.org /var/named/ZSK/Kdns.org.+005+03486.key

Let's break this down. The first parameter, "- 0", tells us which zone to sign (in this case, dns.org). The second parameter, "- k", allows us to point at the Key Signing Key. Next up is the location of our zonefile, followed by the location of the Zone Signing Key.

Note; the IDs in the filenames will vary for your set-up.

Afterwards, you'll notice there are now 2 extra files generated in the /var/named/SET directory. First is "dsset-dns.org.", followed by "keyset-dns.org.". I've placed these in a seperate directory on purpose, to keep the overview in the /var/named/zones directory.

You'll also notice that in /var/named/zones, beside the already existing "dns.org" file, there is now also a file called "dns.org.signed" which holds the signed version of the zonefile. It's also much larger than the original zonefile, because all RRs (Resource Records) are now signed. For reference, here is the original zonefile, the zonefile with dnskey, and the fully signed zonefile.

Now we can change our named.conf to point at the ".signed" version to load this signed zone.

zone "dns.org" IN {
type master;
file "/var/named/zones/dns.org.signed";
};

And reload your nameserver to test the newly signed zone.

# dig @localhost dns.org +dnssec

You'll notice every record is now accompanied by a RRSIG Resource Record as well, which holds the signed version of that Resource Record you requested.

6) Key rotation, zone maintenance

A) Once a zone has been signed, the RRSIG's will have a lifespan of 30 days. After these 30 days, the signatures will expire and cause zones to no longer validate. The only method to "reset" that 30 day timer, is to resign your zones (see step 5 above).

B) Whenever you modify the zone, to add/modify/remove records, you will also have to resign the original zonefile, to re-generate the .signed version.

C) You should re-generate your KSK and ZSK on time. It's advised to re-generate the KSK every year, and the ZSK every 3 months. The longer the key has been in existance, the greater the chance it's been "compromised". To do so, delete the old key (public & private), and re-generate using step 3 of this article.

7) Summary

All in all, enabling DNSSEC for one zone comes down to:

1. Generate a ZSK and KSK (per zone) using dnssec-keygen
2. Include those keys into your zonefile
3. Sign the zone using dnssec-signzone
4. Load your signed zonefile
5. Reload the zone or nameserver

8) Links, Articles & Video

I managed to implement DNSSEC fairly simply, because the following information was made available on the web which explained it very well.

• DNSSEC In 6 minutes (PDF)
• DNSSEC Mini Howto (html)
• Erik Berls - Deploying DNSSEC - LayerOne 2009 (Youtube video, 55min)
• Enabling DNSSEC On Bind (html)
• Practical DNS Setup: How To Implement DNSSEC (practical documents)
• Implementing DNSSEC (dyndns)

If you'd like to add something, please use the comments below. If I've missed some vital information, or published some awful mistake, please let me know.

options {
...
// DNSSEC
dnssec-enable yes;
};

Video: A Day In The Life Of A Facebook Engineer

Obviously, Facebook has seen amongst the most impressive scalability problems ever. And here are some of the interesting bits, which can be used to manage and tune high traffic & high availability websites.

• HipHop for PHP: transforms PHP code to optimized C++ code, for greater CPU efficiency. Started as a "hack" on one of Facebook's Hackatons.
• Memcached: duh ... only downside is it requires some PHP recoding to make use of it (but shouldn't be much if you're already using frameworks or database classes).
• Services: seperate key systems, make them independant of each other (news feed, photos, video, ... - allows you to disable one service, and keep the rest going)
• CFEngine: automating sysadmin tasks (alternatives: Puppet, Chef). Ideal in "clone" environments, a cloud of servers running a similar configuration. I have my doubts on highly customized environments, where each server is configured individually to specifics needs.
• dsh: distributed shell, run commands on any set of hosts in your network

Monitoring will be needed to keep an eye on your infrastructure:

• Ganglia: outdated, but very fast
• Nagios

Take a look at other Open Source contributions made by Facebook's Team!

For sale: Classic Mini, British Open (Special Edition). This version was created in limited stock. Only 1.000 in the UK, and 2.000 more in Europe.

It features an electric sunroof, 1.3l petrol engine, and the driving experience of a lifetime. It's on sale since I lack the time to fully enjoy it. It's among the best preserved British Open's I've ever seen, fully restored by it's previous owner.

It's located on several dealer-sites:

• 2dehands.be
• hebbes.be
• kapaza.be
• vlanauto.be
• autoscout24.be
• autozone.be

And if you're up for some mild reading, I would recommend: The History of Mini. Some more info on the car can be found here.