FillTheGrid.com – Sudoku for the Web (Yes, all the good domains are taken)

Future plans including fleshing it out into a more complete site (contact, about, help, maybe a blog), adding user support especially for remembering which puzzles you’ve played, and adding lots of fun statistics surrounding games won, time taken, difficulty, etc.

What is Spam Log?

Spam Log is a simple WordPress plugin that logs a message every time a comment is marked as spam. Each log message includes the IP address of the poster and the comment’s ID. The log can easily be processed by fail2ban. fail2ban is a daemon that scans log files for misbehaving clients and bans them by IP address. Here is sample output generated by Spam Log:

2009-04-20 04:15:03 comment id=527 from host=83.233.30.32 marked as spam
2009-04-20 04:18:15 comment id=528 from host=83.233.30.32 marked as spam
2009-04-20 04:20:36 comment id=529 from host=83.233.30.32 marked as spam
2009-04-20 04:21:46 comment id=530 from host=83.233.30.32 marked as spam
2009-04-20 04:22:49 comment id=531 from host=83.233.30.32 marked as spam

Why use Spam Log and fail2ban if Akismet/wp-recaptcha/etc. is already catching all the spam?

• Many spammers post 50+ comments a day from a single IP address. Even if every comment is correctly marked as spam, the volume alone means that you can’t easily monitor the spam queue for false positives. Spam Log and fail2ban should considerably reduce the total amount of spam.
• Even if spam comments never appear on your blog, they still waste valuable resources on your server. Low-memory virtual servers need all available resources for serving legitimate users. Banning spammers at the firewall before they ever connect to your web server is very efficient.

Installation

Spam Log

1. Upload the spam-log folder to the wp-content/plugins directory.
2. Active the plugin through the WordPress Admin menu.
3. Set the location of the spam log through Spam Log’s Options page in the WordPress Admin menu. By default, the location is set to wp-content/spam.log. The file or containing directory needs to be writeable by the user that the web server runs as. On Debian or Ubuntu systems, you can do the following:

$ sudo touch /path/to/spam.log
$ sudo chown www-data.www-data /path/to/spam.log

fail2ban Configuration

Create /etc/fail2ban/filter.d/spam-log.conf with the following contents:

[Definition]
failregex = ^\s*comment id=\d+ from host=<HOST> marked as spam$
ignoreregex =

Add the following lines to /etc/fail2ban/jail.local:

[spam-log]
enabled  = true
port     = http,https
filter   = spam-log
logpath  = /path/to/spam.log
maxretry = 5
findtime = 3600
bantime  = 86400

Change logpath to the path you set on Spam Log’s Options page. This configuration will ban an IP address for a day if it’s used to post 5 comments within an hour that are marked as spam. Warning: Some captcha plugins mark comments as spam when a user fails a captcha. Be careful decreasing maxretry if you’re using such a plugin as there’s a risk that you will ban legitimate users.

Download

spam-log-0.1.tar.gz
spam-log-0.1.zip

The Old Way: Use a Fixed Priority for iptables

The traditional UNIX syslog service only has two ways to categorize, and consequently route, messages: facility and priority. Facilities include kernel, mail, daemon, etc. Priorities include emergency, alert, warning, debug, etc. The Linux iptables firewall runs in the kernel and therefore always has the facility set to kern. Using traditional syslog software, the only way you can separate iptables messages from other kernel messages is to set the priority on all iptables messages to something specific that hopefully isn’t used for other kernel logging.

For example, you could add something like the following to /etc/syslog.conf:

kern.=debug -/var/log/iptables.log

and specifically remove the kernel debugging messages from all other logs like so:

kern.*;kern.!=debug -/var/log/kern.log

and in each iptables logging rule use the command line option --log-level debug.

There are two distinct disadvantages to this approach. First, there’s no guarantee that other kernel components won’t use the priority you’ve set iptables to log at. There’s a real possibility that useful messages will be lost in the deluge of firewall logging. Second, this approach prevents you from actually setting meaningful priorities in your firewall logs. You might not care about random machines hammering Windows networking ports, but you definitely want to know about malformed packets reaching your server.

The New Way: Filter Based on Message Content with rsyslog

rsyslog is mostly a drop-in replacement for a tradtional syslog daemon–on Linux, klogd and sysklogd. In fact, on Debian and Ubuntu, you can simply:

$ sudo apt-get install rsyslog

and if you haven’t customized /etc/syslog.conf, logging should continue to work in precisely the same way. rsyslog has been the default syslog on Red Hat/Fedora based systems for a number of versions now, but if it’s not installed:

$ sudo yum install rsyslog

Configure iptables to Use a Unique Prefix

We’ll setup rsyslog to filter based on the beginning of a message from iptables. So, for each logging rule in your firewall script, add --log-prefix "iptables: ". Most firewall builder applications can be easily configured to add a prefix to every logging rule. For example, if you’re using firehol as I am, you could add:

FIREHOL_LOG_PREFIX="firehol: "

to /etc/firehol/firehol.conf.

Configure rsyslog to Filter Based on Prefix

Create /etc/rsyslog.d/iptables.conf with the following contents:

:msg, startswith, "iptables: " -/var/log/iptables.log
& ~

The first line means send all messages that start with “iptables: ” to /var/log/iptables.log. The second line means discard the messages that were matched in the previous line. The second line is of course optional, but it saves the trouble of explicitly filtering out firewall logs from subsequent syslog rules.

When I configured this on my own machines, I did notice one issue that may be a peculiarity of firehol, but it’s probably worth mentioning anyway. It seems that firehol adds an extra single quote at the beginning of log messages that needs to be matched in the rsyslog rule. For example, here’s a log message from firehol:

Apr 17 12:41:07 tick kernel: 'firehol: 'IN-internet':'IN=eth0 OUT= MAC=fe:fd:cf:c0:47:b5:00:0e:39:6f:48:00:08:00 SRC=189.137.225.191 DST=207.192.75.74 LEN=64 TOS=0x00 PREC=0x00 TTL=32 ID=5671 DF PROTO=TCP SPT=3549 DPT=5555 WINDOW=65535 RES=0x00 SYN URGP=0

Notice the extra quote after “kernel: ” and before “firehol: “. So, on my machine I configured the rsyslog filter like so:

:msg, startswith, "'firehol: " -/var/log/iptables.log
& ~

Configure iptables Log Rotation

Finally, since we’re logging to a new file, it’s useful to create a log rotation rule. Create a file /etc/logrotate.d/iptables with the following contents:

/var/log/iptables.log
{
	rotate 7
	daily
	missingok
	notifempty
	delaycompress
	compress
	postrotate
		invoke-rc.d rsyslog reload > /dev/null
	endscript
}

The preceding script tells logrotate to rotate the firewall log daily and keep logs from the past seven days.

In this post, I’ll  cover rules 5-13 and summarize the results of the optimizations that I made to my site.

YSlow’s 13 Rules to Improve Web Site Performance (rules 5-13)

A number of these rules are fairly simple, so I’ll cover some together.

5. Put CSS at the top

6. Put JS at the bottom

These rules are fairly easy to follow and most sites shouldn’t have to change anything. Putting CSS in the document <head> makes the page appear to load faster because of progressive rendering. It turns out that javascript blocks parallel downloads, so ideally you want to load javascript when everything else has already loaded. Also, if the javascript is hosted externally and the external server is slow, the rest of the page should load without problems.

7. Avoid CSS expressions

I’ll be honest here. I didn’t even know CSS had expressions until I saw this rule. YSlow recommends avoiding them because they are evaluated an absurd number of times.

8. Make JS and CSS external

External files can be cached indepently. If your users usually browse to more than one page on your site or you have frequent returning visitors, this should improve load times.

9. Reduce DNS lookups

DNS lookups can still take a fair amount of time even on a broadband connection. Basically, try to keep the number of unique host names fairly low. There is one competing performance benefit to serving page components from multiple hosts. It turns out that most browsers will only make two simultaneous connections to a given host. So, serving content from multiple hosts can speed up page load times especially if there are a lot of components.

10. Minify JS and CSS

Actually, in YSlow this rule says Minify JS, but on the explanation page, they added CSS. Minifying these files simply makes them smaller even if you’re already compressing them. The best minifier I found is the YUI Compressor, which handles both CSS and javascript. Here are the results of minifying my site’s stylesheet:

The YUI Compressor is pretty easy to use, but using it is going to get annoying unless you automate. To solve this problem, I once again modified my publish CSS script to do the minifying automatically. If you read the first part of this article and you’re wondering how many times I’m going to modify my publish script, this is the final version:

#!/bin/sh

SERIAL_FILE=serial.txt
OLD_SERIAL=`cat ${SERIAL_FILE}`
SERIAL=$((${OLD_SERIAL} + 1))
YUICOMPRESSOR_PATH=/home/avery/yuicompressor-2.4.2/build/yuicompressor-2.4.2.jar

cat	style.css.original \
	../../plugins/wp-recaptcha/recaptcha.css \
	../../plugins/deko-boko-a-recaptcha-contact-form-plugin/display/dekoboko.css \
	| java -jar ${YUICOMPRESSOR_PATH} --type css > style-${SERIAL}.css

sed "s/REPLACE_WITH_SERIAL/${SERIAL}/g" < header.php.original > header.php

rm style-${OLD_SERIAL}.css

echo ${SERIAL} > ${SERIAL_FILE}

Using publish scripts for stylesheets and javascript might seem like overkill at first, but that simple script has allowed me to make significant improvements that otherwise would be too time consuming to implement.

11. Avoid redirects

12. Remove duplicate scripts

These two rules are fairly easy to follow. Sometimes you can’t avoid redirects such as when you move to a new domain or restructure your site. Those sort of redirects are good and they maintain the reputation your site has built with search engines. That said, a lot of redirects can be eliminated simply by adding a trailing slash to a URL. Be especially mindful if you’re working on hand-coded sites rather than CMS-based sites. Remove duplicate scripts is pretty obvious. Unfortunately, Google Adsense serves the same scripts for each ad on a page and there’s no way to fix it.

13. Configure ETags

I won’t say much about ETags. This YSlow rule actually means configure ETags or remove them entirely. For a site being served off a single web server, ETags offer no benefits and some drawbacks including the fact that Apache generates invalid ETags for gzipped content. I recommend turning them off unless you’re willing to configure them per Yahoo’s ETags best practices. To disable ETags in Apache add the following to the virtual host configuration or the main .htaccess:

FileETag None

Results

Finally, here are the results that I obtained. First, my YSlow score:

Final YSlow Performance Tab

I’m somewhat disappointed that after all of that, my score barely improved, moving from an F (59) to a D (63). Unfortunately, most of the remaining areas of optimization are either unrealistic as in Use a CDN or are out of my control.

What about real performance? The following tables include the minimum, maximum, and average times out of 10 page loads. First, with a cold cache:

I’m actually fairly satisfied. Most of the optimizations I did would benefit repeat visitors rather than new visitors. A 15% performance improvement is not bad at all. Next, with a warm cache:

These results baffled me. The improvements are only slightly better than the cold cache numbers and I really thought page loads would speed up more for repeat visitors. After some investigation, it turns out that I was loading pages in the wrong way. Specifically, I was hitting the f5 key to reload the page in my warm cache tests. Reloading with f5 has a special meaning in at least Firefox. It tells the browser to check that every item in the cache is the correct version even if the item is set to expire in 10 years. Here are the numbers without using f5:

Now, those are some fast page loads. I’m not interested enough to revert all my changes and re-test, but some statistics from YSlow suggest that I probably sped up warm cache page loads by a fair amount. Here’s the initial Stats tab, prior to any optimization:

Initial YSlow Stats Tab

And here’s the final Stats tab after all the optimizations:

Final YSlow Stats Tab

Of particular note: in the case of a primed cache, I reduced the number of HTTP requests from 25 to 15.

Overall, I’m satisfied with the performance improvements. Even if the improvements are not amazing, working with YSlow is not particularly hard or time consuming. In fact, writing this article took much longer than the actual changes I made to my web site. A 15% reduction in load times is worth an hour or two of writing scripts and changing server configurations.

YSlow’s 13 Rules to Improve Web Site Performance (rules 1-4)

YSlow grades web sites based on 13 different rules. Browse to your site, click on the YSlow icon in the lower right corner of Firefox, and click on the Performance tab. Here’s how one post on my site ranked before making any changes:

Initial YSlow Performance Tab

Clicking on any rule takes you to a page explaining it in more detail.

A quick note: the screenshot above was taken before I disabled Woopra, which is web analytics software similar to Google Analytics. I had tried it out a couple months ago and had simply forgotten that I had it enabled. Disabling Woopra improved my grades very slighty in a number of rules, but I don’t include that optimization in the main article because it’s technically a functional change to my site. As explained below, YSlow doesn’t make functional recommendations.

What follows are the changes that I made to my site, rule-by-rule.

1. Make fewer HTTP requests

This is one of the most important rules and it’s also one of the most difficult to implement. It’s worth mentioning at this point that YSlow’s general strategy isn’t to recommend changes that alter the actual content and functionality of a site, but instead suggest changes that will make the same content load faster. In other words, if you have a gallery with 20 images, the recommendation isn’t to cut the gallery down to 5 images. Consequently, Make fewer HTTP requests focuses on three components of web pages: javascript, stylesheets, and background images.

Expanding the rule will show counts of each component if they exceed YSlow’s thresolds:

Make fewer requests expanded

In the case of my blog, the problem comes down to external javascript and CSS. To get a list of page components, click on the Components tab. They’re sorted by type so, it’s very easy to identify the problem files.

Javascript and CSS components

I can group these into 4 basic categories: Google Adsense, Google Analytics, reCAPTCHA, and WordPress plugins. Adsense has a particularly egregious number of external javascript files. Unfortunately, there’s nothing I can do about it as modifying the code is against Adsense policies. Analytics has just one javascript file and keeping that external is probably a good idea. Almost anyone who browses more than a handful of sites will have ga.js cached as Analytics use is pervasive. Similarly, there’s a good possibility that visitors will have the main reCAPTCHA javascript cached and the other code is dynamic so you don’t want to mess with it. Finally, there are the stylesheets from WordPress plugins.

Combining CSS into one file–especially when it’s all hosted on the same server–is a really good idea. In this case, I’m serving my main style.css file and two other stylesheets from the wp-recaptcha plugin and the Deko Boko plugin. I wrote a simple script to combine all of the files into one, like so:

#!/bin/sh

cat style.css.original \
    ../../plugins/wp-recaptcha/recaptcha.css \
    ../../plugins/deko-boko-a-recaptcha-contact-form-plugin/display/dekoboko.css \
    > style.css

I also modified both plugins to no longer include a link to their original stylesheets in the <head> section of the page. Search the PHP files for references to the original stylesheets and comment those lines out.

Before moving on, it’s worth mentioning that you should use CSS sprites if at all possible. Many sites use tens of icons per page. Combining those tiny icons into one sprite is a huge performance boost.

2. Use a CDN

One persistent criticism of YSlow is that it focuses on performance improvements to major sites. I don’t really agree with that; in fact, nearly every rule applies equally to both small and large sites. Use a CDN is the exception. A CDN allows you to serve content to users from geographically close servers. It’s a decent performance improvement, but it’s also incredibly expensive.

3. Add an Expires header

What’s the fastest HTTP request? One that never takes place. The idea behind far future Expires headers is that you should allow permanent caching of any page component that won’t change or even that is unlikely to change very often. If it does need to be modified, you simply rename it. To accomplish this using Apache, enable mod_expires and add the following to the virtual host configuration or the main .htaccess:

<IfModule mod_expires.c>
    ExpiresActive On
    ExpiresByType image/gif "access plus 10 years"
    ExpiresByType image/jpeg "access plus 10 years"
    ExpiresByType image/png "access plus 10 years"
    ExpiresByType text/css "access plus 10 years"
</IfModule>

Warning: The preceding configuration means that any time you change an image or stylesheet served my Apache, you must rename it. If a browser or proxy server has the component cached, it will never check to see if its version is the same as the version on your server. For images, this is probably not an annoyance; changing an image usually means uploading a new version in which case it will have a different file name anyway. For stylesheets, this can be annoying if you don’t automate the process.

I solved the stylesheet problem with a script. In the same directory as style.css, I created a text file like so:

$ echo 0 > serial.txt

Next, I modified the script that combined the 3 original stylesheets I was using like so:

#!/bin/sh

SERIAL_FILE=serial.txt
OLD_SERIAL=`cat ${SERIAL_FILE}`
SERIAL=$((${OLD_SERIAL} + 1))

cat style.css.original \
    ../../plugins/wp-recaptcha/recaptcha.css \
    ../../plugins/deko-boko-a-recaptcha-contact-form-plugin/display/dekoboko.css \
    > style-${SERIAL}.css

sed "s/REPLACE_WITH_SERIAL/${SERIAL}/g" < header.php.original > header.php

rm -f style-${OLD_SERIAL}.css

echo ${SERIAL} > ${SERIAL_FILE}

The script reads a number in from serial.txt, increments it, creates a new stylesheet with the number appended, and in this case modifies WordPress’ header.php to link to the new stylesheet. Every time I want a different style on my site, I modifiy style.css.original and run this script.

4. Gzip components

Everyone in every circumstance should enable mod_deflate. Way back in the early days of the web there were browsers that didn’t play well with gzip; they basically don’t exist anymore. In Apache, enable mod_deflate–it’s probably already enabled–and add the following to the virtual host configuration or the main .htacess:

<IfModule mod_deflate.c>
    AddOutputFilterByType DEFLATE text/html text/plain text/xml text/css
</IfModule>

Continued Soon

When I started writing this post, I didn’t realize how long it was going to be. Part two, which will include the other 9 easier-to-implement and less important rules, should be up within a few days. I’ll also include some before-and-after performance statistics.

Part two is now finished: Using YSlow to Optimize Web Site Performance Continued.

Firefox 3 native widgets on Linux

As you can see, there’s an ugly white box around the otherwise rounded widgets. Originally, I thought my CSS was lacking, but after hours–literally–of googling I determined that Firefox’s implementation of gtk+ widgets is just shoddy. Of course, on a site with a white background, they look great. I’m reminded of that Henry Ford quote: “Any customer can have a car painted any colour that he wants as long as it is black.”

Applying any styling to the input elements will completely disable the native widgets. I ended up writing some basic CSS:

input, textarea { border: 2px solid #888888; }
input:focus, textarea:focus {
	border-color: #D9D27C;
	background-color: #FFFBC4;
}

They don’t look nearly as good as native widgets on a white background, but they look a whole lot better than native widgets on my site. See below:

Basic styled widgets

I also added some input focus bling to the form:

Text input with keyboard focus

Even though they look alright now, I’ve decided that my next major project on my blog should be a complete theme redesign. Traditional color schemes are easier to work with and I’ll also get familiar with a lot of WordPress PHP code.

About a year ago I bought an external SATA drive for backups. My normal usage consisted of:

1. Power on and connect the drive
2. mount /media/backup
3. Run my backup script
4. umount /media/backup
5. Power off and unplug the drive

This seemed to work pretty well–at the very least, I wasn’t losing data–except the drive made a strange sound when I powered it off. It wasn’t a normal drive spin down sound; it was louder and shorter. So, I googled for authoritative instructions on using external drives with Linux. While most sources suggest doing exactly what I did, it’s not ideal.

It turns out that most cheap external USB/SATA/firewire enclosures don’t properly issue a stop command to the drive when you flick the power switch. Instead, the power switch simply cuts power to the drive, which forces the drive to do an emergency head retract. If you think that sounds bad, you’re right. Emergency retracts aren’t going to brick your drive immediately, but if they occur regularly they’re putting a lot of unnecessary wear and tear on the drive. In fact, some drives monitor how often this happens with S.M.A.R.T. attribute 192. (Check Wikipedia’s S.M.A.R.T. page for a comprehensive list of attributes)

Solution

The solution is to spin down the drive via software before turning it off and unplugging it. The best way to do this is with a utility called scsiadd. This program can add and remove drives to Linux’s SCSI subsystem. Additionally, with fairly modern kernels, removing a device will issue a stop command, which is exactly what we’re looking for. Run:

$ sudo scsiadd -p

which should print something like:

Attached devices:
Host: scsi0 Channel: 00 Id: 00 Lun: 00
  Vendor: ATA      Model: SAMSUNG HD300LJ  Rev: ZT10
  Type:   Direct-Access                    ANSI  SCSI revision: 05
Host: scsi4 Channel: 00 Id: 00 Lun: 00
  Vendor: LITE-ON  Model: DVDRW LH-20A1L   Rev: BL05
  Type:   CD-ROM                           ANSI  SCSI revision: 05
Host: scsi5 Channel: 00 Id: 00 Lun: 00
  Vendor: ATA      Model: WDC WD10EACS-00Z Rev: 01.0
  Type:   Direct-Access                    ANSI  SCSI revision: 05

Identify the drive you want to remove and then issue:

$ sudo scsiadd -r host channel id lun

substituting the corresponding values from the scsiadd -p output. For example, if I wanted to remove “WDC WD10EACS-00Z”, I would run:

$ sudo scsiadd -r 5 0 0 0

If everything works, scsiadd should print:

Attached devices:
Host: scsi0 Channel: 00 Id: 00 Lun: 00
  Vendor: ATA      Model: SAMSUNG HD300LJ  Rev: ZT10
  Type:   Direct-Access                    ANSI  SCSI revision: 05
Host: scsi4 Channel: 00 Id: 00 Lun: 00
  Vendor: LITE-ON  Model: DVDRW LH-20A1L   Rev: BL05
  Type:   CD-ROM                           ANSI  SCSI revision: 05

You can double-check the end of dmesg. You should see:

[608188.235216] sd 5:0:0:0: [sdb] Synchronizing SCSI cache
[608188.235362] sd 5:0:0:0: [sdb] Stopping disk
[608188.794296] ata6.00: disabled

At this point, the drive is removed from Linux’s SCSI subsystem and it should not be spinning. It’s safe to unplug and turn off.

Using scsiadd directly can be inconvenient because it requires looking up the host, channel, id, and lun of the drive. I wrote a short script that will take a normal Linux device file like /dev/sdb, figure out the correct arguments to scsiadd, and run scsiadd -r. I use this script in my larger backup script.

#!/bin/sh

if [ $# -ne 1 ]; then
    echo "Usage: $0 <device>"
    exit 1
fi

if ! which lsscsi >/dev/null 2>&1; then
    echo "Error: lsscsi not installed";
    exit 1
fi

if ! which scsiadd >/dev/null 2>&1; then
    echo "Error: scsiadd not installed"
    exit 1
fi

device=`lsscsi | grep $1`
if [ -z "$device" ]; then
    echo "Error: could not find device: $1"
    exit 1
fi

hcil=`echo $device | awk \
    '{split(substr($0, 2, 7),a,":"); print a[1], a[2], a[3], a[4]}'`

scsiadd -r $hcil

It does require the lsscsi command to be present on the system.

Why Use S.M.A.R.T.?

Basically, S.M.A.R.T. may give you enough of a warning that you can safely backup all your data before your hard drive dies. There is some amount of conflicting information on the internet about how reliable the warnings are. The best source of research that I found is a paper from Google that describes an internal study of hard drive failure. A quick summary: certain events greatly increase the chance of hard drive failure including reallocation events and failed self-tests, but only about 60% of the drives that failed in the study had any negative S.M.A.R.T. attributes. Obviously, nothing replaces regular backups.

A good source for more information is the S.M.A.R.T. wikipedia page.

Installation

On Debian or Ubuntu systems:

$ sudo apt-get install smartmontools

On Fedora:

$ sudo yum install smartmontools

Capabilities and Initial Tests

smartmontools comes with two programs: smartctl which is meant for interactive use and smartd which continuously monitors S.M.A.R.T. Let’s look at smartctl first:

$ sudo smartctl -i /dev/sda

Replace /dev/sda with your hard drive’s device file in this command and all subsequent commands. If there’s only one hard drive in the system, it should be /dev/sda or /dev/hda. If this command fails, you may need to let smartctl know what type of hard drive interface you’re using:

$ sudo smartctl -d TYPE -i /dev/sda

where TYPE is usually one of ata, scsi, or sat (for serial ata). See the smartctl man page for more information. Note that if you need -d here, you will need to add it to all smartctl commands. This should print information similar to:

=== START OF INFORMATION SECTION ===
Model Family:     SAMSUNG SpinPoint T133 series
Device Model:     SAMSUNG HD300LJ
Serial Number:    S0D7J1UL303628
Firmware Version: ZT100-12
User Capacity:    300,067,970,560 bytes
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   7
ATA Standard is:  ATA/ATAPI-7 T13 1532D revision 4a
Local Time is:    Fri Jan  2 03:08:20 2009 EST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

Now that smartctl can access the drive, let’s turn on some features. Run the following command:

$ sudo smartctl -s on -o on -S on /dev/sda

• -s on: This turns on S.M.A.R.T. support or does nothing if it’s already enabled.
• -o on: This turns on offline data collection. Offline data collection periodically updates certain S.M.A.R.T. attributes. Theoretically this could have a performance impact. However, from the smartctl man page:
  

  Normally, the disk will suspend offline testing while disk accesses are taking place, and then automatically resume it when the disk would otherwise be idle, so  in  practice  it has little effect.

• -S on: This enables “autosave of device vendor-specific Attributes”.

The command should return:

=== START OF ENABLE/DISABLE COMMANDS SECTION ===
SMART Enabled.
SMART Attribute Autosave Enabled.
SMART Automatic Offline Testing Enabled every four hours.

Next, let’s check the overall health:

$ sudo smartctl -H /dev/sda

This command should return:

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

If it doesn’t return PASSED, you should immediately backup all your data. Your hard drive is probably failing. Next, let’s make sure that the drive supports self-tests. I have yet to see a drive that doesn’t, but the following command also gives time estimates for each test:

$ sudo smartctl -c /dev/sda

I won’t list the complete output because it’s somewhat lengthy. Make sure “Self-test supported” appears in the “Offline data collection capabilities” section. Also, look for output similar to:

Short self-test routine
recommended polling time: 	 (   2) minutes.
Extended self-test routine
recommended polling time: 	 ( 127) minutes.

These are rough estimates of how long the short and long self-test’s will take respectively. Let’s run the short test:

$ sudo smartctl -t short /dev/sda

On my drive, this test should take 2 minutes, but this obviously varies. You can run:

$ sudo smartctl -l selftest /dev/sda

to check results. Unfortunately, there’s no way to check progress, so just keep running that command until the results show up. A successful run will look like:

=== START OF READ SMART DATA SECTION ===
SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%     21472         -

Now, do the same for the long self-test:

$ sudo smartctl -t long /dev/sda

The long test can take a significant amount of time. You might want to run it overnight and check for the results in the morning. If either test fails, you should immediately backup all your data and read the last section of this guide.

Configuring smartd

We’ve now enabled some features and run the basic tests. Instead of repeating the previous section daily, we can setup smartd to do it all automatically. If your system has an /etc/smartd.conf file, check for a line that begins with DEVICESCAN. If you find one comment it out by adding a ‘#’ to the beginning of the line. DEVICESCAN doesn’t work on my system and specifying a device file is easy. Add the following line to /etc/smartd.conf:

/dev/sda -a -d sat -o on -S on -s (S/../.././02|L/../../6/03) -m root -M exec /usr/share/smartmontools/smartd-runner

Here’s what each option does:

• /dev/sda: Replace this with the device file you’ve been using in smartctl commands.
• -a: This enables some common options. You almost certainly want to use it.
• -d sat: On my system, smartctl correctly guesses that I have a serial ata drive. smartd on the other hand does not. If you had to add a “-d TYPE” parameter to the smartctl commands, you’ll almost certainly have to do the same here. If you didn’t, try leaving it out initially. You can add it later if smartd fails to start.
• -o on, -S on: These have the same meaning as the smartctl equivalents
• -s (S/../.././02|L/../../6/03): This schedules the short and long self-tests. In this example, the short self-test will run daily at 2:00 A.M. The long test will run on Saturday’s at 3:00 A.M. For more information, see the smartd.conf man page.
• -m root: If any errors occur, smartd will send email to root. On my system, mail for root is forwarded to my normal email account. If you don’t have a similar setup, replace root with your normal email address. This option also requires a working email setup. Most Linux distributions automatically have working outbound email.
• -M exec /usr/share/smartmontools/smartd-runner: This last part may be specific to the Debian and Ubuntu smartmontools packages. Check if your system has /usr/share/smartmontools/smartd-runner. If it doesn’t, remove this option. Instead of sending email directly, “-M exec” makes smartd run a different command when errors occur. On Debian, smartd-runner will run each script in /etc/smartmontools/run.d/, one of which emails the user specified by the “-m” option.

If you have more than one hard drive in your system, add a line for each one replacing /dev/sda with a different device file.

Finally, restart smartd:

$ sudo /etc/init.d/smartmontools restart

If this command fails, the end of /var/log/daemon.log should have some diagnostic information. If smartd started fine, we should still test that email notifications are working. Add “-M test” to the end of the configuration line in /etc/smartd.conf. This will make smartd send out a test notification when it’s next started. Once again, restart smartd:

$ sudo /etc/init.d/smartmontools restart

You should receive an email similar to:

This email was generated by the smartd daemon running on:

   host name: polar
  DNS domain: shadypixel.com
  NIS domain: (none)

The following warning/error was logged by the smartd daemon:

TEST EMAIL from smartd for device: /dev/sda

For details see host's SYSLOG (default: /var/log/syslog).

Afterward, you can delete “-M test”.

What To Do If smartd Detects Problems

First, immediately backup everything. Depending on the error, your drive might be close to death or it may still have a long life ahead. Consult the smartmontools FAQ. It has some recommendations for specific errors. Otherwise, ask for help on the smartmontools-support mailing list.

I did apply all the Debian patches that were still relevant so it should be the equivalent of the official Debian package.

First, as is always the case when you’re modifying disk volumes, partitions, or file systems, you should really have a recent backup. A typo in one the following commands could easily destroy data. You have been warned!

All of the required steps must be performed on an unmounted volume. If want to reduce the size of a non-root volume, simply unmount it. For a root volume, you’ll have to boot from a CD. Any modern live or rescue CD should work fine. I prefer SystemRescueCD. It includes almost any disk management programs you might need. After booting from a CD, you may have to issue:

# vgchange -a y

This makes any logical volumes available to the Linux. Most boot CD’s will do it automatically some time during the boot process, but repeating the command won’t hurt. Next, force a file system check on the volume in question:

# e2fsck -f /dev/polar/root

Device names for LVM volumes follow the convention: /dev/<volume group>/<logical volume>. In this case, my volume group is named polar and the volume I’m going to shrink is named root. This is a critical step; resizing a file system in an inconsistent state could have disastrous consequences. Next, resize the actual file system:

# resize2fs /dev/polar/root 180G

Replace 180G with about 90% of the size you want the final volume to be. For example, in this case, I want the final volume to be 200 gigabytes, so I’ll reduce the file system to 180 gigabytes. Why is this necessary? When we reduce the size of the actual volume in the next step, it’s critical that the new size is greater than or equal to the size of the file system. After reading the documentation for both resizefs and lvreduce, I still haven’t been able to find out whether they’re using standard computer gigabytes (1024^3 bytes) or drive manufacturer gigabytes (1000^3 bytes). In this case, the difference is very important. To be on the safe side, we’ll just shrink the file system a bit more than necessary and expand it to use the full space available later. Next, reduce the size of the logical volume:

# lvreduce -L 200G /dev/polar/root

In this case, use the actual size you want to the volume to be. Finally, grow the file system so that it uses all available space on the logical volume:

# resize2fs /dev/polar/root

That’s it. Enjoy your newly acquired free space.