FreeBSD Blog

AT&T Email

KP — Thu, 17 May 2012 06:46:37 PDT

Update: I resubmitted a unblock request with this post, AT&T has confirmed that they will remove the block. It was quick! Thanks!

AT&T blocked all emails sent from one of my servers, which I’m sure was a mistake, because the server hosts several forums and only sends notification emails to registered members, e.g., when a member asked a question on accessforums.net, if someone replied to his question, the forum will send a notification email. To be on the safe side, I have never sent out a single newsletter from these forums.

According to email server log, I submitted unblock requests here two times in the last two months, but no actions, no response whatever, just like a black hole. I’m sure they didn’t look into the problem at all, because anyone who read the emails or checked my server will know it’s a clean server.

Can anyone bring this post to AT&T admin’s attention? Your simple block has brought lots of inconvenience to our forum members (your customers).

(host gateway-f2.isp.att.net[207.115.11.16] refused to talk to me: 550-72.51.41.55 blocked by ldap:ou=rblmx,dc=att,dc=net 550 Error – Blocked for abuse. See http://att.net/blocks)
xxx@bellsouth.net

No custom kernel via ssh

KP — Sun, 21 Aug 2011 08:35:18 PDT

Tried to build a custom kernel on a canceled server, I couldn’t connect to it once the build started, I also couldn’t connect to it after the build finished and server rebooted. I don’t think I will build custom kernels ever again on remote servers.

FreeBSD and Linux

DoS Attacks

KP — Sun, 21 Aug 2011 08:17:39 PDT

If your server was down due to insufficient swap space, usually needed by large amount of MySQL connections, it’s a sign of DoS attacks. My servers started to get attacks earlier this year, and they were getting increasingly frequent. Here are some simple practices I have employed, they have been proved to be effective for my server in the last few months.

1. Lower maximum mysql connections to make sure that the server will be up on attacks. The default value of max_connections is 150 which might be too high for servers that host database-heavy websites. Take my server (2GB ram) as an example, it hosts active vbulletin forums and couldn’t handle 150 connections, even light attacks can bring the server down easily. The server survived attacks after I lowered max_connections to 120, but it was very slow since it used lots of swap space. After I changed it to 100, the server responded fast while being attacked.

100 connections should be enough for 20~30K daily visits. The maximum connections mysql has used can be checked with:

# mysqladmin -u root -p extended-status | grep Max_used_connections
| Max_used_connections | 57 |

mysql> show status where variable_name like ‘max_used_connections’;
+———————-+——-+
| Variable_name | Value |
+———————-+——-+
| Max_used_connections | 57 |
+———————-+——-+

2. When mysql reaches its connection limit, the whole server will be affected, this can be avoided with user-level connection throttle. mysql.user table has a field max_connections, the default value is 0 which should be changed according to each mysql user’s usage. Many attacks targeted a mediawiki application on my server, actually this wiki has less than 100 visits, I don’t care about it at all, after I changed its max_connections to 5, many attacks no longer affect the server.

3. Apache can handle a lot more connections than mysql, but it can reach its limit. I used to set a very high value, but it’s not helpful to detect and fight attacks. I used the following settings on my 4GB-ram server. KeepAliveTimeout can be lower, I used 2 seconds for a while which also worked well. Since my server has enough memory, I used a higher value for better performance (possibly).

TimeOut 60
MaxClients 180
KeepAliveTimeout 5

4. Limit Apache connections per IP. Most attacks on my server were from only several IPs, Apache mod limitipconn can limit each IP’s connections, it can be found in ports www/mod_limitipconn (for Apache 1.3.x) and www/mod_limitipconn2 (for Apache 2). I wasn’t able to see the effect with my limited tests even I set MaxConnPerIP to 3, Apache is too robust.

LoadModule limitipconn_module libexec/apache22/mod_limitipconn.so


    MaxConnPerIP 5
    NoIPLimit images/*

5. Block malicious IPs with firewall. It’s hard to believe but, a certain IP (should be from a compromised server) participated in almost every attack. Here is a very good pf tutorial, follow it strictly and you will be safe.

The above protections should be enough for random attacks.

No related posts.

Backup Script

KP — Sat, 20 Aug 2011 10:32:23 PDT

My simple backup script
It runs daily with cron job.

#!/bin/sh

# directory to backup
BACKUP_DIR=’/home/’

# directory to store backup files
DEST_DIR=’/backup/’

# number of days to store backup files
MAX_DAYS=30

# base backup file name
BASE_FILENAME=’home’

# remove files that are 30 or more days old.
find $DEST_DIR -mtime +$MAX_DAYS -maxdepth 1 -name ‘*.tgz’ -exec /bin/rm -f ‘{}’ +

# generate backup file name with date stamp, no hour/minute info for easier remote backup.
destfilename=$BASE_FILENAME`date “+%Y%m%d”`.tgz

cd $BACKUP_DIR
tar czf $DEST_DIR$destfilename . &

Cross-server backup with scp
This script uses the same path info and runs later than the above one to make sure backup files have been generated, hence no need to check file existence. It runs weekly with cron job.

#!/bin/sh

BACKUP_DIR=’/backup/’
BASE_FILENAME=’home’
BASE_FILENAME_REMOTE=’remotehome’

# generate backup file name, the same with the backup script
local_filename=$BACKUP_DIR$BASE_FILENAME`date “+%Y%m%d”`.tgz
remote_filename=$BACKUP_DIR$BASE_FILENAME_REMOTE`date “+%Y%m%d”`.tgz

# copy remote backup file to local server
scp -i /root/.ssh/id_rsa -P 1234 loginame@hostname:$remote_filename $BACKUP_DIR

# copy local backup file to remote server
scp -i /root/.ssh/id_rsa -P 1234 $local_filename loginame@hostname:$BACKUP_DIR

Cron job

10 4 * * * /root/backup.sh
40 4 * * 0 /root/remote_backup.sh

PHP 5.3 Isn’t That Bad

KP — Fri, 29 Jul 2011 02:21:31 PDT

I complained about php5.3 generating huge log files, also, E_DEPRECATED doesn’t suppress those deprecated errors, as a result, I had to turn off php logs. Actually it doesn’t have to be that bad.

The reason why E_DEPRECATED doesn’t work is that, some scripts set error_reporting value and supersede the setting in php.ini, so, no matter how we configure php, we just couldn’t suppress those errors. Obviously, the best solution is modifying the scripts, but I didn’t want to risk breaking the code and had never given it a try. I was very wrong, the best solution is actually also the easiest solution, there is no risk at all. I fixed a dozen scripts within a couple of hours – just very simple replace.

Here are some common deprecation errors I encountered:

Error: date() [function.date]: It is not safe to rely on the system’s timezone settings. Please use the date.timezone setting. Old versions of WordPress generate crazy logs of this error.
Solution: Set a default timezone in php.ini, my server was changed to
date.timezone = “America/Phoenix”
Here is the list of supported timezones.

Error: PHP Deprecated: Assigning the return value of new by reference is deprecated
Solution: Replace “‘=& new” with “= new”.

Error: Function split() is deprecated.
Solution: Most scripts don’t use the regex version, simply replace “split” with “explode”.

There are some other errors but I don’t remember right now, just search online, there should always be a very easy solution.

Add Zip Support

KP — Fri, 29 Jul 2011 00:14:23 PDT

According to the PHP manual, zip extension uses the functions of zlib (archivers/php5-zlib), but it doesn’t work. Actually there is a separated package dedicated to zip support: archivers/php5-zip, after I installed this package, phpinfo() will have a zip section. Isn’t that confusing?

SSH Tunneling – Easy Secure Proxy

KP — Sat, 19 Mar 2011 12:38:15 PDT

While I was trying to setup OpenVPN as a proxy, I found that SSH tunneling is a much easier and more convenient solution. The configuration only takes a few minutes, and the client side can be configured to use the proxy or not on a program-by-program basis.

Add the following line in /etc/ssh/sshd_config

PermitTunnel yes

Configure Windows client

Disable Remote Access for MySQL

KP — Fri, 18 Mar 2011 11:31:17 PDT

I’m seeing messages like below in mysql error log on my new server:

110318 10:48:02 [Warning] IP address ‘x.x.x.x’ could not be resolved: no reverse address mapping.

I don’t need to access mysql remotely, it should be disabled for either performance or security. This can be changed by adding “skip-networking” in my.cnf, mysql will not listen on a TCP/IP port at all.

[mysqld]
skip-networking

MySQL Backup and Restore

KP — Sat, 12 Mar 2011 02:20:31 PST

Backup single database
# mysqldump -u root -p db_name | gzip > sql.gz

Restore
# gunzip < sql.gz | mysql -u root -p db_name

How to Setup Key Based Authentication in SSH

KP — Tue, 08 Mar 2011 04:19:27 PST

This is for Windows client, please refer to this page for FreeBSD workstation.

Key Based Authentication, step by step

1. Download PuTTYgen (on Windows), generate private/public key pair.

1.1 Save public and private key files.

1.2 Edit public key file according to this page. Basically 3 changes:

Delete the first two and the last line
Join the remaining lines into one single line
Insert ssh-rsa keyword (with one trailing space) in front of the single line

2. Upload public key file to ~/.ssh/authorized_keys, this is the default path for public key file, create .ssh if the directory doesn’t exist. Append public key file content if the file authorized_keys already exists.

3. Edit /etc/ssh/sshd_config
Uncomment or add the following lines:
RSAAuthentication yes
PubkeyAuthentication yes
AuthorizedKeysFile .ssh/authorized_keys
PasswordAuthentication no
ChallengeResponseAuthentication no

Reload sshd
Note: make a test login before reload the sshd config, you may be locked out.
# /etc/rc.d/sshd reload

4. Connect with putty, specify private key in “Connection > SSH > Auth > Browse…”

Upgrading PHP 5.2 to 5.3 – Bad Move

KP — Sat, 04 Sep 2010 22:42:51 PDT

Update: please see my new post: PHP 5.3 Isn’t That Bad

MediaWiki doesn’t like PHP 5.2.8, I had to upgrade PHP to install MediaWiki. PortUpgrade upgraded PHP to 5.3.3 successfully, but it got problems while upgrading PHP modules, it’s mainly caused by the change of PCRE package which became part of PHP on FreeBSD since 5.3. If you need to upgrade to PHP 5.3.x, make sure you read /usr/ports/UPDATING and this thread.

Basically, PCRE needs to be included explicitly when compile PHP, otherwise other packages can’t be compiled due to the missing /usr/local/include/php/ext/pcre/pcre.h. Add the following line to /etc/make.conf if you use ‘make install’, or to /etc/pkgtools.conf if you use portupgrade.

WITH_BUNDLED_PCRE=”YES”

Although I made it work finally, I really don’t like the result. PHP5.3.3 is quite different from its predecessors, I got huge amount of log errors – it’s a pain if you are running all kinds of third party scripts on your server. So far, I think the latest version of PHP 5.2 is a much better solution unless you only run home-made scripts, or all your third-party scripts have been tailored for PHP5.3.

I have all kinds of very old scripts on my server and have no plan to upgrade some of them in the foreseeable future.

The odd thing is that error_reporting didn’t seem to suppress any PHP warnings or deprecated errors regardless of any options, the error logs are so crazy that I had to turn off log_errors, obviously this can only be a temporary solution, I may downgrade PHP soon.

Disable Apache Log Files

KP — Wed, 11 Aug 2010 02:02:32 PDT

Today I was overwhelmed by all kinds of error messages caused by insufficient storage space, this happened once 5 years ago. Since I linked /var/db/ and /var/log/www/ to another big partition, /var usually only uses a few hundred MB and should never use up its 10GB space. It turned to be the master Apache log file (/var/log/httpd-access.log).

I use Google Analytics on many websites, Apache access log files are no long needed, so I disabled them by removing CustomLog lines in httpd.conf, but it’s not the right way, Apache started to write the log info to its master log file instead, which grew up to 9GB over the months.

The correct way to disable Apache log files is changing CustomLog line to:

CustomLog /dev/null combined

Error log files should be kept anyway.

The daily run output email contains the output of command ‘df’, but I didn’t notice the changes.

Welcome to Nginx! What’s This?

KP — Fri, 23 Jul 2010 04:57:27 PDT

When I visited one of my sites, I got a page with only the bold and big text:

Welcome to Nginx!

It looked like my site was hacked, actually I was pretty sure about it at that moment. I almost wanted to roll out my backup, fortunately it’s back to normal 20 minutes later, then I thought it might be my computer’s problem. After some digging, it turned out to be my ISP’s problem, somehow my ISP treated my site as an invalid domain and displayed their own search engine (evil), but their own site didn’t work, hence the default page from their proxy server.

Key Based Authentication in SSH

KP — Thu, 04 Mar 2010 12:27:49 PST

I just changed my SSH authentication from password to key-based. If you are using password authentication, I strongly suggest you change it, it’s a must, the setup is also very easy, there is no any reasons you don’t do this. With all kinds of buggy programs and scripts, our servers are much much weaker than we thought.

Edit: Wrote a step by step guide for easy reference.

Tar: extract one or more files from an archive

KP — Sun, 21 Feb 2010 21:49:31 PST

To extract a single file from an archive:
# tar zxvf tarfile.tgz –include=”desiredfile”

Use patterns:
# tar zxvf tarfile.tgz –include=”desiredfile*”

‘desiredfile’ must use full path. To check file’s fullpath:

# tar tf tarfile.tgz

Note: It’s double ‘-’ before include.