“The Debian project is pleased to announce the third update of its stable distribution Debian GNU/Linux 5.0 (codename lenny). This update mainly adds corrections for security problems to the stable release, along with a few adjustment to serious problems.

Please note that this update does not constitute a new version of Debian GNU/Linux 5.0 but only updates some of the packages included. There is no need to throw away 5.0 CDs or DVDs but only to update via an up-to-date Debian mirror after an installation, to cause any out of date packages to be updated.

Those who frequently install updates from security.debian.org won’t have to update many packages and most updates from security.debian.org are included in this update.

New CD and DVD images containing updated packages and the regular installation media accompanied with the package archive respectively will be available soon at the regular locations.”

Release Announcement: http://www.debian.org/News/2009/20090905

“Amazon VPC enables you to use your own isolated resources within the AWS cloud, and then connect those resources directly to your own datacenter using industry-standard encrypted IPsec VPN connections. With Amazon VPC, you can:

• Create a Virtual Private Cloud on AWS’s scalable infrastructure, and specify its private IP address range from any block you choose.
• Divide your VPC’s private IP address range into one or more subnets in a manner convenient for managing applications and services you run in your VPC.
• Bridge together your VPC and your IT infrastructure via an encrypted VPN connection.
• Add AWS resources, such as Amazon EC2 instances, to your VPC.
• Route traffic between your VPC and the Internet over the VPN connection so that it can be examined by your existing security and networking assets before heading to the public Internet.
• Extend your existing security and management policies within your IT infrastructure to your VPC as if they were running within your infrastructure.”

Besides the regular ec2 prices, we will have to pay for the VPN connection ($0.05 per VPN Connection-hour) and for the data transfer using the VPN tunnel ($0.10 per GB IN – and starting with $0.17 per GB OUT).

This is a great new service from Amazon that takes the cloud computing offerings to a new level. And this just when major competitors thought they were gaining on Amazon with their offerings, Amazon continues to innovate and launch great new services like this one .

The topics are as follows:

• Introduction
• Installation and Configuration
• Using the Nagios Web Interface
• Overview of Nagios Plugins
• Advanced Configuration
• Notifications and Events
• Passive Checks and NSCA
• Monitoring Remote Hosts
• SNMP
• Advanced Monitoring
• Extending Nagios

Nagios is a real veteran in network monitoring with the first version released more than 10 years ago in March 1999. Some would say that not much has changed during the past few years in Nagios, but even so nagios is still the top choice for network professionals and sysadmins when it comes to build a monitoring solution for their infrastructure. The changes that the latest branch 3.x has brought to the table are subtle ones (like the interface is still the same old cgi one), but Wojciech does a good job in showing how we can take advantage of those to better design our nagios installation (using macros, inheritance, dependencies, etc.)

As the book is primary targeted towards new users looking into nagios, it does start by presenting all the usual stuff you would expect: starting with the installation of nagios on all major linux distros (rpm and deb based ones) from OS packages or by compiling it from sources. Next the author shows us how to configure the nagios daemon and start using the web interface. All basic nagios objects and concepts are covered: hosts, services, commands, time periods, contacts. Even from the first part of the book Wojciech will layout a good structure describing how to use templates and object inheritance in a smart way.

The next part presents the real strength of nagios: its plugins. The standard plugins are all presented in great detail, but also it describes how we can find other plugins and use them or even write our own plugins. Also we will learn the basic ways to monitor remote systems using SNMP, NSCA, NRPE or over ssh and where each of these methods is useful.

Chapter 5 “Advanced Configuration” and 6 “Notifications and Events” can be very useful even to experienced nagios sysadmins. You might find yourself trapped using some old configurations and after reading these chapters you will wonder why haven’t you done it like that a long time ago (probably this was not supported at that time ). Basically the author describes very good ideas on how to layout your configuration files so they can scale well and also to be easily automated by external tools. Also dependencies and inheritance if defined in a smart way could save a lot of the configuration and give your setup a better clarity. Notifications, escalations and adaptive monitoring are described in great detail also. The “Distributed Monitoring” part from Chapter 10 should be very useful to people that need to scale nagios beyond a single monitoring machine using several monitoring machines (master, slaves) with real examples. Since nagios doesn’t provide such a functionality out of the box (master/slave) this is quite useful and even if you will not use it quite like that, it might give you some good ideas on how nagios can be scaled.

Overall this is a great book that both new users and advanced nagios admins can benefit from. Having a good monitoring system in place is vital for most organizations these days, and if you choose to use nagios, this book is definitely a good place to start when building your nagios system.

The Amazon documentation explains how to get this information, basically just by using simple http get requests on the ip: 169.254.169.254, like for ex (for the metadata index):
curl http://169.254.169.254/latest/meta-data/
or for the custom data:
curl http://169.254.169.254/latest/user-data

Eucalyptus supports this great feature (starting with v1.4), but we obviously need to target a different ip to retrieve this information (as the amazon ip has nothing to do with our internal cloud ). We need to use the cloud controller IP for the request and the port it is bound (by default 8773 if you have not changed it). This will look like this (you need to run it from inside the actual instance):
curl http://<CC_IP>:8773/latest/meta-data
block-device-mapping/
security-groups
ami-manifest-path
ancestor-ami-ids
public-keys/
reservation-id
ramdisk-id
public-keys/0/
ami-launch-index
kernel-id
instance-type
local-hostname
local-ipv4
hostname
product-codes
public-ipv4
instance-id
public-hostname
ami-id
placement/

For the user data we can start the eucalyptus instance with:
ec2-run-instances <EMI> -d "myhostname" ...other params...
and later we can then retrieve that information from inside the instance using:
curl http://<CC_IP>:8773/latest/user-data
myhostname

While the documentation for Eucalyptus is getting better with every new version there are still missing parts like this one, and hopefully people looking for this information will find this article useful.

Ok, for this particular upload I am sync’ing a few millions files in folders structured like this:
000/000/files..
000/001/files..
…
999/999/file…
the process was taking days with a single s3sync running, so I just put up a small script to run several toplevel folder s3sync’s at the same time. This reduced the time a lot and was a good walkaround for our problem. Here is the script used, in case it might help others:

#!/bin/bash

cd /source_top_folder

id=0
while [  $id -lt 999 ]; do
        sleep 10
        echo "."
        running=$(ps -ef | grep s3sync | grep ruby |wc -l)
        if [ $running -lt 20 ]; then
                lid=`printf "%03d" $id`
                echo "starting a new s3sync - $lid"
                /usr/bin/s3sync -p --no-md5 -v -s -r --progress --delete ./$lid/ my_bucket:$lid/ &
                let id=id+1
        fi
done

This will basically run 20 s3sync instances and start a new one everytime it is needed (if total running go bellow 20). I realize this is not perfect, but it has done its job for us for this particular project. Ideally s3sync would be able to run several parallel upload threads to be much faster, but until then you might use a similar solution if you have such a problem .

Varnish is a state-of-the-art, high-performance HTTP accelerator. It uses the advanced features in Linux 2.6, FreeBSD 6/7 and Solaris 10 to achieve its high performance. I will not go over the installation of varnish here, but I would highly recommend to use the latest version available at this time 2.0.4 as older versions have various issues.

We could try to use something simple like this in a varnish vcl:

backend s3 {
   set backend.host = "my_bucket.s3.amazonaws.com";
   set backend.port = "80";
}

sub vcl_recv {
   if (req.url ~ "\.(css|gif|ico|jpg|jpeg|js|png|swf|txt)$") {
     set req.backend = s3;
     lookup;
   }
}

but unfortunately this will not work. The Amazon S3 servers will look into the hostname passed by the request and this will most likely be different than the amazon bucket (something like static.mydomain.com) and hence will return 403 on any such request.

There are several solutions to make this work correctly, and the first one I will present is going to insert the bucket name in the actual url passed to the S3 backed. This looks like:

backend s3 {
   set backend.host = "s3.amazonaws.com";
   set backend.port = "80";
}

sub vcl_recv {
   if (req.url ~ "\.(css|gif|ico|jpg|jpeg|js|png|swf|txt)$") {
     set req.url = regsub(req.url, "^", "/my_bucket");
     set req.http.host = "localhost";
     set req.backend = s3;
     lookup;
   }
}

this will work fine, inserting the bucket name in the actual url passed to the backend. Still I don’t like this solution very much as it changes the consistency between the urls (direct one and the forwarded one) so here is a much better solution:

backend s3 {
   set backend.host = "s3.amazonaws.com";
   set backend.port = "80";
}

sub vcl_recv {
   if (req.url ~ "\.(css|gif|ico|jpg|jpeg|js|png|swf|txt)$") {
     set req.http.host = "my_bucket.s3.amazonaws.com";
     set req.backend = s3;
     lookup;
   }
}

As we can see, we are setting the http host the the one Amazon S3 servers would expect for our bucket. So we can keep the same url and don’t mess with the actual link we are passing.

A complete varnish vcl configuration to use with the Amazon S3 backend might look like this:

backend s3 {
.host = "s3.amazonaws.com";
.port = "80";
}

sub vcl_recv {
    if (req.url ~ "\.(css|gif|ico|jpg|jpeg|js|png|swf|txt)$") {
        unset req.http.cookie;
        unset req.http.cache-control;
        unset req.http.pragma;
        unset req.http.expires;
        unset req.http.etag;
        unset req.http.X-Forwarded-For;

        set req.backend = s3;
        set req.http.host = "my_bucket.s3.amazonaws.com";

        lookup;
    }
}

sub vcl_fetch {
    unset obj.http.X-Amz-Id-2;
    unset obj.http.X-Amz-Meta-Group;
    unset obj.http.X-Amz-Meta-Owner;
    unset obj.http.X-Amz-Meta-Permissions;
    unset obj.http.X-Amz-Request-Id;

    set obj.ttl = 1w;
    set obj.grace = 30s;
} 

If you found this post interesting, stay tuned for future posts on varnish and how to use it in more complex setups .

Normally on modern linux distributions this is by default set to 022 and you can easily find out what it is on your system by running the umask command:
umask

Contrary to what you might think, this is not enough to have this working for all applications and daemons on the system. This works fine for any files created from a shell session, but the files created by other processes, like the web server for example, will still use the default, unless otherwise configured. In order to have apache use a different umask we can define this inside /etc/apache2/envvars (debian, and ubuntu systems) or /etc/sysconfig/httpd (rhel,centos systems) like this:
umask 002
and restart apache to enable it.

Other daemons will have different locations where you can define this to overwrite the default setting for umask (check their documentation if you are unsure).

“The Debian project has decided to adopt a new policy of time-based development freezes for future releases, on a two-year cycle. Freezes will from now on happen in the December of every odd year, which means that releases will from now on happen sometime in the first half of every even year. To that effect the next freeze will happen in December 2009, with a release expected in spring 2010. The project chose December as a suitable freeze date since spring releases proved successful for the releases of Debian GNU/Linux 4.0 (codenamed “Etch”) and Debian GNU/Linux 5.0 (“Lenny”).”

This doesn’t mean that we will have a time-based release as for example Ubuntu does on a specific date, but it means that we will have a time-based freeze for each new release (in December of every odd year); the release will still become stable “when it is ready”, but after this, we can expect the new releases in general sometimes in the spring of the every even year.

“Time-based freezes will allow the Debian Project to blend the predictability of time based releases with its well established policy of feature based releases. The new freeze policy will provide better predictability of releases for users of the Debian distribution, and also allow Debian developers to do better long-term planning. A two-year release cycle will give more time for disruptive changes, reducing inconveniences caused for users. Having predictable freezes should also reduce overall freeze time.”

This new approach will leave a very short time for the next release Debian 6.0 (“Squeeze”), that will be freezed later on this year (lenny was released earlier this year in February). Here are the major release goals for squeeze: multi-arch support, which will improve the installation of 32 bit packages on 64 bit machines, and an optimised boot process for better boot performance and reliability.

Here are the members of the configuration management panel (from left to right):

• Igal Koshevoy of AutomateIt
• Brendan Strejcek of Cfengine
• Luke Kanies from Reductive Labs for Puppet
• Narayan Desai of bcfg2
• Adam Jacob from Opscode for Chef

Luckily the video of the session (among other videos from Open Source Bridge) was published and anyone can see this great event:

Now, after I sow this I must admit that I was hoping for a little more engagement and controversy. Instead we sow a friendly debate where everyone presented his own tool, without trying to go over the line and tell why it is better than the one of someone else (we have definitely seen several such blog posts from them in the past ). Anyway this was a great event and a great opportunity to have all the major people in this field come together and share their story. I’m sure that after this they will get back to work, we will see new features and improvements in their tools.

I’ve been asked to review this site and give my impressions about it. The truth is that I don’t have much experience with shared hosting as most of my experience is with dedicated servers from various hosting companies, and anytime I had a friend asking about where do I recommend him to host his small site I didn’t knew where to direct him. This is why I thought that such a webhosting directory as FindMyHosting would be a great start for anyone looking for a shared hosting account to host his new site. We can search from a long list of hosting company and get them ranked by users reports (nice).

We can easily search for the best hosting plans by:

• price
• country (would be nice to see some from EU, not only from the US, etc.)
• platform (linux, win, etc.)
• disk space
• data transfer

The hosting directory also lists various plans by their programming languages and features support like:

• FrontPage Web Hosting
• PHP / MySQL Web Hosting
• ASP Web Hosting Plans
• JSP Web Hosting Plans
• ColdFusion Web Hosting Plans

Besides the searchable database of hosting plans FindMyHosting is also providing some very good introductory articles for people new to hosting that can help them better understand this industry and make a better decision on finding their first host.

Conclusion

FindMyHosting is a webhosting directory that can help people find the right hosting plan and hosting company. This is mainly restricted to shared hosting (even if you can see some dedicated server entries you should not rely on that list) and mainly from hosting companies from the States. I would suggest to bring in more hosting companies and their offering from all over the world (Europe for ex., but any country really); normally finding a good host in US is much easier than in other places . Also personally, I would rather remove the dedicated server section as that can be confusing to new users in the domain, or if not try to add some serious companies and be a real directory for dedicated servers also.