Managing your Puppet infrastructure with Puppet is a bit of a chicken-and-egg scenario. This article describes how I bootstrap a server into becoming a Puppet Master server.

Table of Contents

• A Note About the Puppet Module
• The Puppet Master
  • First Steps
  • Bootstrap Files
• Agent Bootstrapping

A Note About the Puppet Module

I used to use my own fork of the puppetlabs/puppet module but have recently discovered the puppetlabs-operations/puppet module. This article uses my own fork of the operations module. Since this module is fairly new to me, there might be errors. Please let me know if you notice any.

The Puppet Master

This section will describe the creation of the Puppet Master server. The Puppet Master will be hosted on Ubuntu 12.04. It will use PuppetDB for stored configuration and Apache/Passenger for the web service.

First Steps

If you’re at a command prompt on a recently provisioned server, first ensure all packages are updated:

Next, install some basic components:

Once this is done, you’ll have access to Puppet 2.7.11. Upgrading to a newer version will be covered later on.

Next clone the puppet module:

Bootstrap Files

Once the repo is cloned, you can utilize a few supporting scripts to help with boostrapping. These scripts can be found here.

The first bootstrap script to run is bootstrap.sh. If you look at the contents of this file, it will simply checkout a collection of other Puppet modules as well as make a production and development directory structure for manifests of those environments.

Once bootstrap.sh has finished, you can optionally apply another manifest to gain access to updated versions of Puppet by way of a custom apt repository:

Creating your own apt repository is covered in this article.

Next, open pmaster.pp in an editor. Review the settings and change anything necessary. If you require more configuration than what is given, please review the puppet module.

Once edited to suit your environment, apply it:

This manifest should fail at the very end due to Apache being unable to start. This is because Ubuntu will automatically start the puppetmaster server as soon as the package is installed. In my opinion, it’s much easier to just stop the puppetmaster service and then re-apply the manifest to check for any more errors:

(Note: the puppet module disables the puppetmaster service from starting at boot by editing the /etc/default/puppetmaster file)

Once that’s done, install and configure PuppetDB:

I’ve been unable to figure out the right logic to install both puppetmaster and puppetdb in a single manifest and single puppet run. It’s easy enough to just apply two manifests and be done with it.

pmaster-puppetdb.pp might fail at the end. This is most likely because PuppetDB takes a minute or two to fully launch. If pmaster-puppetdb.pp fails, wait a minute and try again — chances are it will work the second time.

That’s it. You should now have a fully functional Puppet Master service. You can verify this by running:

and not get any errors.

Agent Bootstrapping

There are a few different ways to bootstrap an agent. If you’re running in a virtual environment, you can make a VM image that has the Puppet agent installed and configured. By using this image, all of your new VMs will launch Puppet on boot and connect to the configured Puppet Master service.

Another way, and this works with bare-metal provisioning, is to install the Puppet agent as a post-install step. For example, using Debian/Ubuntu preseed, this might look like:

puppet.conf is a generic Puppet configuration file that you can make publicly accessible.

Conclusion

Even though you’ll rarely create a Puppet Master service once you have one up and running, it’s good practice to have a stable, repeatable procedure to ensure it’s done correctly. Puppet can be used to create this procedure, but it takes a bit of CLI bootstrapping, too.

Sébastien Han posted a good article on how to use rsyslog with OpenStack. Coincidentally, I recently configured rsyslog for an OpenStack project and figured I’d publish my configuration, too.

Table of Contents

• Configuring rsyslog
  • Configuring the Server
  • Configuring the Client
• OpenStack and syslog
• Sorting and Organizing

Configuring rsyslog

I have three puppet manifests that I use to configure rsyslog:

• base.pp
• client.pp
• server.pp

base.pp is simply declared in both client.pp and server.pp.

Configuring the Server

server.pp is used to configure a server to be an rsyslog-server. To apply it, simply do:

Although that seems simple enough, the heavy work is in the two files that the manifest uses:

• cloud-rsyslog.logrotate is a script used by logrotate to, of course, rotate the logs.
• server.conf.erb is an rsyslog configuration file that has a tremendous amount of stuff going on. I’ll describe this later on.

Configuring the Client

Client configuration is just a simple as server configuration:

OpenStack and syslog

As Sébastien described, each OpenStack component can be configured to send their logs to syslog. You can see this being done in the following manifests (search for “syslog”):

• compute node
• cloud controller

For Swift, the proxy-logging middleware sends the logs to syslog.

Sorting and Organizing

When an OpenStack component sends a log to the rsyslog server, the log is passed through the filters in the server.conf file. Based on if the log is from Swift (special case), the log facility, and the hostname, the log is directed to a specific log file in a specific directory.

I have logs going into the following directories:

• /var/log/rsyslog/%HOSTNAME%/nova.log: all logs related to nova from that host.
• /var/log/rsyslog/%HOSTNAME%/glance.log: all logs related to glance from that host.
• /var/log/rsyslog/%HOSTNAME%/cinder.log: all logs related to cinder from that host.
• /var/log/rsyslog/%HOSTNAME%/keystone.log: all logs related to keystone from that host.
• /var/log/rsyslog/%HOSTNAME%/swift.log: all logs related to nova from that host.
• /var/log/rsyslog/%HOSTNAME%/syslog.log: all other logs from that host.
• /var/log/rsyslog/nova.log: all nova logs from all hosts.
• /var/log/rsyslog/swift.log: all swift logs from all hosts.
• /var/log/rsyslog/cinder.log: all cinder logs from all hosts.

Conclusion

rsyslog provides a great deal of flexibility when it comes to logging. Puppet, as usual, comes to the rescue in making the complexities of such services manageable.

While this solution works for me, I have plans to revisit OpenStack logs in combination with logstash in the future.

This article will explain various methods of controlling apt (the Debian and Ubuntu package manager) across several servers with Puppet and the benefits of doing so.

Table of Contents

• Caching apt Packages
  • apt-cacher-ng
• Mirroring apt Repositories
  • apt-mirror
• Creating Your Own apt Repository
  • reprepro
  • Making Your Repository Available
• Use-case: Puppet 2.7.x
• Conclusion

Caching apt Packages

Each time apt downloads a package, it is locally cached on the server until you explicitly clear the cache. This gives you the ability to reinstall the package without having apt download it again.

On individual servers, this works fine. But what about groups of several servers? If, for example, you ran the following on 15 different servers to install tmux:

tmux would have been downloaded from a public server 15 times. This is very inefficient — especially if the package was significantly large.

A solution to this is to designate a server to be an apt proxy server. All other servers will go through this server to download any packages. If the package does not exist on the apt proxy server, it will be downloaded from a public server. Now when tmux is installed on 15 different servers, it is only downloaded from a public server once.

apt-cacher-ng

The best application for apt proxying is apt-cacher-ng.

To set up acng by using Puppet, either use the puppet module tool or download the module directly from github:

or

Next, apply the following class to the “apt server” (the “apt server” is the server that will provide all apt services throughout this article):

When acng is installed, it will automatically use the contents of your sources.list file as the upstream apt repository. This will work for 90% of use-cases.

On every other server that will use the apt server as a proxy, apply the following manifest:

Mirroring apt Repositories

acng is useful if your servers use the same subset of packages. If you find that your servers are downloading a wide variety of packages — maybe even most of the available packages in a repository — it would make more sense to just mirror the entire repository instead of letting acng inevitably do it.

apt-mirror

apt-mirror is an application that provides an easy way to mirror one or more repositories. To configure with Puppet, first install the apt_mirror module:

or

Next, apply the base manifest:

The apt_mirror module comes with a defined type to add individual mirrors. For example, to mirror main and contrib repositories for Ubuntu Precise, use the following:

apt_mirror installs a cron entry that will run once a day. It creates new mirrors that do not yet exist or updates existing mirrors with any changes.

In order to fully utilize the mirror, you will need to make it accessible to all of your servers. I recommend following this article for instructions on how to do this manually. I’ll also describe how to do this with Puppet later in this article.

Creating Your Own apt Repository

There are several reasons to create your own apt repository:

• To distribute your own packages
• To limit the packages available to servers
• To add packages not available in the standard repositories

reprepro

Creating an apt repository is not the most straightforward task. The following page contains a very long list of tools to assist. The most popular tool seems to be reprepro which is what I have chosen to use.

For a great overview on how to use reprepro, please see this article.

To configure reprepro with Puppet, first install the module:

or

The following manifest will install reprepro and set up the reprepro user:

One reprepro is installed, GPG will need configured. Modern apt versions use GPG to sign both packages and files that maintain the apt repository. In my opinion, it is difficult to use Puppet to maintain GPG keys, so I recommend doing this part outside of Puppet.

First, change to the reprepro user:

Next, generate a GPG key:

Follow and answer the prompts. I recommend not using a passphrase. Although it’s insecure to do such a thing, GPG is just being used to sign arbitrary apt repository files — you can always delete the repository and re-create it if you feel your key has become compromised.

Once the key has been created, export it and store it in the reprepro Puppet module:

Now to continue with Puppet. The reprepro Puppet module has the ability to crate and maintain several apt repositories. Each repository contains one or more distributions. For most cases, and for the purpose of this document, only one of each will be created.

First, the repository:

And next the distribution:

(The sign_with value can be obtained by doing gpg --list-keys)

Once these manifests have been applied to the apt server, you can begin adding packages to your custom repository. The reprepro Puppet module installs a cron entry that will monitor the $basedir/$repository/tmp/$distribution directory for any *.deb files. If it finds any, it will add the files to the repository and clean the tmp dir out.

Making Your Repository Available

Now that you have an apt repository with packages, you will want all of your servers to be able to download those packages. You’ll need to make your repository publicly accessible in order to do this. The easiest way is by installing a web server like Apache:

Next, create a virtual host that uses your repository as its DocumentRoot:

Also make sure your servers can access your GPG key that you generated:

Finally, tell your servers about your repository. You can use the Puppet apt module to do this:

And use the following manifest on each server:

Now you should be able to perform an apt-cache search on each server for packages in your repository.

Use-case: Puppet 2.7.x

One creative way of using both apt-mirror and reprepro is to ensure your servers are running the latest version of Puppet, but keep them at the 2.7.x versions.

Puppetlabs maintains their own apt repository at http://apt.puppetlabs.com. It’s a great repository and it’s always up to date. However, it makes no distinction between the 2.7 and 3.0 branches of Puppet. This means that unless you specifically specify a 2.7.x version of Puppet, you will get a 3.0 version.

Personally, I’m not ready to work with 3.0, but at the same time, I want my servers to have the latest 2.7 version.

The first step in easily doing this is to mirror the Puppetlabs apt repository:

Once your apt server has mirrored the repository, navigate to the mirror’s location (most likely /var/spool/apt-mirror/mirror/apt.puppetlabs.com):

Inside this directory, navigate further to pool/precise/main:

This directory will contain several directories with a title of a single letter. These directories will contain subdirectories of various Puppet-related packages. For example:

• f/facter
• h/hiera
• p/puppet
• p/puppetdb

Sort through these directories and find the latest 2.7.x versions of each. Copy them to your /var/lib/apt/repo/$repository/tmp/$distribution directory. cron will pick them up and add them to your personal repository.

Now all servers will have access to the latest 2.7.x version of Puppet without having to specify a specific version. Whenever a new 2.7.x release comes out, apt-mirror will retrieve it within 24 hours. Just copy the new version to /var/lib/apt/repo/$repository/tmp/$distribution and it will then be added to your personal repository.

Conclusion

This article described several different ways of configuring apt to help control how groups of servers access packages. Implementing these methods yields benefits of faster downloads as well as limiting or widening access to packages when needed. By using Puppet, you can easily configure a central server to host these roles or configure groups of servers to access the designated apt server.

This article will be a step-by-step guide of how to set up a Cobbler server on CentOS 6. Once the server is complete, you will be able to have CentOS 6 automatically installed onto client computers when they are PXE booted on a private network.

Table of Contents

• Server and Environment Attributes
• Installation
  • Network
  • Package Selection
• Post-Install Steps
  • Disable the Firewall
  • Disable SELinux
  • Perform a Software Update
  • Install VMWare tools
  • Install EPEL
  • Enable NAT
  • Install and Configure Cobbler
  • Add a Distro to Cobbler
  • Generate a Base Kickstart File
  • Create a Cobbler Profile
  • Create a Cobbler System
• Extra: Polipo
  • Install and Configure Polipo
• Conclusion

Server and Environment Attributes

I used a VMWare virtual machine for the server. The VMWare environment has two networks: a public and private. The Public network had full access to the Internet while the private was a standard host-only network.

The VMWare virtual machine had two NICs — one for the public and one for the private.

The purpose of the two networks is to allow clients to initially boot up in the private network where Cobbler will handle the installation. Since Cobbler utilizes DHCP and TFTP, making these services listen on a Private network will ensure that they don’t interfere with any other DHCP or TFTP server on the Public network. Once the installation has finished, the client can be moved to the Public network.

Installation

Use any CentOS 6 installation medium you wish.

Network

For the Network Configuration, I renamed System eth0 and System eth1 to Public and Private respectively. I configured each NIC with static information and made sure to check the Connect Automatically checkbox for each — otherwise NetworkManager will not bring the NICs up.

Package Selection

I chose Minimal Desktop.

Post-Install Steps

Disable the Firewall

It’s possible to have the firewall running — you just need to ensure that the proper ports are opened. For simplicity, and since this server is in a private network, I chose to have the firewall turned off.

You can disable the firewall by doing:

$ system-config-firewall-tui

and unselecting “Enabled”

Disable SELinux

I tried to have Cobbler run properly with SELinux, but in the end, I could not get the two to work together. If anyone has success with this, please let me know.

You can disabled SELinux by editing the /etc/sysconfig/selinux file and changing:

SELINUX=enforcing

to

SELINUX=disabled

and then rebooting.

Perform a Software Update

Before proceeding further, make sure everything is up to date:

$ yum update

Install VMWare tools

This step is only required if you are using VMWare.

Create the file /etc/yum.repos.d/vmware.repo with the following contents:

[vmware-tools]
name=VMware Tools for Red Hat Enterprise Linux $releasever – $basearch
baseurl=http://packages.vmware.com/tools/esx/4.1/rhel6/x86_64
enabled=1
gpgcheck=0

Change the x86_64 to i686 if you need to.

Next, run:

$ yum install vmware-tools

Install EPEL

The EPEL RPM repository contains extra packages, such as Cobbler, for RHEL/CentOS.

Run the following:

$ rpm -Uhv http://download.fedora.redhat.com/pub/epel/6/i386/epel-release-6-5.noarch.rpm

Enable NAT

The following steps will turn the server into a network gateway which will allow clients on the Private side to reach the Internet through the Public side. This allows for network-based installs.

Add the following IPTables rules:

$ /sbin/iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
$ /sbin/iptables -A FORWARD -i eth0 -o eth1 -m state --state RELATED,ESTABLISHED -j ACCEPT
$ /sbin/iptables -A FORWARD -i eth1 -o eth0 -j ACCEPT

Next, ensure these rules are saved:

$ /etc/init.d/iptables save
$ /etc/init.d/iptables restart

Finally, edit /etc/sysctl.conf:

net.ipv4.ip_forward = 1

If you do not want to reboot for this to take effect, run:

$ echo 1 > /proc/sys/net/ipv4/ip_forward

Install and Configure Cobbler

Finally it’s time to install Cobbler.

$ yum install cobbler cobbler-web pykickstart

Ensure some required services will start at boot:

$ chkconfig httpd on
$ chkconfig xinetd on
$ chkconfig cobblerd on
$ service httpd start
$ service xinetd start
$ service cobblerd start

For this setup, I will use dnsmasq for DNS and DHCP. The following changes need made in /etc/cobbler/modules.conf

[dns]
module = manage_dnsmasq

[dhcp]
module = manage_dnsmasq

Edit /etc/cobbler/dnsmasq.template and add the following:

server=192.168.1.1
no-dhcp-interface=eth0

Make sure server is set to your upstream DNS resolver. The no-dhcp-interface will ensure the DHCP server does not run on the Public network.

Also change dhcp-range to an appropriate IP range for your Private network.

Finally, restart Cobbler and have it tell you if anything else is needed to be configured:

$ /etc/init.d/cobblerd restart
$ cobbler check

When everything looks good, run

$ cobbler sync

Add a Distro to Cobbler

For this configuration, I will use a CentOS 6 netinst ISO.

First, mount the ISO:

$ mount /dev/cdrom /mnt

Next, import it into Cobbler:

$ cobbler import cobbler import --name=CentOS6 --path=/mnt

This will return an error. This is OK since it is not a full distribution ISO.

Finish the import:

$ cobbler distro add --arch=x86_64 --breed=redhat --name=CentOS6 --initrd=/var/www/cobbler/ks_mirror/CentOS6/isolinux/initrd.img --kernel=/var/www/cobbler/ks_mirror/CentOS6/isolinux/vmlinuz

Generate a Base Kickstart File

In order to perform an automated install, a Kickstart file is needed. You can create an initial file by using the system-config-kickstart utility:

$ yum install system-config-kickstart

This is a graphical utility so you will need to be logged into X.

You can also use my base file. This has been configured to use Cobbler’s network configuration snippets. The encrypted password is just “password”.

Create a Cobbler Profile

Profiles connect Distributions with Kickstart files. Here we can make a standard CentOS6 profile:

$ cobbler profile add --name=CentOS6-Base --distro=CentOS6 --kickstart=/var/lib/cobbler/kickstarts/centos6.ks

Create a Cobbler System

Systems are individual machines complete with IP addresses and MAC addresses. I prefer to use the Cobbler Web Interface to add systems.

If you are using VMWare, create a new virtual machine and either make note of the generated MAC address or generate your own. If you are using a physical server, make note of the MAC. You can then configure your system in Cobbler to use that MAC. This will ensure that any system-specific options (such as IP address) are only configured for that system.

Once the system is added, be sure to sync Cobbler either through the web interface or on the command line:

$ cobbler sync

Now, PXE boot your client on the Private network. DHCP and TFTP should take over and an automated install should begin.

Extra: Polipo

Although it is possible to have Cobbler mirror various repositories, I prefer to only download what is needed. If you configure Kickstart to use a caching proxy server, you can use cached RPMs to install on multiple servers. This way, once the RPMs are cached, future installs will only take a few minutes — no RPMs will need to be downloaded from the Internet.

I am currently using Polipo to handle this since my own automirror will not work with CentOS 6.

Note: As of RHEL6, a new --proxy option is available to Kickstart files to enable the use of a proxy. Before this, if you wanted to use a proxy, you had to use a “tunneling” method as described in automirror.

Install and Configure Polipo

To install Polipo, just do:

$ yum install polipo

To configure, edit the /etc/polipo/config file:

proxyAddress = "::0"
disableIndexing = false
disableServersList = false

Next, create the file /etc/polipo/uncachable with the contents:

\.xml$

This should ensure that XML files are not cached. This way up-to-date RPM repository data is obtained during installs.

Once that’s all done, edit the Kickstart file to utilize the proxy. This can be seen in the sample Kickstart file linked above.

Conclusion

This article walked through the process of setting up a Cobbler server that will be able to provide automatic installations of CentOS 6 clients.

There are two CPAN modules that provide some nice utilities for working with and manipulating lists: List::Util and List::MoreUtils. This article quickly covers a few of these utilities that I have found useful.

Table of Contents

• List::Util
  • List::Util::first
  • List::Util::max
  • List::Util::shuffle
• List::MoreUtils
  • List::MoreUtils::any
  • List::MoreUtils::all
  • List::MoreUtils::uniq

List::Util

List::Util provides a handful of utilities. Some of these are:

List::Util::first

List::Util::first will return the first element in a list that matches a given criteria:

my @list = qw(1 2 3 4 5 6);
my $first = first { $_ > 4 } @list;  # returns 5

List::Util::max

List::Util::max returns the maximum number from a list:

my @list = qw(1 2 3 4 5 6);
my $ = max @list;  # returns 6

List::Util::shuffle

List::Util::shuffle randomizes a list:

my @list = shuffle qw(1 2 3 4 5 6);
say $list[0] # prints random number between 1 - 6

List::MoreUtils

List::MoreUtils, as the name implies, provides some more list utilities.

List::MoreUtils::any

List::MoreUtils::any returns true if any element in the list fits the given criteria:

say "true" if (any { $_ > 5 } qw(1 2 3 4 5 6));

List::MoreUtils::all

List::MoreUtils::all returns true if all elements in the list fit the given criteria:

say "true" if (all {$_ > 0) qw(1 2 3 4 5 6));

List::MoreUtils::uniq

List::MoreUtils::uniq can provide two functions:

• In list context, it will return a list of only unique elements
• In scalar context, it will return the number of unique elements

my @list = qw(1 1 2 3 4 4 4 5 6 6);
my @uniqs = uniq @list;
print Dumper @uniqs; # returns 1 2 3 4 5 6
my $unique_count; = uniq @list;
say $unique_count; # prints 6

Conclusion

As you can see, these two modules provide some nice convenience functions. There are many more to learn, so check out the CPAN page for each.

Over the years, I’ve spent a lot of time with Perl DBI. It’s a great interface for working with databases, but sometimes I find it a little too heavy. A few months ago I found DBIx::Simple and now all is right with the world.

Table of Contents

• DBIx::Simple
  • Return a Single Value
  • Return Multiple Rows from One Column
  • Create a Hash Indexed by a Column
• Conclusion

DBIx::Simple

To start using DBIx::Simple, create a DBI object in a similar manner to normal DBI.

#!/usr/bin/env perl
use strict;
use warnings;
use DBIx::Simple;

my $dbh = DBIx::Simple->connect( 'DBI:mysql:database=foo', 'user', 'password' ) or die DBIx::Simple->error;

Now comes the fun stuff. With DBI, I use a familiar four-step routine:

• Prepare
• Execute
• Fetch
• Finish

While clean and methodical, these four steps can become a pain to use over and over again.

With DBIx::Simple, it can be as easy as:

    for my $row ($db->query('select username, email from users')->hashes) {
        print "Username: $row->{username}, Email: $row->{email}\n";
    }

Here, everything is rolled into one step.

note: You don’t have to combine the Query and Fetch steps — they can be split. I personally like to for the conciseness.

Query results can be returned in a number of different ways. I’ll explain a few ways, but all can be found in DBIx::Simple‘s perldoc or examples doc.

Return a Single Value

If you know that your result will only have a single value, you can do:

my $sum = $dbh->query('select 2 + 2')->list;

Return Multiple Rows from One Column

Building on the previous example:

my @sums = $dbh->query('select 2 + 2 union select 3 + 3')->flat;

Create a Hash Indexed by a Column

This example creates a hash of users with a Key based on the first column and Value based on the second:

my %users = $dbh->query('select id, username from users')->map;

Conclusion

DBIx::Simple is one of my favorite Perl modules. If you frequently use DBI, I would encourage you to take a look at it.

In a previous article, I wrote about how to use pkg-cacher to cache requested RPM files. Since then, the website for pkg-cacher has become unavailable and coincidentally I wrote my own tool that provides the same functionality called automirror. This article describes how to use it.

Table of Contents

• Update – July 2011
• Caching Packages
• Installation
• Using automirror
  • Normal Mode
  • Tunnel Mode
• Conclusion

Update – July 2011

automirror currently does not work with CentOS 6 (and probably RHEL6). It looks like there are some incompatibilities between the HTTP::Proxy Perl module and libcurl (which yum uses).

I have opened a bug report for HTTP::Proxy and hopefully it will be resolved. In the meantime, I would suggest just using a small caching proxy such as Polipo.

Caching Packages

One reason caching can be useful is: if one has 10 servers that require a package update, it makes more sense to download that package once and then distribute it locally to the other servers than to have each server request the remote package. This saves on outside bandwidth as well as makes retrieving the package extremely faster due to the local connection.

automirror‘s caching system is simple: if a server requests a file with a file extension that you have told automirror to watch out for (for example, an .rpm file), it first checks to see if there is a locally available copy. If not, it retrieves the file from the remote source. After retrieving it, it saves a copy. On the next request for the same .rpm file, it will send back the locally saved copy instead of downloading the remote copy again.

Installation

automirror can be downloaded from its BitBucket repository. You can either use mercurial to clone the repository or download a run-of-the mill .zip, .gz, or .bz2 archive.

Inside the archive is the actual automirror script and a sample init script called rc.automirror. This init script has been tested on CentOS.

Install the automirror script by just copying it to any directory:

# cp automirror /usr/local/bin

To install the init script, first edit it and change any settings or variables necessary. You should only have to edit the options in the OPTIONS section.

By default, the EXTENSIONS option is set to .rpm and .img files to support automated RedHat / CentOS installs.

You can install the init script by doing:

# cp rc.automirror /etc/init.d/automirror
# chkconfig --add automirror

You can now start the service by running

# /etc/init.d/automirror start

or

# service automirror start

Using automirror

For reference, the various options to automirror can be read by doing:

# perldoc /usr/local/bin/automirror

or

# /usr/local/bin/automirror --help

But the init script should be all you need to control it.

There are two main ways to use automirror: normal mode and tunneling mode.

Normal Mode

Normal mode is configured by setting TUNNEL TO 0 in the init script. In normal mode, simply configure any application (or global environment variable such as http_proxy) to point to the host and port where automirror is listening. For example if automirror was listening on 192.168.255.1:8080, to configure yum to use automirror as a proxy, edit /etc/yum.conf and add the following:

proxy=http://192.168.255.1:8080

Now when performing any action with yum, it will utilize the proxy and caching service.

Tunnel Mode

Tunnel mode can be turned on by setting TUNNEL TO 1 in the init script.

There are some applications and services, such as Kickstart or an automated run of Anaconda, that do not support a proxy service. You can use automirror in tunnel mode to get around this. For example, if there is an .rpm file located at

• http://some.centos.mirror.org/path/to/package-1.2-3.rpm

you can still download the package via automirror by specifying:

• http://192.168.255.1:8080/some.centos.mirror.org/path/to/package-1.2-3.rpm

In tunnelling mode, automirror will pop the host off of the url and use the next segment as the new host to connect to.

In Kickstart scripts, you can perform an automated install that uses automirror by specifying:

url --url=http://192.168.255.1:8080/some.centos.mirror.org/pub/centos/5/os/i386

Conclusion

automirror is a very flexible proxy script that can perform a variety of caching. This article covered how to install and use automirror to cache .rpm packages.

Anyone who frequently writes utility scripts will eventually run into having to deal with Dates and Time. The type of tasks, though, can vary: sometimes you have to format a date, convert a date, add or subtract time to a date, or find a date. This article covers various ways of having to deal with Dates and Time in Perl.

Table of Contents

• Core Date Tools
• Date::Manip
• DateTime
• Conclusion

Core Date Tools

To start off with, here are some basic ways to deal with dates and time in Perl without the need for external modules. Here’s a basic script:

#!/usr/bin/env perl
use strict;
use warnings;
use 5.010;

# print formatted time
say scalar localtime;

Running the above script will simply print:

$ perl time.pl
Thu Mar 31 16:38:03 2011

How about dissecting the current date/time:

my ($sec, $min, $hour, $mday, $mon, $year, $wday, $yday, $dst) = localtime();

It’s important to note that $mon (for month) is zero-based and $year needs 1900 added to it. So if you want to print 3/31/2011, you need to do:

$mon += 1;
$year += 1900;
say "$mon/$mday/$year";

Dealing with epoch-based time is a frequent occurrence on *nix-based systems:

# get epoch time
my $time = time;

# get yesterday
$time -= 86400;

# get different parts of specified time
($sec, $min, $hour, $mday, $mon, $year, $wday, $yday, $dst) = localtime($time);

The above code gets the current time in epoch-based seconds, subtracts 1 day from it (60 seconds * 60 minutes * 24 hours) and then retrieves the different parts using the localtime function.

You can easily format a date however you want with the POSIX module and the strftime function:

use POSIX qw/ strftime /;

say "Yesterday was ", strftime("%A, %B %dth, %Y", localtime($time));

This prints:

$ perl time.pl
Yesterday was Wednesday, March 30th, 2011

Date::Manip

Sometimes you need to do more advanced date calculations. I like Date::Manip for these tasks.

For example, here’s how to parse a date:

use Date::Manip::Date;
my $date = new Date::Manip::Date;
$date->parse('2011-03-31 12:47:33');
say $date->printf("%F");

The above prints:

$ perl time.pl
Thursday, March 31, 2011

Date::Manip supports a huge array of formats, the above is just a simple example.

If you wanted to find how many days are between two dates, Date::Manip can do that:

my $d1 = new Date::Manip::Date;
my $d2 = new Date::Manip::Date;

$d1->parse('2001-01-01');
$d2->parse('epoch ' . time); # Use keyword 'epoch' for epoch-based time

$delta = $d1->calc($d2, 'approx');
printf("[%s] minus [%s] is [%s] days\n", $d1->printf("%F"), $d2->printf("%F"), $delta->printf("%dyd"));

This prints:

$ perl time.pl
[Monday, January 01, 2001] minus [Thursday, March 31, 2011] is [3743.29875] days

DateTime

As Perl has a huge collection of Date and Time modules, DateTime was created to standardize on a set of functions and way to use them. One thing I do like about Date::Manip is that I only have to install the one module to have access to a wide array of Date and Time utilities. DateTime is a little different, and I can understand the need for that, too.

In order to run the following examples, three separate modules needed installed:

• DateTime
• DateTime::Format::Strptime
• DateTime::Format::HTTP

This example parses and reformats a simple date:

use DateTime;
use DateTime::Format::Strptime;

my $parser = DateTime::Format::Strptime->new( pattern => '%Y-%m-%d' );
my $dt1 = $parser->parse_datetime('2001-01-02');
say $dt1->mdy('/');

This prints:

$ perl time.pl
01/02/2001

If you don’t want to supply your own date format to be parsed, DateParse::Format::HTTP should be able to parse it for you:

use DateTime;
use DateTime::Format::HTTP;
my $dt2 = DateTime::Format::HTTP->parse_datetime('2001-01-02');
say $dt2->mdy('/');

This prints the same result.

This example gets the current time as it is in Helsinki and then adds 17 days to it:

my $dt3 = DateTime->now(time_zone => 'Europe/Helsinki');
say $dt3->datetime;

$dt3->add( days => 17 );
say $dt3->datetime;

The output is:

$ perl time.pl
2011-04-01T01:38:03
2011-04-18T01:38:03

This final example subtracts 5 months from the current epoch time and then prints the resulting year:

my $dt4 = DateTime->from_epoch( epoch => time );
$dt4->subtract( months => 5 );
say $dt4->year;

The output is:

$ perl time.pl
2010

Conclusion

This article covered only a few of the different types of Date and Time manipulations a systems administrator is likely to run into. As you run into different tasks, keep these modules in mind — they’re able to do a lot more than what is described above.

Most scripts or applications involve some type of output and reporting. If the script or application starts growing, the output can become more complex. Mixing the code that formats and displays your output in the same area as the code that performs the logic can become messy. Templating is a popular solution to this problem. In this article I will quickly cover Text::Template.

Table of Contents

• Text::Template
  • The Report Code
  • The Template
  • The Full Output
• Conclusion

Text::Template

Text::Template is a simple to use templating system for Perl. It’s not restricted by output format, so you can use it for text, HTML, XML, or really any type of readable output.

Here’s a quick example that shows how to use Text::Template with a simple report.

The Report Code

This short script emulates a report of suspended users. For the sake of simplicity, I am just referencing the suspended users by a manually created hash. The script then calls the Text::Template function fill_in_file, which plugs in all data to the template (seen later) and then prints the results.

#!/usr/bin/env perl
use strict;
use warnings;
use 5.010;
use Text::Template qw/ fill_in_file /;

my $users = { 
    moe => 'Wed Mar 30 12:23:26 2011',
    larry => 'Tue Mar 29 09:15:55 2011',
    curly => 'Mon Mar 28 20:45:18 2011',
};

my $data = { 
    title => 'Suspended Users Report',
    users => \$users,
};

say fill_in_file('report.tpl', HASH => $data);

The Template

Now here’s the template that will be used, located in a dedicated file called report.tpl. Perl will interpret anything enclosed in { }‘s — this includes variables, functions, and full blocks of code:

{$title}
=======================================
Generated on {localtime}
---------------------------------------

{
    for my $user (keys %{ $users }) {
        $OUT .= sprintf("%-10s %28s\n", $user, $users->{$user});
    }   
}

The use of the $OUT variable is a special feature of Text::Template. In short, it will collect the output of a loop and print the final results to the rendered template.

The Full Output

When the script is run, here is the output:

Suspended Users Report
=======================================
Generated on Wed Mar 30 15:35:09 2011
---------------------------------------

larry          Tue Mar 29 09:15:55 2011
curly          Mon Mar 28 20:45:18 2011
moe            Wed Mar 30 12:23:26 2011

Conclusion

For a simple example such as the one above, including the output and formatting code would not have been that bad. However, as scripts become larger and more complicated, placing the formatting code into a dedicated template file can bring ease of comprehension by more structure. Although Perl has several templating modules available, I’ve found Text::Template to be very flexible and simple to understand.

Logging is an essential tool and feature for System Administrators. With Perl, logging can be as simple as using print, die, or warn, or more advanced such as logging to syslog, a private log file, or even a database. This article quickly covers the more advanced logging.

Table of Contents

• Sys::Syslog
• Log::Log4perl
  • Simple Use
  • Logging to a Logfile
  • Logging to Syslog
  • Log::Log4perl Resources
• Conclusion

Sys::Syslog

Sys::Syslog is a core Perl module. It provides a standard and simple interface to logging to your (*nix-base) server’s syslog.

Here’s a simple example:

#!/usr/bin/env perl
use strict;
use warnings;
use Sys::Syslog;

my $ident = $0; 
openlog($ident, 'pid', 'user');
syslog('info', 'Testing syslog logging');
closelog();

After running this script, you can find the following in /var/log/messages (or other appropriate log file):

Mar 25 13:36:28 server syslog.pl[20075]: Testing syslog logging

The documentation for Sys::Syslog is very well written and I highly encourage reading it — especially “The Rules of Sys::Syslog”.

Log::Log4perl

Log::Log4perl is an extremely robust logging module for Perl. Its use can range from simple logging features to full-fledged application framework logging (including to remote databases). I’ve never been on a project where Log::Log4perl was needed, but wanted to include it in this article as it seems to be the de-facto logging module for Perl.

Simple Use

This example shows how to print logging information to the screen for everything from the INFO level and up:

#!/usr/bin/env perl
use strict;
use warnings;
use Log::Log4perl qw/:easy/;

Log::Log4perl->easy_init($INFO);
my $logger = get_logger();
$logger->info("Testing Log4perl logging");

When this script is run, the following is printed on the screen:

2011/03/25 13:45:52 Testing Log4perl logging

Logging to a Logfile

This next script writes all logs to a logfile rather than on the screen. As you can see, there’s a bit more involved than the previous example — namely the need to begin to configure Log::Log4perl:

#!/usr/bin/env perl
use strict;
use warnings;
use Log::Log4perl;

my $log_conf = q/  
    log4perl.category = INFO, Logfile
     
    log4perl.appender.Logfile = Log::Log4perl::Appender::File 
    log4perl.appender.Logfile.filename = log4perl.log
    log4perl.appender.Logfile.mode = write 
    log4perl.appender.Logfile.layout = Log::Log4perl::Layout::SimpleLayout     
/;

Log::Log4perl::init( \$log_conf );
my $logger = Log::Log4perl::get_logger();

$logger->error("Testing Log4perl logging");

Once this script is run, you should have a log file called log4perl.log in the same directory. The contents will read:

ERROR - Testing Log4perl logging

Logging to Syslog

Log::Log4perl is also able to log to syslog, although it requires an extra module: Log::Dispatch::Syslog. Unless you are doing other logging with Log::Log4perl, I would recommend using the standard Sys::Syslog module:

#!/usr/bin/env perl
use strict;
use warnings;
use Log::Log4perl;

my $log_conf = q/  
    log4perl.category = INFO, SYSLOG
     
    log4perl.appender.SYSLOG = Log::Dispatch::Syslog
    log4perl.appender.SYSLOG.min_level = warn 
    log4perl.appender.SYSLOG.ident = log4perl.pl
    log4perl.appender.SYSLOG.facility = user
    log4perl.appender.SYSLOG.layout = Log::Log4perl::Layout::SimpleLayout
/;

Log::Log4perl::init( \$log_conf );
my $logger = Log::Log4perl::get_logger();

$logger->warn("Testing Log4perl logging");

Once this script is run, you can find the following message in /var/log/messages:

Mar 25 14:00:02 server log4perl.pl: WARN - Testing Log4perl logging

Log::Log4perl Resources

I found the following links to be great resources for Log::Log4perl:

• Log4perl
• 8 Useful Log::Log4perl Recipes
• Retire your debugger, log smartly with Log::Log4perl!

Conclusion

This article covered two Perl modules that deal with advanced Perl logging, Sys::Syslog and Log::Log4perl.