Cron is an essential *nix package that silently runs in the background running scheduled jobs. The most common version of cron is Vixie Cron. Coincidentally Vixie Cron is also the most vanilla version — it doesn’t provide any flashy features that other variants have, just the ability to read from a crontab and run scheduled jobs.

This, of course, is fine for most servers, but if a server is configured to run several cron jobs every minute, the server could easily get overloaded. I recently ran into this problem and rather than replace the standard cron package with something like fcron I decided to hack Vixie Cron for fun to see what I could do myself.

What Changes I Wanted to Make

When cron wakes up every minute, it checks to see what jobs need to be run and executes them all at once. So if there are 100 jobs configured to run, 100 jobs will run at once. I wanted to stop this from happening, so the main feature I wanted to add was for cron to be able to queue jobs.

The second modification is related to the first: while I could hard-code a queue length inside the cron source, I would rather have the administrator be able to specify what the length should be. I wanted to add this feature as a command-line switch.

Disclaimer

I am not an expert at C programming. What follows is simply me fooling around for fun. And while I was able to correctly get this to work, I would highly advise against running this in production. Even I did not.

Also, unfortunately I did this several months ago and have only been able find the time to write about it now. While I was able to trace back my work, hopefully what I dug up is detailed enough.

Tools and Setup

I worked on Vixie Cron 4.1 from CentOS 5.3. The build and test machine was a standard install of CentOS 5.3.

I followed the instructions at Owl River about how to create an RPM patching and development environment. As that page explains how to do this much better than I could, I would recommend just reading it there.

One difference, though, is that RHEL provides 70 patches to the base vixie cron source. I did not realize this when I first made my changes, so when I created my final patch, it did not apply correctly over the other 70. Therefore, please patch the source files with all included patches before proceeding. I applied them in the order that they are listed in the vixie-cron.spec file.

Dissecting Cron

Once I had my environment setup, I proceeded to read through the source code and try to find my way around.

For how widespread and important cron is, vixie cron is very simple.

Adding the Command Line Switch

cron.c is the file that contains the main() function. This file also contains the area where cron parses the command line options. Interestingly, the unpatched version of Vixie Cron does not use a getopt feature — it manually parses the command line options itself. One of the RHEL patches added getopt later.

I decided to use q, for “queuing”, as the switch letter. The original line looked like:

while (-1 != (argch = getopt(argc, argv, "npx:m:"))) {

And my edit changed it to:

while (-1 != (argch = getopt(argc, argv, "npx:m:q:"))) {

Below that, I added my case for the switch statement:

        case 'q':
            NumProcs = atoi(optarg);
            NumProcs += 1;
            break;

Of course, with C, all variables have to be pre-declared, so somewhere at the top, I added:

NumProcs = -1;

I later found I also had to add a line to globals.h to make NumProcs global to all files:

XTRN int       NumProcs INIT(0);

With these changes, my new command line switch was in place. If an administrator wanted to make cron only run 2 jobs in the queue at a time, he or she would specify:

crond -q2

Now it’s time to add the queueing feature.

Queuing

job.c is the file which controls the job execution. The function job_runqueue() is where the execution actually happens. It’s here where you can see why the jobs all run at once. There is no timing control in the for loop — the job queue is looped and all jobs are fired off as fast as the CPU can run.

I did some quick reading on UNIX processes and forking and came up with the following modifications to the function.

Here is the original:

job_runqueue(void) {
    job *j, *jn;
    int run = 0;

    for (j = jhead; j; j = jn) {
        do_command(j->e, j->u);
        jn = j->next;
        free(j);
        run++;
    }
    jhead = jtail = NULL;
    return (run);
}

And here is my modified version:

int
job_runqueue(void) {
    job *j, *jn;
    int run = 0;
    int numRunning = 0;
    WAIT_T waiter;
    PID_T pid;

    for (j = jhead; j; j = jn) {
        numRunning += 1;
        if (NumProcs != -1) {
            if (numRunning >= NumProcs) {
                while ((pid = wait(&waiter)) < OK && errno == EINTR)
                    ;
                numRunning -= 1;
            }
        }
        do_command(j->e, j->u);
        jn = j->next;
        free(j);
        run++;
    }
    jhead = jtail = NULL;
    return (run);
}

First I added a variable to keep track of how many jobs are running. I then created variables for the process control: wait and pid.

When the function runs the loop, it checks to see if NumProcs is not set to -1 — or in other words, if no q switch was supplied. If it is not -1, it then applies the control.

The control simply says if the number of jobs currently running are greater than or equal to the number of processes specified with q, run wait() and wait for one or more jobs to finish. When a job does finish, decrement the number of jobs currently running by 1.

First Run

I created a cron table with some test jobs:

* * * * * sleep 10; echo `date` >> /home/jtopjian/t2.txt
* * * * * sleep 10; echo `date` >> /home/jtopjian/t2.txt
* * * * * sleep 10; echo `date` >> /home/jtopjian/t2.txt
* * * * * sleep 75; echo `date` >> /home/jtopjian/t2.txt

I ran cron on the command line in Debug mode. Everything worked perfectly. If I set the queue to 1, only one job would run at a time.

Unfortunately, I found a problem. If all jobs took more than 60 seconds to run, cron would skip the next minute’s crons. This was very bad as potentially important jobs would never run.

I re-read the code to find the problem. After an hour of reading and researching, it finally made sense: cron is a single-threaded program. There is no possible way it can keep track of its schedule plus run scheduled jobs. The reason why cron executes all jobs at once is because it needs to return to its sleep schedule as quick as possible so it won’t miss the next run.

In order to solve my problem, I would need to turn cron into a multi-threaded program. My idea was that any time the job queue needed to run, a new thread would be created. The thread would act like its own entity and run the queued jobs without worry of any other part of the program. Though changing the architecture of cron in this way sounded daunting, it actually wasn’t hard at all.

Adding Threading Support

YoLinux provides a wonderful tutorial on Linux Threads. It was all I needed to learn how to add in my threading support.

As I mentioned earlier, I did this a few months ago. Unfortunately, since then, I’ve forgotten what I learned. I am just going to list my changes but not provide details.

The first step was to include the threading library into both cron.c and job.c:

#include <pthread.h>

I then had to declare some variables which would control and work with the threads:

pthread_t thread1;
pthread_mutex_t mutex1 = PTHREAD_MUTEX_INITIALIZER;
int rc1;

Next, I modified the way job_runqueue() was called in cron.c. It originally looked simply like:

job_runqueue();

With my modifications:

if ((rc1 = pthread_create(&thread1, NULL, job_runqueue, NULL))) {
    Debug(DSCH, ("[%ld] thread failed\n", (long)getpid()));
}
pthread_join(thread1, NULL);

The next step was to modify the job.c file and add threading support. The final job_runqueue() function looked like this:

int
job_runqueue(void) {
    job *j, *jn;
    int run = 0;
    int numRunning = 0;
    WAIT_T waiter;
    PID_T pid;
    pthread_mutex_t mutex1 = PTHREAD_MUTEX_INITIALIZER;

    Debug(DSCH, ("[%ld] thread number %ld\n", (long)getpid(), pthread_self()));
    pthread_mutex_lock(&mutex1);
    for (j = jhead; j; j = jn) {
        numRunning += 1;
        if (NumProcs != -1) {
            if (numRunning >= NumProcs) {
                while ((pid = wait(&waiter)) < OK && errno == EINTR)
                    ;
                numRunning -= 1;
            }
        }
        do_command(j->e, j->u);
        jn = j->next;
        free(j);
        run++;
    }
    jhead = jtail = NULL;
    pthread_mutex_unlock(&mutex1);
    return (run);
}

The final step was to edit the Makefile so cron compiled with the pthreads library:

LDFLAGS     =   -g -lpthread

Second Run

With all of these changes made, I re-ran my tests. The threading library worked perfectly. Even if a queue of jobs took over a minute to complete, cron would create another thread with a whole new bank of jobs.

Conclusion

By toying around with the cron source code, I was able to learn the detailed workings of vixie cron as well as some basic UNIX C programming.

I’ve uploaded my final patch and my modified vixie-cron.spec file so readers can compile this version of cron themselves.

If anyone has any input, corrections, comments, or ideas on this topic and the changes I made, I would love to hear them.

I realize this article is rather unorganized and missing a lot of details. If anyone needs anything clarified, please let me know.

That view changed the other day, though.

I needed to poll a group of servers for network activity. Since I just needed the NIC’s byte count, I wrote a few custom scripts instead of using a full package like Cacti. The counts were fed into RRD where I’m generating Daily and Weekly graphs.

On a few of the servers, the graphs showed weird spikes in activity. Usually that would mean a lot of bandwidth was being used at the time — except that the data being graphed was a negative value and the spikes were going in the opposite direction.

The reason for this was that the byte counts that ifconfig was reporting were resetting. Since this was only happening on a few of the servers, I was getting pretty confused on the source of the problem.

Finally I noticed that these servers were all running a 32-bit version of CentOS while the working servers are 64-bit. The byte counts were just reaching their 32-bit integer max.

I’m the type that learns from personal experience. Seeing this negative graphing spike in action has made me fully aware of the benefits to 64-bit.

For the past few years, Unix Mages was running on Wordpress. The site has not changed in over two years as I was no longer writing content for it. So instead of leaving a stale and possibly insecure Wordpress installation on the web, I decided to turn it into an old fashioned static website.

What I Used

I enjoy using Coda as an editor due to its excellent remote functionality. I started creating a simple website consisting of two files: index.html and style.css.

I soon became tired of trying to lay out all of the elements manually, so I decided to include Blueprint.

Once I started working on the other pages, I realized copying and pasting text was severely inefficient if any future changes needed made. So I broke the pages up into a body, header, and footer and glued everything together using Apache SSI.

Enabling mod_include for Ubuntu 8.04

Apache mod_include is turned off by default. To enable it, simply do:

a2enmod include

A few configuration changes needed made in order to get SSI to work:

<Directory /path/to/document/root>
    Options +Includes
    AddOutputFilter INCLUDES .html
</Directory>

I’m now able to include my header.html and footer.html files in any webpage by using:

<!--#include virtual="/header.html" -->

Conclusion

Today it seems the standard way to create a website is to install Wordpress, Drupal, or any other type of Blogging or CMS engine — even if the site will only have a few static pages. I thought it was an excellent practice, if not only for nostalgic reasons, to detail how to create a simple, old fashioned website.

Perfect for beginners, Understanding TCP/IP explains the TCP/IP protocol suite in a very understandable text. Even more, the book is wonderfully illustrated with diagrams and flowcharts detailing everything discussed. The discussion on the application protocols (which include HTTP, FTP, DNS, Email and more) is also top-notch. It’s like having a non-sleep-inducing RFC available to learn how each of these protocols work.

While I’ll always swear to TCP/IP Illustrated, I suppose that book becomes more dated as time goes on. If you’re a novice System Administrator who’s looking for a great primer to TCP/IP and networking, I think you should give this book a chance. It covers everything about the suite — including application protocols — in a single, well illustrated book.

The first thing I noticed about this book is that it’s very easy to read. Pages and pages of theory and definitions of what time and management are simply do not exist. Instead, Thomas gives brief statements backed with examples and practice — just what System Administrators like.

Secondly, this book will not teach you how to format a piece of graph paper to chart your lists or how to organize your desk for maximum performance. Screw that. You save the company every day by stopping pesky foreign hackers out to rule the world. Someone telling you how to make a pile of paper is a slap in the face. Instead, you’ll learn about topics like interruptions, automation, and routines.

Mr. Limoncelli also goes over his own personal Time Management System he calls The Cycle. It’s a nice system to learn off of and possibly base your own on. My personal opinion is that time management systems are not compatible from person to person — and if you’re unable to create your own, you’re in more trouble than you think.

This book is worth a read. Time is a wonderful thing if it’s used correctly and the concepts Mr. Limoncelli covers will definitely help you achieve that.

The Definitive Guide to SQLite is the perfect book about SQLite. It covers everything needed to start working with SQLite including installation, using the SQLite shell, and programming with SQLite using six different language extensions. Another feature of this book is the impressive discussion on basic SQL and SQL theory. I’m glad this was included with the book as it gives more depth to the SQLite topics. For example, when the reader learns about database locks and how SQLite’s locks are based on the filesystem, they gain more of an understanding of what’s happening to the actual database file instead of just blindly believing in some SQL theory.

With System Administration, I always try to be as much of a minimalist as I can. Doing so helps keep everything quiet and simple and eliminates several unnecessary problems. When possible, using SQLite instead of a full RDBMS can only strengthen this methodology — and The Definitive Guide to SQLite is a great resource to learn SQLite from.

The Definitive Guide to Apache mod_rewrite is exactly that — the definitive guide. Unless I’m clearly mistaken, everything about the module is discussed. After the usual coverage of installation and configuration (as well as a quick guide on Regular Expressions), Mr. Bowen goes over the three main pieces to mod_rewrite: RewriteRule, RewriteCond, and RewriteMap. Next it’s time to put those pieces to work with two chapters of practice. Advanced usage such as Proxying and Virtual Hosting also make an appearance.

Though there are set chapters covering the usage of mod_rewrite, several other examples exist scattered throughout the book when needed. Further, when there’s a better or alternative way to accomplish an action, Rich comes right out and tells us. For example: after learning how to set Environment Variables, we’re taught how to stop bandwidth-hogging image theft. Or instead of using mod_rewrite to block clients based on their IP, we’re told it’s more efficient to so using a simple Deny directive. By doing this, we’re presented with hands-on, beneficial information while the original topic is still fresh in our head. Or simply put: we learn stuff.

This book has a nice balance of theory and practice. It starts out with the basics of the LDAP protocol in a very readable tone. From there, it slowly blends in each different topic of LDAP very smoothly. For example, one of the first hands-on pieces of the book shows you how to add objects to LDAP via the command line. It then shows you how to search for those same objects. Though the reader doesn’t know it yet, the format these examples are using is standard LDIF — so when it’s time to use LDIF files, the reader thinks, “Oh! I know this already!”

OpenLDAP is the implementation used in the book, but there’s also footnotes throughout the text giving hints to the reader on other LDAP packages. Everything about LDAP is covered here: the basics, networking, replication, system administration, and programming. The programming section was particularly useful with the added bonus of example code in 4 languages.

So overall, I’m extremely impressed with this book. Why it isn’t more popular is beyond me. Maybe it’s the steep price tag, but I personally think it’s worth it.

Fast forward 3 years to when I finally had somewhat of a grasp on what I was originally looking for. Now this book made sense. It even made a ton of other things more clear. This book really is how the Internet works. Anyone who has a full time job that involves a server or networking device needs to read this book. Yes, it will hurt. Yes, you will fall asleep after every third paragraph — but you will come out a better person in the end. So much so that the next time you’re watching a tcpdump stream, you’ll be able to pick out Blonde, Brunette, Redhead.

Some of the more notable parts of the book were the sections on figures, tables, and lists; typograhphy and formatting; and types of documents to write.

Concerning the Tables and Lists, Rosenberg explained clean formatting, headings, and what to do when full sentences proceed lists. The typography and formatting piece gave a nice explanation of serif and sans-serif fonts and when (and when not) to use them. Finally, the types of documents received the biggest portion of the book. Examples of lab reports, manuals, proposals, and even PowerPoint presentations were given. Having reference to all of these documents in one book is really nice. The proposal part was the most detailed of all the other types — with good reason! There are several different types of proposals you can make and anytime you write a proposal, chances are it has a lot of importance.

Like I mentioned in the beginning, this was a nice book. There were a bunch of other little topics covered such as punctuation and vocab, but I think they weren’t given a lot of attention because of the myriad of other books written on these subjects.