The Scheutz Mechanical Calculator

Hi everyone,

A SourceHosting.net client recently asked to create a large number of users in a custom installation of phpDeadlock that he uses to manage access to his Subversion repository here. Since phpDeadlock doesn’t have a user management API, this sounded like a monotonous, error-prone task.  After entering 10-15 users, the chances of misspelling a name or email address or clicking the wrong button in the UI will likely increase dramatically.

I briefly investigated adding records directly to the backend MySQL database, but since various other actions are fired by user creation, including email notifications to the new user and the system administrator, using the application’s UI was the safest choice.

I had run across the WWW::Mechanize Perl module a while back, but hadn’t used it yet. I knew it could be used to automate interaction with a web site, and after reading through the rich list of methods, I promptly began hacking a script together to parse a list of email addresses supplied by the client and use it to drive the user creation UI.

The module implements a headless web browser, including cookie jar, history, form submission and other behaviors that you would expect, except it doesn’t parse and execute Javascript. That was a non-issue for me.

phpDeadlock has an administrator login prompt that requests a password before any privileged pages are accessed. Logging in to the application was a no-brainer in WWW::Mechanize:

use WWW::Mechanize;
use String::Random;
use Text::Capitalize;

my $mech = WWW::Mechanize->new();
$mech->get('https://phpdeadlock.mydomain.com/admin/');
$mech->set_visible('ItsASecret');
$mech->submit_form();

That was too easy! All further page requests will be authenticated. Next, the script enters a loop to process email addresses presented on STDIN, one per line, and populate the required fields:

for my $email (<STDIN>) {
    chomp $email;
    my @fields = split /\@/, $email;
    my $finitial = substr($fields[0], 0, 1);
    my $lname = substr($fields[0], 1);
    my $pw = new String::Random->randpattern("ssssssss");

    $mech->get('https://phpdeadlock.mydomain.com/admin/newuser.php');
    $mech->submit_form(
        fields => {
            firstname => capitalize($finitial),
            lastname => capitalize($lname),
            email => $email,
            username => $fields[0],
            password => $pw,
            password2 => $pw
        }
    );

    print "Added new user for $finitial $lname\n";
}

If the user’s full name had been supplied, I could have easily parsed that and set the firstname and lastname fields appropriately.

The possibilities for using this module are endless, and I encourage you to try it out, especially when you’re dreading a repetitive web application UI task.  Make sure to check the list of other Perl modules that are based on WWW::Mechanize, too!

Photo by: Mark McArdle

Hi everyone,

I came across a helpful list of PHP optimization tips, and I’d like to share it with you here: http://php100.wordpress.com/2009/07/13/php-performance/

In addition, using the XDebug profiler for collecting performance data and KCacheGrind for viewing it is highly recommended to keep your PHP applications running smoothly!

Common result from converting a complex non-Subversion source code repository to a Subversion dump file

Hi everyone,

Many new SourceHosting.net clients have existing repositories that they want to import into a Subversion repository here. If we’re lucky, the client already uses Subversion, and it’s no more difficult to import the repository than dumping and loading it.

Converting from CVS to SVN is quite easy using the cvs2svn script. We rarely run into any problems, and it has a flexible set of command line options. If, however, the client uses a different source code control system, the process can get trickier.

There are several repository converters available for download, including Polarion Importer for SVN and VSS2SVN. These tools are welcome additions to a release engineer’s bag of tricks, especially if different groups in an organization have not standardized on a single SCM system and you’re trying to convert everyone to Subversion.

As we’ve used the tools mentioned above more and more frequently, we’ve found that they work well for new clients with reasonably small and uncomplicated repositories. However, as the complexity grows with more tags, branches and merge points, the likelihood of producing a corrupted or logically-incorrect Subversion dump file increases.

This problem is illustrated by a recent repository conversion in which the new client used a tool to convert from their existing VSS repository to a Subversion dump file, preparing for their migration to SourceHosting.net. Upon cursory inspection, the dump file contents looked reasonable, but some number of revisions into the loading process, svnadmin reported the following error:

<<< Started new transaction, based on original revision 33
svnadmin: File not found: transaction '30-1', path '/src/docs/ChangeLog'
    * editing path : src/docs/ChangeLog ...

After reading through the dump file, we discovered that it contained a sequence of operations on the “/src/docs/ChangeLog” file that occurred before the file had been added to the repository. We’ve also run across negative-numbered revisions and attempts to delete files from locations that don’t exist. All of these situations will abort repository loading.

These errors aren’t difficult to detect, and even fix, by parsing through the dump file and rewriting it slightly with the Perl CPAN module SVN::Dumpfile. This module extracts data from a dump file and also allows new data to be inserted into it.

In order to make it easier to test software that generates Subversion dump files, we are building on the work of SVN::Dumpfile and creating a sequence of dump file validation tests. Not even the svnadmin tests included in the Subversion distribution parse dump files, and perhaps this work will migrate into their suite as well.

If you have any suggestions or ideas for specific tests that you would like included, please let us know.

I have some servers with RHEL4 installed on them, and I’ve noticed a problem every time I start processes that saturate the I/O channels, such as VMware’s vmware-vdiskmanager. This tool performs various operations on VMware virtual disk files, and when creating a new one, the load average on the server tends to spike into the double digits. As you can imagine, this negatively affects virtual machines running at the same time!

After some searching, I found the ionice tool that looked like a perfect solution to the problem. Unfortunately, it doesn’t run on the 2.6.9 vintage kernel supplied with RHEL4. Back to the drawing board!

After more searching, I came across a forum thread and a link to an ionice replacement for RHEL4, written in Perl. I downloaded it and tried it out, and it appears to work as advertised. My heavy I/O operations take longer now (fine), and the load average stays within acceptable limits (great!).

The script has some hard-coded values, and it can be easily tweaked as needed. Thanks to Greg Bell at ServEdge for writing it!

As I’ve been writing over the past couple of weeks, there are a large number of FreeBSD ports that are unmaintained and would benefit greatly from a new maintainer. Today, we’ll visit with a particularly tragic strain of port:

Unmaintained and BROKEN!

Bill Fenner sends out a periodic automated email to the freebsd-ports mailing list with links to ports that do not build: http://people.freebsd.org/~fenner/errorlogs/.

This is a useful page for any port maintainer to visit periodically to make sure all of his/her ports are humming along. If a port remainis unbuildable for too long, portmgr will likely mark it BROKEN after a while, and if it stays BROKEN too long, it will be removed from the tree.

The problem is that when a port is unmaintained, it’s likely that no one will notice that it doesn’t build, and it won’t be fixed in time to save it from removal. One place to start if you’re interested in adopting a port is the page detailing unmaintained ports that are broken on one or more platforms:

http://people.freebsd.org/~fenner/errorlogs/ports@freebsd.org-date.html

There are many different reasons that ports refuse to build, including (sample log follows each):

• Unfetchable upstream distribution files (log)
• GNU configure errors (log)
• Compiler/linker errors (log)
• Package building problems (log)

Fixing a broken port may be as easy as correcting a download URL or as complex as patching source code so it builds correctly on the AMD64 platform. There are many different ways to get your feet wet, so I encourage you to check some of the broken build log files and see if you can fix, and possibly adopt, one of these ports!

I’m here with some more statistics about the FreeBSD ports tree and the unmaintained ports in it. Today, I calculated the top five categories with the largest number of unmaintained ports in each:

There is a huge variety of software to explore in the ports tree, and if you’re interested in maintaining one or more ports, I suggest looking for something you’ve already used in the past or a piece of software in your area of interest.  For instance, there are a number of astronomy and biology-related ports that could use some help!

Another great place to find ports that need some help, be it maintainership or submitting PRs to fix them, is the pointyhat build cluster status page: http://pointyhat.freebsd.org/errorlogs/.

If you’re just getting your feet wet and don’t know where to start, subscribe to the freebsd-ports mailing list or browse its archives. You’ll find a lot of questions and solutions to common problems as you learn more about the ports infrastructure.

To prepare for the upcoming inaugural International FreeBSD Adopt-A-Port Day on June 15th, 2009, I am publishing some interesting (???) statistics about the FreeBSD ports tree.

First up (cue the violins): The most unloved FreeBSD port!

This port has been a workhorse since the year 1996, back in the days before the Internet you now know, when a fast connection was 56k and we often disabled image loading in our browsers to speed up the browser! Those days, there were fewer than 500 ports in the tree, a far cry from the 20,000+ we now enjoy.

And now, I give you (drumroll, please)….

print/ghostview

The CVS log shows that this port has been unmaintained since 17-Nov-96, so it’s been nearly 13 years that ghostview has been wandering aimlessly through the tree, waiting for someone to adopt her/him/it. The ghostview port won’t require a lot of care and feeding, but if a new upstream version is released, the port would like a FreeBSD maintainer that will submit a PR to keep it in sync.

So, wouldn’t you like to help a port regain its standing in the tree and become all that it can be and more? Contact me at glarkin@FreeBSD.org to get involved and help out!

Martin and Ion-Mihai have just been honored as the newest portmgrs. Congratulations, Martin and Ion-Mihai, and thank you for your help to me and all other port maintainers and committers as we learned (and continue to learn) the ropes!

Just for fun, let’s designate June 15th as the inaugural International FreeBSD Adopt-A-Port Day for 2009!

My colleague and co-mentor to Alexander Logvinov, Thomas Abthorpe, posted to the freebsd-ports mailing list in March looking for folks to adopt unmaintained FreeBSD ports and keep them up to date.

I’d like to continue Thomas’ effort and find out who in the FreeBSD community is interested in taking on one or more unmaintained ports. Some current stats:

FreeBSD ports info page (20325 total ports as of June 4th, 2009)

FreeBSD unmaintained port list (4719 as of June 4th, 2009, 77 with build errors)

FreeBSD unmaintained ports that need upgrading (253 as of June 4th, 2009)

If you are interested in maintaining a port, contributing PRs for port upgrades, and perhaps eventually becoming a ports committer, read the following documents to get a good overview of the process:

• http://www.freebsd.org/doc/en/articles/contributing-ports/index.html
• http://wiki.freebsd.org/PortsTasks
• http://www.onlamp.com/pub/a/bsd/2002/01/31/Big_Scary_Daemons.html

I started my own path to becoming a FreeBSD ports tree committer by discovering that a web application (WebCalendar) that we use at SourceHosting.net was not part of the tree.  I read up on how to create a new port, submitted a PR for it, and after review by an existing FreeBSD committer, it was added to the tree.  Now other WebCalendar users can install it on FreeBSD as easily as typing:

cd /usr/ports/www/webcalendar && make install clean

Since submitting that first PR, I have created a number of new ports for tools used at SourceHosting.net or just in my areas of interest.  I have adopted many as well. Most unmaintained ports are very undemanding and just need some minor TLC! It’s a great way to support a project that has contributed directly to the success of my Real Job.

Have any questions or guidance? Email me at glarkin@FreeBSD.org, follow me on Twitter or comment here. We’d love to work with you!

To go along with the FreeBSD 8.0-CURRENT amd64 virtual appliance I released a few days ago, I’ve also prepared one for the i386 architecture.

Get the virtual appliance here: FreeBSD 8.0-CURRENT i386 virtual appliance

As always, comments and feedback to virtualization@sourcehosting.net is welcome. Enjoy!