chrisstreeter.com

Checking for Python Package Updates at PyPI

2012-04-02T03:45:39Z

I frequently find myself wondering if a bug in a Python package has been fixed and whether there is an upgrade for that package that might fix the bug. So I find that I end up running pip freeze and then having to compare the package versions to those on PyPI manually. Well, anytime you say “run X manually”, you’re being a chump.

I just saw down and wrote a script to get the list of currently installed packages in the current environment (so it works with virtualenv). Then it checks to see what the latest version of the package is on PyPI and prints out the status. If you work with Python and packages, this is awesomesauce.

I added the script to my dotfiles on github.

Assuming the script is at ~/bin/pyupgrades.py, you can run it globally or after activating your favorite virtualenv.

Homebrew Package Update Notifications via Growl

2011-09-20T15:53:39Z

Over the weekend, I finally upgraded my system to Mac OS Lion. I also took the opportunity to do a completely fresh install of my system, doing a final TimeMachine backup before erasing the hard drive and then installing Lion off a USB thumb drive.

I have long used MacPorts as my open source package manager, but I’ve had issues recently with certain ports not being updated or being out of date. So I was really interested in Homebrew. The fact that it is all on github, open and actively developed really appealed to me. After getting it up and running, I wanted to port my package update notifier to use Homebrew. Doing so was really quite easy. Here’s what I came up with, which is also in my dotfiles on github.

#!/bin/bash
#
# Notify of Homebrew updates via Growl on Mac OS X
#
# Author: Chris Streeter http://www.chrisstreeter.com
# Requires: Growl Notify Extra to be installed. Install with
#   brew install growlnotify
 
TERM_APP='/Applications/Terminal.app'
BREW_EXEC='/usr/local/bin/brew'
GROWL_NOTIFY='/usr/local/bin/growlnotify'
GROWL_TITLE="Homebrew Update(s) Available"
GROWL_ARGS="-n 'Homebrew' -d $GROWL_NOTIFY -a $BREW_EXEC"
 
$BREW_EXEC update 2&gt;&amp;1 &gt; /dev/null
outdated=`$BREW_EXEC outdated | tr ' ' '\n'`
 
if [ -z "$outdated" ] ; then
    if [ -e $GROWL_NOTIFY ]; then
        # No updates available
        $GROWL_NOTIFY $GROWL_ARGS -m '' -t "No Homebrew Updates Available"
    fi
else
    # We've got an outdated formula or two
 
    # Nofity via growl
    if [ -e $GROWL_NOTIFY ]; then
        lc=$((`echo "$outdated" | wc -l`))
        outdated=`echo "$outdated" | tail -$lc`
        message=`echo "$outdated" | head -5`
        if [ "$outdated" != "$message" ]; then
            message="Some of the outdated formulae are:
$message"
        else
            message="The following formulae are outdated:
$message"
        fi
        # Send to growlnotify
        echo "$message" | $GROWL_NOTIFY $GROWL_ARGS -s -t $GROWL_TITLE
    fi
fi

Assuming the script is at ~/bin/brew-update-notifier, you can install the script to a crontab by running sudo crontab -e, then adding the line 0 12 * * * /Users/<username>/bin/brew-update-notifier to the end of the file (substituting <username> for your username, or wherever you’ve put the script). I’ve chosen to run the script every day at noon because my computer is likely to be on and connected to a network.

World Backup Day: How I Backup

2011-04-01T05:25:28Z

Today was World Backup Day, and, in the footsteps of Ars, decided I’d detail my current backup strategy. I first started backing up regularly after an incident back in college. One day the hard drive in my laptop began making the telltale sounds of imminent failure. Then the computer stopped booting. And I hadn’t backed up the drive in several weeks, which would have caused me to lose a bunch of data. I ordered a new drive, and just before I was about to install it, I was luckily able to boot my laptop and then copy the data off of it, losing nothing. Since then, I’ve been an big advocate of backups.

My primary computer is a Macbook Pro. Since it is also the computer that all my photos and important data is on, I don’t want to lose any of it. Every day I make sure to plug in my 1 TB external hard drive that Time Machine mirrors the contents of drive to. So at all times, I’ve got a backup of my drive and a history of the files. I also have another external drive that I keep at my parent’s house in their safe that is filled with Time Machine backups. So if my house blows up, I have something to fall back on. The problem with that disk though is that it is several months out of date. And that sucks because my data changes all the time.

But thats not all for my laptop. I also make use of CrashPlan. At home, I also have a Linux box with a 4 hard drives in a RAID 5 setup, so if one drive dies, I don’t lose any data, I can just replace the drive. So I have CrashPlan backup my laptop to my Linux box onto it’s RAID storage, and this can happen anywhere my laptop has Internet access. Currently, CrashPlan is configured to run every night.

My email is hosted in the cloud, but I still run hourly backups onto the RAID storage in my linux box using mbsync. I also run backups of my Google Docs every day using gdatacopier (again, saving to my Linux box).

I used to also have Mozy, but decided to scale back costs for a while. Though I would really like to use it again. For some important things, I’ll stick them on Dropbox, which maintains a backup copy in the cloud. However, storage on Dropbox is much more limited than something like Mozy.

Right now, one single point of failure is my Linux box. The drives are getting old and should be replaced soon (the MTTF is way overdue) and I’m relying too much on them these days. And I really should have a standby drive in the array in case of multiple disk failures. The plus side is that there really isn’t anything on that computer that is not somewhere else, or I can’t redownload. But it would be a huge hassle.

I’ve really tried to create a good, easy solution for backups without any single point of failure. Though there are definitely several areas that I need to improve on. You never need the backup until you need it. And at that point, if you don’t have the backup, you may find you’ve lost years of irreplaceable memories. I know that to me, my photos are irreplaceable. I also have all the documents I created from sixth grade all the way through college. And if I lost those, there’s no bringing them back (there’s just no way to recreate the awesome resume I wrote in sixth grade – “Expected graduation from Middle School, 1998″, “experience with many tools and their uses”, “fast runner” – classic).

If you own a Mac, there is no excuse, just buy a cheap external hard drive and plug it in and use Time Machine. Windows users should just buy a Mac. Linux users… well, write some perl script that rsync’s your files somewhere and put it in cron or write some other solution (I mean really, you’re using Linux, you should be able to figure something out). Or look at CrashPlan. Hopefully, if you aren’t backing up your files, you’ll start.

MacPort Package Update Notifications via Growl

2011-01-23T17:30:39Z

I wrote a quick little script this morning to notify myself of updates to MacPorts via the Mac OS X notification app Growl. The script is a bash script that is designed to run as root from a cronjob (as the port sync command requires root permissions). You can find the script committed in my dotfiles repository on github. For convenience, I pasted the script below, though the copy on github will always be the most up-to-date copy.

#!/bin/bash
#
# Notify of MacPort updates via Growl on Mac OS X
#
# Author: Chris Streeter http://www.chrisstreeter.com
# Requires: Growl Notify Extra to be installed (but fails gracefully). Info
#       about how to get the extra is at http://growl.info/extras.php
 
TERM_APP='/Applications/Terminal.app'
PORT_EXEC='/opt/local/bin/port'
GROWL_NOTIFY='/usr/local/bin/growlnotify'
GROWL_TITLE="MacPort Update(s) Available"
GROWL_ARGS="-n 'MacPorts' -d $GROWL_NOTIFY -a $PORT_EXEC"
 
$PORT_EXEC sync 2&gt;&amp;1 &gt; /dev/null
outdated=`$PORT_EXEC outdated`
 
if echo $outdated | grep -q 'No installed ports are outdated.' ; then
    if [ -e $GROWL_NOTIFY ]; then
        # No updates available
        $GROWL_NOTIFY $GROWL_ARGS -m '' -t "No MacPort Updates Available"
    fi
else
    # We've got an outdated port or two
 
    # Nofity via growl
    if [ -e $GROWL_NOTIFY ]; then
        lc=$((`echo "$outdated" | wc -l` - 1))
        outdated=`echo "$outdated" | tail -$lc | cut -d " " -f 1`
        message=`echo "$outdated" | head -5`
        if [ "$outdated" != "$message" ]; then
            message="Some of the outdated packages are:
$message"
        else
            message="The following packages are outdated:
$message"
        fi
        # Send to growlnotify
        echo "$message" | $GROWL_NOTIFY $GROWL_ARGS -s -t $GROWL_TITLE
    fi
fi

Assuming the script is at ~/bin/port-update-notifier, you can install the script to a crontab by running sudo crontab -e, then adding the line 0 12 * * * /Users/<username>/bin/port-update-notifier to the end of the file (substituting <username> for your username, or wherever you’ve put the script). I’ve chosen to run the script every day at noon because my computer is likely to be on and connected to a network.

Python virtualenvwrapper and Logging

2011-01-21T03:51:03Z

I ran into this issue, and couldn’t find a resolution anywhere online. Well, I figured it out, and thought that I’d put it online for others to find.

The issue was that I got exception tracebacks when sourcing the /usr/local/bin/virtualenvwrapper.sh script upon bash initialization. However, the exception happens while Python is running it’s atexit handlers. So the stack was only 3 levels deep. I’m running virtualenv and virtualenvwrapper on my Ubuntu 9.04 server.
Here’s the output of the exception that I got:

[streeter@mail]:~$ source /usr/local/bin/virtualenvwrapper.sh
Traceback (most recent call last):
File "/usr/lib/python2.6/logging/handlers.py", line 72, in emit
self.doRollover()
File "/usr/lib/python2.6/logging/handlers.py", line 129, in doRollover
os.rename(self.baseFilename, dfn)
OSError: [Errno 13] Permission denied
Traceback (most recent call last):
File "/usr/lib/python2.6/logging/handlers.py", line 71, in emit
if self.shouldRollover(record):
File "/usr/lib/python2.6/logging/handlers.py", line 145, in shouldRollover
self.stream.seek(0, 2)  #due to non-posix-compliant Windows feature
ValueError: I/O operation on closed file
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
File "/usr/lib/python2.6/atexit.py", line 24, in _run_exitfuncs
func(*targs, **kargs)
File "/usr/lib/python2.6/logging/__init__.py", line 1508, in shutdown
h.flush()
File "/usr/lib/python2.6/logging/__init__.py", line 754, in flush
self.stream.flush()
ValueError: I/O operation on closed file
Error in sys.exitfunc:
Traceback (most recent call last):
File "/usr/lib/python2.6/atexit.py", line 24, in _run_exitfuncs
func(*targs, **kargs)
File "/usr/lib/python2.6/logging/__init__.py", line 1508, in shutdown
h.flush()
File "/usr/lib/python2.6/logging/__init__.py", line 754, in flush
self.stream.flush()
ValueError: I/O operation on closed file
[streeter@mail]:~$

As you can see, not a lot to go with. So it seems that virtualenvwrapper does some logging, and it looks like the logging module wants to rename that logfile. Digging even deeper, the logging module wants to rotate the logfile because it got too big. Well, where is the log file? Turns out that virtualenvwrapper sets up a logfile called hook.log at the root of the WORKON_HOME environment variable. My WORKON_HOME is set to /usr/local/pythonenv and lo and behold, that directory was not writeable by my logged in user. Thats an easy fix. Just change the permissions.

Blackout Days

2011-01-03T06:42:38Z

What does one do when one has a cabin in Truckee and a season pass to a ski resort that is blacked out? Well, in my case, I still want to be active in the snow. Over the past week, I decided to go snowshoeing up near Donner Lake. Family and friends parked near Donner Summit and then followed part of the Pacific Crest Trail south towards Sugar Bowl until we were able to overlook Donner Lake on a beautiful day.

Forging a Trail

Icicles

Then, a few days later, on New Years Eve day, I went nordic skiing for my first time ever. We started at the State Park at the east end of Donner Lake and followed a trail several miles out, for about a four hour trip. By the end, I was definitely ready to be done with the skis, and probably won’t go that long again until I’m a bit more experienced on skis (downhill and nordic). It definitely didn’t sour my first experience though and I plan on going at least several more times this winter.

Nordic Skiing

Puffy Snow

Looking Back on 2010

2010-12-31T03:50:50Z

As 2010 winds down in the next few days, I decided to write up a summary of my past year in what, some would term, a Christmas letter. Well, it’s after Christmas, so this is officially a Review of 2010 letter. I emailed out a link to quite a number of people that I’ve got email addresses for. So if you aren’t on that list, you can find the body of work and associated pictures at my 2010 in Retrospect page.

You can keep up with me next year here on my site and through my photos at Flickr or at my gallery. I’d love to hear how your 2010 went, so write back. Hope your 2010 was a good one and all the best next year.

My First 30k

2010-12-21T02:41:44Z

A few months back, around the beginning of September, my sister, a friend and I all decided that we wanted to do a longer trail run. We actually decided this while on a run through some redwoods together. Since we enjoyed trail runs, we decided that the best way to do a longer distance run would be to sign up together and commit to finishing instead of focusing on being competitive in the race.

A terrain view of the course route

When we got home, we looked up to find a 30-35 kilometer Pacific Coast Trail Run that was several months out, giving us plenty of time to train. We settled on the Rodeo Beach 30k, on December 18th. It was on the shorter side (30km instead of 35km), it was plenty far out, giving us a lot more time to train, and it was in the Marin Headlands, a beautiful area that we could train at. I signed up almost immediately, as I knew the act of signing up would get me to commit mentally. My sister signed up and so did my friend (at least I thought she did). I also got a couple people at my gym to sign up too. Things were shaping up nicely.

The start of the race led up a couple hills. It was also quite overcast.

I didn’t train as much as I wanted to while on my Balinese vacation, and upon returning, I tried to stick to a more rigid training schedule where I ran more frequently. Well, I didn’t stick to that schedule as well as I should have. As of the day of the race, the farthest distance I had run in the many months prior was about 7.5 miles, not even half of the final distance of 18.64. I wasn’t terribly nervous, I knew I could still finish the race, but I figured I’d be doing a lot more walking that I’m used to, especially up the hills. And there are a lot of hills on this race, giving the 30km race a formidable 3740 feet of elevation gain along the course. However, it turns out my sister and friend also hadn’t been training for the race either.

Then, because of an Omega block sitting in the middle of the Pacific, lots of rain was scheduled to be directed at Northern California. This threat of rainy weather was enough to scare off my sister. Then I found out my friend hadn’t signed up for the race. So much for the idea of a trio of us doing our first 30k together, enjoying the outdoors on single-track. Thus, I was left to do the 30k alone and slightly unprepared.

The weather ended up lucking out and a break in the rain occurred for most of Saturday. The clouds and fog remained, but almost no wind. An 8:30am start at Fort Cronkhite had me excited and ready to get moving. The course went 6.6km to Tennessee Valley, through WWII bunkers in the headlands. At that point, the 30k and 50k runners went on a 10.4km loop up the Miwok trail to Coyote Ridge to the Coastal Trail and back to the Tennessee Valley aid station. before we continued on the final 13.2km return segment and the formidable Marioncello hill (the last, and biggest hill in the profile below), arguably the worst of the 3 main hills.

The elevation profile for the 30km Rodeo Beach race - 3740 feet in gains

At this point, my splits looked like this: 0:46 for the first 6.6km, 2:00 total to get back to Tennessee Valley; a little over half way through the course. Another hour and I made it to the final aid station at Conzelman with 5.9km downhill to get to the finish. By now, my hips were sore, my calves were sore and I was starting to feel more negative. However, the negativity disappeared at the aid station where I was able to inject some more sugar into my system; though the soreness remained all the way to the finish.

Saul and I at the finish. Thanks for waiting and taking pictures Saul!

After running along the valley floor for what seemed like an eternity, I finally reached the finish line feeling decently strong with plenty of non-running energy. I definitely didn’t have any more kilometers in me, but I also had plenty of strength in my legs to stand. My final time: 3 hours, 41 minutes and 3rd in my age group. Really, the race didn’t feel that long and looking back I wonder where the time did go. I really think I was able to take in the course, get into the moment and forget that I was going for such a long distance. My body was pretty sore the day after, but not more so than past long distance runs or triathlons. Before the 30km run, my previous longest run was somewhere around 22km, just over a half marathon distance. This isn’t my last 30k, and isn’t the longest race I will be running. Maybe next time I’ll get some friends to join me for the entire race though.

Post-Thanksgiving Bishop Climbing Trip

2010-12-02T04:38:03Z

After Thanksgiving at my parent’s house, I took off with some friends to Bishop for 3 days of bouldering and some sport climbing. We arrived just after some storms had passed through the area, leaving a light dusting of snow in some places.

Eastern Sierras Under Shadow

Then, right in the middle of the trip, we got a couple inches of snow in the evening, electing to remain at our site camping in it.

Fresh Snow

In the end, Bishop was super cold (got down to the mid teens at night) but provided some great climbing in the Buttermilks, Gorge and Happys. Makes me look forward to another climbing trip soon!

5 hours of images, taken every 30 seconds at 30 second exposures. Camera settings were ISO 1600, f/6.3, +.33ev, 10mm

Facebook Exporter for Aperture

2010-11-25T16:47:01Z

I’ve wanted to recently upload some of my pictures to Facebook. However, there are several things I really dislike about Aperture’s built-in Facebook syncing.

First, order is not preserved. I want pictures ordered in a predicable way. Often, by date. I have yet to figure out how Aperture chooses to export, it seems to be quite random and is infuriating.

Second, tagging friends on Facebook from the built-in Aperture Faces is spotty, at best. Sometimes it works, and other times I get lots of pictures with people’s names tagged, but those names are not linked to the proper Facebook friends, despite being tagged with the exact same name. Again infuriating!

Third, making any change in Aperture (adding a tag, etc), causes the photo to be re-uploaded to Facebook, creating an updated photo album feed story. I don’t want this behavior. I want to export my photos to Facebook and then choose when it should be updated again. I’ll let you guess my emotion.

After finding no other solutions, I wrote my own Facebook exporter

So, what other options are there out there? Well, this guy Sean Farley created a plugin. However, it is broken on my computer. Supposedly, it does work for some people on Aperture 3, but I cannot get it to even show up in the Export menu and I get no Console messages. I’ve even emailed Sean twice trying to contact him and help figure out why it is broken. No response.

Finally, I’ve given up with these options. I’m not a prolific Facebook photo uploader, but there are times when I want to put up pictures on Facebook instead of Flickr. And I couldn’t do it without exporting the photos from Aperture to my disk, and then using Facebook’s photo upload.

So over the past couple weeks, I spent some time creating my own export plugin for Aperture. I don’t want to completely support this project (beyond having it work for me), and so I decided to release it on github for the benefit of the community. My goal is that someone else will find the code and project useful. There are currently several limitations (documented on github) and I’m sure there are bugs, but I’ve got the project to a stable enough point where I can upload photos to Facebook and am happy with where I’ve gotten it. Check out the github page for more information on getting the plugin and using it.