Chris Adams' Weblog

Cross-Post experiment: Deploying Django Sites using RPM

2009-10-16T23:06:52Z

I'm experimenting with Tumblr and used it for my latest entry: Deploying Django Sites using RPM

Site testing using RED Spider

2009-07-25T15:20:45Z

Mark Nottingham recently released redbot, a modern replacement for the classic cacheability tester. I've been using it at work to audit website performance before releases since proper HTTP caching makes an enormous difference in perceived site performance.

redbot is a focused tool and provides a great deal of detail about at most one page and, optionally, its resources. I wanted to expand the scope to testing an entire site and performing content validation and with a little work came up with red_spider.py, which produces a consolidated report like this.

I have a few ideas for the future, which should involve splitting the code into a separate project rather than a fork of redbot as it acquires more validation capabilities such as borrowing from something like collective.validator.css to validate CSS, RSS/Atom, etc., using PIL to verify that images don't have things like wasteful embedded thumbnails, and borrowing from my wk-bench experiment to load pages using WebKit and report JavaScript errors.

Even friendlier shell prompts for version control

2009-04-03T22:51:26Z

I've extended the earlier VCS-friendly shell prompt to add support for Mercurial and you can now get my current .bash_profile from GitHub:

__has_parent_dir () {
    # Utility function so we can test for things like .git/.hg without firing
    # up a separate process
    test -d "$1" && return 0;

    current="."
    while [ ! "$current" -ef "$current/.." ]; do 
        if [ -d "$current/$1" ]; then
            return 0;
        fi
        current="$current/..";
    done

    return 1;
}

__vcs_name() {
  if [ -d .svn ]; then 
    echo " [svn]"; 
  elif [ -d RCS ];  then 
    echo " [RCS]";  
    elif __has_parent_dir ".git"; then
        echo "$(__git_ps1 ' [git %s]')";
    elif __has_parent_dir ".hg"; then
        echo " [hg $(hg branch)]"
  fi
}

PS1='\[\033]0;\u@\h:\w\007\]\u@\h:\w$(__vcs_name) $ '

Cleaning up the web with jQuery and a little help from Google

2008-09-06T01:44:41Z

Recently the topic of enhancing web pages came up at work. It's a lot easier than it used to be thanks to two trends: the rise of modern JavaScript libraries and public CDNs hosting those libraries. This makes a lot easier to enhance content which you can't easily alter (e.g. the forms used by various big companies with marginal web competency) or in situations where you're worried about compatibility with existing code (some squirrelly vertical apps in our case).

Updated 2009-04-03: Moved the template and example scripts to Gist for ease of copying/maintenance: bookmarklet-template.js, enable-autocomplete.js and resize-textareas.js

Updated 2008-10-14: there's a very similar jQuery-lovefest on Sam Ruby's weblog with plenty of useful tips.

To illustrate just how little code this can require, here's an example which uses jQuery to install a function which sanitizes input (we have a legacy app chokes on smart-quotes and people paste text in from Word), copies the submit buttons from the bottom of the form to the top and adds a graphical datepicker for every date field on the page:

jQuery(":text,textarea").bind("change", sanitizer);
jQuery("form").bind("submit",
    function() {
        jQuery(":text,textarea").each(sanitizer);
    }
);

var submit_buttons = jQuery('input[type="submit"]');
submit_buttons.parent().clone(true).prependTo(
    submit_buttons.parents().filter('form')
);

jQuery('input[id*="DATE"]').datepicker();

That's the complete, ready-to-go, “even works with crotchety old Internet Explorer” guts of the code (the take-home lesson is that jQuery is awesome for busy developers). The downside is that this requires a little but of work: you need to have jQuery (and possibly dependencies like the UI plugin I used above) available and you need to jump through some hoops to load jQuery into an existing page efficiently and without conflicts.

Didn't we used to pay for hosting?

One drawback to all of this is that you need somewhere to host your external libraries since you can't fit the core jQuery into a URL, much less UI components or the less svelte libraries. This meant setting up a server, getting an SSL certificate if you need to work on HTTPS sites, etc. Not that much work but it's now a lot easier and quite noticeably faster because Google makes it trivial to get the popular AJAX libraries from their CDN.

Developing with Bookmarklets

The deployment scenario for the major projects where I've used these techniques is a situation where you have some limited access to the page source: perhaps inserting a single script tag into a template or using something like MonkeyGrease or an Apache proxy with mod_substitute to rewrite the generated HTML as it passes through. This is great for making minimal changes but a bit cumbersome to develop and test with, particularly if you need to work on a production site or your instructions begin something like “Go change your browser's proxy settings…”

If I was only working in Firefox I could use GreaseMonkey but I need to test in Safari and Internet Explorer, too. The portable solution is a simple bookmarklet. I use a simple template (bookmarklet-template.js) which loads jQuery from the Google CDN and, after everything is ready to go, runs either a simple function or the external script of my choosing. This makes it easy to prepare an injector bookmarklet which can be used to pull my code into the current page, after which I can run and debug it using Firebug.

Useful Examples

This is also a useful technique for fixing other people's pages. Here are two bookmarklets and the commented source for tools which I use often:

Enable autocomplete - changes autocomplete="off" to on throughout the page (Source: enable-autocomplete.js)
Resizable Textareas - makes all textareas resizable (alá Safari 3) using jQuery UI Resizable (Source: resize-textareas.js)

I keep both of these in my Firefox & IE bookmark toolbar since they come in handy throughout the day and I've created more any time I find myself regularly needing to deal with a cranky legacy site. The process is simple: copy bookmarklet-template.js, add the code which does whatever fixups the target page needs, run the entire thing through JSLint and, finally paste it into Ted Mielczarek's very handy Bookmarklet Crunchinator.

Good Code Injection Practices

Use Anonymous functions

What's the difference between this bit of code and the first example above?

(function(){
    jQuery(":text,textarea").bind("change", sanitizer);
    jQuery("form").bind("submit",
        function() {
            jQuery(":text,textarea").each(sanitizer);
        }
    );

    var submit_buttons = jQuery('input[type="submit"]');
    submit_buttons.parent().clone(true).prependTo(
        submit_buttons.parents().filter('form')
    );

    jQuery('input[id*="DATE"]').datepicker();
})();

It looks almost identical but there's a key difference: this code is inside an anonymous function and that means that all of my variables are local to the function itself, which means that they won't be visible to other JavaScript on the page and I don't have to worry about conflicting variable or function names. Note that this is only true for variables declared using "var" - if you leave that out or do something like window.foo you can still touch the rest of the page if you need to - for example, replacing the broken validation logic on Comcast's forms.

Reliably detecting when external code has loaded

When jQuery has loaded, it's easy to say "Load this .js file and run this function when it's ready" - here's how the text-area resizer works:

jQuery.getScript(document.location.protocol + "//ajax.googleapis.com/ajax/libs/jqueryui/1.5.2/jquery-ui.js",
    function() {
        jQuery("textarea").resizable();
    }
);

Loading jQuery itself requires you to do this the hard way: generate a script tag on the fly, insert it into the document and listen for the load events to tell when it's safe to run code which depends on the library you're loading. This is easy for Safari, Firefox, etc. which support the standard W3C DOM addEventListener: simply run your code after the script tag fires a "load" event. Unfortunately, it's not that simple for Internet Explorer: in theory attachEvent("onload") would be equivalent but unfortunately load events are quite unreliable for script tags with IE and so we need to use an onReadyStateChange handler as seen below and check for either of two events which may be fired:

var s = document.createElement('script');
s.type = "text/javascript";
s.setAttribute('src', document.location.protocol + '//ajax.googleapis.com/ajax/libs/jquery/1.2.6/jquery.min.js');

if (s.addEventListener) {
    s.addEventListener("load", loader, false);
} else if ("onreadystatechange" in s) {
    s.onreadystatechange = function() {
        if (this.readyState == 'complete' || this.readyState == 'loaded') {
            loader();
        }
    };
} else {
    // Chances are if your browser is this old jQuery won't even work but just in case:
    window.setTimeout(loader(), 2500);
}

document.getElementsByTagName('head')[0].appendChild(s);

It's conceivable that a buggy browser could fire the same event twice in an unusual scenario and if you have any sort of user-driven or timer-based code, you'll want to prevent your payload from being run multiple times using a guard like this which allows the function to check whether it has executed before without using the more common approach of relying on a global variable. Besides cleanliness, this also makes it easy if you might inject multiple things onto a page and don't want to have to rely only on a global variable naming convention to prevent chaos:

// Avoid executing this function twice:
if (arguments.callee._executed)  return;
arguments.callee._executed = true;

Avoid HTTP/HTTPS conflicts

If you're injecting code into pages which may or may not use SSL, you have a problem: if you hard-code a URL in your code and the protocol doesn't match you'll either incur the extra overhead of starting an SSL session (which isn't a major problem) by using https even when you don't need to or encounter Internet Explorer's popular mixed-mode security warning. This is easy to avoid by using the current page's protocol for your scripts as long as you're using a server which can handle either protocol (Google's CDN does; Yahoo's does not):

document.location.protocol + '//path.to.example.com/something.js'

Tracking down mod_auth_cas segfaults

2009-03-27T22:47:04Z

Just a quick note to boost Google rank: if you use the mod_auth_cas Apache module for CAS single-signon and have noticed sporadic Apache segfaults (typically a blank page which reloads correctly), mod_auth_cas 1.0.9 includes my patch which traps crashes caused by corrupted XML ticket files. Please let me know if this fixes your problem as we're still trying to isolate the underlying failure.

Inching towards a Pythonic Keychain wrapper

2009-03-21T21:18:42Z

I've started adding some more advanced ctypes wrappers for the OS X Keychain to PyMacAdmin. There's still a bunch of work to do but the upshot is that you can write code like this and expect it to work:

try:
    keychain = Keychain(options.keychain)
    item = keychain.find_generic_password(
        service_name=options.service,
        account_name=options.account
    )

    print "Removing %s" % item
    keychain.remove(item)
except KeyError, exc:
    print >>sys.stderr, exc.message
except RuntimeError, exc:
    print >>sys.stderr, "Unable to delete keychain item: %s" % exc

and get output like this:

chris@Enceladus:~/Development/pymacadmin [git master] $ ./bin/keychain-delete.py -a "acdha"
Removing GenericPassword(service_name='', account_name='acdha', label='Audioscrobbler: acdha')

There's a bunch of stuff going on behind the scenes now to make things easier than the sadly-unimproved state discussed at length by Wil Shipley back in 2006:

PyMacAdmin.Security now defines many of the kSec* defines from SecKeychainItem.h, simplifying many calls which either required magic numbers or complicated struct.unpack() calls.
PyMacAdmin.Security.Keychain now defines a few classes which represent some of the native typedefs: SecKeychainAttribute, SecKeychainAttributeList and SecKeychainAttributeInfo.
Core PyMacAdmin improvements to make it a little easier to work with Carbon-style APIs:
- mac_strerror() provides a way to lookup the error message for Carbon return codes
- carbon_call() simplifies the process of calling a Carbon function by automatically checking its return code and throwing an exception any time the return code is negative.
- carbon_errcheck() can be used as a ctypes errcheck function using the same logic as carbon_call()
- load_carbon_framework() wraps the ctypes LoadLibrary() method to automatically use carbon_errcheck() for every function in the loaded library - this won't work for functions which use negative return codes for non-error results but the Carbon APIs are pretty consistent in that regard.
All of the above combines to allow PyMacAdmin.Security.Keychain's find_generic_password() to call SecKeychainItemCopyAttributesAndData behind the scenes to collect the item's label and will allow arbitrary other attributes in the future.

If you have any interest in wrapping native Mac APIs with Python please join the discussion over on the PyMacAdmin group - any sort of Python-OS X integration discussion is welcome.

Using SystemConfiguration events within Python

2008-02-28T17:45:24Z

Since this post was originally written, I've been working on the PyMacAdmin project with Nigel Kersten. The information below is still correct but the kicker-replacement script has gained the ability to handle filesystem events and workspace notifications and been renamed to crankd.

In a perfect world software would gracefully network transitions. Unfortunately my users have encountered a fair number of things which don't always handle things like a laptop moving from ethernet to WiFi, a DHCP server taking awhile to respond, etc. While many programs have at least reached the point of eventually timing out and retrying it would be nice to automatically restart something as soon as the system network configuration changes. This is unfortunately system-specific and frequently required some hackish approach involving tail -f or equivalent to watch a log file, which is slow and tends to break on upgrades.

OS X has a nice way to query the current system configuration and receive event notifications when things change: the SystemConfiguration Framework (Technical Note TN1145: Living in a Dynamic TCP/IP Environment is also of interest). You can explore this using the scutil command-line tool - in the example below, I've looked at the list of available events and chosen to watch for power-state changes, receiving a notice when I unplugged the power cable from my laptop:


chris@Enceladus:~ $ scutil
> list
  subKey [0] = Plugin:IPConfiguration
  subKey [1] = Plugin:InterfaceNamer
  subKey [2] = Setup:
  subKey [3] = Setup:/
  subKey [4] = Setup:/Network/Global/IPv4
  subKey [5] = Setup:/Network/HostNames
…
  subKey [21] = State:/IOKit/PowerManagement/CurrentSettings
  subKey [22] = State:/IOKit/PowerSources/InternalBattery-0
…
> n.add State:/IOKit/PowerSources/InternalBattery-0
> n.watch
> notification callback (store address = 0x1036c0).
  changed key [0] = State:/IOKit/PowerSources/InternalBattery-0
notification callback (store address = 0x1036c0).
  changed key [0] = State:/IOKit/PowerSources/InternalBattery-0

This is pretty cool stuff but I'd like to do something smarter than scripting a copy of scutil. I could write an Objective-C application but OS X 10.5 included the very handy PyObjC 2.0 which allows access to most of the native APIs directly from within Python. James Reynolds posted a message to the MacEnterprise mailing list which prompted me to stop procrastinating and actually write some code. A little poking around later and I have a Python script which is ready for me to add whatever custom actions I want to take when the network state changes - the version below is abbreviated so you'll want to download the full watch-network-config.py for your own use:

from Cocoa import *
from SystemConfiguration import *

def handleNetworkConfigChange(store, changedKeys, info):
	print "Global network configuration changed: ", changedKeys
	# Kick a change-intolerant service in the head here

store = SCDynamicStoreCreate(None, "global-network-watcher", handleNetworkConfigChange, None)

SCDynamicStoreSetNotificationKeys(store, None, [ 'State:/Network/Global/IPv4', 'State:/Network/Global/IPv6' ])

CFRunLoopAddSource(CFRunLoopGetCurrent(), SCDynamicStoreCreateRunLoopSource(None, store, 0), kCFRunLoopCommonModes)
CFRunLoopRun()

Geoff Franks took the time to have the event handler use a dictionary so you can listen for multiple events and run a specific command for each one; I added a little syslog support and am releasing this version as a replacement for the widely-used Kicker which was removed in 10.5: Download kicker-replacement

Fun lessons from the trenches: in versions of OS X prior to 10.5 there were several nasty bugs due to lookupd and DirectoryService not having real timeouts: we have some rigs which use DHCP on our public network and static IPs on a private experiment network. When the system booted the private interface didn't need to wait for a DHCP lease and thus came up slightly faster than the public interface — this should have been harmless except that DirectoryService immediately attempted to connect to our LDAP server which isn't reachable on the private network and network timeout values aren't actually used in any version prior to 10.5, causing network accounts and NFS mounts to be unavailable until someone manually killed DirectoryService!

LDAP progress

2009-03-10T12:54:01Z

One of the fixes for the LDAP problems I wrote about last year was just integrated into pam_ldap. This would have taken a very brief amount of time except that an updated patch vanished at some point between my mail server and the upstream bugzilla — ah, software!

Friendlier shell prompts for version control systems

2009-02-03T14:49:00Z

Like many developers, I'm using multiple version control systems these days - lots of Subversion, increasingly git and even some old RCS we haven't migrated away from yet. Here's handy little shell addition which includes info in the prompt, combining this git-specific tip with my older checks:


__vcs_name() {
  if [ -d .svn ]; then
    echo " [svn]";
  elif [ -d RCS ];  then
    echo " [RCS]";
  else
    echo "$(__git_ps1 ' [git %s]')"
  fi
}

PS1='\[\033]0;\u@\h:\w\007\]\u@\h:\w$(__vcs_name) $ '

if [ -d ~/.bash_completion.d ]; then
  for c in ~/.bash_completion.d/*; do
    . "$c"
  done
fi

Note that the last bit is needed to make the git part work - depending on your bash distribution you may not need to have the for loop to process ~/.bash_completion.d. In my case, using the Mac git distribution I simply created that directory and added a symlink to /usr/local/git/contrib/completion/git-completion.bash

The final result will look something like this:


chris@Enceladus:~ $ cd Development/pymacadmin.googlecode/
chris@Enceladus:~/Development/pymacadmin.googlecode [svn] $ cd ../pymacadmin
chris@Enceladus:~/Development/pymacadmin [git master] $

In which the GOP surrenders any pretense of economic expertise

2009-02-02T23:32:30Z

From the "We need a credible opposition party" file: Mitch McConnell wasn't laughed out of the GOP for claiming that the stimulus bill is flawed because it needs more cheap mortgages.

Senate Republican leader Mitch McConnell on Monday demanded an amendment to the mammoth economic stimulus package to give government-backed, low-interest loans to homeowners -- a revision that he says will both increase the demand for houses and boost the average household income.
"We believe that a stimulus bill must fix the main problem first and that's housing," McConnell told reporters Monday in introducing a plan to offer fixed mortgages of 4 percent to "any credit-worthy borrower."

What is it about housing that causes this to be treated any more reasonably than, say, the idea that the US government should have offered cheap loans back in 2000 to protect the people who bought Pets.com at its peak? That's effectively the situation in many parts of the country and the housing market isn't going to settle on reasonable valuations any time soon if the federal government gets into the business of encouraging mortgages which no sane bank would otherwise offer. I can offer some recent experience: last week none of our local banks was anywhere near 4% even with excellent credit, a solid job, 50% down and a special employer-negotiated discount — and that's entirely what you would expect if you looked at the historical record.

McConnell is shamelessly pandering to people who don't want to accept reality - and it seems increasingly unlikely that the remaining fiscal conservatives are going to be able to put adults in charge of GOP policy any time soon. It's also likely that the buckets of cash McConnell gets from real estate might have something to do with his zealous bubble-conservation efforts…

PyMacAdmin update

2009-01-08T23:55:49Z

Macworld has inspired some PyMacAdmin activity lately: there's now a Keychain wrapper and a simple tool for updating Airport network passwords.

If you're interested in managing your Macs using Python, check out the presentation slides and join our mailing list.

No volatility in VW TDI mileage

2008-11-25T03:13:58Z

I found this review of the latest VW Jetta TDI curious:

But like many new clean diesels, which meet even California’s tough emissions rules, the Jetta scoffs at its sticker. Hoarding fuel like a mobile Scrooge, I averaged a remarkable 48 m.p.g. over more than 150 miles of freeway driving. That’s the best mileage of any American-market car I’ve tested — gas, diesel or hybrid. I never knew that driving a steady 60 m.p.h. could be so gratifying, and I vowed to try it more often.

Back when I bought my TDI in 1999 48 MPG would have seemed like a bigger deal but it's actually lower than the 52 MPG I averaged on the ~270 mile round-trip from San Diego to Pasadena and back, with an average speed over 70MPH (other trips weren't as precisely measured . I'm curious whether this is simply due to the extra power or the tighter emissions standards - intriguingly, the 42 MPG average for mixed driving matches the more or less constant value I measured for the last couple of years of mixed highway / city commuting. I know Volkswagen tweaked things to favor power-hungry American drivers for years - I'm curious whether we'll see that switch back now that everyone has rediscovered budgeting.

The sky is falling … always

2008-10-15T02:28:47Z

A great take-down for the recent “Elcomsoft made WPA obsolete” freak–out by people who should know better, pointing out that adding a single character to the password will cancel out the attacker's new-found boost. The part which I find scary and my reason for blogging is the frequent recommendation that you use a VPN instead: this is Ronco Spray-On Security and it's downright hazardous because it puts the focus on the network rather than on the insecure systems and applications which have caused almost every security problems you've ever heard about.

I first talked about this back in 2003 and it wasn't a new idea (or original to me) back then that trusting the network is a really bad idea. Unless used with more discipline than most places manage, a VPN simply enables people to continue ignoring their insecure applications. The underlying problem is that a VPN is convenient but very broad: your salespeople will start using one because you told them they couldn't access the fileserver otherwise — and pretty soon your internal network is getting scanned by the malware installed on their home PC and your network admin is getting DMCA notices because of their kids P2P habit. Worse yet, that one user with a habit of opening malware now has the ability of attacking everyone else in the organization over your mistakenly-trusted network.

There's an easier way to configure your wireless network: treat it like the internet and provide protocol-level security for the resources your users need to access, enforced by firewall rules blocking insecure protocols. This enormously reduces the support cost of VPNs, avoids the significant performance and reliability hits and gets you out of the business of trusting the network at all — which is good, because that trust was almost certainly misplaced.

Harness the power of your wingnut relatives

2008-10-18T02:24:26Z

Like many people, I have a few relatives who have persistent honesty problems when it comes to politics. I'm not talking about things like disagreeing over the best way to deal with poverty or trade but people who consider things like “Liberals are driven by Satan and lie constantly” to constitute a legitimate description of the opposition party. This election, I've been receiving these entirely in the form of multiply-forwarded smears directed at Hillary Clinton and now Barack Obama and it's been getting a bit repetitive listening to the same “Cryto-Muslim-terrorist-eats-babies-hates-America” tirade over and over again. Unfortunately, these are also the people who consider forwarding urban legends their best way to combat the dread menace facing our country and they tend not to listen to either fact-checking or requests not to forward unsourced rumours.

It turns out, however, that there's a way to both lower the volume of crap received and do some good: take a page from the people at Planned Parenthood who started the Pledge-a-Picket program and set a bounty for vicious slander: I picked $50 donated to the target (Obama, the ACLU, etc.) — $500 later and my inbox has remained clear for the first time in recent memory.

Update: the flood has gone down but not dried up. At the conservative inability to stick to facts and logic has raised ~$750 for something better…

Not that this is a huge surprise…

2008-09-13T13:19:15Z

Let's be honest: I had already ruled out voting for McCain over his integrity deficit but I was amused by the rather in-character faces of the two major campaign websites this morning:

Unsurprisingly, both Obama and the Red Cross got my dollar, along with the ACLU in honor of their new Constitution Voter campaign.

An Olympic protestor speaks out

2008-09-02T00:03:05Z

Two interesting interviews with an American protester who was held for a week by the Chinese government's in case some Western reporters were feeling particularly motivated to do their job and detract from the Olympics' warm totalitarian glow:

Powderly said that prison guards believed — correctly — that he was the man who assembled the laser but not the protest’s organizer. They accused him of trying to “murder China,” he said. “We know that you didn’t murder China, but you made the knife and you’re going to take credit for it — unless you show us the hand,’ they said,” in an apparent bid to learn the leader’s identity, Powderly said. Safely back in his Grand Street apartment, Powderly told The Brooklyn Paper how he was deprived of sleep, water, food and medicine, cuffed into painful positions, and had $2,000 stolen from his bank account — “non-lethal methods of waging war on people” that he considers “just as insidious as waterboarding.” After hours without sleep and threats against their lives and the lives of their loved ones, Powderly and the other Americans began to crack. “That’s when I started to realize that I’m really good at being a douche-baggy art star, but I’m really bad at this secret agent business,” he said.

A few years ago we'd have raised a huge fuss over stifling dissent, abuse of an American citizen (remember the Singapore caning?), etc. Now we're quiet, in no small part because the Chinese tactics described are less than those which our government considers standard operating procedure. This isn't to equate the two governments or deny that China has engaged in worse but … they also think they're protecting their country from outside threats. Once we've allowed torture and murder under that guise, who are we to say that their threats are less real than the minimally supported “threats” our government has claimed were deadly serious in the past?

I'm somewhat curious how many conservatives are conflicted between their desire to raise a fuss in service of the long-standing goal of promoting China as the new Evil Empire and the deep, abiding belief that liberal performance artists could do with a little waterboarding.

Of particular interest in this regard is McCain's latest reversal: now that he's secured the GOP nomination he's less enthusiastic about ordaining torture than he was earlier this year and, quite unusually for a key player, he isn't lying or resorting to euphemism to describe what we've been doing:

During an assessment of the Bush presidency on "Fox News Sunday," McCain discussed the administration's use over "waterboarding," a technique that has been used to interrogate terrorist detainees. "Waterboarding to me is torture, okay? And waterboarding was advocated by the administration, and according to a published report, was used," McCain said. "I obviously don't want to torture any prisoners."

So much for profiling

2008-09-01T23:40:44Z

Looks like MI5 is reluctantly giving up on profiling:

MI5 has concluded that there is no easy way to identify those who become involved in terrorism in Britain, according to a classified internal research document on radicalisation seen by the Guardian. The sophisticated analysis, based on hundreds of case studies by the security service, says there is no single pathway to violent extremism. It concludes that it is not possible to draw up a typical profile of the "British terrorist" as most are "demographically unremarkable" and simply reflect the communities in which they live. The "restricted" MI5 report takes apart many of the common stereotypes about those involved in British terrorism. They are mostly British nationals, not illegal immigrants and, far from being Islamist fundamentalists, most are religious novices. Nor, the analysis says, are they "mad and bad". Those over 30 are just as likely to have a wife and children as to be loners with no ties, the research shows.

One key point which doesn't get enough attention is that why profiles don't work they also cause problems by making easier for people who don't fit the profile to avoid the attention of the police who are over-worked investigating profile victims

Introducing crankd

2008-07-22T01:32:18Z

Awhile back I posted a kicker replacement which was also a PyObjC promo. Since then, kicker-replacement has advanced and gained the ability to handle filesystem events and workspace notifications and is now available on Google Code:

http://pymacadmin.googlecode.com

Kok-Yong Tan was kind enough to suggest a new name: crankd. I've updated the repository and even engaged in (gasp) random acts of documentation by adding a wiki page describing how to use crankd for policy-based proxy settings, in this case by enabling an OpenSSH SOCKS proxy when on a remote network as demoed at WWDC.

iPhone OS 2.0: The Good, Bad and very Ugly

2008-07-12T00:05:54Z

The iPhone's new Exchange support works really well: bidirectional pushes take only a second or two as promised, the contact app has seamless searching of our corporate directory and so on, exactly as demoed. Unfortunately, there's a major drawback: once you enable Exchange, the iPhone inexplicably disables the local calendar and address book support. Initially I had assumed that this was an unfortunate compromise to get it out the door by July 11th, along with other notable omissions like tasks and attendee information but it's actually inexcusable: the mobile iCal has full support for multiple calendars if you pay Apple $100/year for Mobile Me. ActiveSync isn't the only trick Apple learned from Microsoft…

Unix Quickie: redirecting to a pipe within a script

2008-07-01T13:23:29Z

The bash exec function has an underutilized feature allowing you to redirect stdin or stderr for the entire script, avoiding the need to make sure that the user always runs myscript.sh > output.log:

exec > example.`date +%Y-%m-%d`.log

Unfortunately, when the documentation says "If FILE is not specified, the redirections take effect in this shell" there's a key omission: you can't simply use a normal pipe to have your output go to mail, logger, etc. or in non-interactive environments like grid / batch schedulers which don't allow you to specify a pipe for job output.

Googling won't turn up much useful other than some hacks which redirect to a temporary file/pipe or re-exec the script and I wanted something cleaner. The solution is to use the bash process substitution feature which allows you to create a file handle for a given command and use that where you might normally use a filename. This is normally demoed for cool input hacks such as diff-ing the output of two commands but it's also useful for our purposes:


#!/bin/bash
exec > >(mail -s "$0: normal job output" user@example.edu)
exec 2> >(mail -s "$0: error job output" syadmin@example.edu)

... commands which will be executed normally ...