pseudofish

Timeless Recommendations

2023-11-27T00:00:00+01:00

1. Books
2. Podcasts

So much of what we consume has a shelf life, which, when turned around, means we can use time as a filter for quality. Anything that survives a certain number of years, has utility beyond now.

For me, I try and avoid reading too many recent books (inspired by Tim). I also avoid recommending anything too new, and have found my recommendations settling into a few key books and podcasts. Rather than write them out many times, I’m publishing them here instead.

1. Books

The Effective Executive (Peter Drucker) – This has been my most referred to book in my professional career. The subtitle of “getting the right things done” captures the essence of the book.
7 Powers (Hamilton Helmer) – A highly effective famework for identifying strategic power for an organization. At Spotify we made use of this framework, which is also used by Netflix leadership. The Acquired podcast uses it to analyze various companies and has interviewed Hamilton too.
Creativity Inc (Ed Catmull) – Creativity is rarely repeatable, let alone in a large organization. Pixar is one of the few that has sustained creativity over a long period of time, and this is an inside account of how they retained inspiration.
Mr China (Tim Clissold) – An insiders account to doing business with and in China over many years. While the examples are from a while ago, they highlight the historical perspective and complexities.
The Evolution of Co-operation (Robert Axelrod) – My math mind loves the idea that the incentives of co-operation can be captured algorithmically. The idea that repeated application of Prisoner’s Dilemma explains much human interaction also appeals. Finite and Infinite Games is also a good read here.
High Output Management (Andy Grove) – Much of the why behind how Silicon Valley tech companies work comes from this book. As with most things, it is worth going back to source to understand today.
On Writing Well (William Zinsser) – the best guide to writing non-fiction. Communicating effectively is critical to sharing your ideas, so do it well.

Absent from my book list are biographies, one of the key ways of learning about the history of any industry or human endeavour. The timeless advice is to read widely and deeply about the people who came before. However, it will depend a bit on your own interests as to which ones to recommend.

2. Podcasts

Manager Tools – Thanks to a recommendation from a great friend when I started my management career early in the 2000’s, Mark and Mike have provided weekly council on how to be a great manager. If you are in an organization and in any way responsible for other people, add this to your podcast player of choice.
Founders – Reviews of biographies of founders. No matter how much you read, you are unlikely to keep pace with David Senra. He will help you learn directly from history’s greatest entrepreneurs. While the podcast is amazing, you will get so much more from it if you read some of the books. The episodes will help you figure out your own interests and where you can go deeper.
Acquired – The stories behind the formation of great companies. If you are in the business of building anything new for the world, this will help you understand how it has been done before. Amazing research and compelling storytelling make it a fun listen too.

Org-mode for everything

2016-08-20T00:00:00+02:00

Org-mode is a useful emacs mode for note taking. Recently, I have been extending my use of it to include blogging and my emacs config.

Blogging with org-mode can be done using the org_reader plugin for pelican. There are other blog platforms that integrate org-mode, however this is the one I’m using. This means I can write an org-mode file that is generated into the blog.

Using org-mode for emacs config is a good example of literate programming. It is both easy to modify and self documenting. Github provides support for org docs, so my emacs config is viewable here.

By using org mode for config, it becomes normal to document why you have particular configuration settings. Given how complex configuring emacs usually is, this makes it more sane. The other win is that you can group settings by using org headings.

I use yasnippet to automate creating most documents, such as a blog entry. This helps with not having to remember specific formatting or metadata for different document types. This is useful for conducting interviews. I have created org templates for each role with criteria I want to assess and a range of questions. Then I just create a new doc, add the template, and go.

Choosing a Chinese name

2015-01-06T00:00:00+01:00

My Chinese name is 周浩轩 (Zhōu hào xuān).

Choosing a name is hard. I had early Chinese names, but they were accidental. One from Chinese class, where our teacher randomly gave us names, and one from my translated Chinese business card ( 威尔森 - wēi ěr sēn - Wilson).

For the last year, I have been working with the same tutor, and I asked her to help. Chinese names are typically associated with meaning. To start, I listed out the sorts of traits I valued in a name. From here, she came up with a list of around ten suggestions.

The challenge was which one?

I started by putting them all into a spreadsheet, and expanding out pinyin, as I didn’t know all the characters. I found that you could check names on RenRen (人人). This let me figure out a rough popularity, and gender balance. For a final sanity check, I also searched for the name on Baidu images to see if there were any bad associations.

After gathering the data, I was down to two candidates, and picked the one that I liked the characters of.

I decided on Zhou 周 as a surname by a more random approach. I looked through two lists of popular Chinese surnames. After avoiding names of friends, some musing, reading up on history and famous people with particular names, I picked.

After I had picked, it was pointed out to me that the two characters in my name reference two chengyu:

浩如烟海 — vast as the open sea
器宇轩昂 — having a dignified appearance

This adds a nice extra dimension to my name.

— 周浩轩

Triangle Heatmaps in R using ggplot

2014-10-25T00:00:00+02:00

I recently watched Alex Schultz’s lecture on Growth, and I was curious as to how triangle heatmaps would look with our own cohort data.

Fortunately, ggplot and R make it very easy to build heatmaps. The tricky bit turned out to be creating some fake data to show how they look.

First import (and install if needed) the key libraries.

library(ggplot2)
library(reshape)

Next, generate some test data, or load existing cohort data for your company.

# Generate decending values from 100 to 1 to simulate retention over time
rcohorts <- replicate(15, sort(runif(15, 1, 100), T))

# Make a triangle
rcohorts[lower.tri(rcohorts)] <- NA

# Convert to a data frame, and add tenure labels
rcohorts <- as.data.frame(rcohorts)
rcohorts$tenure <- seq(0,14)

# Reshape to suit ggplot, remove NAs, and sort the labels
rcohorts <- na.omit(melt(rcohorts, 'tenure', variable_name='cohort'))
rcohorts$cohort <- factor(rcohorts$cohort, levels=rev(levels(rcohorts$cohort)))

The data shows a particular value for a given cohort across a number of weeks.

head(rcohorts)

##    tenure cohort    value
## 1       0     V1 95.11190
## 16      0     V2 99.80269
## 17      1     V2 97.54479
## 31      0     V3 97.10274
## 32      1     V3 93.49820
## 33      2     V3 77.72841

Using ggplot’s geom_tile, it is possible to generate a heatmap based on the cohorts.

# Triangle heatmap to compare cohorts
ggplot(rcohorts, aes(cohort, tenure)) +
    ggtitle('Retention by cohort') +
    theme_bw() +
    xlab('Cohort') +
    ylab('Tenure (weeks)') +
    geom_tile(aes(fill = value), color='white') +
    scale_fill_gradient(low = 'white', high = 'darkblue', space = 'Lab') +
    theme(axis.text.x=element_text(angle=90),
          axis.ticks=element_blank(),
          axis.line=element_blank(),
          panel.border=element_blank(),
          panel.grid.major=element_line(color='#eeeeee'))

With real data, it is easy to spot anomolies, leading to areas for further investigation. For example, why did users who signed up in a particular week retain better than other weeks.

Danny Ferante explains further ways of reading the heatmap in the video.

I find the heatmap much easier to read than just plotting the cohorts:

ggplot(rcohorts, aes(tenure, value)) +
    theme_bw() +
    geom_point(aes(color=cohort))

Downloading Google Analytics data from a Python script

2014-09-22T00:00:00+02:00

If you want to regularly extract data from Google Analytics, it is worthwhile scripting the process. However, it was more complex than I expected.

I started with jaco’s post, and then added some detail from there.

Setting up your Google account

Since Google has moved to OAuth2, you need a service account to get a script to work.

There are a few steps to do from the Google Developers Console:

Create a project (eg. GAStatistics). This step may take a minute or two.
From the project page, go to APIs & auth -> APIs, and add permissions for the Analytics API (accepting the T&Cs)
From the Credentials section of APIs & auth, create a new client ID, and pick Service Account when prompted. You should now automatically download a .p12 file that includes your client ids and secret keys.
For your service account, save the email address that is generated. You’ll use in a few places.

Next you need grant your service account access to your Google Analytics account.

From Google Analytics, pick the web property you want to gather data from.
Navigate to the Admin tab at the top, and then to the User Management section (you need to have permissions, otherwise ask your site admin)
Add your service account email to Add permissions for:, and choose the type of permissions. I usually pick Read & Analyze, as I’m paranoid about messing data up with a malformed script.

From Google’s side of things, you’re ready to go.

Starting a Python project

First, you need to install the Google API Python client. I use virtualenv for scripts so I can package dependencies, so my requirements.txt looks like:

pyopenssl
google-api-python-client

For service accounts, you need to use SignedJwtAssertionCredentials, which needs pyopenssl to work. For details, see this, or just include it.

My script isn’t too different from jaco’s example, however I added a few more things to get you started.

#!/usr/bin/env python
# -*- coding: utf-8; mode: python -*-
"""
Extract a snapshot of Google Analytics data.
"""

from apiclient.discovery import build

from oauth2client.client import SignedJwtAssertionCredentials
from oauth2client.file import Storage

import httplib2

PRIVATE_KEY = '<...>.p12'  # where you store your private key
GSERVICE_EMAIL = '<...>@developer.gserviceaccount.com'
SITE = 'ga:<...>'  # eg. ga:1234

SCOPE_FEEDS = 'https://www.google.com/analytics/feeds/'
CREDENTIALS_STORE = 'credentials.dat'  # credentials cache file


def authorize(private_key_filename,
              service_email,
              scope,
              credentials_store=CREDENTIALS_STORE):
    """Authorise a service account against Google APIs"""
    http = httplib2.Http()
    storage = Storage(credentials_store)
    credentials = storage.get()

    if credentials is None or credentials.invalid:
        key = None
        with open(private_key_filename, 'rb') as f:
            key = f.read()

            credentials = SignedJwtAssertionCredentials(service_email,
                                                        key,
                                                        scope=scope)
            storage.put(credentials)
    else:
        credentials.refresh(http)

    http = credentials.authorize(http)
    return http


def print_data_table(results, trim=True):
    """Given the results from a query, print to screen."""
    if trim:
        format_str = '%25.24s'
    else:
        format_str = '%25s'

    if results.get('rows', []):
        # Print headers.
        output = []
        for header in results.get('columnHeaders'):
            output.append(format_str % header.get('name'))
        print ''.join(output)

        # Print rows.
        for row in results.get('rows'):
            output = []
            for cell in row:
                output.append(format_str % cell)
            print ''.join(output)
    else:
        print 'No Results Found'

def print_top_pages(service, site, start_date, end_date, max_results=10):
    """Print out top X pages for a given site."""
    query = service.data().ga().get(
        ids=site,
        start_date=start_date,
        end_date=end_date,
        dimensions='ga:hostname,ga:pagePath',
        metrics='ga:pageviews',
        sort='-ga:pageviews',
        filters='ga:hostname!=localhost',
        max_results=max_results).execute()

    print_data_table(query)


if __name__ == '__main__':
    http = authorize(PRIVATE_KEY, GSERVICE_EMAIL, SCOPE_FEEDS)
    service = build(serviceName='analytics', version='v3', http=http)

    # TODO(gmwils): check if service is valid before continuing
    start = '31daysAgo'
    end = 'yesterday'

    print 'Top 10 pages'
    print_top_pages(service, SITE, start, end, 10)

This structure gives a good starting point for playing around with different queries.

I did run into an issue where credentials wouldn’t refresh from the cache. This has been fixed, but may need to patch manually.

Calling Google Analytics from the API

There is a lot of different things you can extract from GA Core Reporting API. I found it very helpful to start with the common queries provided as examples.

There are also some API explorers that help with trying queries out.

For extracting data out of the results, there is example code, and details on the response fields.

I found that I was often using filters to get at the data I wanted, and the filter syntax is documented.

Some examples I found useful:

filters='ga:hostname!=localhost'  # exclude development data
filters='ga:source=~email'  # only email campaigns
filters='ga:source=~email;ga:campaign=~^\w'  # email campaigns, with a non-blank campaign tag

To avoid date arithmetic in Python, the API also allows relative dates, so you can write things like:

start = '31daysAgo'
end = 'yesterday'

You can use the dimensions tag to split by additional fields, such as ga:campaign or ga:date.

There are various selectors that can be used in queries, including ones for custom variables, goals and dimensions. Our conversion team has several custom goals setup, so I can query for the results using:

metrics='ga:goal07Completions,ga:goal07Starts,ga:goal07ConversionRate'

I feel like I’m only scratching the surface with Google Analytics every time I use it. Figuring out scripting allows me to dig into further integrations for our normal workflows.

The design of work teams

2014-09-07T00:00:00+02:00

Modern work is all about teams. They are at the core of agile development.

To help better understand how design effective work teams, it is worth reading through J. Richard Hackman’s 1987 research paper: The design of work teams.

The paper is dense reading, so I extracted some relevant sections for future reference.

Background research

From the summary of earlier research, I found this interesting:

In sum, research findings regarding process interventions suggest that structured techniques that minimize process losses (or reduce their effects) can be helpful. On the other hand, interventions that attempt to improve the quality of interpersonal relations among members to promote synergistic “process gains” appear not to yield reliable improvements in group task effectiveness.

That is, there is more improvements to gain from the lean approach to reducing waste, and less from the common soft approaches to improving team interactions.

A Normative Model of Group Effectiveness

The goal of the paper is to create a model of group effectiveness, based on research, such that:

The variables used in the model make non-trivial difference to group performance
It is feasible to change them in an organization
People can understand and use them.

To be effective, the model is restricted to work groups in organizations. Fortunately, this is the circumstances most of us find ourselves in.

Hackman goes on to propose that the overall effectiveness of work groups in organizations is a joint function of:

The level of effort group members collectively expend carrying out task work,
The amount of knowledge and skill members bring to bear on the group task, and
The appropriateness to the task of the performance strategies used by the group in its work.

These are the hurdles a group must surmount to be effective, however don’t provide clear ways of modifying group behaviour. Instead, he goes on to examine the impact of the following classes of variables:

The design of the group as a performing unit: the structure of the group task, the composition of the group, and the group norms that regulate member behaviour
The organizational context of the group: the reward, education, and information systems that influence the group, and the material resources that are put at the group’s disposal
Group synergy resulting from members’ interactions as they carry out the task

Conditions that support effort

To expect a group to work hard on the group task, the following should be met:

The group task requires members to use a variety of relatively high-level skills.
The group task is a whole and meaningful piece of work, with a visible outcome.
The outcomes of the group’s work on the task have significant consequences for other people (eg. other organisation members or external clients)
The task provides group members with substantial autonomy for deciding about hwo they do the work — in effect, the group “owns” the task and is responsible for the work outcomes.
Work on the task generates regular, trustworthy feedback about how well the group is performing.

If a group task meets these criteria, it is likely that members will experience their work as meaningful and feel collectively responsible. Improving the design of a group’s work is usually a better way to foster high collective effort than directly addressing group norms about productivity.

Organisational context

Providing a supportive environment for teams becomes the key responsibility of an organisation.

Reward systems that support high effort by teams tend to have the following three features:

Challenging, specific performance objectives
Positive consequences for excellent performance
Rewards and objectives that focus on group, not individual, behaviour

The destructive effects of rewarding individual rather than team performance can be considerable.

Group synergy

Group synergy can contribute in two ways, either by avoiding process losses or by finding ways to create new internal resources that can be used in their work — capabilities that did not exist before the group created them.

It is not clear that building a great “spirit” in the team is sustainable or if that commitment is sustainable if performance conditions remain poor.

Some other considerations:

Minimise inappropriate weighting of member contributions
Foster collective learning

Design of the group

A group’s composition is the most important condition. Well-composed groups have the following characteristics:

Individual members have high task-relevant expertise
The group is large enough to do the work (but no bigger)
Members have interpersonal as well as task skills
Membership is moderately diverse (enough diversity to cover needed talent, yet similar enough to allow understanding and co-ordination)

Task appropriate performance

To support a task appropriate performance strategy, some other factors to consider:

Design of the group
- Group norms support self-regulation
- Group norms suport situational scanning and strategic planning
Organisational context
- Clarity about the parameters of the performance situation
- Access to data about likely consequences of alternative strategies
Group synergy
- Minimizing slippage in strategy implementation (eg. get to value quickly)
- Creating innovative strategic plans

Excellent group performance requires both a good design for the team and a supportive organisation. Group synergy then acts as an amplifier to tune the impact of design and context.

An Action Model for Improving Group Effectiveness

The normative model (above) helps understand what conditions should be present. The next step is a theory of action as to how to create teams that fit the model.

Diagnosis of existing teams

The approach to diagnosis depends on the organisational structure, particularly the distribution of authority, and the tasks assigned to the team.

Most agile teams are self-managing work groups, where the team members themselves are responsible for monitoring and managing their own processes as well as executing on the task.

Their performance depends on the quality of team design, the organisational context, and on the competence of teh group in managing and executing its work.

Some teams are self-designing, where management’s role is limited to the team’s organisational context.

Creating new teams

The short version is that you should create teams that rank high on each of the variables in the model. There is much more detail in the article, broken down into four stages:

Prework
Creating performance conditions
Forming and building the team
Providing on-going assistance

Once a group is functioning as a social system, it will largely control its own destiny. Managers can assist the group by making it easy for members to re-negotiate situations that impede performance, by ensuring members get on-going assistance to operate as a team, and by helping the group learn from its experiences.

Management of teams

On leadership

Most research has focused on what leaders do within groups, however in this context, leadership has the most influence in how to frame the groups task, structure the group, its context, and to help get the group up and running.

It is not required to have an explicit leader within the group, although this may make sense if substantial co-ordination is required. This is something that should arise from within the group rather than be decided in advance.

On Creating Redundant Conditions

There are many ways for a group to be effective, and even more for it to be ineffective. Thus, it is impossible to specify in detail specific behaviours managers should adopt to help groups perform effectively.

Based on the model and research, the key to effective group management may be to create redundant conditions that support good performance, leaving groups room to develop and enact their own ways of operating within those conditions.

Group performance does not have clean, unitary causes. To help a group improve its effectiveness involves doing whatever is possible to create multiple, redundant conditions that together may nudge the group toward more competent task behaviour and, eventually, better performance.

On Managerial Authority

Given the increase in autonomy and empowerment of teams, this suggests that management attention be re-directed towards improving organisational conditions that foster and support effective group behaviour.

Managerial authority should also be used to establish and enforce standards of group behaviour and acceptable performance. Being vague can be as bad to a group as traditional hands-on supervision. To enable groups to use their authority well, managers must not be afraid to exercise their own.

On Knowing Some Things

This approach to managing effective teams may require unfamiliar and seemingly awkard management behaviours. To manage teams well, one needs to know some things, have some skills and have opportunities to practice.

Investment in training, mentoring and coaching of managagers will result in those who are expert in creating work teams, developing them, and harvesting the considerable contributions they have to make to organisational effectiveness.

Analysing Skritter time series data with R

2014-05-31T00:00:00+02:00

Skritter tracks data on my Chinese study while I use it. With their new API, I can make use of R to try and spot some trends.

With emacs, try out ESS mode. C-c C-l loads the current buffer into an ESS buffer running R.

Downloading the data

Using a client for the Skritter API, you can download either progress stats or every item you’ve ever studied. I chose to start with progress statistics.

The script I used is available here.

There is a lot of data returned across three dimensions:

word and character level stats
definition, reading, writing (rune), and tone
learned, learning, remembered, and studied items

In addition, there is the number of days studied, and the time studied in seconds. You can request by day, week or month, with different limits.

I wanted as long a timespan as possible, so downloaded two years of data at month granularity. I then dumped it all into a CSV file, with a header row.

For example:

date,client,days studied,time studied,word defn learned,...
2012-05-01,,0,0.0,0,...
2012-06-01,,18,25669.0,...
2012-07-01,,48,57054.0,...
2012-08-01,,79,96913.0,...

Loading into R

Loading the data into R is straightforward.

skritter.raw <- read.csv("data/skritter_stats.csv")

The data is now all in one place:

colnames(skritter.raw)

##  [1] "date"                    "client"                 
##  [3] "days.studied"            "time.studied"           
##  [5] "word.defn.learned"       "word.defn.learning"     
##  [7] "word.defn.remembered"    "word.defn.studied"      
##  [9] "word.reading.learned"    "word.reading.learning"  
## [11] "word.reading.remembered" "word.reading.studied"   
## [13] "word.rune.learned"       "word.rune.learning"     
## [15] "word.rune.remembered"    "word.rune.studied"      
## [17] "word.tone.learned"       "word.tone.learning"     
## [19] "word.tone.remembered"    "word.tone.studied"      
## [21] "char.defn.learned"       "char.defn.learning"     
## [23] "char.defn.remembered"    "char.defn.studied"      
## [25] "char.reading.learned"    "char.reading.learning"  
## [27] "char.reading.remembered" "char.reading.studied"   
## [29] "char.rune.learned"       "char.rune.learning"     
## [31] "char.rune.remembered"    "char.rune.studied"      
## [33] "char.tone.learned"       "char.tone.learning"     
## [35] "char.tone.remembered"    "char.tone.studied"

Cleaning up the data

Given that this is a time series, that is the first transformation to make:

skritter.cumulative <- ts(skritter.raw[, -1], start = c(2012, 5), frequency = 12)

The ts function converts the data frame into a time series object. I strip out the first column (skritter.raw[,-1]), as the time information is now in the time series.

Examining a column shows that the data is cumulative:

skritter.cumulative[, "word.tone.remembered"]

##        Jan   Feb   Mar   Apr   May   Jun   Jul   Aug   Sep   Oct   Nov
## 2012                             0   269   776  1368  2322  3357  4161
## 2013  5838  6480  7136  8455  9490 10844 12310 13954 16252 18175 20265
## 2014 23703 25574 27319 28972 30694                                    
##        Dec
## 2012  5226
## 2013 22002
## 2014

To get the increment per month, we can subtract the time series from itself with a lag introduced. Time series in R use different functions than data frames, so double check you still have a time series (eg. is.ts(...)).

skritter <- diff(skritter.cumulative, lag = 1, difference = 1)
skritter[, "word.tone.remembered"]

##       Jan  Feb  Mar  Apr  May  Jun  Jul  Aug  Sep  Oct  Nov  Dec
## 2012                           269  507  592  954 1035  804 1065
## 2013  612  642  656 1319 1035 1354 1466 1644 2298 1923 2090 1737
## 2014 1701 1871 1745 1653 1722

is.ts(skritter)

## [1] TRUE

Now all the columns show the increase (or decrease) per month, while remaining a time series.

Visualising trends

The actual numbers aren’t so interesting, as they correlate to the amount of time spent.

To see the relationship between two series, try:

plot(days.studied ~ word.rune.remembered, data = skritter)

However, the number of days isn’t as useful as the actual time spent:

plot(time.studied ~ word.rune.remembered, data = skritter)

The next thing is to compare the ratios between time spent and aspects of studying. To simplify, I used a function to extract parts of the data:

extractsummary <- function(s, aspect = "remembered", type = "word") {
    summary <- ts.union(reading = s[, paste(type, "reading", aspect, sep = ".")], 
        tone = s[, paste(type, "tone", aspect, sep = ".")], rune = s[, paste(type, 
            "rune", aspect, sep = ".")], defn = s[, paste(type, "defn", aspect, 
            sep = ".")])
    return(summary)
}

paste is used to construct the column names:

paste("word", "reading", "remembered", sep=".")

## [1] "word.reading.remembered"

And grabbed data for both words and characters remembered:

word.remembered <- extractsummary(skritter)
char.remembered <- extractsummary(skritter, type = "char")

R can operate across all the columns at the same time. This makes it easy to plot the number of character aspects remembered per hour of study.

The column names need to be re-added, so they show as plot labels.

char.remembered.per.hour <- (char.remembered * 60 * 60)/skritter[, "time.studied"]
colnames(char.remembered.per.hour) <- colnames(char.remembered)
plot(char.remembered.per.hour, main = "Character aspects remembered per hour")

Looking at the plot, I seem to be improving the number of tones that I learn per hour, and slightly improving on how fast I learn new writings. Definitions and reading seem to have taken about the same time across two years.

Repeating for words:

word.remembered.per.hour <- (word.remembered * 60 * 60)/skritter[, "time.studied"]
colnames(word.remembered.per.hour) <- colnames(word.remembered)
plot(word.remembered.per.hour, main = "Word aspects remembered per hour")

There are a few other plots that are useful.

For example, looking at the words learned by season:

seasonplot(skritter[, "word.tone.learned"])

monthplot(skritter[, "word.tone.learned"])

Looking at these, I seem to put more time into Chinese study during August, September and October.

Update on learning Chinese

2014-01-26T00:00:00+01:00

This year I started online Chinese lessons and continue with reading, listening & learning vocabulary.

My goal is to be able to read & write, however I should also be comfortable chatting in Chinese. I am not good at listening and replying in a reasonable time, making conversation stilted.

Online Lessons

My wife challenged me this year to start lessons, sending me links to a few services. After some research, I picked two to try.

The first choice was between an independent teacher or an online school. An independent teacher will be cheaper and more flexible, and the school is more reliable.

I chose to start with a school.

I then did trial lessons with eChineseLearning and Chinese Hour. Each trial lesson went for around 30 minutes followed by a sales person trying to get me to sign-up. I chose to go with Chinese Hour. They have few features, good teachers and flexible pricing.

I have class via Skype once a week for an hour. After general conversation, we go over particular lessons, such as reading a joke in Chinese and then answering questions about it. Early results are positive. I am more confident speaking, and my grammar is improving.

Make sure you know what your goals are, and use them to guide the type of service you choose.

Other services to look into:

Reading

I finished three books last year: a book of Chinese stories for children, Alice in Wonderland (爱丽丝漫游奇境记) and Around the World in 80 Days (80 天环游地球 ). Now, I’m reading Treasure Island (金银岛).

I’ve been reading children’s books that include both characters and pinyin. My challenge for later this year is to switch to reading novels without pinyin.

I find that the first part of the book can be slow going. After a while, I learn the author’s style, key vocabulary, and it becomes faster and more fun.

Finding books

When I started, I found it a challenge to get copies of things to read in Chinese. Now, I’ve moved to using e-books ( 电子书 ).

Amazon has a good range, but has limited catalogue if you are outside of China. Wandoujia seems to have lots of books that are free, although I’m not sure of their quality, and you’ll need an Android device to download them. Sina also has a book store, 新浪读书, that includes both iOS and Android apps.

HackingChinese is focused on reading during January and has some suggestions.

Vocabulary

I doubled my vocabulary across the last year, and I owe that achievement to Skritter.

Skritter fits nicely into my day. Small bits of time while commuting, or between meetings or just before sleep, and I end up with around 12 hours a month.

As I read or hear new words, I add them into my dictionary and then export them across into Skritter word lists. I find that writing the characters helps with learning the structure and improving my reading speed.

Listening

I increased the amount of Chinese listening in my day through podcasts and music.

I started with a rather open approach to exploring any music that happened to be in Mandarin, and then narrowed down to music that I enjoy. Spotify has reasonable coverage of Mandarin language music due to our launches in Taiwan, Singapore and Malaysia.

If you’re interested, check out this playlist, and these articles on Hacking Chinese. I also started using Xiami and Sina Weibo (微博) to discover and follow mainland artists.

Changing Names

2013-12-15T00:00:00+01:00

After filling in many forms, I am now Geoff van der Meer.

The paperwork part of the story is long, uninteresting, and never ending. Changing ones name as an Australian while living in Sweden just isn’t simple.

The why is shorter. My wife and I chose to use van der Meer, my mother-in-law’s maiden name, as the family name for our future children. van der Meer is a Dutch surname, and translates to English as “of the lake”.

My wife changed her name earlier this year. This left me with a few choices: either keep my current name and be different from my family, hyphenate things, or change my name.

The idea of having “Wilson-van der Meer” as a name didn’t resonate. Neither did having a different name from my immediate family.

So, a bunch of bureaucracy later, I have a new name.

I will still keep gmwils. That was allocated to me at university, and has been my username, email address, Twitter name and many other identifiers for more years than I care to count.

A History of Modern China

2013-09-15T00:00:00+02:00

Wealth and Power ( 富强 ): China’s Long March to the Twenty-first Century

Wealth and Power is a fascinating book, covering the philosophies of 11 leading thinkers and spanning from the Opium Wars in the 1800s through to modern times.

As the dramatis personae of this book abundantly illustrate, China’s modern thinkers and leaders, smarting from the humiliation of precipitous decline and foreign incursion, had their own more immediate and urgent goal, namely, the restoration of national wealth, power and greatness.

This book requires that you have a reasonable knowledge of Chinese history, and is stronger for assuming familiarity.

The people covered include:

Wei Yuan (born in 1794, died in 1857)
Feng Guifen (1809-1874)
the Empress Dowager Cixi (1835-1908)
Liang Qichao (1873-1929)
Sun Yat-Sen (1866-1925)
Chen Duxiu (1879-1942)
Chiang Kai-Shek (1887-1975)
Mao Zedong (1893-1976)
Deng Xiaoping (1904-1997)
Zhu Rongji (1928-)
Liu Xiaobo (1955-)

For a people possessed of such an abiding sense of pride and face, China’s fall in the nineteenth century from a place of such centrality and dominance was a special ignominy, and it engendered a very strong and enduring counterreaction.

Many of the ideas were brought in from tours of either Japan or the West, as well as being informed by China’s long history. By taking a long view, you also get to see how people’s ideas shift over time. I found the two chapters on Deng particularly interesting.

If you are looking for a deeper understanding of how China thinks about the world, and where their trajectory is taking them, this is a great book.

Highly recommended.

The Party: The Secret World of China’s Communist Rulers

The Party provides a comprehensive look at how the party maintains influence and control across all levels of Chinese society.

Comprised of distinct articles, covering various areas of the Party, it can be a bit tiring to read. That said, each chapter contains numerous insights into modern China. It is left to the reader to pick them out of the interviews.

The author, Richard McGregor, spent many years living in China as a correspondant for the Financial Times.

Recommended.

Country Driving

Peter Hessler is not likely to be a new name to you, if you are interested in China.

Country Driving continues his brilliant narratives on modern China with three stories. The first covers driving along remote parts of the Great Wall, the second a story of life of a single family in a small town near Beijing, and the third is in a development zone and covers the rise of a new factory.

Each story is beautiful by itself. Combined, you get the feeling of living through different aspects of China’s recent modernisation.

Highly recommended.

Learning iOS programming

2013-08-12T00:00:00+02:00

I am attempting to get into beginner’s mind as I dive into learning iOS programming.

This post includes some of the resources that I’ve found useful so far.

Books

iOS Programming — start here. This is the book to get you going with iOS programming. The remaining resources aren’t helpful without understanding the fundamentals.
Test-Driven iOS Development — my next book to read.
The Core iOS 6 Developer’s Cookbook. — yet to read, with code samples online.
The Advanced iOS 6 Developer’s Cookbook — yet to read, with code samples online.

I’m now fully sold on tech books on Kindle, as I can read on multiple devices and have available on my Mac for searching when needed.

Blogs

objc.io — advanced topics, in magazine/RSS format.
NSHipster — deep dives on a range of topics.
Cocoa with Love — lots of great content in the archives.
NSScreencasts — some free and some subscription screencasts.

Podcasts

iPhreaks — great introductary podcast. Their first episode includes lots of things to get going.
NSBrief — in depth on a range of topics with guests. A mix of iOS & Mac themes.
Identical Cousins — general chat on a range of topics.
Edge Cases — two iOS experts chatting. What’s not to like.

Other tactics

I am making a point of trying to read as much iOS code as I can. I find this helpful in seeing how other programmers have solved problems. I’m using a mix of production and example code, as well as examining particular commits that look interesting.

I am attempting to fill my Twitter feed with useful iOS content, and randomly following people relating to iOS programming. (eg. @mattt, @drance, @chockenberry, @mattjgalloway, @rwenderlich, & @secboffin)

The other thing I’m doing is to have a hobby iOS project. This gives me specific challenges to try and solve in a real app. So far, I have this deployable through TestFlight to try out on a few phones around the house.

Still so much to dig into and learn. I’m not yet comfortable with unit testing for iOS, and this is the next big thing on my list. Things like CocoaPods, AFNetworking, mogenerator, and Kiwi are likely to follow.

Rethinking how I handle email

2013-05-12T00:00:00+02:00

Email started to take too long, so I recently changed my approach.

At work, we try to avoid unbounded queues. In a distributed service architecture, they can be toxic and swamp otherwise well behaving services.

Email had become an unbounded queue. It kept piling up and swamping my day. Time for a change.

My new process

I adapted the recommendations from Manager Tools, and I now follow these guidelines:

Schedule 30mins in the morning for email.
No more than three time periods of up to 30mins during a day for email.
If I start processing email, I must finish (empty INBOX)

The key for me has been pushing myself to finish if I start. This means that a spare five minutes cannot be spent doing email.

For things that I need to do, either I book time in my calendar or put an entry in my task list, possibly with a reminder.

When I end up with something that I want to come back to later, I’ll create a task to refer back to it, and park the email in a “Follow-up” folder.

For random notes to self, I now add a task. Previously, I was cluttering my inbox and compounding my mail problem.

I also switched to using Instapaper for saving articles from Twitter, also reducing emails to myself.

Results

The last two weeks were a big improvement.

I am feeling less stressed by email, and more focused on important activites. I also feel like I have more time to handle unexpected events and to chat with colleagues.

I still catch myself checking email on occasion, however often my inbox is empty, so I’m slowly weaning myself of the habit.

For personal email accounts, I’m using a similar approach, although with a lower frequency of checking.

For my phone, I will occasionally check email there, and am a bit more relaxed about the process. I also set all mail applications to only check mail when manually asked.

Measuring email volume

Below is a chart of the email volume over the last two years for my work account. I was curious how much email I was receiving, and how much I sent.

For the last few months, I received around 3 times the number of emails as the same period last year. This is a factor of being involved in more projects, and the growth of the company. November last year was during a major project, and spiked in the number of emails I received.

I was surprised to find that I still send the same number of emails (~10-15 per day).

The chart was generated via the IMAP interface to Gmail, and this Python script:

#!/usr/bin/env python

import imaplib

def count_of_search(M, query, mailbox='INBOX'):
    M.select(mailbox_name)

    status, email_ids = M.uid('search', None, r'(X-GM-RAW "%s")' % query)
    return len(email_ids[0].split(' '))

mailbox_name = '[Gmail]/All Mail'
user, passwd = 'user@example.com', 'secret'

M = imaplib.IMAP4_SSL('imap.gmail.com', 993)
M.login(user, passwd)

print "Date,All Mail,Sent,Received"
for year in [2012, 2013]:
    for month in xrange(1,13):
        date_query = 'after: %d/%d/1 before:%d/%d/31' % (year, month, year, month)
        all_mail_count = count_of_search(M, date_query, mailbox_name)

        sent_query = '%s from:%s' % (date_query, user)
        sent_mail_count = count_of_search(M, sent_query, mailbox_name)

        received_query = '%s to:%s' % (date_query, user)
        received_mail_count = count_of_search(M, received_query, mailbox_name)

        print "%d/%d/1, %d, %d, %d" % (year, month,
                                       all_mail_count,
                                       sent_mail_count,
                                       received_mail_count)

The tricky part was finding a way to pass the X-GM-RAW parameter through to IMAP to enable Gmail search criteria. This allows you to use the same search expressions as from within Gmail.

Using deft mode for notes in Emacs

2013-03-14T00:00:00+01:00

Deft mode is one of my favorite discoveries of emacs, especially when combined with the magic of org-mode.

I use the combination of both deft and org-mode almost every day. If I’m doing an interview, taking notes in a meeting, keeping a personal backlog or ideas for a project, then this is what I’m using.

Deft solves the problem of where to put text files. Org-mode structures what goes into them.

Deft can be installed via emacs packaging. I configure it like this:

(when (require 'deft nil 'noerror)
  (setq
     deft-extension "org"
     deft-directory "~/Dropbox/Notes/"
     deft-text-mode 'org-mode))

I put the default folder within my Dropbox folder. This ensures notes are backed up and available on any computer where I’m using Dropbox and emacs.

Once in deft mode (M-x deft):

RET — open the current file
C-c C-n / C-RET — create a new file (auto named)
C-c RET — create a new file (prompt for name)
C-c C-r — rename a file
C-c C-d — delete a file

To search for a particular file, just start typing and the results are filtered in the same view.

For me, this has been near perfect for quick note taking. My current notes folder holds over 150 org files and is still fast to search.

When sharing notes from meetings with others, I use org-mode’s export feature (C-c C-e A), then highlight the part I want, and send it to the clipboard (M-| pbcopy) to paste into email.

Getting a solid system for note taking in emacs has helped with how often I use the same text editor. This was also part of my motivation of migrating my blog to text files (although markdown formatted).

Additional tips

To launch emacs from the command line with deft mode open, I set the following alias:

alias de="emacs --eval '(deft)'"

Often I have similar formats for things like notes, minutes or interviews. To avoid lots of typing, I use yasnippet to create templates.

For some examples, see my dotfiles.

Scandinavian Startups

2013-02-21T00:00:00+01:00

There is something in the water up here. The economy seems to be going well, and the press seems to have taken notice.

An example from USA Today:

If music service Spotify landed Stockholm on the high-tech map, companies such as Klarna underscore its growing influence.

In the first quarter of 2012, Sweden took in almost 20% of all venture capital invested in the European Union, trailing only Germany

And Pando Daily on Stockholm:

Thanks to Klarna, Spotify, Rebtel, and King.com, among others, Stockholm has a longer history of tech successes and more big-dollar exits than its German rival [Berlin].

For instance, the Nordic region as a whole — which still represents a much smaller population than Germany — accounted for about 6.5 percent of the world’s billion-dollar exits from 2005 to 2012, according to statistics provided by Stockholm-based venture capital firm Creandum.

There is still room to grow, as European startups in general are undervalued compared to the US:

Listed European technology firms valued at more than $100 million are on average 32 percent cheaper than their North American peers. Bloomberg says they trade at an average ratio of 15 times earnings compared with 22 times earnings in the US and Canada.

The Economist goes into depth on in their special report on The Nordic Lights:

Finland has become much more market- and entrepreneur-friendly. It has produced an impressive number of start-ups, including 300 founded by former Nokia employees.

And:

The Nordic countries have not only largely escaped the economic problems that are convulsing the Mediterranean world; they have also largely escaped the social ills that plague America.

I feel very lucky to have worked for a number of Nordic companies, and to continue my Scandinavian adventure with Spotify here in Stockholm.

New Blog

2013-02-10T00:00:00+01:00

If things went well, this post will be from my new blog.

I migrated my existing blog across to use Pelican, a static site generator written in Python. This is part of attempting to use emacs & python for as much of my workflow as possible.

During the migration, all the code and formatting were cleaned up, so it should be easy to change. Previously, all the content was in a Wordpress database, and not so easy to transform.

I’m not sure how attached I’ll be to Markdown. I am contemplating adding org-mode support to Pelican, as I use it for most other notetaking.

You may notice some duplicate articles via RSS. This is unfortunately due to changing the URL scheme.

Postgres database migrations with Python

2012-11-18T02:41:00+01:00

If you need to setup database migrations that work both locally and on Heroku in Python, these steps may be helpful.

For a recent project, I wanted to be able to manage the database schema. I’m not using an ORM, however the migration features in SQLAlchemy turn out to be quite useful.

Install the SQLAlchemy migration package, by adding the following to requirements.txt.
```
# Migrations
sqlalchemy-migrate>=0.7.2
```
and update:
```
pip install -r requirements.txt
```
Create a repository for database migrations
```
migrate create db "Cihui"
```
This creates a directory db/ where the code and migrations will reside.

Update the migrate script to use Heroku DB url

#!/usr/bin/env python

from migrate.versioning.shell import main
import os

if __name__ == '__main__':
    db_url = os.environ.get('DATABASE_URL', 'postgresql://localhost:5432/cihui')
    db_url = db_url.replace('postgres:', 'postgresql:', 1)
    main(url=db_url, debug='False', repository='db')

This should allow you to run migrations locally or on Heroku. There was a warning message on Heroku for the url starting with postgres rather than postgresql, thus the replace line.

Update Procfile

This sets up a few useful targets to run on Heroku and locally using foreman.

db_init:  python db/manage.py version_control
db_version:  python db/manage.py db_version
migrate: python db/manage.py upgrade

To migrate locally

foreman run init_db # first time
foreman run migrate

To run on Heroku
```
heroku run db_init
heroku run migrate
```
Write migrations as SQL

I prefer to manually write the migrations, and SQLAlchemy has great support for this. Refer to their docs for details.
```
python db/manage.py script_sql postgresql 'add list table'
```
This generates an upgrade and downgrade sql script in db/versions with an appropriate version number.
Test the migration:
```
python db/manage.py test
```
Runs upgrade and then downgrade on a single version.
DB access to Heroku

Console access is available via:
```
heroku pg:psql
```
Useful for checking specific values or fixing broken migrations.

I’m still iterating on this project, so may change some things around as I go. For the moment, this allowed me to manage the database schema easily in multiple databases.

Reading email in Chinese with Python

2012-10-15T16:41:00+02:00

Read email from a mailbox, in Chinese, with Python. Sounded easy.

After playing at the python prompt and many Google searches later, still no success. Then, I found two articles that helped.

Turns out that there are a few tricks to getting email parsing to work properly for multi-byte languages.

So, if you end up with strings something like:

=?GB2312?Q?Qingwen_Word_List:_=D3=D0=D2=E2=CB=BC=B5=C4=B4=CA?=

Or:

=CA=AF[=CA=AF] sh=A8=AA =A2=D9 rock =A2=DA stone =A2=DB stone inscription =A2==DC one of the eight ancient musical instruments =B0=CB=D2=F4[ba1 yin1]=20=C0=BC=BB=A8[=CCm=BB=A8] l=A8=A2nhu=A8=A1 =A2=D9 cymbidium =A2=DA orchid=20

Then, you’ll need two different approaches. One for the header strings, and one for the body text.

Given an email message, either from a string or file:

from email import header
import email
msg = email.message_from_string(str_msg)

Then, headers can be read using something like:

subject, encoding = header.decode_header(msg['Subject'])[0]
print subject.decode(encoding)

Which gives:

Qingwen Word List: 有意思的词

In this case, the encoding can be read from the email message and is ‘gb2312’.

The body text needs a slightly different approach:

print unicode(msg.get_payload(decode=True), msg.get_content_charset(), 'replace')

For:

石[石] shí ① rock ② stone ③ stone inscription ④ one of the eight ancient musical instruments 八音[ba1 yin1]
兰花[?花] lánhuā ① cymbidium ② orchid...

The ‘replace’ option will mark unknown characters using U+FFFD, rather than throw an exception.

This approach should work for most languages. Getting the various decodings sorted out allowed me to move forward on a project I’m working on. Now to re-implement it test first.

I know the format of the emails coming in, so can ignore multi-part MIME messages. If you are looking for some more details on how to handle them, check out this article.

If you were curious about the =DA encoding, it is called quoted-printable (RFC).

Good Strategy Bad Strategy

2012-06-24T22:53:00+02:00

My current reading list is mixed between coding, decision making, agile, and strategy. Richard Rumelt’s Good Strategy Bad Strategy stands out as an interesting and entertaining read.

I liked how he outlined good strategy, and also how to identify bad strategy. This really helps in pushing through from a good enough attempt to a strategy that will make a difference.

Some quotes:

Good strategy almost always looks this simple and obvious and does not take a thick deck of PowerPoint slides to explain. It does not pop out of some “strategic management” tool, matrix, chart, triangle, or fill-in-the-blanks scheme. Instead, a talented leader identifies the one or two critical issues in the situation—the pivot points that can multiply the effectiveness of effort—and then focuses and concentrates action and resources on them.

Good strategy requires leaders who are willing and able to say no to a wide variety of actions and interests. Strategy is at least as much about what an organization does not do as it is about what it does.

On identifying bad strategy:

I have condensed my list of its key hallmarks to the four listed in the beginning of this chapter: fluff, the failure to face the challenge, mistaking goals for strategy, and bad strategic objectives.

On the structure of a good strategy:

The kernel of a strategy contains three elements: a diagnosis, a guiding policy, and coherent action. The guiding policy specifies the approach to dealing with the obstacles called out in the diagnosis. It is like a signpost, marking the direction forward but not defining the details of the trip. Coherent actions are feasible coordinated policies, resource commitments, and actions designed to carry out the guiding policy.

The problem of coming up with a good strategy has the same logical structure as the problem of coming up with a good scientific hypothesis. The key differences are that most scientific knowledge is broadly shared, whereas you are working with accumulated wisdom about your business and your industry that is unlike anyone else’s. A good strategy is, in the end, a hypothesis about what will work.

The book provides insight into the structure of a good strategy. However, it stops short of providing actionable techniques or analysis to help you get there. In this, it advocates strategy work being done by someone with a deep understanding of the business and competitive landscape.

This left me wanting more, and fortunately there are books out there that provide guidelines on this type of analysis. I see a few Michael Porter books in my future reading.

Debugging Python with PDB

2012-05-03T02:51:00+02:00

I suck at using debuggers, largely because I don’t launch them often enough.

Fortunately, this year’s PyCon had a great talk from Chris McDonough on how to get started with Python’s debugger, PDB. The friendly people at Stockholm’s Python meet-up suggested it.

I made notes while watching, to remove excuses from future me on launching pdb.

Where you want to have the debugger start, add the following code:

import pdb; pdb.set_trace()

This drops you into a PDB prompt. This is more helpful than putting in yet more print statements.

Some helpful commands once you’re in:

l - list code
args - arguments to current function
p <var> - print a var
pp <var> - pretty print a var
n - next
s - step
w - where (stack trace for current position)
h - help

While in PDB, you can evaluate python code. The evaluated code won’t impact the running program.

For a bonus trick if you’re using Emacs, try out pdb track mode. Launch your python process to be debugged in an emacs shell (M-x shell). Stepping through the code in pdb will track with a source code buffer.

This worked out of the box with my emacs config. Your milage may vary.

I may actually start to enjoy debugging now!

Unit testing in Python with folder watching

2012-03-14T05:09:00+01:00

Unit testing is good. Running unit tests manually is annoying.

Rspec has a great addition in autospec, which automatically re-runs specs that are changed or have the associated code changed. This model of working has a long history in the ruby community, with autotest.

I wanted a similar thing for python/nose, but hadn’t had much luck.

The solution is py.test, with some plugins added.

This was suggested to me at the Stockholm Python user group, and I am very grateful. My testing life is now much simpler.

To get started, try something like:

$ sudo pip install pytest pytest-xdist pytest-cov

The simple way to get started is to watch your tests folder:

py.test -f test/

Now as you change files, the relevant tests will automatically be re-run. Perfect to display on that second monitor.

As I was working on some legacy code, I was looking to improve the test coverage, so wanted to see how that was going with pytest-cov:

py.test -f --cov package.name test/

Adding the package.name for your main package means you won’t generate coverage for libraries you are using. Makes the output simpler.

Still not quite happy, it wasn’t showing me which lines I needed coverage on. No problem:

py.test -f --cov package.name --cov-report term-missing test/

For comparison, here is how to get similar results from nose, but without watch support:

nosetests test/nosetests --with-coverage --cover-erase --cover-package=package.name test/

One hurdle I had with py.test was working with Twisted based test-cases. You need to ensure that you use a version of py.test later than 2.0. Debian squeeze does not package this by default, so use pip to install.

The annoyance is if you change the code so it doesn’t terminate. This still needs a context switch to kill off the tests and re-run. Keeps me focused on getting it right the first time.

pytest-xdist includes lots of really cool options, such as distribution of tests across multiple cores, versions, hosts and platforms.

I’ve since discovered this comprehensive summary of the options available. I’m happy I found this after selecting py.test, or I’d still be stuck in evaluation mode rather than writing code.

Update: check out sniffer if you want a more generic watcher framework.

Sharpen the Saw: Typing

2012-03-07T00:14:00+01:00

I once explained my job as an overpaid typist.

There is some truth in that, as much of my day is spent typing. Writing code, taking notes for meetings, email, documentation, writing blogs posts.

It would make sense to become very good at typing.

The basics are often overlooked. You can get a lot of benefit from paying attention to them.

For typing, it comes down to three main areas:

Accuracy
Speed
Touch typing

Touch typing is perhaps the most important aspect of this. You need to be able to type without looking at the keyboard. If you get really good, you shouldn’t need to even look at the screen to confirm what you’re typing.

Accuracy and speed can be built together. Focus on accuracy to start with, and then stretch for speed. There is no point going faster if you can’t maintain a high accuracy.

I found two websites to be helpful in my quest for improved typing skills:

Keyzen — a site focused on improving typing for programmers.
10 fast fingers — general typing improvement with natural language exercises.

Of course, the other thing to do to improve typing is to do lots of typing. If you are writing lots of code, email or documents, then focus on your typing while you are doing it.

It helps to have a good environment for typing, such as a comfortable keyboard, good posture, but you know this already. Add it to the list of things to focus on.

Also, make sure that you are using a program that can keep up with you as you type. I find MS Word to lag when in full WYSIWG mode, whereas emacs and vi always manage to keep up!

Update: You may also want to check out Typespeed.

About Face, Edition 3

2012-02-29T17:21:00+01:00

In any field, if you expand your view to know all the ecosystem around you it’s beneficial.

—Davide Casali

If you are a developer, a product owner, or manage a development team, much of the input is going to be about product design. It is very useful to understand a bit about a good design process, so you can better ask questions and help refine the design based on technical constraints.

About Face, by Alan Cooper, is a great book to start with.

I highlighted much of it on my Kindle!

Here are some quotes:

Good design makes users more effective.

Customers, although they might be able to articulate the problems with an interaction, are not often capable of visualizing the solutions to those problems.

One of the most powerful tools designers bring to the table is empathy

One of the most dangerous practices in product development is isolating designers from the users because doing so eliminates empathic knowledge.

narrative is also one of our most powerful creative methods.

What would a helpful human do? What would a thoughtful, considerate interaction feel like?

All ideas are broken down into discrete sections, with lots of detail and examples on how to implement this with your customers.

Goal-Directed Design combines techniques of ethnography, stakeholder interviews, market research, detailed user models, scenario-based design, and a core set of interaction principles and patterns.

This process can be roughly divided into six phases: Research, Modeling, Requirements Definition, Framework Definition, Refinement, and Support

The book discusses the mindset required to be a good designer:

in anything at all, perfection is finally attained not when there is no longer anything to add, but when there is no longer anything to take away

Next time you find yourself crowing about what cool interaction you’ve designed, just remember that the ultimate user interface for most purposes is no interface at all.

The first part of the book focuses on processes to elicit a good design. The latter part of the book covers specific examples found in user interface design and attempts to encode common patterns.

I found the first section much more valuable to how I think about designing a new product. The latter parts I felt were obvious, although that is perhaps their importance.

Well worth a read.

Note: Bret’s Magic Ink is a useful counterpoint to About Face.

Positioning

2011-11-28T20:37:00+01:00

Positioning, by Al Ries and Jack Trout, has long been on my to read list, and I wish I had read it sooner.

The core message of how to position a product is becoming more relevant as the marketplace continues to crowd.

In our overcommunicated society, the name of the game today is positioning. And only the better players are going to survive.

The book draws on a wide range of examples of different products, where a product could be a type of beer, the Catholic Church or your own career.

The key message is that your products position is determined by how it fits into the consumer’s mind. All the things that matter to you, to your company, are irrelevant if you can’t create a distinct position to separate you from everything else.

Experience has shown that a positioning exercise is a search for the obvious. Those are the easiest concepts to communicate because they make the most sense to the recipient of a message.

The authors are quite scathing about “me too” products that attempt to beat out a competitor by going head-to-head. Many examples are provided within the book.

The suicidal bent of companies that go head-on against established competition is hard to understand.

To repeat, the first rule of positioning is: To win the battle for the mind, you can’t compete head-on against a company that has a strong, established position. You can go around, under or over, but never head to head.

The leader owns the high ground. The No. 1 position in the prospect’s mind. The top rung of the product ladder.

The book covers a range of detail levels, and dips into more detail on key areas such as product naming:

As a guide, the five most common initial letters are S, C, P, A, and T. The five least common are X, Z, Y, Q, and K. One out of eight English words starts with an S. One out of 3000 starts with an X.

The authors are often quite blunt as to how they see things. This is a refreshing change from most business books:

Creative people often resist positioning thinking because they believe it restricts their creativity. And you know what? It does. Positioning thinking does restrict creativity.

I gained a lot of insight into how products have succeeded, or not, by reading this book. The examples are clear and will easily map to situations you are experiencing.

I recommend Positioning for anyone involved in creating something new. This is a book that I will re-read.

Emacs or Vim

2011-11-25T01:33:00+01:00

After over 12 years of using Vi(m), and a brief fling with TextMate, I started using Emacs as my primary editor.

Emacs

The switch has been very positive. The learning curve has been relatively steep, as my expectations from a text editor are quite high.

Emacs strength (and weakness) is that it is incredibly extensible. Where I’m finding Emacs a win over Vim is that I don’t have to leave Emacs to get things done. With Vim, I tend to use more of a mix of terminal windows and the editor.

I began by using the starter kit to get going with Emacs configuration. This made it quicker to get moving, but added a lot of things I ended up not needing. After getting more comfortable with elisp, I started from scratch and rebuilt my emacs.d folder.

To jump in quickly, I also purchased the tutorial video from Peepcode. This certainly helped as emacs is a mental shift coming from Vim.

The big benefit I have found with Emacs is the extension packages. These can be installed from the ELPA repository, and include a range of different modes.

Some of my favourite modes include:

paredit - essential for any lisp, it ensures your brackets match.
deft - simplified note taking. (I it sync via Dropbox)
magit - comprehensive Git workflow within Emacs
markdown-mode - my current default for writing notes, although I’m leaning towards org-mode now.
org-mode - highly capable note taking mode, with export options to everything. You can use it to write a book, create slides, or manage your todo list.

On the Mac, I’ve been using Aquamacs for most of my text editing, and used macports to install the command line client.

I have yet to try Emacs on Windows, as I haven’t been using Windows much at all. There is a release available here.

After cleaning up my .emacs.d configuration, I’ve now started using Emacs on Linux servers I use regularly. For temporary servers, I’ll fallback to Vim as Emacs is often not installed.

If you want to improve your emacs skills follow @emacs_knight.

Vim

To get started with Vim, it is worth reading The Vim Learning Curve is a Myth.

Over the years, the popular mythology around vim has become that it’s insanely difficult to learn; a task to be attempted by only those with the thickest of neck-beards. I’ve heard dozens of times from folks who are convinced it will take them months to reach proficiency.

These beliefs are false.

My feeling is that Vim is unrivalled for the simple task of text editing. Even after a day or two of learning, you will be faster.

Where things get a bit more complicated is when you start to realise that text editing isn’t the whole story for a text editor.

Platform support is superb. The first thing I do to a Windows machine is install Vim. It is rare for an application with a Unix heritage to be so comfortable on Windows.

Historically, I hadn’t been a fan of the graphical version of MacVim. This is something that is much better in recent releases. Load times are much improved.

Integration into external tools is where I feel Vim is lacking a bit. With the number of developers migrating from TextMate to Vim, this gap is being rapidly addressed. However, Vim isn’t as extensible as some other editors.

I really noticed this when trying out Clojure programming. If you are dealing with a REPL based environment, Vim has a way to go.

Closing Thoughts

If you aren’t using either of them, it’s not too late to start. You can’t make a bad choice.

If you are already using either Emacs or Vim, enjoy!

Both are great editors that allow you to be incredibly productive at working with text. Try learning a new feature each week. You use your text editor so often that a small improvement is a major payoff.

Calculating earlier dates using a shell script

2011-10-07T01:40:00+02:00

Mongo has a database size limit in 32 bit mode, so I want to purge out items that are less than a certain date in the past. I decided that it would be easy to write a simple shell script to run the query.

The tricky part was calculating dates in shell. This is what I ended up with:

#!/bin/sh

MONTHS_AGO=3
DATE_AGO_EPOCH=$((`date +%s` - $MONTHS_AGO * 31 * 24 * 3600))
OSTYPE=`uname`
FORMAT='+%Y-%m-%d'

if [ "Linux" = $OSTYPE ] ; then
  DATE_AGO_ISO=`date -d "1970-01-01 00:00 UTC + $DATE_AGO_EPOCH seconds" $FORMAT`
else
  DATE_AGO_ISO=`date -r $DATE_AGO_EPOCH $FORMAT`
fi

DB_CMD="db.items.find( { publish_date : { \$lt : \"$DATE_AGO_ISO\" } } ).count()"
# DB_CMD="db.items.remove( { publish_date : { \$lt : \"$DATE_AGO_ISO\" } } )"

echo $DB_CMD | mongo pz

This now provides a simple way of purging out old items that can be called from cron.

Storm - a real time Hadoop like system in Clojure

2011-09-26T02:33:00+02:00

I am very excited about the potential unleashed by Storm.

Previously at BackType, Nathan Marz has now built Storm into Twitter and then open sourced the platform.

Storm opens up a lot of possibilities, by bringing real-time distributed processing together in an elegant way. And it is built in Clojure!

Some useful links:

Rationale
Slides - detailed presentation on Storm by Nathan
Starter project - a few basic demo use cases
Mailing List
Taking the Emmys by Storm - an hour long video of how to use Storm to aggregate data from Twitter in realtime to spot trends.

While playing around I wanted to try from Clojure (most examples are in Java). There is a great DSL for Clojure that makes using Storm super easy. I forked the storm-starter project to add examples based on a Gist from Nathan.

Java interop with Clojure and Mahout

2011-08-24T02:13:00+02:00

While porting a simple example of generating a Hadoop file for Mahout, I came across a number of edge cases of the Clojure Java interop.

Nested Classes

Java allows for classes to be nested. However, Clojure doesn’t provide an obvious way of referencing them.

For example, Hadoop’s SequenceFile includes SequenceFile.Writer and SequenceFile.Reader.

To access these in Clojure, the period is replaced by a dollar sign.

For example:

(ns example
  (:import [org.apache.hadoop.io SequenceFile$Writer]))

...
  writer (SequenceFile$Writer. fs conf path
                               (class Text)
                               (class VectorWritable))
...

This thread on the mailing list has some more details.

Generics

From the Java code, generics add annotations about the type. However, at the JVM level, generics are nowhere to be seen. It turns out that generics are a compiler level feature only.

What this means for Clojure is that you can safely ignore them, provided that you honour the types that are included in the generic.

Thus the following Java:

List<NamedVector> apples = new ArrayList<NamedVector>();

Can be written in Clojure:

(def apples (ArrayList.))

The responsibility is then on the programmer to ensure the types put into the collection.

This thread on the mailing list has some background.

Primitive Arrays

I started out looking at how to construct a double[] to pass to DenseVector.

Turns out that there are a number of things to be wary of when constructing primitive arrays. Bradford Cross has written this up with examples in a blog post.

The considerations I took away were:

double-array function to migrate a Clojure data structure
make-array with Double/TYPE to pre-allocate the array
Type hints should be considered
Odd things are likely to be happening with type coercion, so check run time performance.

McKinsey on testing your strategy

2011-08-22T18:09:00+02:00

In an article from January 2011, McKinsey outlines a series of ten tests to validate your current strategy:

Summary of the tests:

Will your strategy beat the market?
Does your strategy tap a true source of advantage?
Is your strategy granular about where to compete?
Does your strategy put you ahead of trends?
Does your strategy rest on privileged insights?
Does your strategy embrace uncertainty?
Does your strategy balance commitment and flexibility?
Is your strategy contaminated by bias?
Is there conviction to act on your strategy?
Have you translated your strategy into an action plan?

Each test includes a more detailed description, and an example of a company or industry where it is particularly relevant. There are also links through to relevant articles or books.

Although aimed at corporate strategy, this is an interesting list of points to consider at a personal or project level as well.

Calling Mahout from Clojure

2011-08-18T18:07:00+02:00

Mahout is a set of libraries for running machine learning processes, such as recommendation, clustering and categorisation.

The libraries work against an abstract model that can be anything from a file to a full Hadoop cluster. This means you can start playing around with small data sets in files, a local database, a Hadoop cluster or a custom data store.

After a bit of research, it turned out not to be too complex to call via any JVM language. When you compile and install Mahout, the libraries are installed into your local Maven cache. This makes it very easy to include them into any JVM type project.

To help work through the various features, I’m reading the early access edition of Mahout in Action. I am trying out the examples in Clojure as I read through.

To get started, the following steps will setup a Clojure project to work with Mahout:

Build and install Mahout

The process installs the Mahout jars into your local maven repository, making them accessible to lein.
Create a new project
```
lein new mahoutexample
```

Add dependencies to project.clj.

For example:

:dependencies [
  [org.clojure/clojure "1.2.1"]
  [org.apache.mahout/mahout-core "0.5"]
  [org.apache.mahout/mahout-math "0.5"]
  [org.apache.mahout/mahout-utils "0.5"]]

Then load the dependencies into your project:

lein deps

Add your code

For example, ch02.clj shows calling a recommendation engine.

The time consuming part is adding in the right import statements.
Evaluate from the REPL

This is where it gets fun. Clojure makes it easy to try out different data sets and learning algorithms to rapidly iterate.
[Optional] Add a run task

In project.clj, you can assign a class as the main class. Lein will attempt to find a function called “-main” and call that.
```
lein run
```

As an example, I’ve ported the first chapter of the book to Clojure and it is available on github.

Some other articles on using parts of Mahout with Clojure are the opus artificem probat blog:

Book Reviews

2011-08-12T15:54:00+02:00

I’ve read a few books this year, thanks to Kindle for iPhone & Mac as well as a few actual printed books either found or borrowed from work’s extensive library (one of my favourite parts of our Stockholm office).

These are my highlights.

Paradigms of AI Programming

After re-discovering Lisp style programming with Clojure, this book came high on my to-read list and I was not disappointed.

The book covers a wide range of classic AI problems, and walks through solutions in Common Lisp. This makes the book partly about solving AI problems, and partly an advanced book on Lisp programming.

I loved it for both aspects.

Be warned however, Peter Norvig is a master of his tools and reading along you can be seduced into how obvious each next step is. It is worth trying to implement some of these algorithms yourself, or even try solving Sudoku.

This book won’t make you an expert on the latest AI techniques, but it will add to the class of problems you can solve, and make you a better programmer.

Into the Plex

A great insight into the history and workings of Google. There are lots of examples of how Google goes about solving problems and lots of behind the scenes information.

One section I found interesting was on staff allocation:

Around 2005, Google determined a simple formula to distribute its engineering talent: 70-20-10. Seventy percent of its engineers would work in either search or ads. Twenty percent would focus on key products such as applications. The remaining 10 percent would work on wild cards, which often emerged from the 20 percent time where people could choose their own projects.

Animal Spirits

After finishing Robert Schiller’s course on finance via iTunes University, I picked up his book Animal Spirits.

This is an approachable book that doesn’t assume too much finance background, and covers how behavioural finance fits into a macro economic framework. Or rather, how human emotions messes up existing approaches to making decisions for the broad economy.

If you’re interested in how government should be setting policy, or how human psychology impacts economics, this is well worth a read.

Toyota Production System

The Toyota Way

I started by reading Toyota Production System, as the Toyota Way was already loaned out. This turned out to be fortuitous.

The Toyota Production System is written by Taiichi Ohno, who was responsible for putting in place most of the practices that define their production system. The book is translated from the Japanese, and reads like the writings of a Zen master. Everything is very clear and concise with no superfluous words or explanations.

I found it enlightening. However, it is worth pausing often throughout the book to ponder the implications of what has been said, and was it assumes. This is a book to read to understand the fundamentals and the attitude required to implement a similar approach to your business.

The Toyota Way is an easier read, and based on the time Jeffrey Liker has spent studying Toyota. He had access to a range of plants in the US and interviewed many of the key Toyota people. His book is written as a series of stories that act as a set of case studies for how the Toyota Way was applied to a range of engineering challenges.

I particularly loved the chapters explaining how Toyota created the Lexus brand and the Prius.

To get a good understanding of how Toyota works, it is worth reading both books. I feel like it is clearer on how Kanban fits into a wider corporate picture, after having previously read David Anderson’s book.

The Time Traveller’s Wife

This was left by a previous occupant of the apartment we rented for the summer in Paris. It was a great holiday read.

Suspend disbelief for a while, and just let the story flow. The characters are well thought out and carry the story well.

Prolog in Clojure

2011-08-05T16:31:00+02:00

After reading Peter Novig’s classic, Paradigms of AI Programming, I decided to try porting his implementation of Prolog in Lisp to Clojure.

The initial results are available on GitHub.

The interpreter allows solving of simple logic problems:

(<- (witch ?x) (burns ?x) (female ?x))
(<- (burns ?x) (wooden ?x))
(<- (wooden ?x) (floats ?x))
(<- (floats ?x) (sameweight Duck ?x))
(<- (female Girl))
(<- (sameweight Duck Girl))

(?- (witch Girl))

Porting this turned out to be an interesting challenge. There were a few Lisp functions that weren’t available in Clojure, along with some subtle changes in idiom.

This gave me an opportunity to dive further into Clojure and think about how to implement the algorithms and data structures in a way that made sense, yet was still close to the original.

There is still much to do, as the book further develops the interpreter with user interaction and then refactors it into a compiler. Should keep me busy for a while longer!

Note: Please don’t consider this production code. If you want to look deeper at logic programming, consider either a real Prolog implementation, the core.logic library in Clojure or check out Mercury.

Stockholm Tourist Sights

2011-07-30T19:00:00+02:00

This is my current list of things to do and see if you happen to be visiting Stockholm for a weekend or a bit longer. The list is based on things we’ve seen, or that friends have found.

To get up-to-date tips, contact me directly or via Twitter (@gmwils).

Sights

Vasa Museum — an old ship that has been preserved due to failure to launch. Interesting museum on maritime history. There are sometimes queues to get in, so may be worth a morning visit.
Nordic Museum — provides a history of the nordic region, and is just near the Vasa Museum.
Skansen — an open air museum / zoo / cultural centre. This whole park is quite large, so it is worth spending a full day, as there is lots to do. A good chance to see reindeer and moose in Stockholm.
Fotographiska — a photography museum / gallery on Södermalm. Three floors of photography, with a nice café with a view of Stockholm on the top floor.
Modern Art Museum — meant to be quite good, but still on my todo list.
Gamla Stan - well worth half a day wandering around, see the palace, explore the streets. Almost everything is over priced, but then it is the main tourist district. It also has some of the better restaurants in town.
Södermalm — itself is one of the nicer areas of Stockholm. The shore of the island on the south is a popular place for hanging out. There are lots of nice restaurants near Slussen on the northern side too. The cliffs above Slussen provide a good vantage point to see Galma Stan (old town) and the city in general.
Östermalm — a nice area in between the city centre and Djurgården island (where the Vasa & Skansen are). Lots of nice cafés to pass the day in.
Boat trips — there are ferries that shuttle between the various islands and make a nice way to get from Skeppholmen to Djurgården.
Stockholm Archipelago — this city is all about islands, and there are a number of boat cruises that leave for the wider Archipelago. We visited Fjäderholmarna, which is small and close and made for a nice place to wander around. The article has some info on boats, they also depart from near Östermalm.

Restaurants

Rival Hotel — the restaurant here is worth a visit, a short walk from Mariatorget station
Gondolen — quite a cool location for a restaurant, suspended underneath a bridge near Slussen station. We went for drinks at their bar, which gives a good view of Stockholm. If you want a seat at the restaurant, it is worth booking ahead.
Vete-Katten — Downtown is mostly shops, however if you’re wandering around and looking for somewhere for lunch, Vete-Katten has a great range of traditional Swedish cakes.
Den Gyldene Freden — modern take on tradition Swedish food. We found the service to be friendly.
Sturehof — interesting restaurant in Östermalm.
Pelikan — Swedish home style food.
Oliver Twist (House of Ale) — cozy pub in the midst of Södermalm with a good range of local beers and hearty pub food. Bookings recommended, although we’ve been lucky before.
Djuret — means “the animal” in Swedish, and they only serve one animal at a time. Interesting concept, and has been recommended to me.

For suggestions of coffee places on Södermalm, check out this article.

Transport

To get around, it is worth getting tickets for the underground train line (Tunnelbana, abbreviated to large signs with a T). It runs pretty much everywhere and will make it easier to see the city.

To transfer from the airport, taxi works out cost effective if there are more than one person travelling. Otherwise, the train system is fast and easy to navigate.

Swedish News

For news in English about what is happening in Sweden, try The Local.

For Swedish, DN.se and for Swedish restaurant reviews try På Stan (with Google translate to help).

Language Basics

Almost everyone here speaks perfect English as their second language, so knowing some Swedish isn’t important. That said, some pleasantries always helps.

These few phrases will get you far:

Hej (pron. “Hey!) — hello, or goodbye, general greeting.
Hej då (pron. “Hey door”) — goodbye.
Tack — thanks, and please. Pointing at something and saying tack, or saying tack after receiving something is common. You’ll hear this word a lot.
Tack så mycket (pron. “Tack so mickey”) — thank you very much, often used at the end of transactions.

CS229 Machine Learning at Stanford (iTunes U)

2011-07-29T02:40:00+02:00

Andrew Ng’s course at Stanford on Machine Learning provides a great overview of the different types and applications of current machine learning algorithms. The course is available for free either on YouTube or subscribe in iTunes University.

Be warned, this is a course on theory, so is taught through mathematics rather than programming. You will develop a deeper understanding, but it may hurt your brain.

The section notes provide some assistance in reviewing the key concepts that you’ll need.

I’ve watched the course, but I keep forgetting which lesson covered which topic. To remember, I’ve put together a brief summary below:

1 — Introduction and overview.
2 — Linear regression, Gradient Descent, and Normal Equations.
3 — Linear regression & probabilistic interpretation, logistic regression (classification algorithm), Newton’s method and a brief introduction to the Perceptron algorithm.
4 — Logistic regression & Newton’s method, Exponential Family, Generalised Linear Models (GLMs).
5 — Generative learning algorithms, Gaussian Discriminate Analysis (GDA), Generative vs Discriminate algorithms, Naive Bayes, and Laplace Smoothing.
6 — Naive Bayes & Event Models, Neural Networks, Support Vector Machines (SVMs).
7 — Optimal Margin Classifier, Primal/Dual optimisation problem (KKT), SVM dual, Kernels.
8 — SVMs (Kernels, Soft Margin, SMO algorithm)
9 — Learning theory: Bias/Variance, Empirical Risk Minimisation (ERM), Union Bound/Hoeffding inequality, Uniform Convergence.
10 — VC dimension, Model selection (cross validation, feature selection), Bayesian statistics & regularisation.
11 — Bayesian statistics & regularisation, Online Learning, Advice for applying ML algorithms.
12 — Clustering (k-means), Mixture of Gaussians, Jensen’s Inequality, Expectation Maximisation (EM)
13 — Mixture of Gaussians, Mixture of Naive Bayes, Factor Analysis, and more on Gaussian distributions.
14 — Factor Analysis (EM steps), Principal Component Analysis (PCA).
15 — PCA: Latent Semantic Indexing (LSI), Singular Value Decomposition (SVD) algorithm, and Independent Component Analysis (ICA).
16 — Markov Decision Process (MDPs), Value Function, Value Iteration, and Policy Iteration.
17 — Reinforcement learning, using Markov Decision Processes (MDPs), scaled up to continuous variables. Motivating example is controlling an inverted pendulum. Models of system to control can either be derived (eg. physics model) or learned (eg. via a helicopter pilot)
18 — State Action Rewards, Finite Horizon MDPs, Linear Dynamic Systems (Models, Linear Quadratic Regulation (LQR), Riccati equation).
19 — Debugging RL algorithms, LQR (Differential Dynamic Programming (DDP)), Kalman Filters, Linear Quadratic Gaussians (LQG).
20 — POMDPs (Partially Observable MDPs), Policy Search, Reinforce, Pegasus, Conclusion.

iTunes University is a great way to keep up to date with a range of topics. Worth checking out to see what else you can find.

Replacing sublis in Clojure

2011-07-27T04:06:00+02:00

In porting some Lisp code across to Clojure, I was missing the sublis function, or at least something similar.

The closest I could find was replace, however it failed on nested structures:

user> (replace '{?x ?x_123} '(?x (?y ?x)))(?x_123 (?y ?x))

postwalk-replace from clojure.walk solved the problem:

user> (use 'clojure.walk)
nil
user> (postwalk-replace '{?x ?x_123} '(?x (?y ?x)))(?x_123 (?y ?x_123))

I almost started to write out my own recursive solution. However, I thought that walking clojure data should be a solved problem, and found that replacing elements was also a solved problem.

The more I read through the standard libraries for Clojure, the less code I write.

Bitly API via Clojure

2011-07-03T04:51:00+02:00

While playing with links in Clojure for Photozeit, I wanted to shorten them via Bit.ly and to also lookup click counts per link.

The API is quite straightforward, so I wrote a small library, clojure-bitly, to allow access to it from Clojure.

The Github page includes some example code, and the library can be downloaded from Clojars using either Lein or Maven.

Here is how to shorten a url:

user> (bitly/with-auth *bitly-user* *bitly-apikey*
        (bitly/shorten "http://www.google.com/"))
"http://j.mp/lMfE5t"
user>

The Bitly API docs have more details on which methods are available, or check out the source.

Analysis of data with MongoDB and R

2011-05-25T00:59:00+02:00

For my photography news site, I’ve ended up with a data store of a lot of blog articles in MongoDB. To try and look for patterns, I want to run some analysis. R looks like the right tool for the job.

Integration Installation

To link R to Mongo, use the RMongo library. Behind the scenes, this uses a R to Java bridge and the Java MongoDB driver.

Installation is reasonably simple:

Validate
```
$ cd ..$ R CMD check RMongo
```
Build
```
$ R CMD build RMongo
```
Install
```
$ R CMD install RMongo*.tar.gz
```

These steps assume you already have R installed and available from the command line.

Query for Data

Now from within R, you can connect to a local MongoDB. The mongoDbConnect function takes some additional arguments if your database is remote.

> library("RMongo")
> mongo  < - mongoDbConnect("db")
> result < - dbGetQuery(mongo, "items", "", 0, 10)
> result$publish_date
  [1] "2011-03-03" "2011-03-03" "2011-04-20" "2011-01-24" "2011-01-24" "2011-04-11" "2011-03-29"
  [8] "2011-03-28" "2011-03-22" "2011-03-14"

To some extent, it is as simple as that. You can use more complex queries to extract the data that you want.

The main query function takes five arguments:

database connection
collection name
query
skip - how many objects to skip
limit - total number of objects to return

For example, extract all fields from an items collection where an item was published during April 2011:

> result < - dbGetQuery(mongo, "items", "{'publish_date' : { '$gte' : '2011-04-01', '$lt' : '2011-05-01'}}")

To limit data returned to specific fields, use dbGetQueryForKeys instead:

> result < - dbGetQueryForKeys(mongo, "items", "{'publish_date' : { '$gte' : '2011-04-01', '$lt' : '2011-05-01'}}", "{'publish_date' : 1, 'rank' : 1}")

Plotting of Results

First, summarise the results into a table:

> res < - transform(result, day = weekdays(as.Date(publish_date)))
> res.days < - factor(res$day, levels = c('Sunday', 'Monday', 'Tuesday', 'Wednesday', 'Thursday', 'Friday', 'Saturday'), ordered=TRUE)
> table(res.days)
res.days
Sunday    Monday   Tuesday Wednesday  Thursday    Friday    Saturday
    30        74       123       121       131       128          86

Then, plot as a barchar:

> barplot(t(as.matrix(table(res.days))), main='Items per day of week')

Which gives the following graphic:

This makes it easy to see how many articles were available per day across April.

While I am still scratching the surface, the combination of R with MongoDB looks to be quite useful.

Reading a Twitter account from Clojure

2011-05-09T06:52:00+02:00

For my photography news site, I want to use Twitter as a data source for popular links. This involves getting tweets out of Twitter and into MongoDB.

The first step is to be able to read the tweets from an account using Clojure. The library that holds the key is clojure-twitter.

To try this out, I created a new lein project, and added the following to project.clj:

(defproject twitter "1.0.0-SNAPSHOT"
  :description "Sample project to read in a tweet stream"
  :dependencies [[org.clojure/clojure "1.2.1"]
                 [clojure-twitter "1.2.5"]]
  :dev-dependencies [[swank-clojure "1.3.0"]])

Then run lein deps and start working out the code.

The example is a good start, however it the various authentication keys didn’t quite make sense to me on first look.

After registering an application, you will need to copy four strings from your Twitter account:

Consumer key - identifier for your application, found with your app details.
Consumer secret - shown with your consumer key and almost twice as long. Keep this secret.
Access Token (oauth_token) - identifier for your user, found under the “My Access Token” button next to your application details.
Access Token Secret (oauth_token_secret) - shown with your Access Token

The access token took me a little while to find. Yours will be at a URL similar to:

https://dev.twitter.com/apps/123456/my_token

Where 123456 is the identifier of your application.

Armed with these four pieces of information, the code is quite simple. First include the API:

(ns twitter.core
  (:require [twitter :as twitter]
            [oauth.client :as oauth]))

Then setup the various keys and secret phrases:

;; Make a OAuth consumer
(def oauth-consumer
  (oauth/make-consumer "N0tr3a11ym1k3y" ;; consumer key
                       "R23mnasdfphsfjas08ujhasf08aslfkjasfd234asfd" ;; consumer secret
                       "https://api.twitter.com/oauth/request_token"
                       "https://api.twitter.com/oauth/access_token"
                       "https://api.twitter.com/oauth/authorize"
                       :hmac-sha1))
(def oauth-access-token
  "1234456789-asdfjklkjasdfjkllASDFASDF")
(def oauth-access-token-secret
  "CompletelyRandomStringWithNumbersAndLetters")

Obviously, include your own keys and tokens.

The Clojure API is easy to read from the source and follows closely to Twitter’s API.

I chose to use the https version of the API and also scope the authentication. To read the most recent tweets from my followers:

(defn latest-tweets [count]
  (twitter/with-https
    (twitter/with-oauth
      oauth-consumer
      oauth-access-token
      oauth-access-token-secret
      (twitter/home-timeline :count (str count)))))

I’m not sure why the limit parameter only takes a string. Now after running lein swank and connecting up Emacs, I can run:

user> (use :reload 'twitter.core)
nil
user> (first (map :text (latest-tweets 1)))
"OK, that's 20 videos this weekend alone I posted at http://t.co/Vm2sVQV Hope you like (please subscribe if you do!)"

There is quite a lot of information returned for each tweet, such as conversation threads, user info and retweet data. Now I just need to decide how much of it to store for later analysis.

Set the HTTP User-Agent string in Clojure

2011-04-27T02:38:00+02:00

I wanted to be able to set the User Agent string for web requests made from Clojure. This is polite when crawling other sites, otherwise the request appears as a generic Java application.

To interact with the web, I’m using a wrapper around the Apache http client.

To add into your project, put the following in project.clj:

:dependencies  [[clj-http "0.1.2"] ...]

The user agent string can be set on each call:

(def http-agent "crawlbot/1.0")
(defn get-body [url]
  (:body (client/get url {:headers {"User-Agent" http-agent}})))

The next step is to move this out to a property that the wrapper can read, so it can be set once. This refactoring isn’t yet justified, as I make a limited number of calls.

Kanban & Agile Development

2011-04-19T19:16:00+02:00

Kanban is a scheduling system for work items that I’m exploring further.

I heard about it from several sources, but didn’t focus my attention until earlier this year.

I highly recommend Kanban by David J Anderson. He is someone who is a practitioner as well as an author, which really helps in his explanations.

His book is broken up into four sections:

Introduction
Benefits of Kanban
Implementing Kanban
Making improvements

Each section is filled with references to industry theory, both in software and manufacturing, and liberally sprinkled with case studies from his own companies.

One part that really resonated with me is that introducing Kanban layers over the top of an existing development process. That is, you do not need to change your current process. Instead, the focus is on scheduling the work that goes into the process.

This makes Kanban a logical fit for processes such as Agile, as well as more traditional iterative or even waterfall development projects.

From the book:

Kanban is an approach that drives change by optimizing your existing process. The essence of starting with Kanban is to change as little as possible. You must resist the temptation to change workflow, job titles, roles and responsibilities, and specific working practices. Everything from which the team members and other partners, participants, and stakeholders derive their self-esteem, professional pride, and ego should be left unchanged.

The main target of change will be the quantity of WIP and the interface to and interaction with upstream and downstream parts of your business. So you must work with your team to map the value stream as it exists. Try not to change it or invent it in an idealistic fashion

As with any new area of knowledge, I feel like I’ve only scratched the surface. The interesting part is the range of topics to explore further, such as Deming’s writing.

It is also helpful to be working with teams that have already implemented many of the techniques from Kanban process, and to hear first hand stories of their implementation.

Emacs Update

2011-04-14T21:22:00+02:00

Now that I’ve been using Emacs for a while, I’m finding that my reflexes are adjusting to the Emacs way. On the bright side, I can still jump back into Vim and be productive.

A recent blog post highlighted the areas that I still need to explore.

I’m finding the deeper I dive into Emacs use, the more there is to find. The idea of a 30 day challenge to use nothing but Emacs is going a bit far, however I do find myself using it for more and more things.

Originally my focus was on Clojure, due to the great Lisp integration. I even have the keyboard shortcuts for Paredit mode as the background image on my screen. Now, I also use it for note taking and for organising ideas before preparing presentations. Rails coding is also emacs friendly.

A pleasant surprise is just how many controls and tools support emacs key bindings. Almost every textbox on Mac OS X for example. Typically, I make use of Ctrl-A, Ctrl-E to jump to the beginning and ends of lines, and Ctrl-D to forward delete a character.

The one area I still tend to fall back to Vim with is managing servers. I think I need some more practice with keyboard combos on terminals. I am also still to really learn emacs’ lisp.

Overall, I’m very happy with the switch.

Notes on using Maven

2011-03-14T03:38:00+01:00

I’ve been looking at build tools for JVM projects. These are the notes I took while reviewing Maven.

mvn install
mvn site - generate a summary of the project
mvn dependency:analyze - determine missing or unused dependencies
mvn dependency:tree - tree view of dependencies
mvn help:effective-pom - useful for debugging

Refactoring Maven Projects

For a given Maven project, these are some strategies to optimise the process. Mostly this is the DRY principle - Don’t Repeat Yourself.

Pull up common dependencies to “dependencyManagement”, then only refer to groupId:artifactId in the child project
Use built-ins ${project.version} and ${project.groupId} when referring to a sibling project
Use properties to re-factor version numbers across multiple dependencies from the same group

Build profiles

Use build profiles to setup different environments for dev/test/staging/production

Specify profiles as the last things in a POM
Call using -P e.g. mvn clean install -Pproduction -X
Activation parameters automatically select a profile based on a set of selectors. (ie. a JDK6 profile if JDK6 is used)
Profiles can be stored in a separate “profiles.xml” that lives next to “pom.xml”
mvn help:active-profiles
You can set a default profile in your ~/.m2/settings.xml and put passwords into settings files as well

Finding bundles

There is an online repository at http://mvnrepository.com/

Maven Properties

project.* – Maven Project Object Model, that is stuff in your pom.xml. See: http://maven.apache.org/ref/2.2.1/maven-model/maven.html
settings.* – Things from your \~/.m2/settings.xml file. See: online docs
env.* – Environment variables, such as PATH, HOME, JAVA_HOME and M2_HOME. Note: prefer ${user.home} to ${env.HOME}
System properties – things from System.getProperty(), such as java.version, java.home, os.name, os.arch, user.dir, user.home, etc.
User defined – arbitrary properties within your pom.xml

Note: you can use maven properties in resource files, such as a .properties or .xml file under src/main/resources.

You need to enable this in the pom.xml:

<build>
  <resources>
    <resource>src/main/resources</resource>
    <filtering>true</filtering>
  </resources>
</build>

Assigning custom properties per profile can help with deployment into multiple environments.

Maven & Eclipse

Eclipse has very good integration for Maven. Later versions of Eclipse include this in their package. It can also be added from http://m2eclipse.codehaus.org/

Maven Archetypes

Archetypes exist for a range of different project types.

mvn archetype:generate

Custom archetypes can be developed if you have a new type of project or generated from an existing project.

Maven sites

Maven can generate a site for your project showing information about the project and various reports.

mvn archetype:create -DgroupId=com.example -DartifactId=sample-project
cd sample-project
mvn site:run
mvn clean site
mvn site:run
mvn clean site-deploy

Repository Manager

Hosting a local repository for a team provides a number of benefits, including a local cache to avoid excessive network usage. Additionally, work can be shared both internally and externally.

A repository can be as simple as a file system with the appropriate layout, or a full repository manager. Nexus is an option for hosting a repository. It can be downloaded from: http://nexus.sonatype.org/download-nexus.html

Other Resources

Cheat sheet – comprehensive list of useful commands.
Maven - The Definitive Guide
Various Books
Using Maven with Clojure

Note: I’m continuing with lein for my clojure projects. To integrate with Maven, you can generate a pom.xml file for a with “lein pom“.

Tracking Outbound Links with Rails 3

2011-02-28T05:45:00+01:00

To gain some insight on what interests users, I want to be able to track which links they click on.

There were a number of options I considered, and finally settled on using Javascript to update an underlying Click model. For analytics, I intend to aggregate the data offline. Thus the current aim is capture only.

To store the data, I created a simple model using:

rails g model Click url:string request_ip:string user_agent:string user_id:string

I decided not to store the all the request data, and instead just track the following:

URL - these are already normalized
Request IP - I aim to use this to figure out geo-location at a later date
User Agent - which browser used
User ID - a unique hash for the user, kept in a cookie

Rails kindly adds in the created_at column, which is the final key piece of data.

To generate the click data, I added the following into my application.js:

$("a").live("click", function() {
    params = { click: { url: this.href } };
    $.post("/clicks", params);
    setTimeout('document.location = "' + this.href + '"', 100);
    return false;
});

This posts the click through to the Click controller. In the controller, I added in the following:

def create
    @click = Click.new(params[:click])
    @click.request_ip = request.remote_addr
    @click.user_agent = request.user_agent
    @click.user_id = cookies[:uid]

    respond_to do |format|
        if @click.save
            format.js { render :json => @click }
        else
            format.js { render :json => @click.errors }
        end
    end
end

To generate the tracking cookie, I included the following in application.js:

if($.cookie("uid") == null) {
    var tracking_id = $.md5($.now() + "-" + Math.ceil(Math.random() * 314159)); // Current time + random seed
    $.cookie("uid", tracking_id, { expires: 365 * 20, path: "/" });
}

By using an MD5 hash of the current time and a random number, I am comfortable enough that it is unique enough for tracking purposes.

In this example, I am tracking all clicks on the application. For production use, I’ll filter this down to only the outbound article titles.

I’ve included a skeleton Rails 3 app that includes the code on GitHub. I used a jQuery plugin for cookies and MD5 hash generation, and these are included in the public/javascripts directory.

There are a few things that this misses, such as links clicked via RSS, via Twitter, and a heap of tests. I’m looking into Feedburner and bit.ly to see if I they fill in the missing metrics.

Web Application Frameworks

2011-02-03T21:53:00+01:00

I’ve been thinking about building a web application that behaves more like a desktop application. Think MobileMe, GMail, Google Docs, etc.

This post collects together some of the research that I’ve been doing so I can refer back to it.

SproutCore

Cappuccino

Raphaël

There is a few places that have a comparison between some of these, including:

SproutCore vs Cappuccino (stackoverflow)
SproutCore and Cappuccino (stackoverflow) - if you use Cocoa, pick Cappuccino, else use SproutCore.
To SproutCore, or not to SproutCore
Building a Project Planning App (ruby or rails oceania)

From my current reading, my inclination is to dive deeper into SproutCore. Apple and Yehuda seem to be investing a lot into it. That said, Cappuccino looks quite interesting.

Update: This article provides a good summary of large scale development using Javascript.

Making Emacs even more like TextMate

2010-12-23T15:21:00+01:00

Much as I love TextMate, I’m still trying to get back to one editor. Here are a two more things to make Aquamacs even more TextMate like.

TextMate Minor Mode

Go to defunkt’s github page and follow the instructions.

This gives you access to some of the magical TextMate features. The ones I missed the most are:

⌘T - Go to File
⌘/ - Comment Line (or Selection/Region)
⌘L - Go to Line

By adding in these few shortcuts, I find it much easier to jump between the editors.

Cucumber Support

A simple set of scripts to enable cucumber support into emacs. See the installation page for how to set it up. And don’t forget the pre-req.

Once installed, you get syntax highlighting for feature files, and snippet support.

You can also setup run targets, however that requires a few more dependencies.

Other Rails Files

Try the following in your .emacs to include additional syntax highlighting:

;; Rake files are ruby, too, as are gemspecs, rackup files, etc.
(add-to-list 'auto-mode-alist '("\\.rake$" . ruby-mode))
(add-to-list 'auto-mode-alist '("\\.gemspec$" . ruby-mode))
(add-to-list 'auto-mode-alist '("\\.ru$" . ruby-mode))
(add-to-list 'auto-mode-alist '("Rakefile$" . ruby-mode))
(add-to-list 'auto-mode-alist '("Gemfile$" . ruby-mode))
(add-to-list 'auto-mode-alist '("Capfile$" . ruby-mode))
(add-to-list 'auto-mode-alist '("Vagrantfile$" . ruby-mode))

Following Url Redirects with Clojure

2010-11-11T00:33:00+01:00

In attempting to normalize urls for a project, I needed to unravel the redirects.

Urls that are provided could be via a URL shortening service, an RSS feed or from another source. To compare them, I need to determine the final destination of the URL, not what is initially supplied.

To solve this in Clojure, I used the java.net.URL class to open a connection. By calling .getResponse, the connection will follow through the redirects. If this is successful, I return the final URL, otherwise I give back the initial URL.

(def test-url "http://feedproxy.google.com/~r/amateurphotographercouk/feeds/rss/newsxml/~3/ds1Q9VsOjhI/story01.htm")
(defn resolve-redirect [initial-url]
  "Follows any redirects to the supplied url and returns the final destination"
(let [url (java.net.URL. initial-url)
      conn (.openConnection url)]
  (if (= HttpURLConnection/HTTP_OK (.getResponseCode conn))
    (.. conn getURL toString)
    initial-url)))

I’m still to test this out on a wide variety of production uses, so your milage may vary.

Note: You’ll need to include the HttpURLConnection with (:import (java.net HttpURLConnection)) in your namespace declaration.

MapReduce Examples

2010-10-30T21:36:00+02:00

In investigating MongoDB I decided to explore MapReduce to analyse the posts from a heap of RSS feeds. However, the MongoDB documentation is fairly basic. This post has a bit more detail on more complex map/reduce functions.

Looking outside of MongoDB, there are a few other groups working with MapReduce. Google started the whole thing off, and has some good resources:

MapReduce Tutorial - good overview of the process with some examples.
Original Paper - details of the original MapReduce publication
Lecture Series - I didn’t watch this, but it does seem comprehensive.
Slides - See Lecture 3.

Some examples of using MapReduce include:

Distributed Grep - emit(matched line), reduce(identity)
Count of URL Access Frequency - emit(url, 1), reduce(url, sum)
Reverse Web-Link Graph - emit(target, source), reduce(target, cons) => {target, [sources]}
Term-Vector per Host - {hostname, [term vector]}
Inverted Index
Distributed Sort

The reverse web-link graph sounds interesting:

The map function outputs pairs for each link to a target URL found in a page named “source”. The reduce function concatenates the list of all source URLs associated with a given target URL and emits the pair: .

There is also some work on using MapReduce to allow for machine classification to run in parallel. I also found a presentation on Large Scale Data Analysis that was helpful.

The main thing to get my head around is that the map function can return more complicated results than just a simple count. The reduce function then just glues the results together in some way.

And then there is the idea of multi-pass MapReduce. That could get interesting.

For now, I’m doing a lot of reading. The next step will be start to put into practice some of these ideas for analysing data. Actually, the next step is collecting the data, but that’s on a different thread.

Emacs and Colour

2010-10-20T09:53:00+02:00

After spending time with emacs, I really enjoy its support for Clojure. However, the default colour scheme made me sad.

For vim, I’ve been using desert and for TextMate, the railscasts colour theme. Fortunately, someone has already made a Railscast colour theme for emacs!

Installation was painless. Drop a file into a directory under .emacs.d and update my .emacs file with:

(require 'color-theme)
(color-theme-initialize)
(load-file "~/.emacs.d/site-lisp/themes/color-theme-railscasts.el")
(color-theme-railscasts)

Now emacs loads in a sensible colour scheme and I don’t hurt my eyes while programming.

One more thing I found was the option to set Clojure syntax highlighting in the repl window:

(add-hook 'slime-repl-mode-hook 'clojure-mode-font-lock-setup)

Owning an Agile Product

2010-09-19T17:23:00+02:00

One of the challenges in running a successful agile project is having a strong product owner.

In learning about agile processes, this is an area I haven’t read much about. This recent article on the Scrum Alliance site is a valuable addition and thinking about how to build a better product owner.

Some quotes:

Creating any product can be challenge. With a product being developed at the speed of agility, these challenges increase.

What teams need most is a product owner who is able to prioritize the product backlog.

It’s the product owner’s responsibility to know and communicate a product vision for the team. Knowing where you want to go enables you to reach your destination; the details of getting there we all discover along the way.

Product owners must be able to earn the respect of the team, resolve conflicts through conversations, and adjust priorities accordingly. There are whole books dedicated to managing conflicts. Learning when to take a stand and when to give is important. It is more important, though, that everyone has a voice.

Be concerned about creating a stable environment for the team members to learn and contribute. It’s imperative that you keep the team from being distracted.

The product owner is not a dictated hierarchy, but a service role. The customer, the team, and the company are all tuned around the product by the product owner.

It is worth reading in full.

Chinese vocabulary for Social Media

2010-09-19T15:30:00+02:00

Jeremy Webb, over at the Asia Digital Map blog, has a great list of Chinese terms for the social media web:

Location-based service: 定位服务 dìngwèi fúwù

Augmented reality: 现实增强 xiànshí zēngqiáng

Desktop client: 桌面客户端 zhuōmiàn kèhùduān

Mobile phone client: 手机客户端 shǒujī kèhùduān

App (iPhone etc.): 应用程序 yīngyòng chéngxù

…

As I’ve been using iVocabulary to study vocabulary, I added all the terms into the ProVoc (dng) format.

If you have a Mac, the file is available here (zip).

Learning Clojure with Google App Engine and Emacs

2010-09-18T23:30:00+02:00

This is an overview to my attempts at getting Clojure with Emacs. Inspired by this post, I set out to learn three things simultaneously: emacs, clojure and Google App Engine.

For comparison, my current web prototyping combination is TextMate, Rails, and Heroku. Prior to that it was a mix of python, php or perl, with vim.

There are already number of articles on getting GAE and Clojure to work. Consider this me making notes for myself because I keep forgetting the Emacs shortcuts!

Emacs

I settled on Aquamacs to start out with emacs on the Mac. It seems to be more forgiving if I use normal Mac keyboard shortcuts while I learn emacs keys. Also makes it easy to switch between it and TextMate.

The other feature I love is full screen mode: ⌘-Shift-Return

To get command line access, you need to go to Tools->Install Command Line Tools. This gives you an “aquamacs” shortcut in the shell. I aliased it to “aq“.

For some help on learning emacs, consider:

You may also want to setup Clojure to run with Emacs. This article assumes you are using lein.

Google App Engine (GAE)

The GAE distribution is available here: GAE Download.

I setup my shell to include a $GAE_BASE directory for where I unpacked it, and added $GAE_BASE/bin to my path.

Creating a Clojure Project

The first post I read assumed that you were starting with a manual layout of your project. I’m going to use lein, a build manager for clojure, similar to rake for ruby. Behind the scenes lein uses Java’s maven as a package management system (similar to gems).

To make things a bit more interesting, I plan to write a very basic CMS to explore the various APIs. I’m keeping everything on the githubs.

The steps to getting setup look something like:

$ lein new gaecms
$ mkdir -p war/WEB-INF

Then make some changes to your project.clj file:

(defproject gae-cms "1.0.0-SNAPSHOT"
  :description "A basic CMS built on Google App Engine (GAE)"
  :dependencies [[org.clojure/clojure "1.2.0"]
                 [org.clojure/clojure-contrib "1.2.0"]
                 [compojure "0.4.1"]
                 [ring/ring-servlet "0.2.1"]
                 [appengine "0.4-SNAPSHOT"]]
:dev-dependencies [[leiningen/lein-swank "1.2.0-SNAPSHOT"]]
:compile-path "war/WEB-INF/classes"
:library-path "war/WEB-INF/lib")

To save some typing, you can clone: git://github.com/gmwils/gaecms.git

To run the interpreter locally, try the following:

$ lein deps
$ lein swank

To test out that the interpreted mode is working, open up src/gae_cms/core.clj in AQ.

Then connect to the server:

M-x slime-connect

From within the editor, you can now run either of the following:

C-x C-e    % evaluate current line
C-c C-k    % compile the current file

You also have an interactive clojure shell for typing commands into. Try adding the following into your source code and then pressing C-x C-e to evaluate:

(System/getProperty "java.class.path")

You should see your current CLASSPATH printed out.

Setup for GAE

In $HOME/war/WEB-INF/web.xml map the URLs that you want your custom servlet to handle. In this case, send everything into the servlet:

<?xml version="1.0" encoding="ISO-8859-1"?>
<web-app
  xmlns="http://java.sun.com/xml/ns/javaee"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd"
  version="2.5">
  <display-name>CMS on GAE</display-name>
  <servlet>
    <servlet-name>cms</servlet-name>
    <servlet-class>gae-cms.core</servlet-class>
  </servlet>
  <servlet-mapping>
    <servlet-name>cms</servlet-name>
    <url-pattern>/</url-pattern>
  </servlet-mapping>
</web-app>

In $HOME/war/WEB-INF/appengine-web.xml add the following to setup Google App Engine to refer to your servlet above.

<appengine-web-app xmlns="http://appengine.google.com/ns/1.0">
  <application>cms-clj</application>  <!-- GAE app id for your app -->
  <version>v0-1</version>  <!-- Arbritrary Version Id -->
</appengine-web-app>

Next, update src/gaecms/core.clj with:

(ns gaecms.core
  (:gen-class :extends javax.servlet.http.HttpServlet)
  (:use
    compojure.core
    [ring.util.servlet :only [defservice]])
  (:require [compojure.route :as route]))

(defroutes cms-public
  (GET "/" []
    "<html><title>GAE CMS</title><body><h1>Hello World!</h1></body>")  )
(defroutes cms
  cms-public
  (route/not-found "Page not found")) ; 404 error page

(defservice cms)

To test this locally, try:

$ lein deps
$ lein compile
$ $GAESDK/bin/dev_appserver.sh war

Deploying to Google

First go to the GAE console and create an application. The following should then work:

$ lein deps
$ lein compile
$ $GAESDK/bin/appcfg.sh update war

I’m still playing with getting it working with Google App Engine. The above is enough to get hello world working.

For more details, I suggest you follow the Compojure on GAE blog. Specifically, the article on Deploying to App Engineis a good overview.

Next steps include getting interactive development working with GAE, and actually handling dynamic content.

How to fix online advertising

2010-08-13T15:12:00+02:00

Fark’s Drew Curtis is interviewed by the O’Reilly team and has some good insights on how to succeed with internet advertising.

It’s not the model. It’s the pricing. That sounds obvious, but it all comes down to effective sales methods. Digital salespeople are slowly building a narrative that not all ad impressions are equal, and those that are wrapped around compelling content are more valuable than, say, impressions that rip by when you poke someone on Facebook.

…

it’s not really any one metric. It’s more about what ratio of uniques to visits to page views is the best. Ideally, you want a large number of uniques who visit on a regular basis and consume 3-5 page views per visit.

And a few ideas on improving sales:

Change the sales narrative — Not all impressions are created equal. Not all audiences have the same value.

Reduce industry-wide available inventory

Augment with subscription content of some kind — Any kind.

If you have a community, do an annual event.

If you have a content niche, sell product on it.

Cut overhead dramatically — Legacy media was used to having individual regional monopolies with 40-percent profit margins

There is a lot more in the article.

Agile Retrospectives

2010-07-02T16:54:00+02:00

Presentation from the authors of the book “Agile Retrospectives - Making Good Teams Great” at Google. Lots of great tips on how to improve your sprint & project retrospectives.

One tip I picked up on was to ask “how was the energy for this sprint?” rather than the bland “how are you feeling?”.

Their structure for a retrospective is:

Set the Stage
Gather Data
Generate Insights
Decide What to Do
Close the Retrospective

The way to succeed with a process such as agile is through continuous improvement. The forum for reviewing what needs to be improved and to decide on how to improve things is the sprint retrospectives.

Retrospectives are critical for having a successful project.

Change Host for a Git Origin Server

2010-06-28T21:19:00+02:00

Hopefully this isn’t something you need to do. The server that I’ve been using to collaborate on a few git projects with had the domain name expire. This meant finding a way of migrating the local repositories to get back in sync.

Update: Thanks to @mawolf for pointing out there is an easy way with recent git versions (post Feb, 2010):

git remote set-url origin ssh://newhost.com/usr/local/gitroot/myproject.git

See the man page for details.

If you’re on an older version, then try this:

As a caveat, this works only as it is the same server, just with different names.

Assuming that the new hostname is “newhost.com”, and the old one was “oldhost.com”, the change is quite simple.

Edit the .git/config file in your working directory. You should see something like:

[remote "origin"]
fetch = +refs/heads/*:refs/remotes/origin/*
url = ssh://oldhost.com/usr/local/gitroot/myproject.git

Change oldhost.com to newhost.com, save the file and you’re done.

From my limited testing (git pull origin; git push origin; gitx) everything seems in order. And yes, I know it is bad form to mess with git internals.

A/B Testing Overview

2010-06-25T10:41:00+02:00

A/B Testing is how sites like Amazon, Yahoo, Google, Facebook and others became the best in the world. They used data to drive decisions about content, layout, wording and other aspects of their sites.

Smashing magazine covers the details of A/B testing in this post.

At its core, A/B testing is exactly what it sounds like: you have two versions of an element (A and B) and a metric that defines success. To determine which version is better, you subject both versions to experimentation simultaneously. In the end, you measure which version was more successful and select that version for real-world use.

37signals tested the headline on its pricing page. It found that “30-Day Free Trial on All Accounts” generated 30% more sign-ups than the original “Start a Highrise Account.”

Dustin found that “You should follow me on Twitter here” worked 173% better than his control text, “I’m on Twitter.”

Along with its other A/B tests, CareLogger increased its conversion rate by 34% simply by changing the color of the sign-up button from green to red!

The article covers the results from others of testing, as well as details on how to get started for your own site.

What I find interesting is that the tools that used to require significant investment in time and development are increasingly available as easy to use, free solutions.

Using TextMate with Google Docs

2010-06-22T11:19:00+02:00

The new Google Command Line opens up some interesting possibilities. One that I wanted to explore was using TextMate to edit basic Google Docs.

First, install Google’s Command Line tool. Details are on their wiki for each OS. There are lots of tips in the comments and even a packaged installer. (I used ports to install)

The next step is to create a symlink from the “mate” command to “mate_wait”. What this does is default the wait argument to true. This is needed for the Google script to detect changes to your document.

sudo ln -s /usr/bin/mate /usr/bin/mate_wait

You can now open a document in TextMate:

google docs edit --title "Ideas" --editor "mate_wait"

If you select a title for a document that doesn’t exist, it will be created. Once you close the file in TextMate, control returns to the shell and the document is updated on Google’s servers.

You are only able to edit in text only mode. So it works best on documents without any formatting.

To list your existing documents, try:

google docs list

RSpec XHTML Validation of Views

2010-06-19T16:02:00+02:00

A great way to learn is to read other people’s code. And in that vein, Chris Lowe has a great post on suggested OSS Rails apps to look into.

One technique I picked up from [simply_agile][] is to validate the HTML output by your views. That way, bad HTML isn’t introduced.

To start with, you need to install the [assert_valid_xhtml][] plugin and the libxml-ruby gem.

In your [spec/spec_helper.rb][] file, add the following inside the config block:

config.include ValidateXhtml

And at the end of the file, outside the config block, include the following:

describe "a standard view", :shared => true do
  it "should be successful" do
    response.should be_success
  end
  it "should be valid" do
    response.should be_valid_xhtml
  end
end

Now in your view specs, include the following line:

it_should_behave_like "a standard view"

And that’s it. Now each view will be tested for valid xhtml when you run your specs.

There are a lot of other great techniques scattered through the projects mentioned in the original post. And if you want more details on good practices for RSpec, check out the RSpec Book.

Learning Chinese - continued

2010-06-18T15:29:00+02:00

Since my last post, I’ve spent a fair bit more time on improving my Chinese. Some recent changes to my routine have improved the speed at which I’ve been learning.

[][]

Lingt

I already use flash cards, but am now spending more time using Lingt.com. They run an online language learning program, that includes audio and writing. My accuracy is improving on words I swore I already knew, but turns out I didn’t know that well.

The site also includes badges and incentives to keep you coming back each day. They remember what you’ve been studying and only allow you to study a certain amount each day, keeping your memory fresh. This is a key part to their success. They’ll replay words at the point where you forget them, increasing the effectiveness of your time.

Reading

This was so simple I couldn’t believe it. To get good at reading Chinese, start reading Chinese.

It has always been my goal to be able to read Chinese content on the web. However, I had also believed that my Chinese wasn’t yet good enough.

Reading Kató Lomb‘s book was enlightening. She taught herself more than 16 languages. In her book, she outlines her techniques.

The one that stood out for me was her dedication to reading material in the desired language, well before you can fully understand it. Instead, aim for getting the gist of it on the first read. Don’t look up every word in a dictionary, rather look up those that re-occur lots. Then re-read the book.

For the past few months, I’ve been trying to read two books in Chinese. I find both interesting and I’m learning a lot of Chinese as I go. It doesn’t (yet) feel like reading, but more like attempting a crossword puzzle.

When I started, it would take me around half an hour to muddle through a paragraph. Now, I can often get through between a half and a full page.

One caveat, you do need to allocate time to it. I aim for half an hour a day. Kató suggests that you should be spending at least 12 hours a week on learning a new language.

[][]

Talking

I read a great article recently on learning Chinese. The interesting part for me was to look at finding Chinese speakers who’s English is worse than my Chinese. This makes sense once you think about it; I just hadn’t thought about it.

I still have yet to action this particular plan, and am keeping it on my todo list for now.

[]: http://www.flickr.com/photos/gmwils/3106536779/ “Great Wall at Badaling, China by gmwils, on Flickr”

[]: http://www.flickr.com/photos/gmwils/3578792744/ “Shopping, Chinatown, Singapore by gmwils, on Flickr”

Clojure with TextMate on the Mac

2010-06-17T21:41:00+02:00

Clojure has been getting a bit of press recently, such as prominence on the ThoughtWorks Technology Radar. Having spent some time learning Lisp in the past, I thought it could be a good holiday project to play with.

This post pulls together some of the resources I found for getting started. I’m not going to try and sell you on a Lisp dialect that integrates into the JVM, it should be self evident.

TextMate Setup

TextMate has a bundle that makes it very simple to get started. It includes the clojure runtime, and so you can run clojure code after installation. To install (assuming you have git):

cd ~/Library/Application\ Support/TextMate/Bundles
git clone git://github.com/nullstyle/clojure-tmbundle.git Clojure.tmbundle
osascript -e 'tell app "TextMate" to reload bundles'

Then create a folder, open it in TextMate, create a file (ie. test.clj) and put some sample code into it:

(+ 1 2 3)
(println "Hello from the console")
(. javax.swing.JOptionPane (showMessageDialog nil "Hello World"))

Put the cursor into one of the expressions and press Apple-R. TextMate will figure the rest out and show the result. Apple-Shift-R will run all the expressions.

(The Vim setup looks really good, but I’m committed to learning TextMate at the moment)

Tutorials

I’m not going to be exhaustive on this, but thought I’d include a few pointers (see also Alexandre’s summary). There is a video series available on YouTube, and also on BlipTV.

This is an overview in a cheatsheet, and there is a fairly complete language reference here and a longer list of references as well.

Given that Clojure is Lisp based, it would make sense to understand Lisp. The best option is a book called Practical Common Lisp. This is available on the web for free, or can be purchased from Amazon.

Update: Apparently I was off a bit on the utility of the Common Lisp book.

From @cemerick: @gmwils FYI, Clojure is a lisp, but has little to do with Common Lisp. For books, try http://bit.ly/ccSw2Z or http://joyofclojure.com/

Quantitative Trading

2010-05-11T00:36:00+02:00

Quantitative trading is something I know almost nothing about, but is a topic that I find interesting. Mixing computers, math and some luck to automate trading strategies just sounds fun.

When a friend recommended Quantitative Trading by Ernie Chan, I quickly added to my reading list. What surprised me, was that I finished the book within 24 hours!

There are a few things to like about this book. Ernie’s writing style is very no-nonsense. Every chapter has a logical flow to it, and he is crystal clear about the material he is introducing.

The other part I really like is the practical examples. Theory is introduced only as a way of getting to a measurable outcome. Examples are provided in both Excel and Matlab, and could easily be extended to other automation approaches.

Ernie also covers a the wider business aspects of getting setup as an independent quant, such as the trade-offs between retail and proprietary accounts, risk management, finding, tuning and backtesting trading strategies.

At each point, references are made to further material on each topic, such as books, research papers, online articles and implementations. I certainly felt that I had enough knowledge to go about setting up a simple quantitative trading business.

Even though I do not intend to actually set up as a trader, I found it very informative to understand the industry better. It is something I may come back to in a few years.

Agile projects – From Concept to Product Backlog

2010-04-30T18:13:00+02:00

Agile is something that I’ve previously found to work very well with in-house R&D projects, and rather more challenging when negotiating projects with customers.

The challenge is getting from the sales cycle through to the agile development iterations, be they sprints or otherwise.

By using an agile process, the advantage is that many smart people have been here before. Some of the knowledge is locked up in consultancy houses, while some of it is shared.

Gerard Meszaros has a good presentation highlighting some of the challenges and what often gets missed. The slides are available here.

The section on adding more time into the Product Planning phase fits well with how customers often see projects. By providing scope at the feature level, it allows boundaries to be placed around time and money.

This is certainly an area of agile that I would like to understand more.

Machinist, AuthLogic & Cucumber

2010-03-21T02:40:00+01:00

For a Rails app, I wanted to define rules about authentication using Cucumber and a bit of BDD. This went well, until I added in Pickle, Machinist, and AuthLogic; a few too many new things at once.

I first ran afoul of defining users and then password confirmation. This is easy to fix, as a Machinist spec is Ruby code:

Sham.email { Faker::Internet.email }
Sham.username { Faker::Internet.user_name }
User.blueprint do
  username
  email
  password 'secret'
  password_confirmation { password }
end

I also created a named admin user:

User.blueprint(:admin) do
  username { Sham.username + " -admin-"}
  admin { true }
end

I can then add some user specific step definitions for Cucumber using Webrat:

def user
  @user ||= User.make
end

def admin_user
  @admin_user ||= User.make(:admin)
end

def login(user)
  user
  visit '/login'
  fill_in("Username", :with => user.username)
  fill_in("Password", :with => user.password)
  click_button("Log in")
end

def logout
  visit '/logout'
end

Given /^I am a logged in user$/ do
  login(user)
end

Given /^I am logged in as an admin$/ do
  begin
    login(admin_user)
  rescue
    save_and_open_page
    raise
  end
end

Given /^I am not logged in$/ do
  logout
end

The three behaviors I want to define are for guest, non-admin and admin users. This gives me the following:

Scenario: Can't see new entry as guest
  Given I am not logged in
  When I go to path "/entries"
  Then I should not see "New entry"

Scenario: Can't see new entry as non admin
  Given I am a logged in user
  When I go to path "/entries"
  Then I should not see "New entry"

Scenario: Can see new entry as admin
  Given I am logged in as an admin
  When I go to path "/entries"
  Then I should see "New entry"

In writing this blog post, I cleaned up some of the features and found a few missing authentication paths. One further thing to ponder is what happens when someone visits your controller directly. I think you should also add defined behaviours for this, such as:

Scenario: Can't add new entry as non admin via controller
  Given I am not logged in
  When I go to path "/entries/new"
  Then I should be on the home page

A big thank you to Railscasts. Ryan has done several casts on Cucumber and AuthLogic that really helped get me started.

If you need to sham a Paperclip file with Machinist, check out Tim Riley’s blog. This articlewas also useful, although I found after I had most things working.

Strawberry Swing - Amazing Hand Drawn Animation

2010-03-09T16:40:00+01:00

Read the interview with Shynola’s Chris Harding for some hints on how they made it.

Learning Ruby on Rails

2010-02-15T13:30:00+01:00

This year, my technology goal is to learn Ruby on Rails.

I often try to learn new programming languages, averaging around one a year. Sometimes it is a matter of coming back to a language I haven’t used in a while, other times it is learning a new framework.

After using Ruby for some scripting last year, I decided that Rails was a good choice to learn as a framework this year. Two things convinced me.

Firstly, lots of web innovation appears to arise from the Ruby community. Cucumber is one that has particularly caught my interest.

Secondly is Heroku. This is simply an awesome way to bootstrap a project. No mess, no fuss, code - deploy - repeat. It means I can launch an early version of the app with minimal investment of time or funding.

To dive in deep on a platform like Rails, I’ve taken a few steps:

Books – Admittedly still on order, I’m investing in some Rails books, as I find books provide broader coverage of topics than blogs alone.
Blogs – My RSS reader is being filled out with Ruby and Rails blogs. I subscribed to 20 odd to start with and will curate them over time.
Podcasts – The Rails community is served by a number of great podcasts and screencasts. Railscasts is a great entry point and I added others for broader coverage.
Twitter – Where I find bloggers or people that are influential in the Rails community, I add them to my Twitter list.
Friends – To start the process, I also asked a few friends who were Rails experts for suggestions of where to start. Pete put together quite a good list.
Tools – Installing Rails was a good start. I also added TextMate to my tool collection. There are some nice features that may cause me to make changes to my .vimrc one day.

All of that is the easy part. The next step is to actually write a bunch of Rails code. So far I’m tinkering with example code. However, I do have a larger project in mind which should provide the incentive to carry through. Details on that are for a later date.

If there are sources of good Rails info that you can recommend, please let me know either in comments, Twitter or email.

Top 10 Photos of 2009

2010-01-24T16:20:00+01:00

With much consideration, this is my selection of my top 10 photos from 2009. (in no particular order)

[][]

[][]

[][]

[][]

[][]

[][]

Banff, Alberta, Canada

[][]

Waiting

[][]

Parisian Light

[][]

Gingerbread Christmas Trees

[][]

The larger selection of my favourite images from 2009 is on Flickr.

For comparison, last year’s photos are here.

[]: http://www.flickr.com/photos/gmwils/3451714283/ “Canoes by gmwils, on Flickr”

[]: http://www.flickr.com/photos/gmwils/4084397069/ “Conspiracies – Fi & Benn Wedding by gmwils, on Flickr”

[]: http://www.flickr.com/photos/gmwils/4034941248/ “Tulip Festival – 1 by gmwils, on Flickr”

[]: http://www.flickr.com/photos/gmwils/3417231719/ “Karen & Phil by gmwils, on Flickr”

[]: http://www.flickr.com/photos/gmwils/4146401689/ “Issabella by gmwils, on Flickr”

[]: http://www.flickr.com/photos/gmwils/3762032574/ “Cupcakes by gmwils, on Flickr”

[]: http://www.flickr.com/photos/gmwils/3286185687/ “Banff, Alberta, Canada by gmwils, on Flickr”

[]: http://www.flickr.com/photos/gmwils/3824998303/ “Waiting by gmwils, on Flickr”

[]: http://www.flickr.com/photos/gmwils/4299713026/ “Parisian Light by gmwils, on Flickr”

[]: http://www.flickr.com/photos/gmwils/4157139687/ “Gingerbread Christmas Trees by gmwils, on Flickr”

Online Training with Lynda.com

2009-09-07T16:41:00+02:00

With my CS4 registration, Adobe gave me one month of a Lynda.com subscription. I didn’t think this would be useful, and was expecting to learn a few things. Turns out, it’s been immensely useful.

I recommend trying it for a month. My current plan is to buy a month subscription one every year or so. Or maybe every six months.

I consider myself reasonably effective with Photoshop and the digital darkroom, yet still learnt many new processes and techniques for producing a completed photograph. The following courses were interesting or inspirational:

I also updated a few other Adobe applications, such as InDesign and learnt some Flash basics too:

There are other online video learning sites, such as Kelby Training, that are also worth a look. The important bit is to keep updating your skills and improving your workflow.

Two New Blogs

2009-07-23T22:07:00+02:00

M & I are starting a food blog here: Dreaming of Winter.

After spending the start of this year doing lots of wonderful cooking and shooting a bunch of photos to go with it, turning the passion into a blog made lots of sense.

The theme is based on our mutual love of winter. Hearty winter soups, rich roast meats and delightful desserts. There may be some lighter summer fare, but don’t count on it.

I also started another photography blog where I intend to post about recent shoots, something I’ve avoided going into too much detail with on this blog.

The Man in the High Castle

2009-06-28T16:02:00+02:00

Dilemma of civilized man; body mobilized, but danger obscure

- page 161

While on holidays I finished Phillip K Dick’s book, The Man in the High Castle, a story of how the world might look if the Allies lost the war. Like all great science fiction, this is a biting commentary on our own culture. Well worth a read.

External Accessories on iPhone 3.0

2009-06-19T19:54:00+02:00

My initial reaction to the teasers for 3.0 was excitement. My understanding of the opening up of the accessory port to developers would allow for fun innovation. I was wrong.

The dream I had was to be able to connect my iPhone, via USB, to my camera(s). This would allow my phone to act as a remote control, and as an external screen for photos & video. The gPhoto libraries looked like they could provide a starting point.

What is actually provided doesn’t allow for straight USB connectivity.

The External Accessory support requires a validated accessory, with appropriate hardware protocols, to work. You also need to be a member of two programs, iPhone Developer and the Made for iPod program.

I can’t see Canon or Nikon releasing cameras with iPod dock connectors or protocols.

This doesn’t leave many options. One is to build a hardware device that is USB in one end and a custom protocol out the other. This gets tricky fast as the hardware device needs to figure out the camera support. And I really liked the idea of a simple cable connection being all that is required.

The second option still has some merit: WiFi. There are a few downsides. Firstly, you need a camera that supports WiFi or an expensive adaptor. This limits the audience for the app. Secondly, you need a wireless network available and to also manage pairing between your iPhone and your camera. This starts to be more equipment than desirable.

If you have a laptop lying around, onOne software released such an iPhone app.

Life in the 21st Century

2009-06-02T09:29:00+02:00

On the flight, I get to watch lectures on iPhone development from Stanford, along with talks from a conference in California.
Catch up with family and friends in Singapore, as a result of sharing my location on Facebook.
Photos I took of Singapore are instantly shared with other friends and family around the world.
I keep in touch with my girlfriend back home using free telephony via the internet (thanks Skype).
A friend at work recommends a book to me, and I order it from America to arrive at my Melbourne office, along with some other recommended books to read.
Reviews, freely published on the internet, guide me through a tricky lens purchase. Global warranty allows me to haggle in Singapore’s Chinatown for a lower price than back home.
Hear about a new album by Fat Freddy’s Drop, a New Zealand band, on Twitter on my phone, purchase the album and instantly(ish) download via wireless internet
My camera (5D2) gets updated with new features today, via the internet, making it more video capable.
Subscriptions via NetNewsWire and Tweetie on the iPhone guide me through the bankruptcy intricacies as two 20th Century car companies fall apart.
I can share this with you, as publishing is freely available.

Thinking back only a few decades, most of this would seem science fiction. I’m looking forward to the next few!

ps. I still hold out hope for personal flying devices in my lifetime.

Twitter & The Modern News Cycle

2009-05-19T08:50:00+02:00

Yesterday, Monday 18th May, an earthquake measuring 5 on the richter scale struck California.

I first heard about the quake from John Grey (@[jg_rat][]), the editor of couriermail.com.au. He posted the following:

RT @bartman6: FLASH: Quake hits LA http://bit.ly/jtA6M

The Courier Mail site published a small teaser with a few paragraphs of text and a map of the areas affected. The site also included a reference to this content at the top of their homepage.

As the story unfolded, the article was expanded to include more information.

The article was the first mention in Australian press of the earthquake. John’s team beat the other major news sites, including ABC.com.au, SMH.com.au and even their parent site, news.com.au. At the time of posting, even CNN had not updated their site with news of the earthquake.

I asked John why his team got the news out so early:

@[jg_rat][]: @gmwils The answer is Twitter. And a fast, smart, connected news team.

LA Times was also quick to respond to the story. Their first post, in addition to their own copy, included a link to a Twitter search for earthquake.

Twitter is adding a lot of value to the breaking news gathering and publishing processes.

High Tea in Sorrento

2009-05-18T09:24:00+02:00

After being overly busy with work, a weekend in Melbourne allowed for relaxing travel. Michelle and I went down to Sorrento to visit a friend for high tea.

Lots and lots of cakes, some beautiful sandwiches and then more cakes. All mixed in with a wide variety of tea. Conversations were diverse and drifted between deep and idyllic.

A wonderful way to unwind.

It was also a good opportunity to play more with my new camera. I am really pleased with the results. The colors out of camera are more inspiring than from my old camera.

There are a few more on my Flickr page.

A New Theory on Brains

2009-05-09T16:02:00+02:00

Jeff Hawkins‘ TED talk on brain science got me thinking. He posits that the act of thinking is about predicting, rather than about behavior. This model of reasoning makes sense, and will simplify the computer modeling of thought.

What I find interesting is that our own brian is not great at prediction. We are making progress in the study of our own biases and their impact on prediction and decision making.

For discussion, “Judgement under Uncertainty” is a must read; a collection of papers on our known biases.

When we computerize this model, do we introduce the same or similar biases? Does it make sense in silicon, to recreate our own limitations? What happens when we invent a new form of consciousness with bias removed?

Jeff’s talk is here:

ps. TED talks are a great way to pass time on flights.

An Experience Economy

2009-03-30T18:00:00+02:00

Joseph Pine’s talk at TED really has me thinking.

This is a table outlining the difference between the types of economic output, using the example of coffee.

Economic Output	Business Imperative	Consumer Sensitivity	Coffee Example
Experience	Render	Authenticity	$5.00 per cup (Starbucks)
Service	Improve	Quality	$1.00 per cup (coffee shop)
Goods	Control	Costs	$0.10 per cup (supermarket)
Commodity	Supply	Availability	$0.02 per cup (coffee beans)

The drive to software-as-a-service makes the software industry jump to service. It takes Apple to move it to an experience. Disney is another example of experience based economics.

I like this framework for thinking about what business I am working in. The trick will be to figure out how move up the ladder to render an authentic experience.

His book is now on my to-read list.

50 Years of Style

2009-03-25T08:17:00+01:00

“Strunk and White’s The Elements of Style” has sold more than 10 million copies since its initial publication in April 1959.

Newsday highlights the 50^th anniversary of the classic text on English grammar and writing style.

This is the book on the subject.

The best book of writing advice I’ve ever read. I read it cover-to-cover about once a year.

– John Gruber

I considered offering my copy out for loan. However, I would not be happy without a copy in my possession. It is a small book, and you owe yourself an edition of it.

It saddens me that writing style seems a fading art. This book is not read enough.

Orbis Ring Flash for Events

2009-03-03T22:18:00+01:00

The Orbis Ring flash is a low budget option for using portable strobes to power a ring flash. There are two big advantages to the Orbis: light & cheap. That said, the build quality is very solid and more than I was expecting.

I decided to try mine out at a friend’s housewarming. This was the result:

![Housewarming][]

To shoot with, it was a breeze. My SB-24 mounted snug in the adaptor. An old sync cord linked my 5D2 to the flash and I was away. The results were impressive.

For shooting an event, such as a party or a wedding, the ring flash gives a nice fall off of light without too much in the way of highlights. For candid portraits, it was simply wow.

I’m looking forward to trying it out in a more staged environment. There are a few lighting options I’d like to try, such as using it for fill or using it from off-axis.

The Orbis team has put a few videos on YouTube, and there is a group of Flickr showing what people are doing with it.

Gestalt Theory & Photography

2009-02-24T19:28:00+01:00

The Gestalt effect refers to the form-forming capability of our senses (the word Gestalt in German literally means “shape” or “figure”), particularly with respect to the visual recognition of figures and whole forms instead of just a collection of simple lines and curves.

from Wikipedia

Basically, a bunch of psychologists spent lots of research to figure out how we view images and derive meaning from them.

Turns out, this research is very useful for thinking about photography. Jeff Curto put together two podcasts explaining the connections:

Go With the Gestalt (part 1, part 2)

The four main visual ideas to add meaning to your photos are:

Closure
Continuance
Similarity
Proximity

Reading the wikipedia article with a view to photographic composition can help open up some visual opportunities. Jeff’s podcast is very useful for thinking about the creative side of photography.

Photo Editing Workflow

2009-02-19T15:52:00+01:00

Pressing the shutter on the camera is the simple bit. Once the photo is taken there are a large number of options around workflow. This is my current process.

Transferring the Photos

The obvious way to get photos from the camera to the computer is via USB cable. I don’t like doing this for a few reasons. One is that it drains the camera battery, the other is that it can be slow.

I use a card reader to mount the memory card on my computer and then use Lightroom (LR) to copy the files across.

Depending on my camera or the shoot, I may assign a pre-set on import. This matters more on my Point & Shoot (P&S) cameras, as the default processing is fairly bleh. Adobe’s defaults for my Canon SLRs is usually pretty good as a starting point. I also have a metadata pre-set that assigns various contact & copyright information to the image. I add additional metadata, such as keywords, as I import.

With folder structures, I’ve been through a few. My current layout is to place everything from one “Event” into a date prefixed folder. This can be assigned in the Import dialog. For example, if I went to Banff, it would be “2009-02-15 Banff”. The date indicates when I off loaded the photos, so the folder can span multiple days for an event. I can also have multiple folders per day.

The advantage of this layout is that if I stop using Lightroom, I still have a vague chance of figuring out where a particular photo is. When it was just by date, there wasn’t enough information in the file system to be helpful.

Initial Edit

After LR has built the previews, and sometimes while it is, I will do an initial edit. At this point, I’m viewing the photos for the first time on a computer. If I consider an image acceptable, it gets one star (‘1’ on the keyboard). If a photo is beyond usable, such as bad focus or lens cap on, I’ll set it as a reject (‘x’). Depending on the shoot, I’ll delete the rejects from disk (Ctrl-Backspace / Cmd-Delete). Occasionally, if I really like an image on first viewing, I will give it two stars (‘2’).

From this point, I use the filtering in LR so that I will only see 1 star photos or higher ratings.

Second Edit

The second time through, I’m more critical of the photos. Each much be interesting in its own right to receive two stars (‘2’). While I’m viewing the photos I’m in the Loupe view (‘e’). I typically have LR maximized (‘f’) and with the sidebars hidden (Shift-Tab). I also darken the user interface. I set the dim level in preferences to 50% and use this to hide most of the UI.

When there are several images that are very similar, such as in a portrait or sports shoot, I’ll use the compare (‘c’) mode so I can see images side by side. Pressing the up key while in this mode promotes an image to the left, while allowing you to continue through. Promoted images will get two stars.

If I want to see more than two images at once, I use the survey view (‘n’) after selecting the images from the grid (‘g’).

Post Production

After filtering down to two star images, I start to consider candidates for post-production. It is very rare that I’ll output an image without some level of post. Much of this I now do in LR.

Still with dim set, I switch to the Develop module (‘d’) and start applying corrections in LR. This can include things like fine tuning white balance, exposure, saturation, sharpening or applying subtle vignettes. Much of this I save in pre-sets, so can apply with a single click and preview before applying. Sometimes I use pre-sets from other sites to try out different looks.

If I have a series of similar photos, I’ll use LR to adjust one of them and then sync the settings to the other photos in the shoot. I do this often to set the white balance from a gray card (or by eye) and then make it the same for all the shots.

With the adjustment brush and gradient filter in LR2, there is much less time I spend in Photoshop (PS). There are still some images that I’ll finish in PS. These are typically composites, or portraits that require more retouching. I use LR to manage the PS files and to group the PS file in a stack with the original image.

Once I finish in Photoshop, I’ll occasionally re-adjust things further in Lightroom. At this point, I consider the image done.

Final Edit & Presentation

Once post is complete I try to leave some time before doing a final edit. At this point I’ve spent too long with the photos and am not very objective. I find more clarity after a night’s sleep. The images that are in my mind when I wake are often the stand out ones. Editing right away often means that the one I spent the most time on is foremost in my mind, and may not be the best image.

In the final edit, I’ll assign three star images (‘3’). These are ones that I consider the best of the shoot.

If I want to group images for export to a particular destination, I’ll use one of the colour labels (‘6’) or the Quick Collection (‘b’) to select them. Then I can filter based on these properties and export to external systems or print. Flickr, Gallery or Facebook all have plug-ins for LR, so can be configured as export pre-sets. Make sure to set the colour space to sRGB and set the appropriate size and sharpening for the destination. Having the destination web site re-size and (not) sharpen results in less great images.

I also have some pre-sets for exporting with the right file format and colour space for the print labs.

Now that I have a photo printer at home, I will often print a proof before sending to an external lab. This allows me to see what the photo will look like printed. It is a different experience to hold the photo in your hands than see it on the screen, and I’ll often go back to post and make changes.

Archival

Up until this point, the workflow has been carried out on one of my laptops. I have two copies of the image (one on the card, one on the computer). As I use the camera to format the cards prior to shooting, this gives me some redundancy. However, the images need to be archived.

This is an area that I can improve. My simple strategy is to copy across to an external USB mounted hard drive. Ideally, I should backup to a second hard drive and/or burn to DVD.

Future Usage

My pictures now are easy to find using Smart Collections. Any long term projects that I’m working on get specific keywords assigned and turn up in a specific Smart Collection that searches for the keyword and a particular star rating. I also have some collections based on year, so I can see how I’m progressing.

I can use search to locate images for particular projects, or select images for trying out a new Photoshop technique.

Ever few months, I’ll go back over my 3 star images to consider candidates for 4 stars. The 4 star images are ones I would consider for portfolio usage or fine art sale. To date, I have less than a handful of 5 star images. These are ones that I consider ‘wow’.

My aim is to shoot more images that make it to 5 stars.

Media 09 Conference - Summary

2009-02-18T12:56:00+01:00

Media 09 is a new media conference held in Sydney. I attended last year’s, and missed this year due to being in Canada. Fortunately, Brad from lagrangepoint was there and blogged many of the talks.

Below are some of the highlights from his coverage (click through for full articles):

Brian McCarthy, Fairfax CEO:

In discussing the media environment, he pointed out the differences between the US and Australia, including the greater investment in colour printing and the more discreet geographic distribution of our population that has protected Australian publishers.

He also reiterated his faith in content - and quality journalism - being the key to survival, as the company continues to merge its print and online offerings and diversify into new markets

Caroline Little, from The Guardian News and Media USA:

The role of the newspaper as a record of yesterday’s news has gone, and newsrooms need to adapt.

The newspaper is now no longer the most effective means for an advertiser to reach an audience, with the market becoming more fragmented.

She said there are four key areas - multimedia storytelling, database journalism, reader engagement and citizen journalism. Fear of failure must not hold proprietors back.

Understanding Twitter and Facebook are vital for understanding engagement and their viral nature, and it is important to understand how content is being distributed away from the publisher’s website.

Nic Newman, BBC:

Content must also be made sharable, within the boudaries of content rights, via widgets and other portals so that content gets to where the audience actually is.

Ben Self, used social media for the Obama campaign:

There were more than 1800 videos distributed to supporters, and 50 million views of this content. 68 million Americans were reached either online or in person throughout the campaign. More than US$770 million was raised - US$500 million raised online.

He said that having a dynamic and interesting email program was crucial, with more than 13 million on the distribution list by the end of the campaign. They also made it easier for people to communicate themselves through the web site.

He also looked at tools to target and organise voters geographically, to help get people in neighbourhoods together who want to be part of the campaign.

Meg Pickard, The Guardian Community manager:

“We didnt make any money, but we generated an enormous amount of goodwill,” Pickard said. “We sold several hundred, but we had seveal thousand people linking to it on Facebook.”

There is a wholly trinity of technology, people and editorial. She also talked about the transition from casual users to connected users, committed users and finally catalysts - the final category being the ones who represented less than 1 percent of the audience, but are blogging and twittering about you, saving stories on del.ico.us and creating pages on Facebook, that you need to find a way to engage with.

Richard Cardran, Emmy award winner for his work in digital media:

Cardran says we have reached a point of maturity, to the point where if a business doesn’t have strategies in business for Facebook, MySpace or mobile, then they really should, because all it takes now is willpower.

Cardran says you need a strategy for ecosystem and for syndication, and that leads to a meta strategy. He spoke about creating value around an object, service or presence, not the object, service or presence itself.

End users should be allowed to design custom virtual newspapers and magazines, and they should be able to virally distribute their custom publications. He says newspapers should also create hyper-localised applications to exploit local advertising revenue.

Browser Wars Hot Up; Who Cares?

2009-02-03T06:38:00+01:00

Recent reports highlight Internet Explorer’s drop in percentage of internet users. Given the huge war between Mozilla and IE, it sounds like big news, but is it?

These days I don’t think browsers are as important. The innovation in one is rapidly copied to another. The UI differences between Firefox, IE, Safari and Chrome are minimal. Each of them does a good job of rendering standards based HTML, CSS and Javascript.

Each vendor has a use for their browser. Microsoft has a component version of IE it can embed in most of Windows. Apple has a component version of Safari that it can use across Mac OS X and iPhone OS. Google’s browser is optimized to run Google’s style of application. Firefox is a great platform for building browsers, while also being a browser.

The important thing today is to own a destination worth browsing to. And that is a much harder challenge.

Top 10 Photos of 2008

2009-01-27T20:56:00+01:00

After several hours of editing, these are my top 10 from 2008.

Great Wall at Badaling

Maheno Wreck, Fraser Island

The candidates for my best pictures of 2008 are in this set; lots of depth from a photographic year.

Published in the Age

2009-01-22T20:51:00+01:00

I was lucky and in the right place at the right time with a camera.

Phil needed a portrait submitted to the Age for an article about Astronomy. While we were down at camp last week, I took the opportunity to make a few of him.

We tried out a few styles, and the photo that ran didn’t make it into my final selects. But then I wasn’t looking at the portrait purely for an editorial piece.

This is the photo used:

These are my two favorites from the shoot:

Timelapsed Tents

2009-01-21T21:06:00+01:00

My challenge for 2009 is to become better at video production. So far I’ve spent a bunch of time learning to use Final Cut Pro, and reading up on editing techniques. However, the best way to learn is to try.

The video above is the result of a bit of an experiment.

For camp, we pitched 9 large A-Frame tents. This seemed a good event to try out some timelapse photography.

Three still cameras were used to capture the footage. My Ricoh GX100 was set to take the wide angle at a rate of two frames per minute. This was let run for the entire event. Phil took the close up video with a Canon G9. It captures time lapse video straight to Quicktime. I also used my DSLR to capture some stills.

The wide angle stills were processed in Lightroom to improve the colour, add a vignette and crop slightly. Exported to JPG and imported into a FCP with 2s per frame. The G9 footage was cut in over the top, along with the still frames.

To tie it together, transitions were added to each cut and motion added to the stills to create a sense of movement. The final image includes a text overlay created using Title 3D. Compressor then rendered the final output, that YouTube then crushes.

Beijing Restaurants ( 北京饭馆 )

2008-12-14T23:25:00+01:00

While in Beijing, I did a lot of dining out. There were some great experiences and one questionable one.

Huajia Yiyuan Restaurant - frequented by Cathay Pacific flight crews when in Beijing, this restaurant is close to the top of my list for Beijing. With a wide variety of food on the menu and a tea list longer than most wine lists, there is much to like. The atmosphere is wonderful, nestled in the midst of Ghost Street, without succumbing to the surrounding commercialism.
Da Dong - Modern way to experience Peking Duck, Beijing style. The mini “hamburger” variant was interesting to try, with its heavy garlic. The classic duck was classic. Extensive menu allows for a wide array of choice for accompanying dishes, beautifully photographed. I considered enquiring to buy a copy of their menu, given the print quality, typography and photo budget; it bordered on food porn.
Quanjude - Classic roast duck. The most famous in Beijing and for a good reason. Established over 130 years ago, this is where government officials go to eat, with many private rooms available, if you book in advance. We had a set menu that included just about every part of the duck. Duck hearts were the big surprise in how tasty they were. Duck feet were edible, but not quite to my taste. The roast duck itself was classic and delicious.
Sichuan Restaurant - if you’re staying near the Hard Rock Cafe in Beijing, keep walking. There is a great Sichuan style restaurant just around the corner from it. They have English menus (with pictures), but not much English for the staff. Very good food considering the lack of price. The staff were fascinated by Super Monkey Ball, which was playing on my iPhone while I waited for my meal. Not sure of the name of it though, so may be tricky to find.
Local food - there were many local food places I dined at that had wonderful food. Requires a bit of enthusiasm to try some of it, but many more hits than misses. Don’t turn your nose up at something without at least trying it first. You may find something you really love.
Hard Rock Cafe - ok, so I caved on my last night in Beijing. After many hours walking around the Great Wall, I was not feeling adventurous. I regret this decision. I thought a burger would cure my woes. The burger was average and overpriced. The service was classic American style; not a good thing. I was interrupted three times while trying to eat, each time to ask if my meal was okay. Certainly wasn’t improved by hassling me about it.

I’m open to suggestions for future visits. I’m not staying in Ningbo long enough to write too much about it, although I did stumble across a place with traditional local food that struck me as quite strange; they did not serve tea! The food here is quite good from the little I’ve tried. I am not holding out much expectation for Shenzhen; hopefully I will be surprised.

Learning Chinese

2008-12-12T02:18:00+01:00

After almost two years of study, there are many different approaches I’ve tried for learning Chinese. These are the ones that stuck the most. I’m putting them into order of terms effectiveness, however I do get excited about ones I most recently discovered. My aim is to be able to speak Chinese via MSN with friends. This is going to take time.

I am biased towards the iPhone for software. Access to a dictionary, to learn on the go, is invaluable.

ACCS - I recommend attending class. It helps for motivation. It helps to learn with a teacher. It helps to be encouraged by peer pressure if you fall behind your classmates. ACCS helps me learn to speak and to learn how to read & write characters. Try and find a class in your local area.
ChinesePod - useful for both their regular podcast and forums. The best podcast on the internet, although now mostly hidden behind a paid wall. I’m yet to consider paying as am still working through their back catalogue. Jenny and Ken provide a lively dynamic that make learning fun.
Dianhua Dictionary - comprehensive Chinese/English dictionary on a phone for free. Hard not to like. Combined with native input methods and can search using pinyin or by drawing characters. I also use a hard copy Oxford Chinese/English dictionary on occasion, but it mostly stays on the shelf.
iVocabulary / ProVoc - flash card system with a Mac based desktop app(free) and an iPhone client. Creating paper flash cards was never going to happen. Putting this on the computer helped. Getting it to my phone means I actually study.
Visiting China - at the moment I feel quite swamped by almost everything being in Chinese. That said, two weeks in China is doing great things for my Chinese. Hopefully, it provides motivation to continue. Spending time without a translator helps too.
Chinese movies - watch many, in Mandarin. If you’re enthusiastic, watch with Mandarin subtitles instead of English. You understand less of the film, but learn more of the language.
Chinese books / newspapers / etc - Aim to read something written by native speakers that doesn’t live in a text book or language course. I find the educational material can be contrived.
italki.com - social networking site for meeting people to exchange language with. Still not fully sure of the usefulness, but seems to be an easy way to meet people who want to learn English and are native Chinese speakers. There are other sites that offer similar features.
Lonely Planet - phrase book is helpful, iPhone app is useful if you’re stranded or forget the basics. To be honest though, I rarely refer to it anymore.
Twitter - Adding people from China is helpful, but hasn’t led to as much learning of language as I had thought. That said, I’m learning a heap about culture and current events. Try following @YoukuBuzz for video content.

Note: I just found some MIT courses that are published online that include a few with Chinese. One even comes with a free text book in PDF form. I have not spent much time with it, but it looks promising.

Some of the resources will be useful for whichever language you are learning. If you aren’t learning a language other than your own, ask yourself why not? Learning a language will teach you a new culture and broaden your horizons. Worth the time. My challenge is not getting confused between English, German and Chinese.

Off to China

2008-12-08T08:35:00+01:00

I start a two week trip through China today; mostly for business, and mostly not sharable in public (pesky NDAs again). However, I will post a bunch of things as I go.

We start in Beijing for a week. Since I was here two years ago, things have changed. Arriving last night into the new terminal at the airport was an experience of space and light. Two things missing during the construction going on during 2006. The new structure is vast and impressive, however does invite comparison to Hong Kong and Singapore airports. Not a bad thing, given those are two of my favourite.

After meeting with a colleague from HK, we grabbed a late dinner at a HK diner near the hotel. It felt good to taste HK food again, especially milk tea. Not really a China experience, but it reminds me of many trips to Hong Kong visiting friends and working with SCMP.

The flight over was long, but not too long. As always, Cathay makes for a pleasant flight. Dragon Air was a bit more chaotic to Beijing, but was at least on time unlike anything Qantas can do of late.

Trying to stay awake, I lasted through four films:

Babylon AD - Good distopian future action romp. Nice sci-fi trappings, and good byplay between the characters. More than I was expecting from the trailer. Vin Diesel & Michelle Yeoh work well together. Reminded me a bit of a flashier Children of Men.
Wall-E - Unhappy watching this film. I missed it at the cinemas and now regret it. Absolutely brilliant. Classic Pixar and well worth a watch. Amazing what happens when film makers take risks. A feature film with limited dialogue? Yes, it absolutely works.
Kung Fu Hip Hop - Good filler on a plane, but not that great by HK film standards. An odd mix of the recent American dance type movies with a classic HK style kung fu movie.
The Love Guru - Mike Myers has done better. Felt a little too much like his previous films, which made for entertaining filler but no real brilliance. Go watch an old Austin Powers movie instead.

Internet connects work, with obvious firewall fun. Might be good to skip Facebook for a week. My Australian mobile also works, although I’ll get a local SIM card today. Not sure if I get internet with a local SIM, so Twitter updates are likely to be more sporadic.

How to be an Innovator for Life

2008-11-27T14:24:00+01:00

Tom Kelly, of the design and development firm IDEO, gives a brilliant speech at Stanford’s ETL program.

He breaks innovation down into five habits:

Think like a traveller; be child like with a sense of wonder & exploration.
Treat life as an experiment; be willing to fail provided learning is attached.
Nurture an attitude of wisdom.
Use your whole brain (and your tortoise mind).
Follow your passions.

The ETL series is available as a podcast on iTunes and continues to be a valuable resource.

5D Mark II Shipping Today, Somewhere

2008-11-26T08:15:00+01:00

Canon’s new 5D has started shipping in the US.

Australian shipping is a bit of a guess. Rumor is for the end of the week (28^th^ Nov). Mine is on pre-order from Cameras Direct. They stopped taking any more pre-orders. Hopefully, they are close with Canon and can ship this week.

China is a bit luckier, as it is already available in selected camera stores.

Bonus: Video editing tutorials for FCP and the 5D.

Adobe has also updated Camera Raw to support the 5D Mark II. Lightroom update to follow in December.

Financial Crisis and Blogging

2008-11-21T19:57:00+01:00

From the delightful Indexed blog.

Web 2.0 and the Newspapers

2008-10-31T20:02:00+01:00

(Based on my speech at PANPA this year)

Twitter – Web 2.0’s Poster Child

Web 2.0 was coined at an O’Reilly conference back in 2004. The aim was to capture the trend of new web applications that were emerging. Things like Flickr, Google Maps, Delicious and many others.

At its core, Web 2.0 is not about new technology. It is about using existing technologies to provide a richer, more interactive experience on the web for consumers.

The poster child for this new interaction is the web service, Twitter.

A simple concept. Let people broadcast a 140 character message. Didn’t sound new. Didn’t sound all that interesting. Yet, it has taken the content world by storm. Why?

It started with blogs. Blogs opened up publishing to the masses. They made it easy to regularly publish content. Joined with RSS to allow consumers to subscribe to the content in a way that is convenient for them, the blogosphere exploded. Suddenly, anyone could have a voice online. Google helped, driving traffic to the micro sites.

A person blogging from their bedroom could be ranked higher in Google than a major newspaper for specific topics.

Competition resounded.

Now, some blog networks gather more traffic than major US newspaper sites.

Conversation – An Old Idea

The secret of their success? Conversation.

Blogs provide for conversation in two major forms. The original approach was between blogs. Responses to a post were posted on another blog. This led to trackbacks, and support in blogging platforms for tracking them.

The view was that everyone who wanted to respond could have their own blog. The reality is that most people don’t. This led to comments on the blog post itself.

Fred Wilson, a Silicon Valley venture capitalist claims that the highest quality content on his site is the comments; the conversation between him and his readers and amongst his readers themselves.

Think about that. One of the most successful bloggers claims that his content is not the best on his site. That is game changing. He is no longer pushing content to readers, he is creating a conversation.

Twitter

Twitter is a micro-blogging platform, only this time, everyone can blog. To post content, you write a message that is no more than 140 characters long. This is an arbitrary limit imposed by the system. This creates a social dynamic that keeps content brief and to the point.

The key to Twitter is that there is no difference between posting content and posting a comment. To comment on someone else, you reply to them with content of your own. This has led to everyone on Twitter being a content creator as well as a participant in a conversation.

One of the other smart things Twitter did was move beyond the web. Web 2.0 in its purest is about two way interaction with your audience in a way that suits them.

Twitter gets less than a tenth of its traffic from its website. Other use comes from mobile devices, from third party applications, from its open API. Twitter goes out of its way to provide open access to the conversation between its participants.

Newspapers & Web 2.0

If this is the kind of audience interaction that is working on the internet, how do newspapers compare?

Not well.

Recent statistics suggest that newspaper blogs fail to increase the public dialogue; the conversation. When hosted by a newspaper, 80 percent of bloggers do not respond to comments. If content is about conversations, this is not a good statistic.

Think about your own newsroom. If your web site allows comments, does the original reporter see them?

The early print only newspapers tried very hard to solicit feedback from their readers. This shows up in the Letters to the Editor page. There is an overhead to filter down the feedback and select which nuggets are suitable for print in the next edition.

However, the lead time is slow. Responding can take days or weeks, depending on publication schedule.

The web lets you take letters not just to the editor, but to any journalist. And you don’t need to wait for the next publication date to publish the letter or to respond. It can happen in an instant. Comments are a natural part of getting consumers involved in the news process, and they need to be included into your newsroom process.

Newspapers have generally seen the web as another distribution channel for content. A pure push. The stories, once vetted by the subs, were mass published to the site and then not altered.

Much has been done to enhance this new output channel. However, from the newsroom it is still that: an output channel. Not many have feedback from the web site back to the newsroom.

Newsrooms with a Web 2.0 Future

Let me describe a newsroom of the future, and how content should integrate.

I receive the stream of comments from the article into an Inbox, allowing me to comment on them. The system gathers information about which other sites are linking to my story. Are other news outlets picking it up? Has the blogosphere started reporting on it? This information is freely available by blog search engines such as Technorati. If a competitor publication or a small site wrote a response I can continue the conversation, by posting a comment on their site, or by writing a follow-up story linked to my original story and their response.

Now that I have a number of conversations, this feeds my news gathering process. The web provides metrics on how popular my content is. How many people have read the story, how many people have commented, how many other sites are linked in to my content.

This gives me a finger on the pulse of what my audience is interested in and assists in selecting new topics for research and discussion.

My newsroom also provides me with up to date information on topics that I’m interested in. This includes not only feeds from wire services, filtered by my topics, but also feeds from bloggers, interactions with leads and contacts via instant messages or social services such as Twitter. This allows me to copy taste a stream of information tailored to the kinds of stories I am reporting on.

If a story breaks half way around the world, my desktop provides me a stream of official content from the location as well as citizen journalists and general participants’ contributions. This allows for me to gather quotes and research background information.

I can quickly publish to the web and alert other journalists in my organization of the story. I can start to gather additional media such as images and video to build up my story package. I’m still receiving comments on the story I’ve just posted, allowing me to continue to update the story with new and relevant information.

No longer a single publishing event for the story, it is now an ongoing news event; a conversation.

Expanding Your Reach

Similar to Twitter’s API support, news content is being repurposed across multiple channels. Some of these are purely web based, such as portal sites integrating headlines and summaries or Google News. Others are becoming more diverse. The iPhone is showcasing what is possible for a mobile device. eReaders such as the Kindle in the US, the Sony PRS eReader or the iRex iLiad provide for a different reading experience.

This leads to more opportunities to syndicate your content, more options for expanding the conversation, but also a much more complex media landscape.

Reuters has led the way, providing an open API allowing third parties to remix its content into innovative ways of presentation and distribution.

The new audience

There is no single audience that can be reasoned about in aggregate. Instead there is a collection of participants in a series of conversations. What works for one may not work for another.

One approach is personalization: Allowing the reader to customize the site to their needs. This is something the portal providers, such as Yahoo, have focused on. Their theory is that if they can provide a one stop shop for the consumer, then they will see more of the traffic. They not only integrate their own content, but provide summaries and links to third party content. This gives the user a dashboard of what is hot in their version of the internet.

This becomes more interesting is when this information can be inferred. Say that I’ve been reading the newspaper site for the last year. The web platform can track what I’ve been reading and start to build a profile to infer what else I’m interested in.

Amazon leads the way in this inference approach. They know which products I’ve bought, which products I’ve browsed and serve up suggestions as to what kind of products I might like to consider.

There is no reason that a content site cannot do the same.

Metrics

In the current incarnation of the print newspaper, we go out of our way to track circulation metrics to determine our audience. On the web, this can be handed to us but we’ve not been asking for it.

A good web platform can track an individual reader of your site and infer what kinds of content they are interested in.

This impacts the newsroom in several ways. Firstly by requiring a more interactive approach to content. Secondly, by providing more specific and targeted content. The niche is in. Have journalists specialize and allow consumers access to that specialty.

Classification

Consumers are used to being their own editors on the web. Their own copy tasters. They mix together content from multiple sources, from newspapers, from blogs, from their friends, from companies to produce a stream of information relevant for them. The content you produce needs to be able to fit into this new model. And it is about producing content. Not selecting content. Your readers can do that for you, and want to do it for themselves.

Google News is popular not because of their journalists, but because it provides access to content in a way the consumer can shape.

Caveats

Be wary of your readers’ trust. Newspapers are brands that are trusted. It is trust that has been built up over generations. It is a trust that can be rapidly lost.

Make sure that you are open with what you are trying, why you are trying it and what implications it has for your users. There are no secrets on the web.

The other concern is that of quality. This I have no easy answer for as there is a trade off.

Bloggers have solved this by swinging to the conversation side. The tendency is to post often and be involved in the discussion, although with spelling and grammar errors that would make a newspaper cringe.

Newspapers have historically gone the other way. Pushing content to the web and not being involved in the conversation. There is space in the middle.

Recommendations

Web 2.0 is a two-way medium. Print ways don’t always work on the web.

Some other things to think about:

It is all about conversation. Engage your readers in discussion.
Try new channels for hosting the conversation. Mobile. APIs. Devices.
Let your readers personalize their experience with your content.
Make use of the rich metrics available to drive editorial decisions.

Tracking Technology Trends

2008-10-13T12:41:00+02:00

Robert Scoble recently posted his top list of interesting people for technology. He is a vocal advocate for Friend Feed. It got me thinking about how I go about tracking technology news.

I joined Friend Feed a while ago and added a few of the usual suspects. With Scoble’s list, I set out to add many more to my feed. Using the filtering on the site, I’ve grouped them into a few categories.

I can see the theory behind Friend Feed. Aggregate all these sources, allow comments on items, and provide basic voting. I also like how it will show me interesting items from friends of friends. It is now about improving the signal to noise ratio to get me back more often.

Most of my news still comes in the old fashioned way: blogs.

Newsgator does the magic of keeping them in sync between my Macs, iPhone & PCs. I’ve found it useful to subscribe to some developer’s Delicious feeds; more reliable than waiting for a blog post.

I’m also a big fan of Twitter for ad-hoc discovery and Techmeme for following the overall industry.

Things off my radar? Newspapers and dedicated media outlets. This is a big shift from a few years ago, when I used to read the Green Guide (a section of the Age) and Ars Technica religiously.

Thus my consumption of news shifts further towards being personal and aggregated.

Become a more effective leader

2008-10-08T21:22:00+02:00

Manager Tools (MT) is a weekly podcast that provides free training on how to be a more effective manager and leader. The hosts have an extensive background in coaching managers and executives.

Management is a skill and it can be learned. Typically it is not taught well, if at all; MT teaches it effectively.

If you’re thinking, “but I’m not a manager, I won’t benefit”, then you’d be wrong. Even if you don’t manage other people, you manage your own time and interact into a larger organization. The skills taught by MT are beneficial to anyone who works for a corporation.

These guys have been podcasting since 2005. There is some benefit to going right back to the start, but if you want to catch up there is a subset of casts that are considered “core”.

I’ve found this podcast to be the best source of on-going knowledge about management, leadership, and communication. Each cast is designed to teach specific actions you can take today.

MT was originally recommended to me be a friend at CSIRO. Since then I’ve listened to most of the episodes. It is that good.

Vanity and Rupert

2008-10-06T08:17:00+02:00

Rupert Murdoch opens up to Vanity Fair’s Michael Wolff as background for an upcoming biography. The article provides great insight to the man and, by proxy, the companies he runs.

But his odd lack of seductiveness or felicitousness—contributing to his aura of villainy—became after a while alluring in itself. There’s no spin, because he really can’t explain himself. Rather, what you see is what you get. He’s transparent. The nature of the beast is entirely evident.

…

There is at News Corp. never a discussion of Murdoch’s exit. It is referred to only as “in 30 or 40 years,” when he is gone—which may have started as an amusing locution, but is now a practiced and even official one. His existential predicament is, in other words, his own.

The Guardian also has a good piece on the Battle Royale that is brewing between the new WSJ and the New York Times, and why it may no longer matter.

Murdoch fell into conversation with Arthur Sulzberger, publisher of the Journal’s arch-rival, the New York Times. By buying the Journal, Murdoch had parked his tanks directly on the NYT’s lawn. What, people wondered, would the two make of each other?

Canon 5D Mark II And Wow

2008-09-18T21:29:00+02:00

The 5D Mark II is one heck of a camera and puts Canon back in the game at the semi-pro level.

I was all set to buy the original 5D after its price dropped with the release of the new one. I also considered selling all my Canon gear to get the D700. I’m glad I waited.

The new revision has some grand features:

21 megapixel, full-frame CMOS sensor
ISO 100 - 6400 (extendable to ISO 50 - 25600)
HD Movie recording and HDMI out
3.9 frames per second continuous shooting

The addition that would have been nice is an improved max sync speed. 1/200 is a bit slow. ISO 50 helps, but it would be even better if max sync could get to 1/250 or 1/320.

External links:

The video samples are not exciting, but are beautiful quality. If only Chase Jarvis was shooting Canon gear. Not that it matters, the new 5D will sell by the truck load.

Olympics in Pictures

2008-08-16T22:18:00+02:00

I wasn’t hugely interested in the Olympics from a sporting perspective. I’m more a Winter Olympics kinda guy. However, I’ve found myself enthralled by the lengths taken by photographers to capture the games.

The main entry point is Vincent Laforet‘s blog. He’s an experienced photographer and is sharing his thought processes as he covers the games for Newsweek.

Newsweek have taken an interesting approach and sent three photographers to shot the games and to blog about it. Their combined blog is here. This article is a good example of the kinds of information being posted. The editor also selects a photo for the day.

Reuter’s photo blog is also featuring an Olympic photo of the day from their photographers.

I’m learning a lot about sport photography and photography in general by reading through these, especially Vincent’s musings. I now have enthusiasm for the Olympics.

VMWare Movie Capture & Conversion

2008-08-01T10:11:00+02:00

On a PC, I want to be able to easily capture screencasts for the products we use at work. The ulterior motive is to be able to show a video rather than repeat the demo hundreds of times.

VMWare gives a tantalizing option: Capture Movie

It is under the VM menu in the menu bar. Prompts for a filename and you’re away.

Slight catch. Playing back the movie that is recorded leads to not much but trouble. Turns out that VMWare encodes the movie into a non-standard video format. Details of it are here. (It is using the VNC protocol to encode the video with some VM specific headers)

If you’ve installed Workstation, the decoder for the codec is included. Otherwise, you can download the codec here. I found you need to use Microsoft Media Player to actually play back.

That still leaves the problem of editing the video. With the video recorded and the codec installed, you can convert the video into a suitable format. I used Prism’s conversion program to produce an AVI that can be read by Final Cut Pro.

Final Cut Pro took a bit of effort, but produced a very slick result. You can use simpler options for the video editing piece.

The result is that I can now capture video of demonstrations that are run in a VMWare environment, and re-mix them into various screencasts.

Photography Marketing Tips

2008-07-14T09:25:00+02:00

Be Aggressive

Diversify

Look for new resources for finding customers

Pay attention to your primary message — don’t let it get lost

Look for new promotional partners

Build your brand

Pay attention to your website

Use your own work for your holiday cards and stationary

Look for new promotional opportunities

Stay on top of technology

This is a very short synopsis of Skips’ normal four-hour presentation.

From Skip Cohen (via Jeff Greene).

I started out in photography as I love taking photographs. I’m sticking with photography as there is so much to learn; business strategy, marketing, technology & more.

Melbourne Pro Photo Labs

2008-07-08T23:18:00+02:00

I’m recently become quite excited about printing my photos. So much so that I’ve bought an A4 photo printer for “proofing”. For high quality prints, I’d rather outsource to a professional lab. The information on which ones are in Melbourne is a bit scattered, thus this list.

Melbourne labs:

Bond Imaging – Richmond (quote on request)
CPL Digital – St Kilda (price list)
Digital Works – Hallam (price list)
Lab X – St Kilda (price list)
NuLab – Braeside (price list on registration)
Vanbar Imaging – Carlton / North Melbourne (price list)

To get the most out of a good photo lab, or any photo lab, you need to be using a colour managed workflow. This should include proofing in Photoshop based on the colour profile of the print device. If the lab can not supply a way of colour matching to their printer, you may want to consider another lab.

Disclaimer: I’m not affiliated with any of these labs. I actually haven’t used most of them, so if you have recommendations, please leave a comment.

Extreme Photographing

2008-06-13T08:50:00+02:00

This edition of Michael Clark‘s newsletter is a doozie. Aside from reminding me of my annoyance with Canon for not having a decent competitor to the D300, he has an inspiring article on shooting the Patagonia Expedition Race. This is wow.

The race is described on the official site as:

It is a journey to the end of the world, to the unknown, a trip to a virgin and wild territory, to southern Patagonia, the tip of the American continent. It is a race marked by the mystique of exploration, of discovery and facing the untamable Mother Nature. It is an adventure at the end of the world…

Reading that in isolation, I’m thinking “yeah, sure”. Reading that after Michael’s story and seeing his photos, I’m thinking “wow, Wow!”.

He quotes Kent Kobersteen, a famed photo editor from National Geographic:

The really strong photos come from those situations where the last thing you want to do is take pictures – where everything is going to hell, where the storms are raging and everyone is trying to hang on.

They were definitely in the right place:

And we were in an alpine jungle with trees so dense that not even a helicopter could find us in this valley.

Well worth a read (pdf). Or even just a download to look at the pictures!

A closing thought:

There are two basic ways to find inspiration. One is to go on a trip and the other is to put the camera down for a while.

– Jay Maisel, page 20

Favorite Things

2008-05-25T23:02:00+02:00

Snowboarding, rally driving and Primus. Too good!

HaskellDB Performance

2008-05-18T23:47:00+02:00

Things did not go well with HaskellDB when I ran my query across a large data set. MySQL as a query optimizer, sucked. Postgres seems to be doing better.

I started with a mysql database containing 11 years of day data from the ASX. This includes the closing price (and some other info) of all stocks traded on the Australian Stock Exchange (ASX) since 1997.

The tables are organized with unique keys and, based on some testing, include appropriate indexes.Things still ran slowly.

I tried tuning mysql. Then, based on the slow queries logged, tried running them by hand to see if I was missing any indexes. No such luck. The query just didn’t perform.

I simplified the query and ran it again (see below). Now it ran fast, but doesn’t help with the generated query. Looks like MySQL’s optimizer doesn’t optimize enough.

MySQL timing:

Query 1: 6.25s (anywhere from 24s - 6s)
Query 2: 0.15s

Next step. Install Postgres.

Postgres timing:

Query 1: 0.03s
Query 2: 0.03s

To avoid the cache impacting the results, both servers were stopped and re-started between each query run. With caching, Postgres gets down to 0.2s. MySQL will still have 6s/0.3s. Tests were run on a MacBook with 2Gb of memory.

Both queries were run with the command line tool of the respective database engines, mysql and psql. The end_of_day table has 4,749,025 records and the stock table has 25,863 records.

Query 1 – HaskellDB generated

SELECT closing_price2 as closing_price,
       trade_date2 as trade_date
FROM (SELECT stock_id as stock_id2,
             trade_date as trade_date2,
             closing_price as closing_price2
      FROM end_of_day as T1) as T1,
     (SELECT stock_id as stock_id1,
             ticker as ticker1
      FROM stock as T1) as T2
WHERE stock_id1 = stock_id2
  AND ticker1 IN ('FXJ','NWS','WAN')
  AND trade_date2 = '2008-02-14 00:00:00';

Query 2 – Hand written

SELECT e.closing_price as closing_price,
       e.trade_date as trade_date
FROM
    stock s, end_of_day e
WHERE
    s.stock_id = e.stock_id
    AND s.ticker IN ('FXJ','NWS','WAN')
    AND e.trade_date = '2008-02-14 00:00:00';

So don’t use HaskellDB and MySQL if you are doing joins across large tables. Use Postgres instead.

The Haskell query:

indexStock db tkrs stockDate = do
  let q = do
      s < - table S.stock
      e <- table E.end_of_day
      restrict (s!S.stock_id .==. e!E.stock_id .&&.
                s!S.ticker `_in` tickers .&&.
                e!E.trade_date .==. constant stockDate)
      r <- project (closing_price << e!E.closing_price #
                    trade_date << e!E.trade_date)
      return r
      where
        tickers = map (constant) tkrs
rs <- query db q
return $ sum $ map (\\r -> read (r!closing_price)) rs

pseudofish

Timeless Recommendations

Table of Contents

1. Books

2. Podcasts

Org-mode for everything

Choosing a Chinese name

Triangle Heatmaps in R using ggplot

Downloading Google Analytics data from a Python script

Setting up your Google account

Starting a Python project

Calling Google Analytics from the API

The design of work teams

Background research

A Normative Model of Group Effectiveness

Conditions that support effort

Organisational context

Group synergy

Design of the group

Task appropriate performance

An Action Model for Improving Group Effectiveness

Diagnosis of existing teams

Creating new teams

Management of teams

On leadership

On Creating Redundant Conditions

On Managerial Authority

On Knowing Some Things

Further reading

Analysing Skritter time series data with R

Downloading the data

Loading into R

Cleaning up the data

Visualising trends

Further reading

Update on learning Chinese

Online Lessons

Reading

Finding books

Vocabulary

Listening

Changing Names

A History of Modern China

Wealth and Power ( 富强 ): China’s Long March to the Twenty-first Century

The Party: The Secret World of China’s Communist Rulers

Country Driving

Learning iOS programming

Books

Blogs

Podcasts

Other tactics

Rethinking how I handle email

My new process

Results

Measuring email volume

Using deft mode for notes in Emacs

Additional tips

Scandinavian Startups

New Blog

Postgres database migrations with Python

Reading email in Chinese with Python

Good Strategy Bad Strategy

Debugging Python with PDB

Unit testing in Python with folder watching

Sharpen the Saw: Typing

About Face, Edition 3

Positioning

Emacs or Vim

Emacs

Vim

Closing Thoughts

Calculating earlier dates using a shell script

Storm - a real time Hadoop like system in Clojure

Java interop with Clojure and Mahout

Nested Classes

Generics

Primitive Arrays

McKinsey on testing your strategy

Calling Mahout from Clojure

Book Reviews