pseudofish

Rethinking how I handle email

2013-05-12T00:00:00+02:00

Email started to take too long, so I recently changed my approach.

At work, we try to avoid unbounded queues. In a distributed service architecture, they can be toxic and swamp otherwise well behaving services.

Email had become an unbounded queue. It kept piling up and swamping my day. Time for a change.

My new process

I adapted the recommendations from Manager Tools, and I now follow these guidelines:

Schedule 30mins in the morning for email.
No more than three time periods of up to 30mins during a day for email.
If I start processing email, I must finish (empty INBOX)

The key for me has been pushing myself to finish if I start. This means that a spare five minutes cannot be spent doing email.

For things that I need to do, either I book time in my calendar or put an entry in my task list, possibly with a reminder.

When I end up with something that I want to come back to later, I’ll create a task to refer back to it, and park the email in a “Follow-up” folder.

For random notes to self, I now add a task. Previously, I was cluttering my inbox and compounding my mail problem.

I also switched to using Instapaper for saving articles from Twitter, also reducing emails to myself.

Results

The last two weeks were a big improvement.

I am feeling less stressed by email, and more focused on important activites. I also feel like I have more time to handle unexpected events and to chat with colleagues.

I still catch myself checking email on occasion, however often my inbox is empty, so I’m slowly weaning myself of the habit.

For personal email accounts, I’m using a similar approach, although with a lower frequency of checking.

For my phone, I will occasionally check email there, and am a bit more relaxed about the process. I also set all mail applications to only check mail when manually asked.

Measuring email volume

Below is a chart of the email volume over the last two years for my work account. I was curious how much email I was receiving, and how much I sent.

For the last few months, I received around 3 times the number of emails as the same period last year. This is a factor of being involved in more projects, and the growth of the company. November last year was during a major project, and spiked in the number of emails I received.

I was surprised to find that I still send the same number of emails (~10-15 per day).

The chart was generated via the IMAP interface to Gmail, and this Python script:

#!/usr/bin/env python

import imaplib

def count_of_search(M, query, mailbox='INBOX'):
    M.select(mailbox_name)

    status, email_ids = M.uid('search', None, r'(X-GM-RAW "%s")' % query)
    return len(email_ids[0].split(' '))

mailbox_name = '[Gmail]/All Mail'
user, passwd = 'user@example.com', 'secret'

M = imaplib.IMAP4_SSL('imap.gmail.com', 993)
M.login(user, passwd)

print "Date,All Mail,Sent,Received"
for year in [2012, 2013]:
    for month in xrange(1,13):
        date_query = 'after: %d/%d/1 before:%d/%d/31' % (year, month, year, month)
        all_mail_count = count_of_search(M, date_query, mailbox_name)

        sent_query = '%s from:%s' % (date_query, user)
        sent_mail_count = count_of_search(M, sent_query, mailbox_name)

        received_query = '%s to:%s' % (date_query, user)
        received_mail_count = count_of_search(M, received_query, mailbox_name)

        print "%d/%d/1, %d, %d, %d" % (year, month,
                                       all_mail_count,
                                       sent_mail_count,
                                       received_mail_count)

The tricky part was finding a way to pass the X-GM-RAW parameter through to IMAP to enable Gmail search criteria. This allows you to use the same search expressions as from within Gmail.

Using deft mode for notes in Emacs

2013-03-14T00:00:00+01:00

Deft mode is one of my favorite discoveries of emacs, especially when combined with the magic of org-mode.

I use the combination of both deft and org-mode almost every day. If I’m doing an interview, taking notes in a meeting, keeping a personal backlog or ideas for a project, then this is what I’m using.

Deft solves the problem of where to put text files. Org-mode structures what goes into them.

Deft can be installed via emacs packaging. I configure it like this:

(when (require 'deft nil 'noerror)
  (setq
     deft-extension "org"
     deft-directory "~/Dropbox/Notes/"
     deft-text-mode 'org-mode))

I put the default folder within my Dropbox folder. This ensures notes are backed up and available on any computer where I’m using Dropbox and emacs.

Once in deft mode (M-x deft):

RET — open the current file
C-c C-n / C-RET — create a new file (auto named)
C-c RET — create a new file (prompt for name)
C-c C-r — rename a file
C-c C-d — delete a file

To search for a particular file, just start typing and the results are filtered in the same view.

For me, this has been near perfect for quick note taking. My current notes folder holds over 150 org files and is still fast to search.

When sharing notes from meetings with others, I use org-mode’s export feature (C-c C-e A), then highlight the part I want, and send it to the clipboard (M-| pbcopy) to paste into email.

Getting a solid system for note taking in emacs has helped with how often I use the same text editor. This was also part of my motivation of migrating my blog to text files (although markdown formatted).

Additional tips

To launch emacs from the command line with deft mode open, I set the following alias:

alias de="emacs --eval '(deft)'"

Often I have similar formats for things like notes, minutes or interviews. To avoid lots of typing, I use yasnippet to create templates.

For some examples, see my dotfiles.

Scandinavian Startups

2013-02-21T00:00:00+01:00

There is something in the water up here. The economy seems to be going well, and the press seems to have taken notice.

An example from USA Today:

If music service Spotify landed Stockholm on the high-tech map, companies such as Klarna underscore its growing influence.

In the first quarter of 2012, Sweden took in almost 20% of all venture capital invested in the European Union, trailing only Germany

And Pando Daily on Stockholm:

Thanks to Klarna, Spotify, Rebtel, and King.com, among others, Stockholm has a longer history of tech successes and more big-dollar exits than its German rival [Berlin].

For instance, the Nordic region as a whole — which still represents a much smaller population than Germany — accounted for about 6.5 percent of the world’s billion-dollar exits from 2005 to 2012, according to statistics provided by Stockholm-based venture capital firm Creandum.

There is still room to grow, as European startups in general are undervalued compared to the US:

Listed European technology firms valued at more than $100 million are on average 32 percent cheaper than their North American peers. Bloomberg says they trade at an average ratio of 15 times earnings compared with 22 times earnings in the US and Canada.

The Economist goes into depth on in their special report on The Nordic Lights:

Finland has become much more market- and entrepreneur-friendly. It has produced an impressive number of start-ups, including 300 founded by former Nokia employees.

And:

The Nordic countries have not only largely escaped the economic problems that are convulsing the Mediterranean world; they have also largely escaped the social ills that plague America.

I feel very lucky to have worked for a number of Nordic companies, and to continue my Scandinavian adventure with Spotify here in Stockholm.

New Blog

2013-02-10T00:00:00+01:00

If things went well, this post will be from my new blog.

I migrated my existing blog across to use Pelican, a static site generator written in Python. This is part of attempting to use emacs & python for as much of my workflow as possible.

During the migration, all the code and formatting were cleaned up, so it should be easy to change. Previously, all the content was in a Wordpress database, and not so easy to transform.

I’m not sure how attached I’ll be to Markdown. I am contemplating adding org-mode support to Pelican, as I use it for most other notetaking.

You may notice some duplicate articles via RSS. This is unfortunately due to changing the URL scheme.

Postgres database migrations with Python

2012-11-18T02:41:00+01:00

If you need to setup database migrations that work both locally and on Heroku in Python, these steps may be helpful.

For a recent project, I wanted to be able to manage the database schema. I’m not using an ORM, however the migration features in SQLAlchemy turn out to be quite useful.

Install the SQLAlchemy migration package, by adding the following to requirements.txt.
```
# Migrations
sqlalchemy-migrate>=0.7.2
```
and update:
```
pip install -r requirements.txt
```
Create a repository for database migrations
```
migrate create db "Cihui"
```
This creates a directory db/ where the code and migrations will reside.

Update the migrate script to use Heroku DB url

#!/usr/bin/env python

from migrate.versioning.shell import main
import os

if __name__ == '__main__':
    db_url = os.environ.get('DATABASE_URL', 'postgresql://localhost:5432/cihui')
    db_url = db_url.replace('postgres:', 'postgresql:', 1)
    main(url=db_url, debug='False', repository='db')

This should allow you to run migrations locally or on Heroku. There was a warning message on Heroku for the url starting with postgres rather than postgresql, thus the replace line.

Update Procfile

This sets up a few useful targets to run on Heroku and locally using foreman.

db_init:  python db/manage.py version_control
db_version:  python db/manage.py db_version
migrate: python db/manage.py upgrade

To migrate locally

foreman run init_db # first time
foreman run migrate

To run on Heroku
```
heroku run db_init
heroku run migrate
```
Write migrations as SQL

I prefer to manually write the migrations, and SQLAlchemy has great support for this. Refer to their docs for details.
```
python db/manage.py script_sql postgresql 'add list table'
```
This generates an upgrade and downgrade sql script in db/versions with an appropriate version number.
Test the migration:
```
python db/manage.py test
```
Runs upgrade and then downgrade on a single version.
DB access to Heroku

Console access is available via:
```
heroku pg:psql
```
Useful for checking specific values or fixing broken migrations.

I’m still iterating on this project, so may change some things around as I go. For the moment, this allowed me to manage the database schema easily in multiple databases.

Reading email in Chinese with Python

2012-10-15T16:41:00+02:00

Read email from a mailbox, in Chinese, with Python. Sounded easy.

After playing at the python prompt and many Google searches later, still no success. Then, I found two articles that helped.

Turns out that there are a few tricks to getting email parsing to work properly for multi-byte languages.

So, if you end up with strings something like:

=?GB2312?Q?Qingwen_Word_List:_=D3=D0=D2=E2=CB=BC=B5=C4=B4=CA?=

Or:

=CA=AF[=CA=AF] sh=A8=AA =A2=D9 rock =A2=DA stone =A2=DB stone inscription =A2==DC one of the eight ancient musical instruments =B0=CB=D2=F4[ba1 yin1]=20=C0=BC=BB=A8[=CCm=BB=A8] l=A8=A2nhu=A8=A1 =A2=D9 cymbidium =A2=DA orchid=20

Then, you’ll need two different approaches. One for the header strings, and one for the body text.

Given an email message, either from a string or file:

from email import header
import email
msg = email.message_from_string(str_msg)

Then, headers can be read using something like:

subject, encoding = header.decode_header(msg['Subject'])[0]
print subject.decode(encoding)

Which gives:

Qingwen Word List: 有意思的词

In this case, the encoding can be read from the email message and is ‘gb2312’.

The body text needs a slightly different approach:

print unicode(msg.get_payload(decode=True), msg.get_content_charset(), 'replace')

For:

石[石] shí ① rock ② stone ③ stone inscription ④ one of the eight ancient musical instruments 八音[ba1 yin1]
兰花[?花] lánhuā ① cymbidium ② orchid...

The ‘replace’ option will mark unknown characters using U+FFFD, rather than throw an exception.

This approach should work for most languages. Getting the various decodings sorted out allowed me to move forward on a project I’m working on. Now to re-implement it test first.

I know the format of the emails coming in, so can ignore multi-part MIME messages. If you are looking for some more details on how to handle them, check out this article.

If you were curious about the =DA encoding, it is called quoted-printable (RFC).

Good Strategy Bad Strategy

2012-06-24T22:53:00+02:00

My current reading list is mixed between coding, decision making, agile, and strategy. Richard Rumelt’s Good Strategy Bad Strategy stands out as an interesting and entertaining read.

I liked how he outlined good strategy, and also how to identify bad strategy. This really helps in pushing through from a good enough attempt to a strategy that will make a difference.

Some quotes:

Good strategy almost always looks this simple and obvious and does not take a thick deck of PowerPoint slides to explain. It does not pop out of some “strategic management” tool, matrix, chart, triangle, or fill-in-the-blanks scheme. Instead, a talented leader identifies the one or two critical issues in the situation—the pivot points that can multiply the effectiveness of effort—and then focuses and concentrates action and resources on them.

Good strategy requires leaders who are willing and able to say no to a wide variety of actions and interests. Strategy is at least as much about what an organization does not do as it is about what it does.

On identifying bad strategy:

I have condensed my list of its key hallmarks to the four listed in the beginning of this chapter: fluff, the failure to face the challenge, mistaking goals for strategy, and bad strategic objectives.

On the structure of a good strategy:

The kernel of a strategy contains three elements: a diagnosis, a guiding policy, and coherent action. The guiding policy specifies the approach to dealing with the obstacles called out in the diagnosis. It is like a signpost, marking the direction forward but not defining the details of the trip. Coherent actions are feasible coordinated policies, resource commitments, and actions designed to carry out the guiding policy.

The problem of coming up with a good strategy has the same logical structure as the problem of coming up with a good scientific hypothesis. The key differences are that most scientific knowledge is broadly shared, whereas you are working with accumulated wisdom about your business and your industry that is unlike anyone else’s. A good strategy is, in the end, a hypothesis about what will work.

The book provides insight into the structure of a good strategy. However, it stops short of providing actionable techniques or analysis to help you get there. In this, it advocates strategy work being done by someone with a deep understanding of the business and competitive landscape.

This left me wanting more, and fortunately there are books out there that provide guidelines on this type of analysis. I see a few Michael Porter books in my future reading.

Debugging Python with PDB

2012-05-03T02:51:00+02:00

I suck at using debuggers, largely because I don’t launch them often enough.

Fortunately, this year’s PyCon had a great talk from Chris McDonough on how to get started with Python’s debugger, PDB. The friendly people at Stockholm’s Python meet-up suggested it.

I made notes while watching, to remove excuses from future me on launching pdb.

Where you want to have the debugger start, add the following code:

import pdb; pdb.set_trace()

This drops you into a PDB prompt. This is more helpful than putting in yet more print statements.

Some helpful commands once you’re in:

l - list code
args - arguments to current function
p <var> - print a var
pp <var> - pretty print a var
n - next
s - step
w - where (stack trace for current position)
h - help

While in PDB, you can evaluate python code. The evaluated code won’t impact the running program.

For a bonus trick if you’re using Emacs, try out pdb track mode. Launch your python process to be debugged in an emacs shell (M-x shell). Stepping through the code in pdb will track with a source code buffer.

This worked out of the box with my emacs config. Your milage may vary.

I may actually start to enjoy debugging now!

Unit testing in Python with folder watching

2012-03-14T05:09:00+01:00

Unit testing is good. Running unit tests manually is annoying.

Rspec has a great addition in autospec, which automatically re-runs specs that are changed or have the associated code changed. This model of working has a long history in the ruby community, with autotest.

I wanted a similar thing for python/nose, but hadn’t had much luck.

The solution is py.test, with some plugins added.

This was suggested to me at the Stockholm Python user group, and I am very grateful. My testing life is now much simpler.

To get started, try something like:

$ sudo pip install pytest pytest-xdist pytest-cov

The simple way to get started is to watch your tests folder:

py.test -f test/

Now as you change files, the relevant tests will automatically be re-run. Perfect to display on that second monitor.

As I was working on some legacy code, I was looking to improve the test coverage, so wanted to see how that was going with pytest-cov:

py.test -f --cov package.name test/

Adding the package.name for your main package means you won’t generate coverage for libraries you are using. Makes the output simpler.

Still not quite happy, it wasn’t showing me which lines I needed coverage on. No problem:

py.test -f --cov package.name --cov-report term-missing test/

For comparison, here is how to get similar results from nose, but without watch support:

nosetests test/nosetests --with-coverage --cover-erase --cover-package=package.name test/

One hurdle I had with py.test was working with Twisted based test-cases. You need to ensure that you use a version of py.test later than 2.0. Debian squeeze does not package this by default, so use pip to install.

The annoyance is if you change the code so it doesn’t terminate. This still needs a context switch to kill off the tests and re-run. Keeps me focused on getting it right the first time.

pytest-xdist includes lots of really cool options, such as distribution of tests across multiple cores, versions, hosts and platforms.

I’ve since discovered this comprehensive summary of the options available. I’m happy I found this after selecting py.test, or I’d still be stuck in evaluation mode rather than writing code.

Update: check out sniffer if you want a more generic watcher framework.

Sharpen the Saw: Typing

2012-03-07T00:14:00+01:00

I once explained my job as an overpaid typist.

There is some truth in that, as much of my day is spent typing. Writing code, taking notes for meetings, email, documentation, writing blogs posts.

It would make sense to become very good at typing.

The basics are often overlooked. You can get a lot of benefit from paying attention to them.

For typing, it comes down to three main areas:

Accuracy
Speed
Touch typing

Touch typing is perhaps the most important aspect of this. You need to be able to type without looking at the keyboard. If you get really good, you shouldn’t need to even look at the screen to confirm what you’re typing.

Accuracy and speed can be built together. Focus on accuracy to start with, and then stretch for speed. There is no point going faster if you can’t maintain a high accuracy.

I found two websites to be helpful in my quest for improved typing skills:

Keyzen — a site focused on improving typing for programmers.
10 fast fingers — general typing improvement with natural language exercises.

Of course, the other thing to do to improve typing is to do lots of typing. If you are writing lots of code, email or documents, then focus on your typing while you are doing it.

It helps to have a good environment for typing, such as a comfortable keyboard, good posture, but you know this already. Add it to the list of things to focus on.

Also, make sure that you are using a program that can keep up with you as you type. I find MS Word to lag when in full WYSIWG mode, whereas emacs and vi always manage to keep up!

Update: You may also want to check out Typespeed.

About Face, Edition 3

2012-02-29T17:21:00+01:00

In any field, if you expand your view to know all the ecosystem around you it’s beneficial.

—Davide Casali

If you are a developer, a product owner, or manage a development team, much of the input is going to be about product design. It is very useful to understand a bit about a good design process, so you can better ask questions and help refine the design based on technical constraints.

About Face, by Alan Cooper, is a great book to start with.

I highlighted much of it on my Kindle!

Here are some quotes:

Good design makes users more effective.

Customers, although they might be able to articulate the problems with an interaction, are not often capable of visualizing the solutions to those problems.

One of the most powerful tools designers bring to the table is empathy

One of the most dangerous practices in product development is isolating designers from the users because doing so eliminates empathic knowledge.

narrative is also one of our most powerful creative methods.

What would a helpful human do? What would a thoughtful, considerate interaction feel like?

All ideas are broken down into discrete sections, with lots of detail and examples on how to implement this with your customers.

Goal-Directed Design combines techniques of ethnography, stakeholder interviews, market research, detailed user models, scenario-based design, and a core set of interaction principles and patterns.

This process can be roughly divided into six phases: Research, Modeling, Requirements Definition, Framework Definition, Refinement, and Support

The book discusses the mindset required to be a good designer:

in anything at all, perfection is finally attained not when there is no longer anything to add, but when there is no longer anything to take away

Next time you find yourself crowing about what cool interaction you’ve designed, just remember that the ultimate user interface for most purposes is no interface at all.

The first part of the book focuses on processes to elicit a good design. The latter part of the book covers specific examples found in user interface design and attempts to encode common patterns.

I found the first section much more valuable to how I think about designing a new product. The latter parts I felt were obvious, although that is perhaps their importance.

Well worth a read.

Note: Bret’s Magic Ink is a useful counterpoint to About Face.

Positioning

2011-11-28T20:37:00+01:00

Positioning, by Al Ries and Jack Trout, has long been on my to read list, and I wish I had read it sooner.

The core message of how to position a product is becoming more relevant as the marketplace continues to crowd.

In our overcommunicated society, the name of the game today is positioning. And only the better players are going to survive.

The book draws on a wide range of examples of different products, where a product could be a type of beer, the Catholic Church or your own career.

The key message is that your products position is determined by how it fits into the consumer’s mind. All the things that matter to you, to your company, are irrelevant if you can’t create a distinct position to separate you from everything else.

Experience has shown that a positioning exercise is a search for the obvious. Those are the easiest concepts to communicate because they make the most sense to the recipient of a message.

The authors are quite scathing about “me too” products that attempt to beat out a competitor by going head-to-head. Many examples are provided within the book.

The suicidal bent of companies that go head-on against established competition is hard to understand.

To repeat, the first rule of positioning is: To win the battle for the mind, you can’t compete head-on against a company that has a strong, established position. You can go around, under or over, but never head to head.

The leader owns the high ground. The No. 1 position in the prospect’s mind. The top rung of the product ladder.

The book covers a range of detail levels, and dips into more detail on key areas such as product naming:

As a guide, the five most common initial letters are S, C, P, A, and T. The five least common are X, Z, Y, Q, and K. One out of eight English words starts with an S. One out of 3000 starts with an X.

The authors are often quite blunt as to how they see things. This is a refreshing change from most business books:

Creative people often resist positioning thinking because they believe it restricts their creativity. And you know what? It does. Positioning thinking does restrict creativity.

I gained a lot of insight into how products have succeeded, or not, by reading this book. The examples are clear and will easily map to situations you are experiencing.

I recommend Positioning for anyone involved in creating something new. This is a book that I will re-read.

Emacs or Vim

2011-11-25T01:33:00+01:00

After over 12 years of using Vi(m), and a brief fling with TextMate, I started using Emacs as my primary editor.

Emacs

The switch has been very positive. The learning curve has been relatively steep, as my expectations from a text editor are quite high.

Emacs strength (and weakness) is that it is incredibly extensible. Where I’m finding Emacs a win over Vim is that I don’t have to leave Emacs to get things done. With Vim, I tend to use more of a mix of terminal windows and the editor.

I began by using the starter kit to get going with Emacs configuration. This made it quicker to get moving, but added a lot of things I ended up not needing. After getting more comfortable with elisp, I started from scratch and rebuilt my emacs.d folder.

To jump in quickly, I also purchased the tutorial video from Peepcode. This certainly helped as emacs is a mental shift coming from Vim.

The big benefit I have found with Emacs is the extension packages. These can be installed from the ELPA repository, and include a range of different modes.

Some of my favourite modes include:

paredit - essential for any lisp, it ensures your brackets match.
deft - simplified note taking. (I it sync via Dropbox)
magit - comprehensive Git workflow within Emacs
markdown-mode - my current default for writing notes, although I’m leaning towards org-mode now.
org-mode - highly capable note taking mode, with export options to everything. You can use it to write a book, create slides, or manage your todo list.

On the Mac, I’ve been using Aquamacs for most of my text editing, and used macports to install the command line client.

I have yet to try Emacs on Windows, as I haven’t been using Windows much at all. There is a release available here.

After cleaning up my .emacs.d configuration, I’ve now started using Emacs on Linux servers I use regularly. For temporary servers, I’ll fallback to Vim as Emacs is often not installed.

If you want to improve your emacs skills follow @emacs_knight.

Vim

To get started with Vim, it is worth reading The Vim Learning Curve is a Myth.

Over the years, the popular mythology around vim has become that it’s insanely difficult to learn; a task to be attempted by only those with the thickest of neck-beards. I’ve heard dozens of times from folks who are convinced it will take them months to reach proficiency.

These beliefs are false.

My feeling is that Vim is unrivalled for the simple task of text editing. Even after a day or two of learning, you will be faster.

Where things get a bit more complicated is when you start to realise that text editing isn’t the whole story for a text editor.

Platform support is superb. The first thing I do to a Windows machine is install Vim. It is rare for an application with a Unix heritage to be so comfortable on Windows.

Historically, I hadn’t been a fan of the graphical version of MacVim. This is something that is much better in recent releases. Load times are much improved.

Integration into external tools is where I feel Vim is lacking a bit. With the number of developers migrating from TextMate to Vim, this gap is being rapidly addressed. However, Vim isn’t as extensible as some other editors.

I really noticed this when trying out Clojure programming. If you are dealing with a REPL based environment, Vim has a way to go.

Closing Thoughts

If you aren’t using either of them, it’s not too late to start. You can’t make a bad choice.

If you are already using either Emacs or Vim, enjoy!

Both are great editors that allow you to be incredibly productive at working with text. Try learning a new feature each week. You use your text editor so often that a small improvement is a major payoff.

Calculating earlier dates using a shell script

2011-10-07T01:40:00+02:00

Mongo has a database size limit in 32 bit mode, so I want to purge out items that are less than a certain date in the past. I decided that it would be easy to write a simple shell script to run the query.

The tricky part was calculating dates in shell. This is what I ended up with:

#!/bin/sh

MONTHS_AGO=3
DATE_AGO_EPOCH=$((`date +%s` - $MONTHS_AGO * 31 * 24 * 3600))
OSTYPE=`uname`
FORMAT='+%Y-%m-%d'

if [ "Linux" = $OSTYPE ] ; then
  DATE_AGO_ISO=`date -d "1970-01-01 00:00 UTC + $DATE_AGO_EPOCH seconds" $FORMAT`
else
  DATE_AGO_ISO=`date -r $DATE_AGO_EPOCH $FORMAT`
fi

DB_CMD="db.items.find( { publish_date : { \$lt : \"$DATE_AGO_ISO\" } } ).count()"
# DB_CMD="db.items.remove( { publish_date : { \$lt : \"$DATE_AGO_ISO\" } } )"

echo $DB_CMD | mongo pz

This now provides a simple way of purging out old items that can be called from cron.

Storm - a real time Hadoop like system in Clojure

2011-09-26T02:33:00+02:00

I am very excited about the potential unleashed by Storm.

Previously at BackType, Nathan Marz has now built Storm into Twitter and then open sourced the platform.

Storm opens up a lot of possibilities, by bringing real-time distributed processing together in an elegant way. And it is built in Clojure!

Some useful links:

Rationale
Slides - detailed presentation on Storm by Nathan
Starter project - a few basic demo use cases
Mailing List
Taking the Emmys by Storm - an hour long video of how to use Storm to aggregate data from Twitter in realtime to spot trends.

While playing around I wanted to try from Clojure (most examples are in Java). There is a great DSL for Clojure that makes using Storm super easy. I forked the storm-starter project to add examples based on a Gist from Nathan.