Kyle Banker | Hwaet!

MongoDB in Three Minutes

2009-11-05T20:59:32-05:00

Couple nights back, I was given three minutes to demonstrate MongoDB before a somewhat large group of people who’d never heard of it. Source code is on github. At one minute each, the highlights are these:

1. Schema-free documents.

MongoDB is schema-free; this means that the structure of MongoDB data need not be defined up front. MongoDB stores data as collections of documents. A document can be thought of as a JSON object, Python dictionary, or Ruby hash (among other things). Documents are natural, elegant ways of representing data, and are of the essence of MongoDB.

Suppose we want to download a few tweets.

# DB connection and collection
@db  = Mongo::Connection.new.db(DATABASE_NAME)
@nyc = @db.collection('nyc')

(1..5).each do |page|
  Twitter::Search.new('nyc').page(page).each do |tweet|
    @nyc.save(tweet)
  end
end

That gets us the first five pages of ‘nyc’-related tweets and saves them to the our ‘nyc’ collection in the db. For each tweet, the Twitter gem returns a Ruby hash, which saves naturally to the database.

2. Dynamic queries.

MongoDB speaks the language of documents, enabling expressive queries. To take a few examples:

We can query for a specific key:

  @nyc = DB.collection('nyc')
  @nyc.find(:username => "hwaet")

Or on a nested key pointing to an array:

  @nyc.find('user.followers' => {'234352343'})

We can search a field using a regular expression:

  @nyc.find('text' => /z.+/})

Or query across a date range:

  @nyc.find('created_at' => {'$lte' => Time.now - (60*60*24)}})

And all of this can be efficient because each collection can define up to 40 secondary indexes:

  @nyc.create_index(['text', 1])
  @nyc.create_index(['user.followers', 1])
  @nyc.create_index([from_user, 1], ['created_at', -1])

3. Binary Storage

What if we want to store images, videos, or music? MongoDB stores arbitrary binary data, too.

This code goes through our collection of tweets, fetches each user’s profile image, and saves it to the database using GridFS (i.e., MongoDB’s specification for storing larger binary objects).

@nyc.find.each do |tweet|
  filename = tweet['from_user'].downcase + ".jpg"
  next if GridStore.exist?(@db, filename)

  GridStore.open @db, filename, 'w+' do |file|
    data = open(tweet['profile_image_url']).read
    file.content_type = 'image/jpeg'
    file.puts data
  end
end

If we want to serve those images with sinatra:

get '/images/:id' do
  content_type "image/jpeg"
  filename = params[:id].downcase + ".jpg"
  GridStore.read(DB, filename)
end

Speed, scalability, ease of use…

All this is to say nothing of production case studies or paths to scalability. Interested readers are encouraged to download a binary and start experimenting.

Review: Finite and Infinite Games

2009-08-17T12:57:06-04:00

Western thinkers are fond of dividing the world in two. Though simplistic, it’s effective. The flesh vs. spirit characters in the novels of Nikos Kazantzakis are memorable. The modes of having and being in Erich Fromm are illustrative. And when James P. Carse wrote about finite and infinite games, we were given another dichotomous lens for peering into the mysteries of human being.

In Finite and Infinite Games, Carse asks us to construe human activities – working, loving, walking, cooking, reading, driving – in fact, anything you can imagine – as kinds of games, and invites us to consider whether we play these games finitely or infinitely.

Finite games are meant to come to an end. They are played for the purpose of winning, vanquishing an opponent, or bringing about some desired outcome: a student gains admission to the Ivy League, a worker receives a promotion, a country goes to war and wins or loses.

Finite games are theatrical. They require an audience to say who has won, to agree upon an end, and to confer titles upon the winners, which is all but certain for finite players:

The outcome of a finite game is the past waiting to happen. Whoever plays toward a certain outcome desires a particular past. By competing for a future prize, finite players compete for a prized past.

Infinite players, on the other hand, play with the spirit of continuing play. Such play is not theatrical but dramatic. The drama lies in choosing an undefined path that is not ultimately for oneself. Artists and visionaries, those who revel in spontaneity and laughter, who are aware of their places in society but do not take their places seriously—these are infinite players.

It soon becomes clear that finitude and infinity are not intrinsic to particular games but to the attitude of the player. The simple act of eating a meal can be either a finite or infinite game. I might be eating with a certain end in mind, to gain energy or lose weight or make a political statement. I might also eat spontaneously, for no reason other than to eat.

To read this book is to subject our attitudes to the finite-infinite paradigm, discover that we’re engaged in both modes of play, and consider to what extent we’re comfortable with this. Our conclusion may depend on our response to the book’s compelling final chapter, whose subject is myth and whose main idea is illustrated by the story of Copernicus:

Copernicus was a traveler who went with a hundred pairs of eyes, daring to look again at all that is familiar in the hope of vision. What we hear in this account is the ancient saga of the solitary wanderer, the peregrinus, who risks anything for the sake of surprise. True, at a certain point he stopped to look and may have ended his journey as a Master Player setting down bounded fact. But what resounds most deeply in the life of Copernicus is the journey that made knowledge possible and not the knowledge that made the journey successful.

Simple Proxy with Squid, SSH, and a VPS

2009-08-15T20:59:24-04:00

Sometimes, while at home browsing the web, I’ll enter an erroneous address and receive my ISP’s custom search page – with suggested links – as a response. This is always a little unsettling because it reminds me that all of my network traffic is being mediated, and perhaps recorded, by a powerful media conglomorate.

If you have a VPS, it’s easy to set up your own proxy server, connect securely using SSH, and know that any traffic sniffed by your ISP will be encrypted.

1. Install and configure Squid.

If you’re running Ubuntu, you can install Squid as follows:

sudo apt-get install squid squid-common

Before starting the server, you’ll need to make some changes to the squid configuration file (/etc/squid/squid.conf).

First, decide upon a visible_hostname. Squid won’t start unless this directive is set.

visible_hostname my_funky_proxy_server

Next, make sure that you allow for all http traffic.

http_access allow all

Finally, if you’d like Squid to run on a port other than the default, search for the http_port directive:

http_port 3128

Save your settings and start the squid server.

sudo /etc/init.d/squid start

If Squid starts without any errors, you’ll have yourself an http proxy server running on port 3128 of your VPS.

2. Port forwarding with SSH.

The next step is to initiate a secure connection between a port on your home computer and port 3128 (or whichever port Squid is running on) on your VPS. Suppose we choose a local port 2600. Then we can use SSH to connect like so:

ssh -N -L 2600:localhost:3128 me@myvps.com

This command connects to myvps.com with the username me and, when connected, forwards port 2600 on my local computer to port 3128 on myvps.com.

3. Point your web browser to the new secure port.

If you’re using Firefox, open up the preferences, choose Advanced > Network > Settings. Choose Manual proxy configuration, enter localhost as the > HTTP proxy and 2600 (or whichever local port you configured in the last > set) as your port.

Nginx Subdomains Gotcha

2009-07-18T17:15:03-04:00

I was trying to set up a subdomain, photos.kylebanker.com, using nginx. No problems, until I realized that *.kylebanker.com (e.g., whatever.kylebanker.com) was resolving to the content at photos.kylebanker.com. Here’s my initial configuration. Do you see the what’s wrong?

   1  
   2  # Redirect all requests containing 'www.kylebanker.com' 
   3  # to 'kylebanker.com' 
   4  server {
   5      listen 80;
   6      server_name  www.kylebanker.com;
   7      rewrite ^(.*) http://kylebanker.com$1 permanent;
   8  }
   9  
  10  # Listen for kylebanker.com
  11  server {
  12          listen   80;
  13          server_name  kylebanker.com;
  14  
  15          access_log  /var/log/nginx/blog.access.log;
  16  
  17          location / { 
  18                  root   /home/blog/public; 
  19                  index  index.html index.htm; 
  20          } 
  21  }
  22  
  23  # Listen for photos.kylebanker.com
  24  server {
  25          listen   80;
  26          server_name  photos.kylebanker.com;
  27  
  28          access_log  /var/log/nginx/photos.access.log;
  29  
  30          location / { 
  31                  root   /home/photos/public; 
  32                  index  index.html index.htm; 
  33          } 
  34  }
  35

Turns out that I needed to add a wildcard alias. Line 6 should read as follows:

        server_name  kylebanker.com *.kylebanker.com;

Was this simply too obvious to be documented elsewhere?

Javascript, Tetris, and Hanami

2009-04-27T20:05:17-04:00

Javascript and Hanami are the topics of this post, though they really have nothing in common aside from being at the forefront of my mind.

See, just the other day, I gave myself the challenge of writing a Tetris clone using Javascript and CSS. It also happens that I had recently visited the Brooklyn Botanic Garden to witness the great blossoming (photos).

The result? A Hanami-themed version of Tetris, to play in your broweser. Try it out!.

Incidentally, Javascript’s functional paradigm is slick. Genuinely wowed by this elegant little language. The code is on github.

The Perils of Simplicity

2009-03-31T20:08:33-04:00

Michael Osinski’s software helped create the morgage-backed securities that have undermined our banking system. He reflects on the easy manipulation of systems we don’t wholly understand:

The power we all hold in our hands is shocking, yet it’s controlled by a few swipes of a finger. The drive to simplify the user’s contact with the machine has an inherent side effect of disguising the complexity of a given task. Over time, the users of any software are inured to the intricate nature of what they are doing. Also, as the software does more of the “thinking,” the user does less.

How do we reconcile this “shocking” power with the simplicity enabled by modern software? As more and more human systems submit to control by software, how do we avoid rendering what’s supposed to be too difficult or too dangerous all too doable?

Osinski’s story is worth reading.

Decluttering

2009-03-29T22:31:44-04:00

A cable modem, wireless router, and power strip can still equal a mess under the desk. A few cheap materials and two hours of enjoyable labor yielded this from the front:

And this underneath:

Alas, this clever idea is not my own. Check out Declutter Your Desk for the original implementation.

The Tyranny of the Fixed Price

2009-03-19T21:56:25-04:00

From Philip Greenspun’s book report on The Rise and Decline of Nations: Economic Growth, Stagflation, and Social Rigidities. The book’s author, Mancur Olson, argues that national economies fall with the rise of special interest groups, which prevent markets from correcting themselves:

How could the Great Depression have lasted so long? [Mancur] Olson suggests assuming that a lot of prices are fixed by colluding business cartels and/or by government regulation. The prices are fixed higher than they would be in a free market, which imposes costs on society and guarantees supranormal profits to cartel members. If there is inflation, the losses to the economy from the cartel are ameliorated. The fixed price is no longer much higher than what would have been the market price. In the event of deflation, however, the fixed price is now ridiculously high, demand for such an overpriced product plummets, and production plummets. Investment in new factories will fall to zero almost immediately.

The government bailouts seem all the more deplorable and misguided.

Recipe: Salmon and Sweet Potatoes

2009-03-18T00:45:12-04:00

A simple, relatively inexpensive, fancy dinner (and, no, I can’t believe I’m posting a recipe to my blog.)

Gourmets can still eat well, even as we enter recession times. Try on this salmon recipe instead of going out; it won’t break the bank.

Salmon

2 Salmon steaks
4 tbsp mustard (dijon, spicy brown)
1 tbsp honey
1/2 cup chopped fresh herbs (mostly parsley, with some thyme)
Parchment paper

Preheat oven to 450F. Mix the mustard and honey. Line a cookie sheet with the parchement paper, placing the Salmon on top. Spread the honey-mustard mixture over the surface of the Salmon steaks and sprinkle with the herbs. Bake around 20 minutes, or until flaky.

Sweet Potatoes

4 tbsp unsalted butter (cut into four pieces)
2 tbsp heavy cream
1/2 tsp salt
1 tsp sugar
2lbs sweet potatoes, peeled, quartered, cut into 1/4-inch-thick slices.

Combine the butter, cream, salt, sugar, and potatoes in a medium saucepan. Cover and cook over low heat until the potatoes become mashable, 30-45 minutes. Remove from heat and mash.

Review: No Country for Old Men

2009-03-04T22:15:13-05:00

Several scenes from Cormac McCarthy’s No Country for Old Men made me sick to my stomach. Yet somehow I finished this novel, impelled by a deep respect for McCarthy’s other works (e.g., The Border Trilogy, The Road) and the inescapable allure of violence.

For this novel is indeed horrific. The remorseless Chigurh, unmoved by any power save the caprice of a coin flip, casts a dark pall across this story. Intermixed are the reflections of an old sheriff trying to make sense of the downfall of American civilization, of civility itself, and the turning away from the simple truths of right and wrong.

Sure, there’s also a narcotics plot featuring a suitcase stuffed with millions of dollars, decaying bodies, Ford Broncos, innumerable dreary hotels, border crossings, cafe diners, and anonymous powerful men working behind the scenes from Houston skyscrapers. But the novel seems indifferent to these details in the end. Even certain sympathetic characters, when brutally murdered, are apparently forgotten, the narrative not skipping a beat.

What stand out are Sheriff Bell and Chigurh; their names cannot be unintentional. Bell serves as a reminder, a call to ponder, and not ignore, the brutality we witness in the news daily and the horror that may be to come.

That horror would be Chigurgh, the amorphous and frightening embodiment of civilization as it stands taken its logical extreme. I don’t believe I’ve ever encountered a character so heinous in all of literature, an eerie and sobering portrait I won’t soon forget.