From cells to a cities, visualizing the world as a series of recurring patterns which can be understood is awe inspiring to me.

To think that once we unlock these patterns we’ll find nothing is random, that makes me optimistic.

Behind the scenes at Key Ring we also have a large Rails footprint. For the past year we’ve been using New Relic to monitor application performance up and down the stack. When it comes to monitoring and troubleshooting a web app I can’t recommend these guys enough. They’re constantly making improvements, adding features, and the product has become indispensable to our team.

So I just finished building some data services in CherryPy and was psyched to hook up the New Relic agent to this app. The docs on the New Relic site are pretty thorough but I wasn’t able to piece together how to hook the agent into the CherryPy app. Also, could’t find anything on the web. Through trial and error I was able to get it working…. it’s actually extremely easy if you just follow these steps.

1. Download the latest Python agent. Unpack.

 wget http://download.newrelic.com/python_agent/release/newrelic-1.0.5.156.tar.gz

2. Run the setup

python setup.py install

3. Generate a New Relic agent config file using your API key.

newrelic-admin generate-config $YOUR_API_KEY newrelic.ini

4. Change your app name in the New Relic config. There are some other logging bells and whistles which are all self-explanatory

5.  Add these 2 lines to your sites startup script. Make sure you do it early in the script before any database calls, etc…

import newrelic.agent

newrelic.agent.initialize('/path/to/newrelic.ini')

And that’s it. In the next minute or so you’ll begin to see your stats streaming into your New Relic dashboard.

I kept searching for issues related to the actual phone, Samsung USB drivers,  etc – but as these things sometimes turn out it was a simple setting I finally stumbled upon in this post. I just needed to increase the ADB timeout to 10000 and everything worked fine. Side note… these Galaxy S phones are painfully slow to work with, in the end I wasn’t suprised by the solution.

Anyway, here’s hoping that anyone else running into this issue comes upon this post quickly and can get back to productive work. Happy hacking!

While the term “Real Time” has taken off over the past 6 months most realize that our existing communications infrastructure already operates at near real time. You send an email, it arrives in seconds. You place a call, someone picks up. Blog posts, satellite television, GPS, IM, etc etc etc.

I’d say the fundamental shift in behavior we are seeing on the web today is related to “Always On”. It’s ubiquitous network connectivity that makes us feel the already real time nature of the web even more.

So what’s up with Real Time Search and the Real Time Web? Basically it’s about content being indexed and presented in search results as fast as it’s being produced. This is a certainly a step in the right direction towards the larger goal of instant and ubiquitous human knowledge – “when I know, you know”.  The problem is there’s just too much noise when you turn on the stream and the only filter in place are keywords.

The technology is important though; data must be collected and indexed before it can be filtered/ranked. We’re getting there.

What gets me excited about the Real Time Web are the ways it can be used to augment existing methods for consumption of news and entertainment. Imagine the ways that the PubSub model combined with Real Time Search will allow people to “tune-in” to personalized data feeds during sporting events, tv shows, breaking news.

For example, when I am watching the Dallas Cowboys on TV I don’t want to type “Dallas Cowboys” into a search engine and be flooded by results. I want to tune-in to a list of people that I’ve selected (or have been recommended).  These people may be professionals, they might be my neighbor. It’s these people that will be providing insight, analysis, and commentary. Troy Aikmen and Joe Buck? Nope. I want comedy. I want bias. I want camaraderie. Then when the game is over I want to tune out, I want it all to go away.

To me the Real Time Web is not about speed, it’s about moving past the period where Social Networks are persistent. The Real Time Web will introduce Social Networks that are dynamic.  Networks that emerge and disappear in short spans of time. These networks will be asynchronous – increasingly the Real Time Web will look more like the Real World.

So the task here was to find a way to specify the IP address, aka socket, aka network interface when making an http request using Python’s urllib2. Why would you want to do this you ask? Well for many web API’s the request rate is limited by whitelisting the IP address – such is the case with Twitter. In the event that you want to be able to use the same machine (with multiple network interfaces) to run jobs in parallel you need to be able to specify where the requests should be routed.

The problem is Python’s urllib2 is based on the httplib library which doesn’t let you specify which address to bind to. This person tried to get around the problem in 2005 without any luck, another guy created a patch for httplib in 2008 which hasn’t been accepted, and finally someone else created a subclass for httplib which unfortunately I couldn’t get hooked up to the urllib2 class.

The best solution I found was this “monkey patch” from Alex Martelli over on Stack Overflow.  In his example he attacks the problem using the socket library instead of the httplib. By his own admission stuff like this is not ideal, but the solution is actually very simple and elegant. I like it.

I wrapped the snippet up into a function which can be called in a Python script anytime before you invoke a urllib2 request.

def bind_alt_socket(alt_ip):

true_socket = socket.socket
def bound_socket(*a, **k):
     sock = true_socket(*a, **k)
     sock.bind((alt_ip, 0))
     return sock
socket.socket = bound_socket

Hope this can be of help to someone in the future who’s searching for the same thing I was.

When upgrading from 5.0 to 5.1 using apt everything will install normally. Then when the MySQL service tries to restart you’ll see and init.d error and an error that looks something like this:

Errors were encountered while processing:mysql-server-5.1mysql-server

Not a lot to go on here but as it turns there is a deprecated entry in the my.cnf file called skip-bdb.  Comment this line out and you should be good to go.

It was unfortunate that the post was written in such a polarizing manner and that the comment thread quickly eroded into a shouting match because there is another important point to be made here.

Something else I want to put out there to developers is that the evolution of programming should be focused less on desktop vs web vs embedded or choice of language/platform, and more on how lowering the barriers to entry for new programmers is a positive and not a negative.

To someone writing device drivers or kernel patches the idea of writing a JavaScript function to manipulate the DOM may seem “uninteresting”, but the fact is that more and more people are getting started with programming this way. All they need is a text editor and a web browser and they are on their way. This is a very good thing.

Programming is about automation and automation is about improving efficiency. The more people we can somehow involve in this process the better because in the end the web provides not only the largest number of potential users, but also the largest number of potential programmers. The exciting thing is we are just getting started.

Let me start by saying that the examples Martin uses in the first post are technically accurate if we were all disk manufacturers. Whether bitmap (audio, video) or text (email, html), it’s true all of these file types use a structured format when being processed by a computer. That being said, we are not all disk manufacturers.

As a data architect I’ve always felt the true spirit of the phrase “unstructured data” corresponds to the modeling and analysis of the data. If you have a collection of objects in an email, an image, or web page… then these things are unstructured. They tell you nothing without the context of the structured model.

If this were simply a preference in terminology then I wouldn’t think too much of it, but when a relational database vendor claims that “nontraditional” (unstructured) data is easily converted to “traditional” data by running fact/entity extraction routines and loading a table it makes me stop and question the true intent of the original message. It’s not as simple as pushing a button, and an RDBMS is most often not your best option. This isn’t something which should be glossed over.

The problem is that when using a relational database schema the relationships, attributes, and quantities must be defined before running any extraction routines. That’s ok when running against a fixed set of data looking for a known set of attributes/measures – but when you are mining millions of images or billions of web pages all of the edges don’t start to show up until you actually start to extract and analyze the data. In this situation a relational database actually makes it harder to consume unstructured data due to the high cost associated with schema changes

To me the term unstructured makes sense… it’s simply the inverse of structured. Data without a model if you will.  And remember, the larger and more diverse the data set, the less you will know about it’s characteristcs ahead of time.

What I am thinking is this…. as system architects or software engineers this is a critical question to ask not only during design, but also as our solutions age over time. When asked this way it challenges us to think about dependencies from a different perspective. Instead of “are things working? If so, add new feature“, we should equally be looking for ways to remove, replace, or consolidate components for the benefits of efficiency, simplicity and performance. Sounds simple right, then why is it so hard to do?

I don’t think we go through this exercise often enough because politically it can be a very difficult. The sheer inertia of a large project can prevent people from even thinking about the question. The thing is that writing code, choosing a framework, establishing an API…. these are all sunk costs. We often attach emotional value to the time and effort that went into doing the work, and have a hard time imagining what things would look like without them.

It’s certainly a challenge, but to build better systems we need to be able to let things go, to scrap code, and to replace components without emotions or personal bias.

Oh and for the record, I do think RDF and Linked Data are the right combination to build the Web of Data. I just hope that we can keep asking questions, challenging assumptions, and continue to have constructive debates about the future of the Web like the one which took place yesterday on Paul’s website. Great stuff.

What is really rocking my world is the whole concept of throw-away servers. The idea that a discrete process can spin up a new server that gets built at run time, does some work, then disappears is amazing.  I see this as turning the whole concept of linear scale on it’s head. You don’t scale an app, you scale individual threads. Powerful stuff, especially when dealing with data mining and event processing.

Much more coming soon…..