Zen and the Art of Programming

What is DB2 pureScale?

2009-10-21T14:38:20Z

There are two main types of scalability: vertical and horizontal. Vertical scalability consists of potentiating the hardware specs of a given server. This is typically done by increasing the number (and to a minor extent, speed) of the processors, adding more RAM, and so on. Commodity hardware tends to impose a strong limit on the resources that can be augmented. There is only so much that can be stuffed into your typical x64_86 Dell server, even replacing the entire machine with the top of the line model (having the same architecture). For this very reason, many customers opt to purchase more expensive hardware that has the advantage of facilitating vertical scalability, by allowing much more headroom for additional resources. This is why Unix-based solutions like IBM System p + AIX and Sun’s Solaris + SPARC are common choices in demanding environments.

For all their merits, even premium hardware solutions can only do so much in terms of vertical scalability. They may postpone the problem, by allowing a large number of processors and a huge amount of RAM to be installed, but they are still not limitless. And that’s when it becomes cheaper and much more feasible to attempt horizontal scalability, or the process of adding extra servers that share data and the overall system load.

Commercial databases, particularly DB2, excel at vertical scalability, which incidentally is one of the weakest aspects of open source RDBMS like MySQL. When it comes to horizontal scalability though, even commercial databases tend do struggle a little. Scalability is achieved, but a lot of work is required to configure complex setups involving masters, slaves, replication, failover servers, and so on. Until now.

IBM has just introduced a new technology for DB2 called pureScale, which accomplishes three incredible feats:

Horizontal scalability: Virtually unlimited capacity by simply adding new nodes to the cluster.
Application transparency: No code changes needed.
Continuos data availability: regardless of which server fails, the system will continue to operate albeit with a proportionally reduced throughput.

DB2 pureScale applies to DB2 running on Unix (IBM System p running AIX only, for the moment), the same principles and rock solid technology employed by DB2 for z/OS Data Sharing and SYSPLEX. This technology happens to be employed by some of the largest companies in the world and it’s considered the golden standard when it comes to easily create OLTP database clusters, providing virtually unlimited scalability capabilities.

You can read more about it here and here. If you are going to IOD this year, don’t forget to attend this overview by one the architects of DB2 pureScale. And even if you can’t head over to Las Vegas next week, you can still attend online our Chat with the Lab today at 11:30am EST. Database scalability is a hot topic and this is going to be one of the most popular chats with the lab ever, so register for free now.

Bookmark and share:

Add code highlighting to your Google Waves

2009-10-20T00:53:46Z

Google Wave is still rough around the edges, but it has a lot of potential in terms of becoming a great collaboration tool. As a developer, your first question will probably be: “How do I add code highlighting to my waves?”. The answer is straightforward, however not very easy to find if you google it. I hope this post will help fellow developers who are experimenting with Google Wave.

The following steps are required to obtain syntax highlighting for your code:

Create a new wave and add the Syntaxy robot to your wave. Use the wave address: kasyntaxy@appspot.com.
Reply to your first message or within it, thereby creating a reply (called “blip” in Google lingo).
Specify your code’s language, prefixing the name with a hash and exclamation mark, like #!python or #!ruby.

At this point, as you type the code in your blip it will be highlighted by the Syntaxy bot as shown in the picture below:

More advanced automatic syntax highlighting bots will probably appear as Google Wave progresses, but this one should do the trick for now. On a side note, if you copy and paste code from XCode, the code formatting will be kept in your waves and blips without the need for bots.

Bookmark and share:

Getting MacRuby’s compiler to work

2009-10-08T20:49:32Z

There is major news in Rubyland today. MacRuby’s team just released their fist beta of version 0.5 (an experimental, still incomplete version of Ruby), which brings JIT, removal of the dreaded GIL (Global Interpreter Lock), native threads, GCD (Grand Central Dispatch) for multicore computing, and a whole new set of features found in the release announcement to the table.

The most important new feature is the presence of a compiler. That’s right, thanks to this release, Ruby code can now become highly optimized executable code. How awesome is that? I can sense that you’re pumped by this news, so why not head over to MacRuby.com and download the installation file for yourself? After you’ve done that, the next thing you’re going to want to do is run a small test like the following:

$ macrubyc world_domination.rb -o world_domination
Can't locate program `llc'

Oh noes! llc is a tool that ships with the LLVM (upon which MacRuby is built), however it’s not included with MacRuby’s installer (it will be in the future). But fear not my friends, there is a solution:

$ svn co -r 82747 https://llvm.org/svn/llvm-project/llvm/trunk llvm-trunk
$ cd llvm-trunk
$ ./configure
$ UNIVERSAL=1 UNIVERSAL_ARCH="i386 x86_64" ENABLE_OPTIMIZED=1 make -j2
$ sudo env UNIVERSAL=1 UNIVERSAL_ARCH="i386 x86_64" ENABLE_OPTIMIZED=1 make install

If your machine does not have 2 cores, remove the -j2 option from the fourth line or adjust the number accordingly.

The compilation phase may take a couple of centuries, depending on your machine’s speed, but it should eventually build the LLVM. llc will be placed in your PATH, and you’ll finally be able to compile Ruby code and obtain an executable to help you carry out your world domination plans.

$ macrubyc world_domination.rb -o world_domination
$ ./world_domination
MUAHAHAHAHA!

Bookmark and share:

The CRTC tries to eliminate ISP competition in Canada

2009-09-24T16:18:21Z

The CRTC is at it again. This time they want to allow Canadian phone companies to cut off their wholesale offering, essentially killing ISP competition in Canada. Please express your concerns here, where you can quickly email Industry Minister Tony Clement, Prime Minister Harper, the Opposition Leader (Michael Ignatieff) and your local MP. Let them know that you care about this important issue.

For more details, I’m attaching an informative email I received from TekSavvy:

Hello,

We need your help — we need you to lend us your voice and state your opinion on a recent CRTC decision that could allow the incumbent phone companies (such as Bell and Telus) to discontinue offering services (Internet and phone) to wholesale providers which in effect would eliminate competition in Canada.

With no competition to help fuel the market; Internet innovation could come to stand still and prices for broadband access could skyrocket out of control. The incumbents would also be free to do with the lines as they please. This would open up future privacy invading Deep Packet Inspection (DPI) technologies and direct marketing/advertising based on *you* and *your families* internet activities and habits.

They could also use this same technology to turn the Internet into a pay-per-website service the same way that TV is a pay-per-channel service. Can you imagine a world where you have to pay to use the Google website to search for information? Then, if you find what you are looking for you may be restricted to the content because your ISP doesn’t have a “peering” agreement with said company.

Please help us stop this major potential catastrophe.

All concerned consumers please visit:
http://www.competitivebroadband.com/consumer/

All concerned businesses please visit:
http://www.competitivebroadband.com

Read the information presented and decide for yourself.

Once you’ve decided what you think is right, you can click the “Make a Difference” link, fill out the form with your details and send a pre-written message to Minister of Industry Tony Clement, Prime Minister Harper, the Opposition Leader, and your local MP or choose to write your own.

Together we can make a difference.
Thank you,
TekSavvy Solutions, Inc.

Bookmark and share:

Benchmarking Tornado vs. Twisted Web vs. Tornado on Twisted

2009-09-14T07:10:11Z

FriendFeed, which was recently acquired by Facebook, just released an interesting piece of open source software.

Tornado is an open source version of the scalable, non-blocking web server and tools that power FriendFeed. The FriendFeed application is written using a web framework that looks a bit like web.py or Google’s webapp, but with additional tools and optimizations to take advantage of the underlying non-blocking infrastructure.

The story so far

This release generated widespread interest among the Python and open source development communities. Rightfully so. There are many reasons to like Tornado. To begin with, it’s fast — and that’s fundamental for a web server. By using nginx as a load balancer and a static file server, and running a few Tornado instances (usually one per core available on the machine) it’s possible to handle thousands upon thousands of concurrent connections on relatively modest hardware; and this isn’t just theory. Tornado has already proven its worth in the field, by allowing FriendFeed to scale graciously.

Tornado is not only a fast web server, it acts as a very lightweight application framework as well. As such, it’s an appealing alternative to well established frameworks to the growing group of developers who’d like to develop “closer to the metal” and avoid the baggage associated with full-fledged web frameworks. The two things combined make Tornado ideal for developing “real time” web services and applications.

The feedback so far hasn’t been all positive though. Criticism of the project has mainly focused on the lack of test coverage and the fact that FriendFeed has opted not to contribute to, and improve on, the existing Twisted Web project (which has similar goals). To make things worse, there were a few nonchalant comments about it as well. Performance issues and lack of ease of use were the reported motivations for starting a new project from scratch.

Dustin Sallings started working on a hybrid solution (henceforth Tornado on Twisted) that would reportedly keep the good parts that Tornado introduced, while using Twisted as its core for networking and HTTP parsing.

At this point I became naturally curious about the speed of these three web servers. Is Tornado really faster than Twisted Web? And what about Tornado on Twisted, would it be faster or slower? Let’s find out.

Benchmark results

I ran a simple Hello World app for all three web servers. All the web servers were run in standalone mode without a load balancer. I stress tested the web servers with httperf using a progressively larger amount of concurrent requests. 100,000 requests were generated for each test. The web servers were run on a desktop machine with an Intel® Core™2 Quad Processor Q6600 (8M Cache, 2.40 GHz, 1066 MHz FSB) processor and 8GB of RAM. The operating system of choice was Ubuntu 9.04 (x86_64).

Without further ado, here are the results:

As you can see Tornado turned out to be faster than the rest of the Python web servers. Handling a peak of almost 3900 req/s with a single front-end and on commodity hardware is nothing to sneer at.

Twisted Web didn’t do too bad either (max. 2703.7 req/s), but the difference in performance is noticeable. Likewise, the performance of Tornado on Twisted was virtually identical to that of Twisted Web.

There you have it. I was curious about the possible outcome and now I know. Remember, this is a report on the numbers I got on my machine, not a research paper. But I hope that you find them interesting nevertheless.

Show me the code

Tornado:

import tornado.httpserver
import tornado.ioloop
import tornado.options
import tornado.web
import logging

from tornado.options import define, options

define("port", default=8888, help="run on the given port", type=int)

class MainHandler(tornado.web.RequestHandler):
    def get(self):
        self.write("Hello, world!")

def main():
    tornado.options.parse_command_line()
    application = tornado.web.Application([
        (r"/", MainHandler),
    ])
    http_server = tornado.httpserver.HTTPServer(application)
    http_server.listen(options.port)
    tornado.ioloop.IOLoop.instance().start()

if __name__ == "__main__":
    main()

Twisted Web:

from twisted.internet import epollreactor
epollreactor.install()
from twisted.internet import reactor
from twisted.web import server, resource

class Simple(resource.Resource):
    isLeaf = True
    def render_GET(self, request):
        return "Hello, world!"

site = server.Site(Simple())
reactor.listenTCP(8888, site)
reactor.run()

Tornado on Twisted:

from twisted.internet import epollreactor
epollreactor.install()
from twisted.internet import reactor

import tornado.options
import tornado.twister
import tornado.web
import logging

from tornado.options import define, options

define("port", default=8888, help="run on the given port", type=int)

class MainHandler(tornado.web.RequestHandler):
    def get(self):
        self.write("Hello, world!")

def main():
    tornado.options.parse_command_line()
    application = tornado.web.Application([
        (r"/", MainHandler),
    ])

    site = tornado.twister.TornadoSite(application)
    reactor.listenTCP(options.port, site)

    reactor.run()

if __name__ == "__main__":
    main()

UPDATE (September 14, 2009):

The original version of this post included Unicorn as well. This wasn’t fair however, since it’s not an asynchronous web server.
EventMachine HTTP Server was added, but I have since decided to remove it as I prefer to let the article be a fair comparison between asynchronous Python web servers.
I initially used Apache Benchmark (ab). The results were misleading at best. I re-ran the tests with httperf and updated the results above.
Stock Tornado couldn’t be tested with httperf because their HTTP Server doesn’t implement getClientIP(). I had to manually modify a method to return the remote ip address. This may introduce a very minimal advantage for Tornado, but it should be negligible in this context.
I modified the examples for Twisted and Tornado on Twisted, to ensure that both took advantage of the epoll-based reactor.

Bookmark and share:

Installing and configuring DB2 Express-C 9.7 on Ubuntu 9.04 Server

2009-09-11T01:53:00Z

DB2 Express-C 9.7 can easily be installed on Ubuntu 9.04 Desktop edition by simply issuing sudo ./db2setup. This will open up a launchpad and you’ll be able to install the product through a wizard. But what about setting up DB2 Express-C 9.7 on Ubuntu 9.04 Server edition? When you are ssh-ing into your VPS or dedicated sever, there are no GUIs to help you out.

IBM provides all the information on how to proceed through a series of detailed documents, which explain why each command is required and what your options are. But if you are looking for a single document that is slightly more concise in nature, you can read this excellent step by step guide that was published today on the blog bits.of.info.

Bookmark and share:

Improve the speed and security of your SQL queries

2009-09-09T13:48:10Z

An easy way to improve the performance and security of SQL queries is to replace literals with parameters. By replacing literal values with parameters, advanced relational databases will be able to compile your queries and have their execution plans cached. This saves time and precious resources when the same query (minus the actual values) is executed over and over.

Consider the following series of queries:

SELECT * FROM users WHERE karma BETWEEN 100 AND 499;
SELECT * FROM users WHERE karma BETWEEN 500 AND 999;
SELECT * FROM users WHERE karma BETWEEN 1000 AND 1999;
SELECT * FROM users WHERE karma BETWEEN 2000 AND 4999;
SELECT * FROM users WHERE karma BETWEEN 5000 AND 9999;
SELECT * FROM users WHERE karma BETWEEN 10000 AND 50000;

These each represent the same query and can be transformed into a single parameterized query:

SELECT * FROM users WHERE karma BETWEEN ? AND ?;

Trying to use clever tricks with quotes in order to inject arbitrary SQL code becomes futile. Parameters are considered values, and have no effect on the structure of the query itself.

Parameterized queries are therefore efficient and go a long way towards preventing SQL injection attacks in your applications. They have virtually no downside.

Newbie developers often ignore the existence of this feature and end up irritating seasoned DBAs who have to deal with the consequences of their incompetence. Leon Katsnelson argues that this is such an important matter, that every DBA should forward this Computerworld article to their developers. I tend to agree with how important of an issue that is.

That article provides the following example in Java:

String lastName = req.getParameter("lastName");
String query = "select * from customers where last_name = ?"
PreparedStatement pstmt = connection.prepareStatement(query);
pstmt.setString(1, lastName);
try { ResultSet results = pstmt.execute(); }

Here I’ll show you an example of how to work with parameterized queries from Ruby and Python. I’ll use the Ruby and Python drivers for DB2.

Ruby first:

require 'ibm_db'

conn = IBM_DB.connect("mydb", "db2inst1", "mypassword")

query = "SELECT * FROM users WHERE karma BETWEEN ? AND ?"
pstmt = IBM_DB.prepare(conn, query)

values = [500, 999]
IBM_DB.execute(pstmt, values)

while row = IBM_DB.fetch_array(pstmt)
  puts "#{row[0]}:#{row[1]}"
end

We load the driver (use mswin32/ibm_db on Windows, and ibm_db.bundle on Mac), create a prepared statement, and then bind the two parameter values to it through the execute method. We then fetch the resultset one row at a time and print the value of the first two fields for each record. For fine-tuned control we could have used the IBM_DB::bind_param method.

The Python version is very similar:

import ibm_db

conn = ibm_db.connect("mydb", "db2inst1", "mypassword")

query = "SELECT * FROM users WHERE karma BETWEEN ? AND ?"
pstmt = ibm_db.prepare(conn, query)

values = (500, 999)
ibm_db.execute(pstmt, values)

tuple = ibm_db.fetch_tuple(pstmt)
while tuple:
    print tuple[0] + ":" + tuple[1]
    tuple = ibm_db.fetch_tuple(pstmt)

As you can see, working with parameterized queries is not any harder than dynamically generating SQL queries. Yet the benefits of doing so are huge.

Unfortunately, despite being a very sound choice to base an Object-Relational Mapper (ORM) on, ActiveRecord does not use parameterized queries. Even when it looks like you are passing parameters to a given method, these are actually used to dynamically form an SQL query. Of course you are still free to use parameterized queries in your Rails applications by employing the driver directly. But I really think this is something ActiveRecord should be built upon.

Luckily for Django developers, Django’s ORM uses parameterized queries, thus improving both performance and security with a single design choice. In the Python world you couldn’t get away with ignoring parameterized queries.

For those of you using Rails, all is not lost. DB2 Express-C 9.7 has a killer feature known as the Statement Concentrator, which caches similar queries allowing them to use a shared access plan. It’s not as efficient as using prepared statements in your code, but it’s the best you can do when, as in the case of ActiveRecord, you can’t use parameterized queries directly. Leon’s article explains in greater detail how this feature actually works.

Bookmark and share:

Enabling support for DB2 and Python/Django/SQLAlchemy on Mac OS X Snow Leopard

2009-09-08T23:39:34Z

This is the Python version of a post I made about Ruby a few days ago.

Now that Mac OS X 10.6 is out, it’s time to leave the world of 32 bit computing behind. The pre-installed Python interpreter will run in 64 bit mode by default, so you may need to pay attention when installing some C-based eggs.

Assuming you have DB2 Express-C installed already, the ibm_db Python egg for DB2 can easily be installed by following these simple steps:

$ sudo -s
$ export IBM_DB_LIB=/Users/<username>/sqllib/lib64
$ export IBM_DB_DIR=/Users/<username>/sqllib
$ export ARCHFLAGS="-arch x86_64"
$ easy_install ibm_db

This will install the ibm_db C driver, and the ibm_db_dbi Python module that complies to the DB-API 2.0 specification.

You can verify that the installation was successful my running the following:

$ python
>>> import ibm_db
>>>

Now, for the Django adapter, install Django first (if you haven’t done so already):

$ sudo easy_install django

The Django adapter can then be installed as follows:

$ sudo easy_install ibm_db_django

Finally, if have installed SQLAlchemy and wish to install the DB2 adapter for it, run:

$ sudo easy_install ibm_db_sa

Please let me know if you encounter any issues, I’d be glad to help you.

Bookmark and share:

Cell Phone Cost Calculator Killed In Canada. Enough Is Enough!

2009-09-06T23:15:00Z

Having been born and raised in Europe, I find the Canadian Telco sector appalling. In what is an otherwise outstanding country, the monopolistic tendencies and de facto cartels of the phone companies are screwing over Canadian residents, and there is very little being done to counteract this.

Recently Bell got its way again, and UBB (Usage Based Billing) was introduced regardless of what thousands upon thousands of taxpayers had to say about it. Countless complaints were forwarded to the CRTC (Canadian Radio-television and Telecommunications Commission), and these were promptly disregarded.

On Slashdot today there’s a story entitled Cell Phone Cost Calculator Killed In Canada. Upon further inspection it’s revealed that cell phone carriers have successfully lobbied public officials to stop a taxpayer funded initiative that would publish an online cellphone cost calculator. (”OMG competition! Think of the shareholders!” As someone pointed out on Slashdot).

Stopping this service is a waste of our money, but more importantly, it clearly highlights the fear of transparency and competition that is typical of companies like Bell and Rogers. It shows the power that phone companies have over the government. And it demonstrates how elected officials like Tony Clement (Industry Minister) are far more concerned with the bottom line of public companies than the interest of Canadian citizens and residents alike.

This is outrageous. How can we fight back? For once, I’m a believer in voting with my dollars. I do not currently own a cell phone, and I surf the web on a (factual) 3 Mbps connection from TekSavvy, having switched from Rogers’ 10 Mbps plan more than a year ago.

The two are an impediment, because I need a cell phone and a fast Internet connection. But the lack of competition doesn’t leave you with many alternative options if you want to avoid giving your hard earned money to the types of carriers mentioned above. And in the case of the calculator, my tax money ended up being wasted to protect these companies’ obsolete pricing models.

The (always excellent) Michael Geist makes a worthwhile suggestion:

With public dollars having funded the mothballed project, the government should now consider releasing the calculator’s source code and enable other groups to pick up where the OCA (Office of Consumer Affairs) left off.

Today I registered MyPhoneBill.ca. Should they ever release the source code, I will host it and deploy it at this handy URL. After all, we’ve already paid for the code.

In truth, I don’t see this happening. An open source solution that companies and individuals could build upon would frighten cell phone carriers even more than a closed source project deployed by the government.

If they won’t open source the code of the cell phone cost calculator, as I suspect the case will be, I’ll take a stand. I’ll implement and offer the service myself at MyPhoneBill.ca (site not active yet).

Such a site may very well have a solid business model, and I’m not opposed to that idea. But that’s not the main reason why I’d create such a project. Canadians have the right to learn about what the most advantageous phone plan for their usage pattern is, without having to spend days on end doing research. And companies have no right whatsoever to prevent this from happening. Similar sites exist in almost every other industrialized country and cell phone companies usually collaborate with them.

Enough is enough!

Bookmark and share:

Installing the Ruby driver for DB2 on Mac OS X Snow Leopard

2009-09-03T19:23:35Z

Now that Mac OS X 10.6 is out, it’s time to leave the world of 32 bit computing behind. The pre-installed Ruby interpreter will run in 64 bit mode by default, so you may need to pay attention when installing some C-based gems. The ibm_db Ruby gem for DB2 can easily be installed or updated to the latest available version by following these simple steps:

$ sudo -s
$ export IBM_DB_LIB=/Users/<username>/sqllib/lib64
$ export IBM_DB_INCLUDE=/Users/<username>/sqllib/include
$ export ARCHFLAGS="-arch x86_64"
$ gem install ibm_db

You can verify that the installation was successful my running the following:

$ irb
>> require 'ibm_db.bundle'
=> true

Please let me know if you encounter any issues, I’d be glad to help you.

Bookmark and share: