Very high on my list of things to fix I have an item to look into Vaurien, and make some tweaks to the service to cope better with outages in other parts of the system.

As you probably realized by now if you’re somehow involved into the design and maintenance of any kind of distributed systems, network partitions are a big deal, and we’ve had some of those affect our service in the past, some with very interesting effects.

Take our application servers for example. They’ve been through many generations of rewrites, and switches from one WSGI server to another in the past (not long before I joined the team), each of them with a particular issue. Either they didn’t scale, or crashed constantly, or had memory leaks. Or maybe none of those (it was before I joined the team, so I wouldn’t know for sure). By the time I joined, Paste, of all the things, was one of the WSGI servers in use in part of the service, and Twisted WSGI was used in another part.

(The actual setup of those services is very interesting on itself. It’s a mix of Twisted and Django (and many others have done this before, so it’s not very unique. But there are internal details which are quite interesting. More below.)

Having moved from another team that used Twisted heavily, I decided to call it out and settle on Twisted WSGI, which seemed just fine.

As for the stability and memory issues, we started ironing them out one by one. Turns out the majority of the problems had nothing to do with the WSGI server itself, but everything to do with not cleaning up resources correctly, be it temporary files, file descriptors, and cycles between producers and consumers.

And everything was perfect.

But then we got a few networking issues and hardware issues. Some of the servers were eventually moved to a different datacenter and things got even more interesting. I’ll go into the details of the specific problems that I’m hoping to approach with Vaurien on a different post, but suffice to say that talking to many external services in a threaded server doesn’t get pretty when there’s a network blip.

So speaking of threaded and Twisted, and coming to the subject of this post.

In front of a subset of our services we currently have 4 HAProxy instances in different servers. They are all set up to use httpchk every 2 seconds, which by default sends an OPTIONS request to ‘/’. If you’re still following, we have a Django app running there, and depending on how you have your Django app configured, it might just take that OPTIONS request and treat it just like a GET, effectively (in our case) rendering a response just as if a normal browser had requested it. Turns out that page is not particularly lean in our case.

So you take 4 servers, effectively doing a GET request to your homepage every 2s each one, times many processes serving that page across a couple hosts, and you have a full plate for someone looking for things to optimize.

To make it more fun, early on I added monitoring of the thread pool used by Twisted WSGI, sending metrics to Graphite. Whenever we had a network blip we saw the queue growing and growing without bound. This was actually a combination of a couple things, which I’m still working on fixing:

1. HAProxy will keep triggering the httpchk after the service is taken out of rotation.
2. Twisted WSGI will keep accepting requests and throwing them in the thread pool queue, even if the thread pool is busy and the queue is building up.
3. We do a terrible job at timing out connections to external services currently so a minor blip can easily cause the thread pool queue to build up.

As a strategy to alleviate that problem I came up with the following solution:

1. Implement a custom thread pool that triggers a callback when going from busy -> free and from free -> busy (where busy is defined as: there are more requests queued than idle threads).
2. Changed the response to the HAProxy httpchk to simply check that busy/free state.
3. Changed the handling of that HAProxy check to *not* go through the thread pool.

(There’s a few more details that I won’t get into in this post, but that’s the high-level summary.)

I have good confidence that this will fix (or at least alleviate) the main issue, which is the queue growing without bounds in the thread pool, and it will instead move the queueing to HAProxy. But after looking through the metrics today I saw an unintended consequence of the changes.

In retrospect, it seems fairly obvious that this was to be one of the expected outcomes. I was simply surprised to see it since it was not the immediate goal of the proposed changes, but simply a side effect of them.

I hope you enjoyed this glimpse into what goes on at the heart of my job. I expect to write more about this soon, and maybe explore some of the details that I didn’t get into, since this post is already too long.

What I want to share with you today is the joy of being able to connect with old friends and recollect memories, as I mentioned I was longing for in my last post. In those few days I was in California I managed to catch up with Limi and Philipp, said an en passant hi to Rob Miller at the Mozilla SF office, had dinner with Gustavo, walked around the city with Fernando, Alberto, and Geoff, ending up at an amazing Chinese restaurant pretty much by accident, paid a visit to Marlon, who took me on a guided tour of the Facebook HQ followed by lunch at The Cheesecake Factory which I couldn’t refuse. It was exausting, but really great catching up with everyone!

A recurring topic between all of us was the general issues that all of our companies (Mozilla, Canonical, Facebook) have with general public perception. Most interestingly perhaps is the similarity between Canonical and Facebook when it comes down to privacy matters, how there seems to be a disconnect between the internal and external messaging on those matters, and how much the public perception is biased by the media and the very loud minority of privacy tinfoil hat zealots. I wish I could do more to help with solving that. Perhaps pushing for more transparency, better communication at least from the technical side of things could be a way to improve that.

Tech talks aside, I was simply overwhelmed by how much my kids’ pictures and videos are popular amongst friends. Every single person that I talked to was quick to mention that as the very first thing. Oddly, that generally does not reflect in likes and comments on those Facebook posts, which is an interesting observation. Are people generally afraid of clicking that Like link or is it too much effort for them? I’m sure it would do for a great usability study.

I hope to explore a bit more on the outcome of the sprint on a later post. Suffice to say that I was really glad to be present and contribute some feedback to all the planning that’s going into the next cycle, and the opportunity to meet some old friends while at it was invaluable. Looking forward to be doing more of that in the coming months, at FISL and PythonBrasil.

As an article I’ve read yesterday mentioned, we tech heads seem to live on a bubble that mostly bounces between social networks and having post work hours drinks with colleagues, usually from the same company. I wish we could all be more social in the physical world, and talk more about things that are not so tech-related. About life, and family, and non-work things, and enjoy ourselves more.

And headed straight into the shining sun.

I’ve been carrying on with life. Heads down into work. Enjoying my kids, which just started at daycare (or is afternooncare a word, even?) this week. But it’s going to hurt for quite some time. I can tell.

I feel like I should be writing more. I feel the need to hang out with old friends again. I might end up going to this year’s PythonBrasil/PloneConf one-two combo just for that. Hopefully some of the old-timers like Alan and Limi will be there. It would sure be great to see Paul again, which is apparently confirmed for one of the keynotes.

I actually started this post feeling very nostalgic, and the paragraph above shows. This was after stumbling on Gustavo’s 12 years ago post. Then I looked at the archive of this blog and found that my oldest post here is from July 20th 2003. Reading forward from there I’ve revived some really happy memories, and tons of now-broken links. One of the broken links was to one of the oldest traces of me out there, asking for help with Including TCL scripts in the CSLU Toolkit, back in 1999. Here’s another, from 2005, where I mention meeting Gustavo in person for the first time. And, the irony. I linked to his LiveJournal blog. :)

I feel much better now.

I should be writing more, and soon. I have lots of things to share, both personally and professionally. I really wish to become more like Dorneles. Not only fun to work with, but also fun to be around. I want to share more, not only about the things that I do, but also the things that I am. I hope to become a better person in the process, and I hope you can share and enjoy that journey with me.

The realization of that had a profound impact in my life. As programmers, technically minded people that we are, we tend to focus on ourselves and rarely share our knowledge, except to show it off at conferences here and there. We do it mostly to please our ego. Dorneles was nothing like that. He didn’t present at many conferences, though he was always there as either a participant or an organizer. He didn’t write lengthy blog posts full of technical details. In fact, he didn’t blog at all since 2007. He didn’t create brilliant frameworks. Instead, his contributions were mostly invisible, through IRC, IM or Skype, directly between him and the person seeking his help. It was very common for him to carry about five parallel conversations over IM, helping people all over the world without breaking a sweat. He would happily pull out his laptop or phone at a table in a blink to teach someone a trick. The endless stream of messages on twitter following his death should serve as proof that his way was much more effective in causing an impact in the world than a horde of highly skilled programmers ever could.

As other people well put it, Dorneles could never lose that smile on his face. Nothing could make him curse or complain. He was a mixture of cheerfulness, innocence, tranquility and awkwardness that would immediately conquer people’s hearts with no effort. He was like that young little brother that makes you blush at a family dinner for being excessively goofy, but that you love so much you would never reprehend.

To me and my wife, Dorneles was closer than family. My wife suffered more from his loss than for the loss of her lovely grandma, or from her uncle that died a slow death from brain cancer a couple years ago.

In retrospect, I can now see that he did treat me like an older brother, always eager to take on my advice. When he was alone in a big empty office in Caxias do Sul, following the break with the other partners at X3ng I told him to move to Garibaldi because we could rent a similar office for much cheaper. I suggested that he should reunite with his wife and two kids, which were living with her mom, and that raising them in Garibaldi would be much better for their education and overall quality of life. I said it would be nice if he had a car so that he didn’t need to call a cab at 3am when his kids were sick (he refused to buy a car for years, since he had cars stolen several times). At a time he was having trouble getting regular pay, I suggested he should work for Enfold. I told him to go to more conferences and sprints, so that he could put faces to the names on #plone. He was easily convinced. And all of these things had a great impact in his life.

I regret not spending as much time with him as I could in the last year. But many are the good memories that we’ve shared. Back in November 2009, when my backpack with laptop and cameras containing all the pictures from my vacation in New York got stolen on our first day in Houston he calmly told me: “Don’t worry, you can buy all those things again. They didn’t take from you the most precious thing: your life.”

I regret that he didn’t find the time to visit my 3mo old twins. But I’m glad that after my visit last Saturday, he went to visit his 2 weeks old nephew and his dad on the Sunday. I’m glad that we spent a good 2h last Saturday sitting around, chatting about the most diverse subjects while I was at his place waiting for the rain to stop.

Dorneles was a sushi lover. He once walked for tens of blocks in Houston on a cracking hot summer day just to eat sushi, and lived to tell the tale. That’s perhaps the most un-American thing he could do. The first thing we did when we met in Houston in November 2009 was to go out for sushi. When Nate came to Brazil, he took him out for sushi, and that’s perhaps the most un-Brazilian thing he could do. His love for sushi passed on to his kids. For several weeks I told my wife that we should invite his family out for sushi. We were waiting for the right occasion, and it never happened. In his memory, I think we should do it anyway.

Today I’m going to share with you one of the tricks that we’ve discovered almost by accident and that can help with making your website faster, if YUI is your Javascript framework of choice.

We’ve started using YUI in Landscape around March of 2009, by rewriting all of our ExtJS-based scripts (which weren’t that many) to use YUI. Since our application requires HTTPS due to potentially sensitive information, we had to self-host YUI, to avoid mixed-content warnings. Initially we had little experience with it, so we went with the easiest option available: loading the YUI seed file and letting everything else be loaded on-demand by the YUI Loader, without a combo handler (more on that later).

Soon enough we noticed that it wasn’t such a great idea. First-time access was terrible, with load times over 30 seconds in some cases, due to the combination of HTTPS connection overhead, no static resource caching and large number of requests to fetch all the resources. It was worse than that even, because most browsers either don’t cache HTTPS content by default, or only cache it on memory, unless you set the proper caching headers. To give you an idea of what it looked like, here’s what a trivial page using the ‘overlay’ module looks like in terms of loading.

No combo loading at all

It was pretty clear that this kind of performance would not be acceptable, so I started looking for options to reduce the number of requests. It was around that time that I heard about Steve Souders work on web performance, through a long time friend from the Plone community that happened to be working at Google and shared a link to Souders’ blog post on ‘@import’.

Turns out that the YUI Loader supports using a combo handler for reducing the number of requests, when loading modules on demand. What this ‘combo handler’ does is basically take a GET request with a bunch of filenames as parameters and return a single response with the respective files concatenated. Here’s an example using the combo server from YUI’s CDN.

Combo loading with a single 'Y.use()'

The problem was that we couldn’t use YUI’s combo server on the CDN because it doesn’t support HTTPS, which was requirement for us.

In November of 2009, at UDS-Lucid in Dallas, we had our first YUI-only cross-team sprint, totalling about 20 people from Launchpad, Landscape, ISD and Ubuntu One. The main goals were to spread knowledge on YUI and improve support for IE on our shared codebase (lazr-js), but I managed to sneak in two things for the Landscape team: implementing a Python-based combo loader (served by Twisted, but it’s just a plain WSGI app) and extracting YUI module metadata, so that we could make it easier to load extension modules on-demand, the code for which ended up as part of the lazr-js project.

At this point however we realized that there’s a significant issue with using the combo loader on an application as diverse as Launchpad or Landscape: depending on how you group your ‘Y.use()’ calls, the combo URL generated by the YUI Loader can vary significantly, which means that re-use from the browser cache is less than optimal. Here’s an example of loading both the ‘anim-color’ module and the ‘overlay’ module within a page. Compare it to the previous example and notice how the parameters passed to the combo are completely different, since the second request (to load the ‘overlay’ module) only requests modules which haven’t been loaded by the first request (to load the ‘anim-color’ module).

Combo loader can be a caching problem

In a hurry to get the situation solved ASAP, and taking a clue from our fellow developers on the Launchpad team, we went with a simpler alternative: manually combining all of YUI, including the seed file, and loading all of that in a ‘script’ tag in the document ‘head’. It turns out that the situation improved quite dramatically, but there was still more to be done. Although the number of requests was much smaller, the total page size was actually larger, since we were loading a lot of stuff that wasn’t even being used. Worse, since we were putting that ‘script’ tag in the ‘head’, it was blocking the whole page from loading until the script was finished downloading, which I after following Souders’ blog for a while I realized wasn’t such a great idea either. Here’s an example of what that looked like.

Combined modules in the head

At this point we had a significant codebase of YUI-based modules, and so did the Launchpad team, but we had gone from solely relying on the YUI Loader to not using it at all (by just loading all the code in one big honking file).

Once we got the module metadata extraction figured out though, I resumed working on improving the first-view load time, by reducing the number of modules manually combined to the bare minimum shared by all of our pages. Also, by having a single file with the common modules being loaded on all the pages, we can be sure that it is highly cacheable. And then we let the YUI Loader kick in and load the missing modules on less-used pages, which was a really nice improvement in page weight and reducing the time to the ‘onload’ event.

At this point Souders started talking about asynchronous loading of scripts and the ‘defer’ attribute, and I got my copy of Even Faster Web Sites. I started pondering if there was something that could be done to defer the loading of the combined modules while still letting the YUI Loader do it’s job.

The problem lies in the fact that YUI asynchronously loads modules that are not on the page yet, calling a callback function (the last argument to ‘Y.use()’) once all the scripts containing the required modules have finished loading. If all the modules are already on the page, it just calls the callback right away. So by just tacking a ‘defer’ attribute on the ‘script’ tag for the combined modules there would be a race condition between the loader checking if the modules were loaded and the script with the combined modules being loaded.

Then, it struck me. If the last argument to ‘Y.use()’ is just a callback function, could we queue those functions to be called at a later time and load the combined modules ourselves, before letting the YUI loader proceed? After a little back and forth with Dav Glass and some pseudocode exchange, I got an implementation of this idea(which I’m calling the ‘Prefetch YUI Loader Hack‘). And thus we managed to move the loading of all the combined modules from a ‘script’ in the ‘head’ to use asynchronous script loading, shortening even further our time to the ‘onload’ event. Here’s an example of what it looks like.

Prefetch with a single request

But wait, there’s more! Souders then started talking about parallel loading, tickling my speed bug again. After thinking for a while about the problem, enlightenment came again: when an YUI module is loaded, it is not executed right away, but added to an internal registry of modules. What that means is that regardless of what other modules it depends on, they can all be loaded independently from each other! That means you can break up a manually combined file into N smaller files and have them be loaded in up to N parallel connections (where N varies by browser and by browser version) and when they are all done you can let the YUI Loader kick in. Here’s an example with two parallel downloads plus an extra module not prefetched.

Prefetch, with parallel download

So that’s where things stand now. There are still optimizations that can be done in Landscape, like using the combo loader to reduce the number of requests on less common pages, or even using fixed URLs to the combo loader with the prefetch hack, to simplify the build process. We also need to start using Caridy’s Event Binder module, since now the pages load so fast that the users start clicking around before the event handlers are in place (ha!).

I am also pushing to get those kinds of tips documented and passed around Canonical, through a project codenamed Dare2BFast, which has the goal of coming up with a set of Web Performance Guidelines that all Canonical web sites should follow, both on the frontend and backend. Stay tuned!

I’m sitting here at Canonical’s office in Montreal, where we are having another Landscape sprint. There’s a couple things that make this sprint special, one of them being that Frank Wierzbicki has joined us! Welcome to the team, Frank, and I hope you enjoy!

It has also coincided with the end of our 1.5.5 milestone, which rolled out yesterday, with some interesting highlights:

Speed!

Contrary to what other people might tell you, YES speed does matter. We’ve been slowly (ha!) rolling out improvements to page load speed, and I’m happy to say that the improvements are very noticeable. Things we’ve done to improve this range from setting proper caching headers for static resources to heavily abusing the YUI Loader. I should write more about that on a more appropriate occasion. There’s always things to improve on, so keep an eye on it!

Faster Trials

Over the last four years, the trial registration for Landscape hasn’t been… enjoyable to say the least. This week we’ve finally rolled out the first iteration of our simplified registration process, which removes a manual approval step which used to take several weeks to be done. That means you can now register for a trial account in Landscape and start using it right away! Quite a novel concept, isn’t it? :) We’ll planning some more updates to it, so that when starting the registration process you have a better sense of where you are in the process and what are steps involved.

We have more exciting updates coming up before the end of the year, so keep an eye on it. But in the meantime, hit that ‘Registration’ link in Landscape, get it like you like it, and enjoy!

After several months of silence, I finally managed to get all the needed pieces back together to build lxml again. More than once people asked over on the lxml mailing list about what happened to the said eggs. The good news is that an unnoficial release of lxml 2.2.6 for Python 2.6 on Windows is now available for testing, in 32 and 64 bit flavors. The 64 bits version was compiled without iconv support because that’s the only thing I haven’t figured out how to compile.

In case you didn’t hear about what happened, this delay was caused by a robbery that occurred on the same day I arrived in Houston, November of last year. I was with my wife and we left for a couple hours for dinner, and when we came back the place we were staying at (Robin’s Nest) had been broken into my backpack was taken, with my laptop, two cameras (with all the pictures we had taken in New York) and two cellphones. We have already replaced everything that was robbed and moved on. Thankfully Robin had insurance, which covered about half of our losses.

That was the bad news. The good news is completely unrelated to all of this: my wife is pregnant, and the babies are due in somewhere between late November/early December. Note I said babies. Yes. Twins. Apparently two girls (last ecography was a little bit to early to say, but the doctor gave us about 90% confidence). We’ve tentatively picked Laura and Rafaela for her names.

So if you’re as happy as we are about the news and want to show some gratitude, we’ve set up a Baby Registry at Babies’R'Us, and the shipping address is set to Enfold System’s office in Houston. I will be visiting my old friends there around September 10, on my way back from another Landscape sprint in Montreal to pick up the gifts and to say Hi.

Thanks everyone for your patience!

Over the last year (wow, time flies by!) I’ve been working at Canonical, as part of the Landscape team. This is a very diverse team with lots of different skills, and somehow I found myself naturally gravitating towards working more closely on frontend-related issues, of which I could highlight writing YUI3 widgets, speeding up page loading experience and creating a nice testing infrastructure. There’s a ton of things I could write about that, and I really plan to. But today’s entry will be pretty short.

As part of a brain-break task I fixed some of our Javascript tests today so that they would run on Google Chrome. We haven’t been targeting Chrome so far, but that might change soon, driven by Google Analytics stats of people using Landscape.

But, the thing that really caught my attention was the difference in speed between Chrome and Firefox.

For comparison:

Google Chrome 5.0.307.7 beta

$ BROWSER=google-chrome ./bin/test -1vpc --layer=JsTestDriverLayer
Running tests at level 1
Running canonical.testing.javascript.JsTestDriverLayer tests:
Set up canonical.testing.javascript.JsTestDriverLayer in 1.020 seconds.
Running:

Ran 318 tests with 0 failures and 0 errors in 9.545 seconds.
Tearing down left over layers:
Tear down canonical.testing.javascript.JsTestDriverLayer in 0.366 seconds.

Firefox 3.6.3pre

$ BROWSER=firefox ./bin/test -1vpc --layer=JsTestDriverLayer
Running tests at level 1
Running canonical.testing.javascript.JsTestDriverLayer tests:
  Set up canonical.testing.javascript.JsTestDriverLayer in 1.014 seconds.
  Running:
                                                                                                                             
  Ran 318 tests with 0 failures and 0 errors in 15.032 seconds.
Tearing down left over layers:
  Tear down canonical.testing.javascript.JsTestDriverLayer in 0.349 seconds.

Firefox 3.7a3pre

$ BROWSER=firefox-3.7 ./bin/test -1vpc --layer=JsTestDriverLayer
Running tests at level 1
Running canonical.testing.javascript.JsTestDriverLayer tests:
  Set up canonical.testing.javascript.JsTestDriverLayer in 0.804 seconds.
  Running:
                                                                                                                           
  Ran 318 tests with 0 failures and 0 errors in 13.433 seconds.
Tearing down left over layers:
  Tear down canonical.testing.javascript.JsTestDriverLayer in 0.379 seconds.

Disclaimer: Both instances of Firefox were started with the “-safe-mode” flag, which disables all plugins and extensions. Also, as they say around here at Canonical: NOT A METRIC. But interesting still.

If you look closely at this post you might find some hints about things we’ve been working on and which I hope to write about, in addition to general tips and tricks about page speed optimization from experiences in both Landscape and Launchpad.