My time at Canonical could only be described as excellent, and with the kids just past 3 years old, personal life has been getting back on track again. We moved to a larger apartment mid-2012. Kids started going to day care in early 2013, which gave me some calm afternoons to focus on work. We finished paying our car. Things suddenly got… too comfortable.

The loss of Dorneles, just short of three years ago, affected me in ways that were not fully apparent. I became more introspective and spent way too much time thinking about how short our life truly is, and how our plans, if not executed, end up being just that… plans. Not too long before his accident, Dorneles was talking about moving to Norway with his family, a plan that never had the chance to be executed.

For many years I, too, planned to move abroad, but never had the courage to do so. And then other things go in the way. Finishing university, having kids, Daiane being too attached to her parents, and so on.

Suddenly, an opportunity popped up. A chance to reset everything. Start a new life, in a new country, with a new job. It seemed too good to be true. And indeed, we spent countless nights debating the pros and cons to no avail. Nothing could tip the scale in either direction.

Truth is, more than not having a good reason to take the opportunity, we lacked a really good reason not to take the opportunity, other than fear. Mighty, paralyzing fear. We went over the pros once more: kids are the right age, our parents are still young and can care for themselves for a few years, if everything goes wrong we still have a home we can come back to, and so on. And then we thought of Dorneles again, and that tipped the scale big time.

Since last Tuesday, we’re now living in Zürich. As of next Monday, I’ll be one more piece in the well oiled Chocolate Factory.

As someone that’s quite adverse to change, a small town boy that always lived in the same city, I still get shivers down my spine every time I think about the effects of this decision on my own life, and the life of my kids and relatives. There’s really no way to know if it was the right decision or not, but my wife keeps telling me that it is much better to try and fail, than not try and regret.

So there it is: life, reset. A clean slate. Let’s make this a beautiful and memorable journey.

Subject: You have been added to canonical

Hello Sidnei da Silva,

James Troup (elmo) added you as a member of Canonical (canonical).
<https://launchpad.net/~canonical>

Regards,
The Launchpad team

In looking at the various storage backends supported by LXC and experimenting with each of them, I’ve stumbled on various issues, from broken inotify support in overlayfs to random timing issues deleting btrfs snapshots. Eventually I’ve discovered the holy grail of LVM Thin Provisioning and started working on a patch to LXC which would allow it to create volumes in a thin provisioned pool. In the meantime, Docker announced their intent of adding support for LVM Thin Provisioning too. I’m happy to announce that the work I started in August is now merged into LXC as various separate pull requests (#67, #70, #72, #74) and is already available in the LXC Daily PPA. I’ve even created a Gist showing how to use it.

As pointed out by a colleague today, Thin Provisioning support should soon land in Docker itself. I applaud the initiative, and am really looking forward to seeing this feature land in Docker. I wish though there was a little more coordination going on to get that work upstreamed into LXC instead, to the benefit of everyone. Regardless, I’m committed to make sure that the differences between whatever ends up landing in Docker and what I’ve added to LXC eventually converge. One such difference is that I’ve simply shelled out to the ‘lvs’ command while Alexander is using libdevmapper directly, something I’ve looked at doing but felt a little over my head. I’ve also haven’t got around to making the filesystem default to ext4 with DISCARD support yet, but that’s on the top of my list.

So without much ado, let’s look at an example with a loopback-mounted device backed by a sparse-allocated file.

$ sudo fallocate -l 6G /tmp/lxc
$ sudo losetup -f /tmp/lxc
$ sudo pvcreate /dev/loop0
$ sudo vgcreate lxc /dev/loop0

$ sudo lvcreate -l 95%FREE --type thin-pool --thinpool lxc lxc
$ sudo lvs
LV VG Attr LSize Pool Origin Data% Move Log Copy% Convert
lxc lxc twi-a-tz- 5.70g 0.00

$ sudo lxc-create -n precise-template -B lvm --fssize 8G -t ubuntu-cloud -- -r precise
$ sudo lvs
LV VG Attr LSize Pool Origin Data% Move Log Copy% Convert
lxc lxc twi-a-tz- 5.70g 17.38
precise-template lxc Vwi-a-tz- 7.81g lxc 12.67

$ time sudo lxc-clone -s precise-template precise-copy
real 0m0.276s
user 0m0.021s
sys 0m0.085s

Plenty fast I say. What does the newly created volume looks like?

$ sudo lvs
LV VG Attr LSize Pool Origin Data% Move Log Copy% Convert
lxc lxc twi-a-tz- 5.70g 17.44
precise-copy lxc Vwi-a-tz- 7.81g lxc precise-template 12.71
precise-template lxc Vwi-a-tz- 7.81g lxc 12.67

Not only cloning the container was super fast, but it also barely used any new data in the ‘lxc’ thin pool.

The result of this work should soon make it’s way into Juju and, I hope, Docker, either via Alexander’s work or some other way. If you’re into Docker or a more hardcore LXC user, I hope you enjoy this feature and make good use of it.

Very high on my list of things to fix I have an item to look into Vaurien, and make some tweaks to the service to cope better with outages in other parts of the system.

As you probably realized by now if you’re somehow involved into the design and maintenance of any kind of distributed systems, network partitions are a big deal, and we’ve had some of those affect our service in the past, some with very interesting effects.

Take our application servers for example. They’ve been through many generations of rewrites, and switches from one WSGI server to another in the past (not long before I joined the team), each of them with a particular issue. Either they didn’t scale, or crashed constantly, or had memory leaks. Or maybe none of those (it was before I joined the team, so I wouldn’t know for sure). By the time I joined, Paste, of all the things, was one of the WSGI servers in use in part of the service, and Twisted WSGI was used in another part.

(The actual setup of those services is very interesting on itself. It’s a mix of Twisted and Django (and many others have done this before, so it’s not very unique. But there are internal details which are quite interesting. More below.)

Having moved from another team that used Twisted heavily, I decided to call it out and settle on Twisted WSGI, which seemed just fine.

As for the stability and memory issues, we started ironing them out one by one. Turns out the majority of the problems had nothing to do with the WSGI server itself, but everything to do with not cleaning up resources correctly, be it temporary files, file descriptors, and cycles between producers and consumers.

And everything was perfect.

But then we got a few networking issues and hardware issues. Some of the servers were eventually moved to a different datacenter and things got even more interesting. I’ll go into the details of the specific problems that I’m hoping to approach with Vaurien on a different post, but suffice to say that talking to many external services in a threaded server doesn’t get pretty when there’s a network blip.

So speaking of threaded and Twisted, and coming to the subject of this post.

In front of a subset of our services we currently have 4 HAProxy instances in different servers. They are all set up to use httpchk every 2 seconds, which by default sends an OPTIONS request to ‘/’. If you’re still following, we have a Django app running there, and depending on how you have your Django app configured, it might just take that OPTIONS request and treat it just like a GET, effectively (in our case) rendering a response just as if a normal browser had requested it. Turns out that page is not particularly lean in our case.

So you take 4 servers, effectively doing a GET request to your homepage every 2s each one, times many processes serving that page across a couple hosts, and you have a full plate for someone looking for things to optimize.

To make it more fun, early on I added monitoring of the thread pool used by Twisted WSGI, sending metrics to Graphite. Whenever we had a network blip we saw the queue growing and growing without bound. This was actually a combination of a couple things, which I’m still working on fixing:

1. HAProxy will keep triggering the httpchk after the service is taken out of rotation.
2. Twisted WSGI will keep accepting requests and throwing them in the thread pool queue, even if the thread pool is busy and the queue is building up.
3. We do a terrible job at timing out connections to external services currently so a minor blip can easily cause the thread pool queue to build up.

As a strategy to alleviate that problem I came up with the following solution:

1. Implement a custom thread pool that triggers a callback when going from busy -> free and from free -> busy (where busy is defined as: there are more requests queued than idle threads).
2. Changed the response to the HAProxy httpchk to simply check that busy/free state.
3. Changed the handling of that HAProxy check to *not* go through the thread pool.

(There’s a few more details that I won’t get into in this post, but that’s the high-level summary.)

I have good confidence that this will fix (or at least alleviate) the main issue, which is the queue growing without bounds in the thread pool, and it will instead move the queueing to HAProxy. But after looking through the metrics today I saw an unintended consequence of the changes.

In retrospect, it seems fairly obvious that this was to be one of the expected outcomes. I was simply surprised to see it since it was not the immediate goal of the proposed changes, but simply a side effect of them.

I hope you enjoyed this glimpse into what goes on at the heart of my job. I expect to write more about this soon, and maybe explore some of the details that I didn’t get into, since this post is already too long.

What I want to share with you today is the joy of being able to connect with old friends and recollect memories, as I mentioned I was longing for in my last post. In those few days I was in California I managed to catch up with Limi and Philipp, said an en passant hi to Rob Miller at the Mozilla SF office, had dinner with Gustavo, walked around the city with Fernando, Alberto, and Geoff, ending up at an amazing Chinese restaurant pretty much by accident, paid a visit to Marlon, who took me on a guided tour of the Facebook HQ followed by lunch at The Cheesecake Factory which I couldn’t refuse. It was exausting, but really great catching up with everyone!

A recurring topic between all of us was the general issues that all of our companies (Mozilla, Canonical, Facebook) have with general public perception. Most interestingly perhaps is the similarity between Canonical and Facebook when it comes down to privacy matters, how there seems to be a disconnect between the internal and external messaging on those matters, and how much the public perception is biased by the media and the very loud minority of privacy tinfoil hat zealots. I wish I could do more to help with solving that. Perhaps pushing for more transparency, better communication at least from the technical side of things could be a way to improve that.

Tech talks aside, I was simply overwhelmed by how much my kids’ pictures and videos are popular amongst friends. Every single person that I talked to was quick to mention that as the very first thing. Oddly, that generally does not reflect in likes and comments on those Facebook posts, which is an interesting observation. Are people generally afraid of clicking that Like link or is it too much effort for them? I’m sure it would do for a great usability study.

I hope to explore a bit more on the outcome of the sprint on a later post. Suffice to say that I was really glad to be present and contribute some feedback to all the planning that’s going into the next cycle, and the opportunity to meet some old friends while at it was invaluable. Looking forward to be doing more of that in the coming months, at FISL and PythonBrasil.

As an article I’ve read yesterday mentioned, we tech heads seem to live on a bubble that mostly bounces between social networks and having post work hours drinks with colleagues, usually from the same company. I wish we could all be more social in the physical world, and talk more about things that are not so tech-related. About life, and family, and non-work things, and enjoy ourselves more.

And headed straight into the shining sun.

I’ve been carrying on with life. Heads down into work. Enjoying my kids, which just started at daycare (or is afternooncare a word, even?) this week. But it’s going to hurt for quite some time. I can tell.

I feel like I should be writing more. I feel the need to hang out with old friends again. I might end up going to this year’s PythonBrasil/PloneConf one-two combo just for that. Hopefully some of the old-timers like Alan and Limi will be there. It would sure be great to see Paul again, which is apparently confirmed for one of the keynotes.

I actually started this post feeling very nostalgic, and the paragraph above shows. This was after stumbling on Gustavo’s 12 years ago post. Then I looked at the archive of this blog and found that my oldest post here is from July 20th 2003. Reading forward from there I’ve revived some really happy memories, and tons of now-broken links. One of the broken links was to one of the oldest traces of me out there, asking for help with Including TCL scripts in the CSLU Toolkit, back in 1999. Here’s another, from 2005, where I mention meeting Gustavo in person for the first time. And, the irony. I linked to his LiveJournal blog. :)

I feel much better now.

I should be writing more, and soon. I have lots of things to share, both personally and professionally. I really wish to become more like Dorneles. Not only fun to work with, but also fun to be around. I want to share more, not only about the things that I do, but also the things that I am. I hope to become a better person in the process, and I hope you can share and enjoy that journey with me.

The realization of that had a profound impact in my life. As programmers, technically minded people that we are, we tend to focus on ourselves and rarely share our knowledge, except to show it off at conferences here and there. We do it mostly to please our ego. Dorneles was nothing like that. He didn’t present at many conferences, though he was always there as either a participant or an organizer. He didn’t write lengthy blog posts full of technical details. In fact, he didn’t blog at all since 2007. He didn’t create brilliant frameworks. Instead, his contributions were mostly invisible, through IRC, IM or Skype, directly between him and the person seeking his help. It was very common for him to carry about five parallel conversations over IM, helping people all over the world without breaking a sweat. He would happily pull out his laptop or phone at a table in a blink to teach someone a trick. The endless stream of messages on twitter following his death should serve as proof that his way was much more effective in causing an impact in the world than a horde of highly skilled programmers ever could.

As other people well put it, Dorneles could never lose that smile on his face. Nothing could make him curse or complain. He was a mixture of cheerfulness, innocence, tranquility and awkwardness that would immediately conquer people’s hearts with no effort. He was like that young little brother that makes you blush at a family dinner for being excessively goofy, but that you love so much you would never reprehend.

To me and my wife, Dorneles was closer than family. My wife suffered more from his loss than for the loss of her lovely grandma, or from her uncle that died a slow death from brain cancer a couple years ago.

In retrospect, I can now see that he did treat me like an older brother, always eager to take on my advice. When he was alone in a big empty office in Caxias do Sul, following the break with the other partners at X3ng I told him to move to Garibaldi because we could rent a similar office for much cheaper. I suggested that he should reunite with his wife and two kids, which were living with her mom, and that raising them in Garibaldi would be much better for their education and overall quality of life. I said it would be nice if he had a car so that he didn’t need to call a cab at 3am when his kids were sick (he refused to buy a car for years, since he had cars stolen several times). At a time he was having trouble getting regular pay, I suggested he should work for Enfold. I told him to go to more conferences and sprints, so that he could put faces to the names on #plone. He was easily convinced. And all of these things had a great impact in his life.

I regret not spending as much time with him as I could in the last year. But many are the good memories that we’ve shared. Back in November 2009, when my backpack with laptop and cameras containing all the pictures from my vacation in New York got stolen on our first day in Houston he calmly told me: “Don’t worry, you can buy all those things again. They didn’t take from you the most precious thing: your life.”

I regret that he didn’t find the time to visit my 3mo old twins. But I’m glad that after my visit last Saturday, he went to visit his 2 weeks old nephew and his dad on the Sunday. I’m glad that we spent a good 2h last Saturday sitting around, chatting about the most diverse subjects while I was at his place waiting for the rain to stop.

Dorneles was a sushi lover. He once walked for tens of blocks in Houston on a cracking hot summer day just to eat sushi, and lived to tell the tale. That’s perhaps the most un-American thing he could do. The first thing we did when we met in Houston in November 2009 was to go out for sushi. When Nate came to Brazil, he took him out for sushi, and that’s perhaps the most un-Brazilian thing he could do. His love for sushi passed on to his kids. For several weeks I told my wife that we should invite his family out for sushi. We were waiting for the right occasion, and it never happened. In his memory, I think we should do it anyway.

Today I’m going to share with you one of the tricks that we’ve discovered almost by accident and that can help with making your website faster, if YUI is your Javascript framework of choice.

We’ve started using YUI in Landscape around March of 2009, by rewriting all of our ExtJS-based scripts (which weren’t that many) to use YUI. Since our application requires HTTPS due to potentially sensitive information, we had to self-host YUI, to avoid mixed-content warnings. Initially we had little experience with it, so we went with the easiest option available: loading the YUI seed file and letting everything else be loaded on-demand by the YUI Loader, without a combo handler (more on that later).

Soon enough we noticed that it wasn’t such a great idea. First-time access was terrible, with load times over 30 seconds in some cases, due to the combination of HTTPS connection overhead, no static resource caching and large number of requests to fetch all the resources. It was worse than that even, because most browsers either don’t cache HTTPS content by default, or only cache it on memory, unless you set the proper caching headers. To give you an idea of what it looked like, here’s what a trivial page using the ‘overlay’ module looks like in terms of loading.

No combo loading at all

It was pretty clear that this kind of performance would not be acceptable, so I started looking for options to reduce the number of requests. It was around that time that I heard about Steve Souders work on web performance, through a long time friend from the Plone community that happened to be working at Google and shared a link to Souders’ blog post on ‘@import’.

Turns out that the YUI Loader supports using a combo handler for reducing the number of requests, when loading modules on demand. What this ‘combo handler’ does is basically take a GET request with a bunch of filenames as parameters and return a single response with the respective files concatenated. Here’s an example using the combo server from YUI’s CDN.

Combo loading with a single 'Y.use()'

The problem was that we couldn’t use YUI’s combo server on the CDN because it doesn’t support HTTPS, which was requirement for us.

In November of 2009, at UDS-Lucid in Dallas, we had our first YUI-only cross-team sprint, totalling about 20 people from Launchpad, Landscape, ISD and Ubuntu One. The main goals were to spread knowledge on YUI and improve support for IE on our shared codebase (lazr-js), but I managed to sneak in two things for the Landscape team: implementing a Python-based combo loader (served by Twisted, but it’s just a plain WSGI app) and extracting YUI module metadata, so that we could make it easier to load extension modules on-demand, the code for which ended up as part of the lazr-js project.

At this point however we realized that there’s a significant issue with using the combo loader on an application as diverse as Launchpad or Landscape: depending on how you group your ‘Y.use()’ calls, the combo URL generated by the YUI Loader can vary significantly, which means that re-use from the browser cache is less than optimal. Here’s an example of loading both the ‘anim-color’ module and the ‘overlay’ module within a page. Compare it to the previous example and notice how the parameters passed to the combo are completely different, since the second request (to load the ‘overlay’ module) only requests modules which haven’t been loaded by the first request (to load the ‘anim-color’ module).

Combo loader can be a caching problem

In a hurry to get the situation solved ASAP, and taking a clue from our fellow developers on the Launchpad team, we went with a simpler alternative: manually combining all of YUI, including the seed file, and loading all of that in a ‘script’ tag in the document ‘head’. It turns out that the situation improved quite dramatically, but there was still more to be done. Although the number of requests was much smaller, the total page size was actually larger, since we were loading a lot of stuff that wasn’t even being used. Worse, since we were putting that ‘script’ tag in the ‘head’, it was blocking the whole page from loading until the script was finished downloading, which I after following Souders’ blog for a while I realized wasn’t such a great idea either. Here’s an example of what that looked like.

Combined modules in the head

At this point we had a significant codebase of YUI-based modules, and so did the Launchpad team, but we had gone from solely relying on the YUI Loader to not using it at all (by just loading all the code in one big honking file).

Once we got the module metadata extraction figured out though, I resumed working on improving the first-view load time, by reducing the number of modules manually combined to the bare minimum shared by all of our pages. Also, by having a single file with the common modules being loaded on all the pages, we can be sure that it is highly cacheable. And then we let the YUI Loader kick in and load the missing modules on less-used pages, which was a really nice improvement in page weight and reducing the time to the ‘onload’ event.

At this point Souders started talking about asynchronous loading of scripts and the ‘defer’ attribute, and I got my copy of Even Faster Web Sites. I started pondering if there was something that could be done to defer the loading of the combined modules while still letting the YUI Loader do it’s job.

The problem lies in the fact that YUI asynchronously loads modules that are not on the page yet, calling a callback function (the last argument to ‘Y.use()’) once all the scripts containing the required modules have finished loading. If all the modules are already on the page, it just calls the callback right away. So by just tacking a ‘defer’ attribute on the ‘script’ tag for the combined modules there would be a race condition between the loader checking if the modules were loaded and the script with the combined modules being loaded.

Then, it struck me. If the last argument to ‘Y.use()’ is just a callback function, could we queue those functions to be called at a later time and load the combined modules ourselves, before letting the YUI loader proceed? After a little back and forth with Dav Glass and some pseudocode exchange, I got an implementation of this idea(which I’m calling the ‘Prefetch YUI Loader Hack‘). And thus we managed to move the loading of all the combined modules from a ‘script’ in the ‘head’ to use asynchronous script loading, shortening even further our time to the ‘onload’ event. Here’s an example of what it looks like.

Prefetch with a single request

But wait, there’s more! Souders then started talking about parallel loading, tickling my speed bug again. After thinking for a while about the problem, enlightenment came again: when an YUI module is loaded, it is not executed right away, but added to an internal registry of modules. What that means is that regardless of what other modules it depends on, they can all be loaded independently from each other! That means you can break up a manually combined file into N smaller files and have them be loaded in up to N parallel connections (where N varies by browser and by browser version) and when they are all done you can let the YUI Loader kick in. Here’s an example with two parallel downloads plus an extra module not prefetched.

Prefetch, with parallel download

So that’s where things stand now. There are still optimizations that can be done in Landscape, like using the combo loader to reduce the number of requests on less common pages, or even using fixed URLs to the combo loader with the prefetch hack, to simplify the build process. We also need to start using Caridy’s Event Binder module, since now the pages load so fast that the users start clicking around before the event handlers are in place (ha!).

I am also pushing to get those kinds of tips documented and passed around Canonical, through a project codenamed Dare2BFast, which has the goal of coming up with a set of Web Performance Guidelines that all Canonical web sites should follow, both on the frontend and backend. Stay tuned!

I’m sitting here at Canonical’s office in Montreal, where we are having another Landscape sprint. There’s a couple things that make this sprint special, one of them being that Frank Wierzbicki has joined us! Welcome to the team, Frank, and I hope you enjoy!

It has also coincided with the end of our 1.5.5 milestone, which rolled out yesterday, with some interesting highlights:

Speed!

Contrary to what other people might tell you, YES speed does matter. We’ve been slowly (ha!) rolling out improvements to page load speed, and I’m happy to say that the improvements are very noticeable. Things we’ve done to improve this range from setting proper caching headers for static resources to heavily abusing the YUI Loader. I should write more about that on a more appropriate occasion. There’s always things to improve on, so keep an eye on it!

Faster Trials

Over the last four years, the trial registration for Landscape hasn’t been… enjoyable to say the least. This week we’ve finally rolled out the first iteration of our simplified registration process, which removes a manual approval step which used to take several weeks to be done. That means you can now register for a trial account in Landscape and start using it right away! Quite a novel concept, isn’t it? :) We’ll planning some more updates to it, so that when starting the registration process you have a better sense of where you are in the process and what are steps involved.

We have more exciting updates coming up before the end of the year, so keep an eye on it. But in the meantime, hit that ‘Registration’ link in Landscape, get it like you like it, and enjoy!