If you are going to LISA11 in Boston next week, we should definitely meetup. Contact me on twitter or email.

Also my colleagues from the LISA11 blogging team (Ben, Rikki and Matt) have done some very interesting interviews with some key people from LISA11 to get you prepared for the event. Check out the USENIX blog for more from us in the next week.

Here is also a quick intro of our team: “LISA11 Next Week – Meet your blog team!“

Luckily, Jordan Sissel has built a tool called FPM (Effing Package Management), exactly for this: to ease the pain of building new packages; packages that you will use for your own infrastructure and you want them customized based on your own needs; and you don’t care about upstream rules and standards and other limitations when building such packages. This can be very useful for people deploying their own applications as rpms (or debs) and can simplify a lot of the process of building those packages.

FPM can be easily installed on your build system using rubygems:
gem install fpm

Once installed you can use fpm to build packages (targets):

• deb
• rpm
• solaris

from any of the following sources:

• directory (of compiled source of some application)
• gem
• python eggs
• rpm
• node npm packages

Use the command line help (fpm –help) or the wiki to see full details on how to use it. I’ll show some simple examples on how to build some packages from various input sources that I’ve found useful myself.

1. Package a directory – output of a ‘make install’ command

This is how you would usually package an application that you would install with:
./configure; make; make install
For example, here is how you can create an rpm of the latest version of memcached:
wget http://memcached.googlecode.com/files/memcached-1.4.7.tar.gz
tar -zxvf memcached-1.4.7.tar.gz
cd memcached-1.4.7
./configure --prefix=/usr
make
so far everything looks like a normal manual installation (that would be followed by make install). Still we will now install it in a separate folder so we can capture the output:
mkdir /tmp/installdir
make install DESTDIR=/tmp/installdir
and finally using fpm to create the rpm package:
fpm -s dir -t rpm -n memcached -v 1.4.7 -C /tmp/installdir
where -s is the input source type (directory), -t is the type of package (rpm), -n in the name of the package and -v is the version; -C is the directory where fpm will look for the files.
Note: you might need to install various libraries to build your package; for ex. in this case I had to install libevent-dev.

If you are packaging your own application you can do this just by pointing to your build folder and set the version of the app. Here is an example for an deb package:
fpm -s dir -t deb -n myapp -v 0.0.1 -C /build/myapp/0.0.1/

There are various other parameters that you can use but basically this is how simple it is to build a package from a directory.
Here is an example on how to define some dependencies on the package you are building (using -d; repeat it as many times as needed):
fpm -s dir -t deb -n memcached -v 1.4.7 -C /tmp/installdir \
-d "libstdc++6 (>= 4.4.5)" \
-d "libevent-1.4-2 (>= 1.4.13)"

2. Ruby gems or python egg – converted to packages

You can create a deb or rpm from a gem very simple with fpm:
fpm -s gem -t deb <gem_name>
this will download the gem and create a package named rubygem-<gem_name>
For example:
fpm -s gem -t deb fpm
will create a debian package for fpm: rubygem-fpm_0.3.7_all.deb

You can inspect it with dpkg –info and you can notice that in this case it will fill nicely all the fields with the maintainer, and dependencies on various other gems. Very cool.

If you use python and want to package various python eggs this will work exactly the same and you will use -s python (it will download the python packages with easy_install first).

Overall FPM is a great tool and can help you simplify the way you are building your own packages. Check it out and let me know what you think and if you found it useful. And if you found this useful don’t forget to thank Jordan for his great work on this awesome tool.

So if you have an idea for a chef cookbook, now it’s the time to start working on it. I’m offering my help for free for all my blog readers: I will help you write a cookbook by implementing your ideas; help reviewing it or suggest improvements, or whatever else you might need help with. Use the contact form to email me (or DM me on twitter) and let me know how I can help.

If you don’t have time to write a new cookbook but you have a great idea for a cookbook that is missing from the opscode community site, please post it bellow in the comments section and I’m sure some of my blog readers will help create it.

Again this is a brilliant idea from Opscode and it creates a win-win situation for everyone. I’m just curious, is this the first idea from their new community manager? If this is the case, great job Jesse .

So let’s see how we can use veewee. I’m assuming you already have vagrant installed (and virtualbox), but if you don’t please install them first. To install veewee we just have to install the veewee gem:
gem install veewee
once you installed veewee you can see a new task added to vagrant: basebox.

Here is the list of the templates we get out of the box once we install veewee:
vagrant basebox templates
The following templates are available:
vagrant basebox define '' 'archlinux-i686'
vagrant basebox define '' 'CentOS-4.8-i386'
vagrant basebox define '' 'CentOS-5.6-i386'
vagrant basebox define '' 'CentOS-5.6-i386-netboot'
vagrant basebox define '' 'Debian-6.0.1a-amd64-netboot'
vagrant basebox define '' 'Debian-6.0.1a-i386-netboot'
vagrant basebox define '' 'Fedora-14-amd64'
vagrant basebox define '' 'Fedora-14-amd64-netboot'
vagrant basebox define '' 'Fedora-14-i386'
vagrant basebox define '' 'Fedora-14-i386-netboot'
vagrant basebox define '' 'freebsd-8.2-experimental'
vagrant basebox define '' 'freebsd-8.2-pcbsd-i386'
vagrant basebox define '' 'freebsd-8.2-pcbsd-i386-netboot'
vagrant basebox define '' 'gentoo-latest-i386-experimental'
vagrant basebox define '' 'opensuse-11.4-i386-experimental'
vagrant basebox define '' 'solaris-11-express-i386'
vagrant basebox define '' 'Sysrescuecd-2.0.0-experimental'
vagrant basebox define '' 'ubuntu-10.04.2-amd64-netboot'
vagrant basebox define '' 'ubuntu-10.04.2-server-amd64'
vagrant basebox define '' 'ubuntu-10.04.2-server-i386'
vagrant basebox define '' 'ubuntu-10.04.2-server-i386-netboot'
vagrant basebox define '' 'ubuntu-10.10-server-amd64'
vagrant basebox define '' 'ubuntu-10.10-server-amd64-netboot'
vagrant basebox define '' 'ubuntu-10.10-server-i386'
vagrant basebox define '' 'ubuntu-10.10-server-i386-netboot'
vagrant basebox define '' 'ubuntu-11.04-server-amd64'
vagrant basebox define '' 'ubuntu-11.04-server-i386'
vagrant basebox define '' 'windows-2008R2-amd64-experimental'

This means that we can build a box based on any of the above templates. That’s awesome! Let’s say we want to build a debian squeeze box using veewee; we would have to run:
vagrant basebox define 'debian-60' 'Debian-6.0.1a-amd64-netboot'
and this will create a folder definitions/debian-60 with the following files (the content of the veewee template):
definition.rb
postinstall.sh
preseed.cfg
we can modify/tune any of those files based on our custom needs. The file definition.rb is the main definition of the template. Here you would define the memory size, disk size, iso file, etc. The content is very easy to understand, but you would normally not have to change many things here. preseed.cfg is just a standard preseed file where you would customize the actual install process (you could change here the partitions or their type, timezone setup, etc). And finally postinstall.sh that is a bash script that will run at the end of the installation process and it will install ruby, gems , chef and puppet and also the virtualbox guest additions (needed for shared folders).

If you have the iso already place it in ‘currentdir’/iso. If not, veewee will download it and place it in the appropriate folder before starting the install process:
vagrant basebox build 'debian-60'
this will start the installation and you can see all the steps it takes (the keystrokes as they are entered, etc.). This can take a while… Once it is done you can validate the build with:
vagrant basebox validate 'debian-60'
(this will run a few basic tests to see if it can connect to the vm as user vagrant, if chef and puppet were installed, if the shared folders are accessible, etc).

And finally you can export it as a vagrant box with:
vagrant basebox export 'debian-60'
and add it to vagrant:
vagrant box add 'debian-60' debian-60.box
and now you can use it in vagrant with:
vagrant init 'debian-60'

That’s it. Very simple and now we have our own box built from scratch. As a side note, I found this very useful to test and troubleshoot preseed configurations . As you can see there are plenty of templates available in veewee but if you create a new one please consider to share it with others and send it to Patrick on github. I’m sure he will be happy to include it in newer versions of veewee. And if you found this useful don’t forget to thank Patrick for his great work on this awesome tool.

@LSPEMeetup made available the video on justin.tv; unfortunately the quality of the video/sound is not the best; you can find it here.

vmbuilder kvm ubuntu --suite=lucid --flavour=virtual --arch=amd64 --mirror=http://en.archive.ubuntu.com/ubuntu -o --libvirt=qemu:///system --ip=10.0.0.11 --gw=10.0.0.1 --part=vmbuilder.partition --templates=mytemplates --user=username --pass=password --firstboot=/var/vms/vm1/boot.sh --mem=1024 --hostname=myhost --bridge=br0

Now even if we haven’t tuned anything I would have expected it to perform at least the same level or even better compared with a Xen instance. Still, this was not the case, and the performance was really horrible and any kind of IO bound tasks would effectively lock the instance. Looking into this and trying to understand what was the problem I was able to isolate this issue happening only on instances that had ext4 as the filesystem (the default for lucid), but strangely enough this didn’t happen for an older instance that was build with ext3 (actually a debian lenny instance). All the images build with the above command will use qcow2 sparse format as the default format for the disk.

In order to achieve good IO performance we had to use cache=’writeback’ for the instances and this will significantly increase the IO performance and bring it almost to host level performance, but in anycase much better compared with the old xen instances we had. Here is how you can enable writeback for an instance: stop the vm; edit the guestdomain and add cache=writeback in the driver section, save and start back the vm:
virsh --connect qemu:///system
stop guestdomain
edit guestdomain   <-- add cache='writeback' in the driver section
start guestdomain

Here is the how the disk part of my guest domain looks like after adding the cache writeback:
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2' cache='writeback'/>
<source file='/var/vms/vm2/ubuntu-kvm/tmphAUcOB.qcow2'/>
<target dev='hda' bus='ide'/>
</disk>

In the process of debugging and searching for a fix for this issue, I’ve found out that it can also be useful to use elevator=noop as the default kernel io scheduler; this definitely helps, but not to the same extend as the cache writeback setting on the virtio disk. You can add elevator=noop to your kernel command line in your grub config, and I have this by default on all the instances.

Hopefully this will help you greatly improve IO performance for your KVM guests and will save you the time I’ve lost while trying to find a solution to this problem. Please feel free to share your experiences using the comment form bellow; also I’m curious if you have any other tips on how to improve this even more.

1. Stop all the chef related services

Here is a handy command that will stop all the possible chef server related services:
for svc in server server-webui solr expander
do
sudo /etc/init.d/chef-${svc} stop
done

2. Upgrade the chef-server gems

Simply run:
sudo gem update chef chef-server --no-ri --no-rdoc
and this should upgrade all the other gems it needs to. A sample output will look like this:
gem update chef chef-server --no-ri --no-rdoc
Updating installed gems
Updating chef
Successfully installed chef-0.10.2
Updating chef-expander
Successfully installed chef-expander-0.10.2
Updating chef-server
Successfully installed chef-server-api-0.10.2
Successfully installed chef-server-webui-0.10.2
Successfully installed chef-solr-0.10.2
Successfully installed chef-server-0.10.2
Gems updated: chef, chef-expander, chef-server-api, chef-server-webui, chef-solr, chef-server

Optional: if you want you can cleanup the system from old, unused gems with:
sudo gem cleanup

3. Start back the chef server services

Again in a single command, now to start them:
for svc in server server-webui solr expander
do
sudo /etc/init.d/chef-${svc} start
done

That’s it, now you should be running the latest and greatest chef server version 0.10.2.

• first of all it is a free event (compared with a regular O’Reilly conference where prices usually start at $1k).
• it is much more interactive: while Velocity is a classic conference where you normally have a presenter showing off something (hopefully not selling or hiring), and maybe some questions at the end, DevOpsDays is more like an open discussion, with people either on a panel or open spaces.
• the food was way better at DevOpsDays, no question about it. And the ice cream on Saturday added an extra special touch .

The first day, Friday, started with the “Devops State of the Union” by John Willis. This was a very good introduction on what DevOps means and a look back on what happened during the past couple of years, especially considering the fact that many people where there for the first time. For example I met someone from Microsoft that was sent here to find out “what is this devops thing” and how they can use it, and this just shows what a huge progress the devops movement has made in such a short amount of time, and how many people are now interested in the movement. (in this particular case I’m not sure he returned at Microsoft with something useful, but just the fact that they are interested in this demonstrates my point).

Next, we had some very interesting panels (4-5 people in general) like: “To Package or not to Package“, “Orchestration at Scale“, “DevOps Metrics and Measurement“, “DevOps..Where’s the QA?” and finally “Escaping the DevOps Echo Chamber“. Even though I believe some moderators could have done a better job (not leaving for 10-15mins people standing/waiting to ask a question) I believe this is a great format, very informal and interactive promoting an open discussion and people sharing their experiences. We had also some great ignite presentations and by far the most interesting and unexpected one was David Lutz with his DevOps song.

The second day was in the format of an unconference, with several open spaces and some short presentations around lunch time. Many people left as they probably wanted to spend the weekend home with their families, but many stayed for Saturday also (about half). From the sessions I attended, I really enjoyed a lot the one about Kanban; very useful to see how others used it in operations teams and what problems they had implementing it. I enjoyed also Patrick’s presentation about vagrant (I’m playing already with veewee). Also was very proud of Nate and Rich releasing their product Reactor8 with this occasion.

Overall I think this was an awesome event. Very much improved compared with last year: two days compared with only one, and I liked a lot the format (day 1 panels & day 2 open spaces). Personally I will probably skip Velocity next year (unless I have a talk accepted) and stick only with DevOpsDays. If you are in the area there is no reason to miss DevOpsDays and I would highly recommend it, or any of the other DevOpsDays events close to your area.

ps: They recorded all the event (very professional, with multiple angles, etc.) and the content will probably come up online very soon, and once that happens I will link it here also.

First impression was that this was much bigger than last year, and the numbers just released by O’Reilly demonstrate it was almost double with about 2,000 participants and another sell out (compared with 1,200 in 2010). Considering the high price of the conference (~1k) this is very impressive and interesting to see how they will host next year even more people. From a question a speaker had many people where at their first Velocity, but even so I met many friends and people I met last year. From the content point of view there was a lot of focus on mobile performance that I haven’t seen last year. Myself I mostly followed the operations track, and only a few talks in the performance track.

Tuesday was the “workshop” day, and I feel that they should change that name as none of the workshops I attended (OpenStack, Chef, Postmortem, etc.) were nothing else that talks (some quite interesting ones) that were maybe longer than a usual conference talk, but not workshops. This is confusing and people I spoke didn’t really liked it either. Not even the Chef workshop that could have had a practical exercise (and even if they sent instructions for people to configure their laptops to prepare for this), had nothing for the people in the audience to test and get their hands dirty. Even if this was not a real workshop either the talk by John Allspaw: “Advanced Postmortem Fu and Human Error 101” (slides) was the best of the conference in my opinion and I highly recommend it to anyone interested in the topic (if of course O’Reilly will publish the video). The day ended with a very good Ignite session (including a karaoke part that was quite entertaining).

Wednesday and Thursday were the main days of the conference with even more people arriving in Santa Clara. I’ve met people that traveled from all over US, but also from other parts of the world (Europe in general). Each day started with plenaries and sponsored talks in the morning until lunch. Personally I would cut down on these plenaries and sponsored talks, or at least give people some alternatives; even allowing for BoFs or any other talks would be great. From the actual presentations, I liked Mark Imbriaco talk “Building for the Cloud: Lessons Learned at Heroku“, Patrick Debois and Andrew Shafer “Measuring the devops gap” (slides).
In general all the talks I attended were very good. I had high hopes for Adam Jacob’s “Choose Your Own Adventure 2” (slides) but this was not even close as funny as the one from last year (not sure why; maybe the room was too big, but also the content was not quite so funny).

On another note, Velocity Europe was announced to be happening sometimes in November in Berlin. Also Jesse Robbins passed the co-chair duties he had for the last 4 years to John Allspaw.

Overall, I would say that Velocity is still the conference to go for anyone serious about operations (even though there are some other good options appearing: DevOpsDays, Surge). Velocity is a great, high quality conference that gathers the best people in operations and web performance field. Still for me, I will probably follow next year only the ‘hall/bar track’ as that is what I found to be the most valuable; and of course DevOpsDays that followed Friday and Saturday, is something I wouldn’t miss for anything.