Even if I have not attended any workshop (they had 2 flavors, one targeted towards a sysadmin workflow and one for developers) the general feeling from people I talked with and attended them was that it was a very good experience, with a lot of hands-on practical examples. Tuesday afternoon, myself I attended the “ChefConf Pre-event Hackday: TEST ALL THE THINGS!!!“ organized by Bryan Berry and it was great, and showed how many people are interested in testing their infrastructure as code; it was focused on cookbook testing (unit testing and integration testing), continuous integration with jenkins, and other things like that

The first full day of ChefConf was Wednesday. The conference was structured with main presentations during the mornings and breakout sessions in the afternoon (with 2 main tracks and also a vendor one). From the beginning you could tell that this will be a very well run conference, and even if this was the first one, people like Jesse Robbins have a lot of experience running such events. Not surprisingly ChefConf kicked off with Adam Jacob‘s “State of the Union Part 1: Chef, Past and Present” (video) ; Jesse Robbins talked about the community around chef and how this is a key part of Opscode strategy and their efforts to take this to the next level. He showed this very nice visualization of the commits to the chef github repo.

There were many interesting talks during the day, and they recorded most of them and hopefully will make them available online soon so you can see them if you didn’t had the chance to be here (or you want to review them again). I particularly enjoyed:

• Ron Vidal – Operations Secret Sauce: Incident Management (video); similar to Jesse Robbins GameDay talk and it was a very nice addition, inspirational and full of interesting points.
• Jim Hopp‘s – Test-driven Development for Chef Practitioners (video); very well prepared and presented. I hope to have Jim to our Chef Bay Area meetup group to present something similar on the subject and run a testing hackaton.
• Patrick McDonnell‘s – Lessons from Etsy: Avoiding Kitchen Nightmares; people seem to love everything Etsy is doing and they are sharing a lot of their workflow with chef and open sourcing various tools they write.
• and many others…

In the evening we had a great Ignite event ran by Andrew Shafer in his unconfundable way. We had 10 ignite speakers and in the middle there was a fun karaoke ignite that had 10 volunteers rambled on some slides they never sow before. If they recorded this, and will show it online look up the ones by Stephen Nelson-Smith and John Vincent as they were very entertaining.

The second day of the conference started with Christopher Brown‘s “State of the Union Part 2: Chef, the Future” where he outlined some of the future features and main focuses of Opscode for Chef: becoming easier to install and use (omnibus installer), enterprise ready, focus on Windows and also a lot of focus on quality. Opscode is working on a project called kitchen chef that will allow to test the functionality of cookbooks on various environments and platforms, and quickly ensure the quality of the cookbook is maintained during various iterations. Also a lot of work has been put into reporting and handlers. The server side also has been completely rewritten in erlang and sql (from ruby and couchdb) and we should see this soon in the open-source and the private chef server. From the work done you can easily tell that a lot of work has been done on private chef and this is quickly becoming an important asset for Opscode going forward.

There were many great talks during the day from speakers like Artur Bergman, Ben Rockwood, Jason Stowe, John Esser, Rob Hirschfeld, Theo Schlossnagle, etc. I finished my day just like I started Tuesday with another event focused on testing: “Test Driven Development Roundtable“, ran by Stephen Nelson-Smith on a panel with Seth Chisamore, Jim Hopp and my friend Rob Berger. They went over the tools people are using these days and what are the things that are still missing and need to be worked on regarding testing.

Overall, I think this was an awesome event and I hope to be able to attend the next one also (hopefully at the same place). My impression is that Opscode is ready to move forward and make the next step and grow the community even bigger: “The revolution will not be televised – it will be coded with chef”.

First we need to identify the file that is causing this issue; and for this we will verify all the packed objects and look for the biggest ones:
git verify-pack -v .git/objects/pack/*.idx | sort -k 3 -n | tail -5
(and grab the revisions with the biggest files). Then find the name of the files in those revisions:
git rev-list --objects --all | grep <revision_id>

Next, remove the file from all revisions:
git filter-branch --index-filter 'git rm --cached --ignore-unmatch <filename>'
rm -rf .git/refs/original/

Edit .git/packed-refs and remove/comment any external pack-refs. Without this the cleanup might not work. I my case I had refs/remotes/origin/master and some others branches.
vim .git/packed-refs

Finally repack and cleanup and remove those objects:
git reflog expire --all --expire-unreachable=0
git repack -A -d
git prune

Hopefully these steps will help you completely remove those un-wanted files from your git history. Let me know if you have any problems after following these simple steps.

Note: if you want to test these steps here is how to quickly create a test repo:
# Make a small repo
mkdir test
cd test
git init
echo hi > there
git add there
git commit -m 'Small repo'
# Add a random 10M binary file
dd if=/dev/urandom of=testme.txt count=10 bs=1M
git add testme.txt
git commit -m 'Add big binary file'
# Remove the 10M binary file
git rm testme.txt
git commit -m 'Remove big binary file'

If you are going to LISA11 in Boston next week, we should definitely meetup. Contact me on twitter or email.

—–

The Limoncelli Test, was a very interesting presentation by Tom Limoncelli based on a blog post he wrote earlier this year. If you haven’t done it already I would strongly recommend to take the test and see how does your sysadmin team rank on “The Limoncelli Test”.

Recovering From Linux Hard Drive Disasters is Theodore Ts’o signature training material on what to do if you have any sort of hard drive failure and covers in depth details on how to recover from such disasters caused by software or hardware failures.

GameDay: Creating Resiliency Through Destruction (slides): I enjoyed very much Jesse Robbins presentation, where he draws parallels between two of his greatest passions: firefighting and operations. Watch the video.

SRE@Google: Thousands of DevOps Since 2004: Tom Limoncelli, describes the technologies and policies that Google uses to do what is (now) called DevOps. Watch the video.

Also my colleagues from the LISA11 blogging team (Ben, Rikki and Matt) have done some very interesting interviews with some key people from LISA11 to get you prepared for the event. Check out the USENIX blog for more from us in the next week.

Here is also a quick intro of our team: “LISA11 Next Week – Meet your blog team!“

Luckily, Jordan Sissel has built a tool called FPM (Effing Package Management), exactly for this: to ease the pain of building new packages; packages that you will use for your own infrastructure and you want them customized based on your own needs; and you don’t care about upstream rules and standards and other limitations when building such packages. This can be very useful for people deploying their own applications as rpms (or debs) and can simplify a lot of the process of building those packages.

FPM can be easily installed on your build system using rubygems:
gem install fpm

Once installed you can use fpm to build packages (targets):

• deb
• rpm
• solaris

from any of the following sources:

• directory (of compiled source of some application)
• gem
• python eggs
• rpm
• node npm packages

Use the command line help (fpm –help) or the wiki to see full details on how to use it. I’ll show some simple examples on how to build some packages from various input sources that I’ve found useful myself.

1. Package a directory – output of a ‘make install’ command

This is how you would usually package an application that you would install with:
./configure; make; make install
For example, here is how you can create an rpm of the latest version of memcached:
wget http://memcached.googlecode.com/files/memcached-1.4.7.tar.gz
tar -zxvf memcached-1.4.7.tar.gz
cd memcached-1.4.7
./configure --prefix=/usr
make
so far everything looks like a normal manual installation (that would be followed by make install). Still we will now install it in a separate folder so we can capture the output:
mkdir /tmp/installdir
make install DESTDIR=/tmp/installdir
and finally using fpm to create the rpm package:
fpm -s dir -t rpm -n memcached -v 1.4.7 -C /tmp/installdir
where -s is the input source type (directory), -t is the type of package (rpm), -n in the name of the package and -v is the version; -C is the directory where fpm will look for the files.
Note: you might need to install various libraries to build your package; for ex. in this case I had to install libevent-dev.

If you are packaging your own application you can do this just by pointing to your build folder and set the version of the app. Here is an example for an deb package:
fpm -s dir -t deb -n myapp -v 0.0.1 -C /build/myapp/0.0.1/

There are various other parameters that you can use but basically this is how simple it is to build a package from a directory.
Here is an example on how to define some dependencies on the package you are building (using -d; repeat it as many times as needed):
fpm -s dir -t deb -n memcached -v 1.4.7 -C /tmp/installdir \
-d "libstdc++6 (>= 4.4.5)" \
-d "libevent-1.4-2 (>= 1.4.13)"

2. Ruby gems or python egg – converted to packages

You can create a deb or rpm from a gem very simple with fpm:
fpm -s gem -t deb <gem_name>
this will download the gem and create a package named rubygem-<gem_name>
For example:
fpm -s gem -t deb fpm
will create a debian package for fpm: rubygem-fpm_0.3.7_all.deb

You can inspect it with dpkg –info and you can notice that in this case it will fill nicely all the fields with the maintainer, and dependencies on various other gems. Very cool.

If you use python and want to package various python eggs this will work exactly the same and you will use -s python (it will download the python packages with easy_install first).

Overall FPM is a great tool and can help you simplify the way you are building your own packages. Check it out and let me know what you think and if you found it useful. And if you found this useful don’t forget to thank Jordan for his great work on this awesome tool.

So if you have an idea for a chef cookbook, now it’s the time to start working on it. I’m offering my help for free for all my blog readers: I will help you write a cookbook by implementing your ideas; help reviewing it or suggest improvements, or whatever else you might need help with. Use the contact form to email me (or DM me on twitter) and let me know how I can help.

If you don’t have time to write a new cookbook but you have a great idea for a cookbook that is missing from the opscode community site, please post it bellow in the comments section and I’m sure some of my blog readers will help create it.

Again this is a brilliant idea from Opscode and it creates a win-win situation for everyone. I’m just curious, is this the first idea from their new community manager? If this is the case, great job Jesse .

So let’s see how we can use veewee. I’m assuming you already have vagrant installed (and virtualbox), but if you don’t please install them first. To install veewee we just have to install the veewee gem:
gem install veewee
once you installed veewee you can see a new task added to vagrant: basebox.

Here is the list of the templates we get out of the box once we install veewee:
vagrant basebox templates
The following templates are available:
vagrant basebox define '' 'archlinux-i686'
vagrant basebox define '' 'CentOS-4.8-i386'
vagrant basebox define '' 'CentOS-5.6-i386'
vagrant basebox define '' 'CentOS-5.6-i386-netboot'
vagrant basebox define '' 'Debian-6.0.1a-amd64-netboot'
vagrant basebox define '' 'Debian-6.0.1a-i386-netboot'
vagrant basebox define '' 'Fedora-14-amd64'
vagrant basebox define '' 'Fedora-14-amd64-netboot'
vagrant basebox define '' 'Fedora-14-i386'
vagrant basebox define '' 'Fedora-14-i386-netboot'
vagrant basebox define '' 'freebsd-8.2-experimental'
vagrant basebox define '' 'freebsd-8.2-pcbsd-i386'
vagrant basebox define '' 'freebsd-8.2-pcbsd-i386-netboot'
vagrant basebox define '' 'gentoo-latest-i386-experimental'
vagrant basebox define '' 'opensuse-11.4-i386-experimental'
vagrant basebox define '' 'solaris-11-express-i386'
vagrant basebox define '' 'Sysrescuecd-2.0.0-experimental'
vagrant basebox define '' 'ubuntu-10.04.2-amd64-netboot'
vagrant basebox define '' 'ubuntu-10.04.2-server-amd64'
vagrant basebox define '' 'ubuntu-10.04.2-server-i386'
vagrant basebox define '' 'ubuntu-10.04.2-server-i386-netboot'
vagrant basebox define '' 'ubuntu-10.10-server-amd64'
vagrant basebox define '' 'ubuntu-10.10-server-amd64-netboot'
vagrant basebox define '' 'ubuntu-10.10-server-i386'
vagrant basebox define '' 'ubuntu-10.10-server-i386-netboot'
vagrant basebox define '' 'ubuntu-11.04-server-amd64'
vagrant basebox define '' 'ubuntu-11.04-server-i386'
vagrant basebox define '' 'windows-2008R2-amd64-experimental'

This means that we can build a box based on any of the above templates. That’s awesome! Let’s say we want to build a debian squeeze box using veewee; we would have to run:
vagrant basebox define 'debian-60' 'Debian-6.0.1a-amd64-netboot'
and this will create a folder definitions/debian-60 with the following files (the content of the veewee template):
definition.rb
postinstall.sh
preseed.cfg
we can modify/tune any of those files based on our custom needs. The file definition.rb is the main definition of the template. Here you would define the memory size, disk size, iso file, etc. The content is very easy to understand, but you would normally not have to change many things here. preseed.cfg is just a standard preseed file where you would customize the actual install process (you could change here the partitions or their type, timezone setup, etc). And finally postinstall.sh that is a bash script that will run at the end of the installation process and it will install ruby, gems , chef and puppet and also the virtualbox guest additions (needed for shared folders).

If you have the iso already place it in ‘currentdir’/iso. If not, veewee will download it and place it in the appropriate folder before starting the install process:
vagrant basebox build 'debian-60'
this will start the installation and you can see all the steps it takes (the keystrokes as they are entered, etc.). This can take a while… Once it is done you can validate the build with:
vagrant basebox validate 'debian-60'
(this will run a few basic tests to see if it can connect to the vm as user vagrant, if chef and puppet were installed, if the shared folders are accessible, etc).

And finally you can export it as a vagrant box with:
vagrant basebox export 'debian-60'
and add it to vagrant:
vagrant box add 'debian-60' debian-60.box
and now you can use it in vagrant with:
vagrant init 'debian-60'

That’s it. Very simple and now we have our own box built from scratch. As a side note, I found this very useful to test and troubleshoot preseed configurations . As you can see there are plenty of templates available in veewee but if you create a new one please consider to share it with others and send it to Patrick on github. I’m sure he will be happy to include it in newer versions of veewee. And if you found this useful don’t forget to thank Patrick for his great work on this awesome tool.

@LSPEMeetup made available the video on justin.tv; unfortunately the quality of the video/sound is not the best; you can find it here.

vmbuilder kvm ubuntu --suite=lucid --flavour=virtual --arch=amd64 --mirror=http://en.archive.ubuntu.com/ubuntu -o --libvirt=qemu:///system --ip=10.0.0.11 --gw=10.0.0.1 --part=vmbuilder.partition --templates=mytemplates --user=username --pass=password --firstboot=/var/vms/vm1/boot.sh --mem=1024 --hostname=myhost --bridge=br0

Now even if we haven’t tuned anything I would have expected it to perform at least the same level or even better compared with a Xen instance. Still, this was not the case, and the performance was really horrible and any kind of IO bound tasks would effectively lock the instance. Looking into this and trying to understand what was the problem I was able to isolate this issue happening only on instances that had ext4 as the filesystem (the default for lucid), but strangely enough this didn’t happen for an older instance that was build with ext3 (actually a debian lenny instance). All the images build with the above command will use qcow2 sparse format as the default format for the disk.

In order to achieve good IO performance we had to use cache=’writeback’ for the instances and this will significantly increase the IO performance and bring it almost to host level performance, but in anycase much better compared with the old xen instances we had. Here is how you can enable writeback for an instance: stop the vm; edit the guestdomain and add cache=writeback in the driver section, save and start back the vm:
virsh --connect qemu:///system
stop guestdomain
edit guestdomain   <-- add cache='writeback' in the driver section
start guestdomain

Here is the how the disk part of my guest domain looks like after adding the cache writeback:
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2' cache='writeback'/>
<source file='/var/vms/vm2/ubuntu-kvm/tmphAUcOB.qcow2'/>
<target dev='hda' bus='ide'/>
</disk>

In the process of debugging and searching for a fix for this issue, I’ve found out that it can also be useful to use elevator=noop as the default kernel io scheduler; this definitely helps, but not to the same extend as the cache writeback setting on the virtio disk. You can add elevator=noop to your kernel command line in your grub config, and I have this by default on all the instances.

Hopefully this will help you greatly improve IO performance for your KVM guests and will save you the time I’ve lost while trying to find a solution to this problem. Please feel free to share your experiences using the comment form bellow; also I’m curious if you have any other tips on how to improve this even more.

1. Stop all the chef related services

Here is a handy command that will stop all the possible chef server related services:
for svc in server server-webui solr expander
do
sudo /etc/init.d/chef-${svc} stop
done

2. Upgrade the chef-server gems

Simply run:
sudo gem update chef chef-server --no-ri --no-rdoc
and this should upgrade all the other gems it needs to. A sample output will look like this:
gem update chef chef-server --no-ri --no-rdoc
Updating installed gems
Updating chef
Successfully installed chef-0.10.2
Updating chef-expander
Successfully installed chef-expander-0.10.2
Updating chef-server
Successfully installed chef-server-api-0.10.2
Successfully installed chef-server-webui-0.10.2
Successfully installed chef-solr-0.10.2
Successfully installed chef-server-0.10.2
Gems updated: chef, chef-expander, chef-server-api, chef-server-webui, chef-solr, chef-server

Optional: if you want you can cleanup the system from old, unused gems with:
sudo gem cleanup

3. Start back the chef server services

Again in a single command, now to start them:
for svc in server server-webui solr expander
do
sudo /etc/init.d/chef-${svc} start
done

That’s it, now you should be running the latest and greatest chef server version 0.10.2.