flame.org - Home

attr_accessible vs accepts_nested_attributes_for

2009-11-10T12:23:00Z

First, I just have to say that I never thought I’d be in Japan, let along blogging from here.

Today I ran into a little gotcha with restricting access to models when trying out the Rails 2.3 method of nested models. In my model, I have:

  has_many :dnskeys, :dependent => true
  accepts_nested_attributes_for :dnskeys,
    :allow_destroy => true,
    :reject_if => proc { |attrs| attrs.all? { |k, v| v.blank? }}

  attr_accessible :name

I was surprised, although I should not have been, that this prevented me from posting a zone and its dnskeys in the same action. Adding the proper attr_accessible line made the magic happen again:

  attr_accessible :dnskeys_attributes

Experiments with HAML

2009-11-05T09:09:00Z

I decided to try HAML today, rather than the standard ERB templates. The jury is still out on if it is better or worse, but like many things Rails, it is better to just dive in and try it.

I have a little project (not yet released) which I’m converting over to use HAML. Since I also use rspec for testing, along with Cucumber and Webrat, I thought I had a long day ahead of me.

It turns out most of it can be made close to automatic, with just a little cleanup afterwards.

Using the rake task below, you can issue these two commands to convert your existing *.html.erb views into *.html.haml. The second target will convert the rspec view tests to expect haml. Renaming the spec tests is not required, but I like sanity.

12	$ rake haml:convert:html $ rake haml:convert:spec

If you use a SCM (I still like Subversion best) you’ll want to remove the old files and add the new ones. For Subversion, I do:

12	$ svn rm app/views//.html.erb spec/views//.html.erb_spec.rb $ svn add app/views//.html.haml spec/views//.html.haml_spec.rb

The main problems were that not all of the templates were properly indented, and “- end” lines appeared in the HAML templates. Fixing this was fast, as I really don’t have much in my templates yet as this is a new project. This step too could probably be done in an automated way. However, the indentation is probably going to require a bit of manual work.

While there, I also fixed up multi-line constructs generated with a much nicer compact format:

  %h1
    Title
  %tr
    %th
      Heading
    %th
      Heading
...

Compare with this:

  %h1 Title

  %tr
    %th Heading
    %th Heading
...

And now for the rake task:

# RAILS_ROOT/lib/tasks/haml_convert.rake

require 'haml'
require 'haml/exec'
 
namespace :haml do
  namespace :convert do
    desc "Convert all *.html.erb files to *.html.haml"
    task :html do
      Dir.glob("app/views/**/*.html.erb").each do |html|
        puts html + "..."
        Haml::Exec::HTML2Haml.new([html, html.gsub(".html.erb", ".html.haml")]).process_result
        File.delete(html)
      end
    end

    desc "Convert all *.html.erb_spec.rb files to *.html.haml_spec.rb"
    task :spec do
      Dir.glob("spec/views/**/*.html.erb_spec.rb").each do |oldname|
        puts oldname + "..."
        newname = oldname.gsub(".html.erb", ".html.haml")
        nf = File.open(newname, "w")
        File.open(oldname) do |of|
          of.each_line do |line|
            nf.puts line.gsub(".html.erb", ".html.haml")
          end
        end
        nf.close
        #File.delete(oldname)
      end
    end
  end
end

ActiveResource and Shallow Nested Routes

2009-11-04T11:36:00Z

I recently tried to dive into ActiveResource, which allows a Ruby (usually Ruby on Rails) application to connect to a remote RESTful API and treat it almost as if it were local. There were some problems with my API from ActiveResource’s point of view.

Firstly, I use nested routes. This isn’t really a problem as it is possible to specify a prefix to add to the path, and even to substitution on this path. However, Rails added the concept of a “shallow nested route” a while back.

A shallow nested route is a short-hand notation which allows one to appear to scope out collections (that is, a list of things) from the actual thing itself. For example, in my application (https://dlv.isc.org/) I have:

/users/123 <– specific userid

/users/123/zones <— list of all zones for user 123

/zones/456 <— specific zone

This allows me to look at a specific user’s zones (or them, their own zones) without having to do some sort of special back-end processing which relies on something not in the path, such as the @current_user instance variable. After all, how would an admin list a user’s zones if /zones returned the current user’s zones only? Through an admin interface probably, but that seems messy. Shallow routes are seemingly more clean.

However, they break ActiveResource’s concept of the world.

With a little trickery, however, I managed to get shallow nested routes to work without having to do much additional work. I did have to pass in an additional argument to the collection list(s) however.

class Zone < ActiveResource::Base
  self.site = SITE
  self.user = USERNAME
  self.password = PASSWORD
  
  def self.collection_path(prefix_options = {}, query_options = nil)
    prefix_options, query_options = split_options(prefix_options) if query_options.nil?
    "/users/#{USERID}#{prefix(prefix_options)}#{collection_name}.#{format.extension}#{query_string(query_options)}"
  end
end

class Dnskey < ActiveResource::Base
  self.site = SITE
  self.user = USERNAME
  self.password = PASSWORD
  
  def self.collection_path(prefix_options = {}, query_options = nil)
    prefix_options, query_options = split_options(prefix_options) if query_options.nil?
    z = ''
    if query_options.has_key?(:zone_id)
      z = "/zones/#{query_options[:zone_id]}"
      query_options.delete(:zone_id)
    end
    "#{z}#{prefix(prefix_options)}#{collection_name}.#{format.extension}#{query_string(query_options)}"
  end
end

In my case, I used a constant USERID to the Zone’s collection, but left the specific item (aka “elementpath”) alone. For Dnskeys, where I needed to pass in different zoneid values, I had to do a small trick.

I call this as:

all_zones = Zones.find :all # returns a list of all zones for the USERID
z = Zone.find(1) # returns only Zone with the id of 1
z.destroy # destroys the zone, uses the element_path()

z = Zone.find(2)
keys = Dnskey.find(:all, { :zone_id => z.id }) # disgusting, but works

#
# This is a hack.  I want to use Dnskey.create here, but it won't work since I cannot
# pass the zone_id along, and there are no association hints so I can't use
# zone.dnskeys.create() like I can with ActiveRecord.
#
d = Dnskey.new(:flags => key.flags, :algorithm => key.algorithm.to_i, ...)
d.prefix_options[:zone_id] = zone.id
d.save

First steps at "real" web bling

2009-08-14T18:56:00Z

You’d not know it from the looks of this blog, but I’m starting to play with “real” web design stuff. You know, fonts that aren’t available on the web for title bars, buttons, etc. Yea, image replacement. Ahh well, I guess it’s time to stop thinking that purist views can be pretty too.

Along these lines, I’ve cracked out Illustrator. It pains me to do stuff in Photoshop for some reason – too limited, things don’t scale properly, etc.

I’d love to hear about any open source tools that can do the steps of “make it pretty”, “edit it easily”, and end with “save slices out to individual files.” Ideally with “write all the XHTML and css files for you in a sane and reliable way.”

D-Link VTA-VR with Asterisk

2009-06-05T06:11:00Z

While walking around a local store, I came upon the clearance isle and found a D-Link VTA-VR in an opened package. Seeing the price was $25 (which read as $15 at the register) I thought I’d give it a try. The goal is to rip it from its Vonage branding and make it speak to my Asterisk server.

It worked, mostly. There are many guides on the net on how to deal with this device, but it turns out the default password was already open, so perhaps someone had already cracked it for me. Thanks, if so.

The box has two phone jacks, but my first goal was to get just one of them working. This turns out to be trivial – just configure the username, password, and the proxy and away it goes.

The second line turned out to be a problem. This client, like so many other cheap devices, seems to break the SIP protocol. For one, both lines share a single UDP agent port (defaults to 10000, I set it to 5060 for packet capture filter sanity.) This is ok but, when registering at least, Asterisk would often (but not always) report “expired nonce.”

What is an “expired nonce” you ask? It is part of the registration protocol. Basically, the SIP device sends a REGISTER requeest without any login information, and it receives an UNAUTHORIZED response. In that response, however, is some information which can be combined with a username and a password that the client and server know, to generate an authentication token. Part of this is called a “nonce” which is really just a little bit of random data that prevents certain security attacks.

What I was seeing is this:

VTA:  REGISTER for line 1
Asterisk:  UNAUTHORIZED retry with nonce [123456]
VTA:  REGISTER for line 2
Asterisk:  UNAUTHORIZED retry with nonce [abcdef]
VTA:  REGISTER for line 1 with nonce [123456]
Asterisk:  Expired nonce [123456] retry with [deadbeef]
VTA:  REGISTER for line 2 with nonce [abcdef]
Asterisk:  Expired nonce [abcdef] retry with [feedme]
...

The VTA would never recover from this. A bit of random jitter between registration attempts, or only attempting one at a time, would fix this. Using different ports on the VTA device would fix this, but there is no configuration option for that. If the VTA used a different Call-ID for each registration request, it would work.

Unfortunately, there seems to be no way to get both lines working reliably with a single Asterisk server.

There are several hacks I can think of, one is to install a SIP proxy that can register the second line, but would really just proxy the connection to the Asterisk server. Another would be to cause Asterisk to listen on more than one port, and use different ports per line. I don’t think that’s easy or even possible.

Another option would be to hack Asterisk to make it understand that a very popular firmware is rather broken, and it should just expire a nonce based on more criteria than it uses now. Right now, for a given Call-ID, there can be but one nonce. If a new one is issued, the old goes away. If Asterisk were to maintain a linked list of possible values and expire them all when one is found to be working, this might work. It would delay line 2 registration, but that’s quite acceptable.

Myth Busting

I’d also like to dispel some myths that many VTA-VR hacking pages are saying, usually referring to each other in the process.

The VTA-VR uses four UDP ports: One for the shared “agent port”, one for each line’s SIP proxy (usually both are 5060, this is the port the Asterisk server listens on), and one for TLS (which I do not use, but it defaults to 5061.) You cannot set the proxy port to something other than what the asterisk server is listening on, so any comments like “you need to use different ports” is just wrong, as unless your Asterisk server is accepting connections on more than one port, it won’t work.

Setting the “user agent” port to 5060 is handy. If you do this, you can set the defaultip=10.42.1.2 in sip.conf for that line, and even if the device is not registered, Asterisk will still send calls there. This is somewhat scary, but it does seem to work. Sometimes.

Changing the timers does not help a great deal. They are defaults, it’s best to leave them alone.

DNSSEC vs Firewall

2009-03-27T03:19:00Z

A very common cause for DNSSEC validation failure under BIND 9 is firewall issues. Specifically, a firewall that blocks fragments.

To work around this, limiting the packet size one is willing to accept so to avoid fragmentation is a good, but temporary, solution.

options {
  edns-udp-size 1460;
};

This has the side-effect of causing TCP retries on large packets, which are often the DNSKEY responses. However, it also causes DNSSEC to work, so overall it’s a good thing.

A Real XML API and Rails

2009-03-06T21:53:00Z

I recently implemented an XML API that I intended to be used outside of a web browser. Much of the words others have written on the topic are ways to get a Javascript framework to use the authentication_token magic. Some others show the GET side and mention the DELETE but omit the PUT and POST methods.

Here are things I’ve learned:

To ensure XML data is returned, use the correct HTTP header: Accept: application/xml
Rails assumes that multipart forms and url-encoded forms are from browsers. You can’t use them in a default Rails setup if you want to avoid the authentication_token check.
To POST or PUT, use a header of Content-Type: application/xml and include XML data.

It is rather unfortunate that Rails assumes that the encoding ties into a browser. It should be possible to use any encoding so long as the XML data types in the headers are correct. This is probably a bug, but it might be that if you receive XML through an API, you should send XML too.

Example

curl --user user:password -H 'Accept: application/xml' \
  -H 'Content-Type: application/xml' \
  -X POST -d '<?xml version="1.0" encoding="UTF-8"?>
<item>
  <name>foo</name>
  <description>bar</description>
  <price>100</price>
</item>' http://localhost:3000/items/create

Dell and World Record Customer Support

2009-03-03T03:05:00Z

Recently a message started appearing that told me my A/C adapter was not recognized by the system. Knowing that the A/C adapter and laptop were covered by a next-day on-side warranty, I called Dell. I carefully explained the problem: the batteries don’t charge any longer, the boot-time error message, and that the green LED on the adapter seems dimmer than it should be.

The technician at Dell decided that, with this set of problems, the motherboard needs to be replaced. I asked if perhaps an adapter should be tried first… No, he assured me, that would not be the problem.

Two days later (which is service contract for “next business day service”) a very friendly and helpful technician arrived, and replaced the motherboard. Same problem. He and I had a good old laugh at Dell for that, and he asked Dell to send a replacement A/C adapter.

Two days later, it arrived. No warning message! Success! But wait… now when I move the laptop it looses power for a brief moment. If there is no battery, it turns off. This was not happening before…

Calling Dell resulted in a mess. The first technician at Dell I spoke with was, to be as kind as possible, a moron. He had me run hardware tests. He had me set the brightness on the laptop LCD to full on and off battery, and since it didn’t flicker anymore was fully prepared to declare the problem resolved. I slowly and carefully explained that this is a physical problem and that moving the computer at all causes it to turn off if the batteries are too low or removed.

True to the incompetence that I have come to expect from nearly every computer company’s tech support, he declared that since no errors show up in the BIOS self-tests, it MUST be software. I asked to speak to his manager.

The manager declared that this was indeed a problem, part of the ongoing issue, but that I needed to ship the computer to Dell. This is because, in the mean time, my service contract has expired. They admit that while this was an on-going problem, and they will fix it, it won’t be done with on-site, and that I have to ship the laptop. I called bullsh*t.

After about 20 minutes of hold-time, two agents, and over 45 minutes of on-the-phone time they finally decided that it was indeed still covered under the same warranty replacement terms as when the problem started, and they will ship another motherboard out and have it replaced again, next-day (in service-contract terms), which means to the rest of us 3 days.

So, so far, to replace a $100 A/C adapter, Dell has wasted five hours of my time and probably two to three times what the laptop is worth in service calls.

Ruby and OpenSSL

2009-02-28T21:56:00Z

I recently had to do some DNSSEC-type (somewhat low-level) cryptography work, and found the seeming lack of Ruby OpenSSL documentation a big pain. I found numerous examples of how OpenSSL is commonly used with PEM-encoded keys, but precious little information on low-level key loading. To save others the trouble of having to dig up some of this, I’ve collected some short examples of how to do low-level RSA and DSA building from a lower level than most use.

This table summarizes the variables which need to be set to use an RSA public and private key.

RSA Keys

Key TypeItemDescription

RSA Public	e	Public Exponent
RSA	n	Modulus
RSA Private	d	Private Exponent
RSA Private	p	Prime 1
RSA Private	q	Prime 2
RSA Private	dmq1	Exponent 1
RSA Private	dmp1	Exponent 2
RSA Private	iqmp	Coefficient

Thus, in order to make a working RSA public key (so the method key.publicencrypt() or key.publicdecrypt() work) you must set at least n and e. For a working private key, you would need to load all of the items. Exposing any of the items marked as “RSA Private” above will cause a key compromise.

RSA Example

In this example, a 128-bit RSA key is loaded from numerical values. In DNSSEC, the public key is stored in the DNSKEY record for the zones. Don’t use these numbers for real crypto; the short key length is used only to make the numbers short enough to fit in the screen width. For real work, 1024 is probably a reasonable minimum length for short-lived uses, and 2048 for longer-term use.

require 'openssl'

#
# Build a RSA public key.  We only need to load two things
# here in order to use the public key to use it to encrypt,
# sign, or verify.
#
pub = OpenSSL::PKey::RSA::new
pub.e = 65537
pub.n = 216457604585180710748301099018726389113

# At this point, this will work:
crypted = pub.public_encrypt("test")

#
# Build an RSA private key.  For the private key to work, we need
# to load the entire key, private and public components.  As we
# should have access to both, this is not really a problem.
#
prv = OpenSSL::PKey::RSA::new
prv.e = 65537
prv.d = 178210827022942698143906513631075003381
prv.n = 216457604585180710748301099018726389113
prv.p = 15294921647876231099
prv.q = 14152253249053866587
prv.dmp1 = 6806715058393856237
prv.dmq1 = 637679537428568107
prv.iqmp = 6672106206837437412

# Now we have a working private key.
puts prv.private_decrypt(crypted) # prints "test"

DSA keys

A DSA key is more or less the same, just with different variable names. It is also split into a public and private part, and the key can be loaded from individual components just as easily.

Key TypeItemDescription

DSA Public	pub_key	Public Key
DSA	q	Prime 1
DSA	p	Prime 2
DSA	g	Multiplicative order modulo p is q
DSA Private	priv_key	Private Key

DSA Example

Unfortunately, this example has some numbers which are too long to display nicely. I have used a trick to convert them from strings into integers so they will fit here. Normally you would not need to do this.

require 'openssl'

#
# Build an DSA private key.  For the private key to work, we need
# to load the entire key, private and public components.  As we
# should have access to both, this is not really a problem.
#
prv = OpenSSL::PKey::DSA::new
prv.pub_key = ("899167044393666062859565588228279347268072456516837337" +
               "963353916587148226144760114643916732975837345856985656" +
               "3340384802383806137452386519280693373122367959").to_i
prv.p = ("952649509730281181203079535805855260554748337655197352471196" +
         "869232197576949258404031665397657842790773780623545384978542" +
         "6685417827665656974405272756289291").to_i
prv.q = 903197981571669745498020976355730183999507610553
prv.g = ("535694721480531756072717909769318961974692885092552247120424" +
         "749877864650255208980198391972633196543370921493242375015765" +
         "755160911031468160738717891191998").to_i
prv.priv_key = 557886499717422048101097620625259920363848888840

# At this point, this will work:
signature = prv.sign(OpenSSL::Digest::DSS1.new, "test")

#
# Build a DSA public key.  We only need to load two things
# here in order to use the public key to use it to encrypt,
# sign, or verify.
#
pub = OpenSSL::PKey::DSA::new
pub.pub_key = ("899167044393666062859565588228279347268072456516837337" +
               "963353916587148226144760114643916732975837345856985656" +
               "3340384802383806137452386519280693373122367959").to_i
pub.p = ("952649509730281181203079535805855260554748337655197352471196" +
         "869232197576949258404031665397657842790773780623545384978542" +
         "6685417827665656974405272756289291").to_i
pub.q = 903197981571669745498020976355730183999507610553
pub.g = ("535694721480531756072717909769318961974692885092552247120424" +
         "749877864650255208980198391972633196543370921493242375015765" +
         "755160911031468160738717891191998").to_i

# Now we have a working private key.  Verify the signature
if pub.verify(OpenSSL::Digest::DSS1.new, signature, "test")
  puts "Signature verified."
else
  puts "Signature verification failed."
end

Find conditions from params[]

2009-02-28T20:05:00Z

I have often wished to add a trivial search capability to my controllers which would allow searching on different fields. Without getting into a mess of many different if statements each of which has a different set of conditions, I use something much like the following:

def index
  cond_hash = {}
  cond_strings = []

  if params[:search]
    cond_hash[:login] = "%#{params[:search]}%"
    cond_hash[:email] = "%#{params[:search]}%"
    cond_strings << "(login ILIKE :login OR email ILIKE :email)"
  end
  if params[:search_address]
    cond_hash[:address] = params[:search_address]
    cond_strings << "(address == :address)"
  end

  conditions = cond_strings.join(" AND ")
  @users = User.all :conditions => [ conditions, cond_hash ]
end

It may be safer to use users.login, users.email, and users.address in the SQL-like strings above.

Rails 2.0 and cool error handling

2009-02-28T05:09:00Z

Rails is a great framework, but it has gained something of a bad reputation in terms of error reporting. Everyone has seen them – those ugly 500-status “we’re sorry, but something has gone wrong” messages.

In a finished application, this just doesn’t cut it. It used to be necessary to trap the error using something like this:

def rescue_action_in_public(exception)
  case exception
  when ::ActiveRecord::RecordNotFound
    render :file => "#{Rails.public_path}/404.html", :status => 404
  else
    render :file => "#{Rails.public_path}/500.html", :status => 500
  end
end

The problem is that, as error causes begin to pile up, this method gets messy. Plus, you might want to render something different in one controller than in another, and would either need to duplicate all the logic, put controller-specific cases in the ApplicationController, or call ApplicationController::rescueactionin_public directly.

Enter Rails 2.x error handling. Drop this in your controller or ApplicationController.

123	rescue_from(ActiveRecord::RecordNotFound) do \|e\| render :file => "#{Rails.public_path}/404_record_not_found.html", :status => 404end

By building up a rich set of rescue_from handlers in the ApplicationController, and only overriding the ones you wish to make per-controller, you have fine-grained control. And they work exactly the same in development and testing as they do in production.

Ruby Regular Expression Gotchas

2009-02-26T06:05:00Z

I love Ruby. I love Ruby on Rails. Rarely have I found a language or a framework that just works.

However, you still have to know the finer details sometimes. I recently made a model for a DNS zone. The name in the model is the “front part” of a fully qualified domain name. For instance, if zone.name = “foo” then I would write the name into my name server’s configuration files as “foo.example.com.”

Knowing that people were evil, I saw that if a user put a string in like “example.com. NS hackerz-will-someday-rule-the-earth.ru.\nfoo” I would happily write out two strings, one being rather bad.

Knowing how easy this sort of data validation is in Rails, I made my model look like:

class Zone < ActiveRecord::Base
  validates_presence_of :name
  validates_uniqueness_of :name
  validates_format_of :name,
    :with => /^[a-zA-Z0-9\-\_\.]+$/,
    :message => "contains invalid characters."
end

Happy, I ran a few tests using my browser and found that I could not insert names with spaces, colons, tabs, etc. Then, several days later, I decided it was time to write tests for this.

require 'test_helper'
class ZoneTest < ActiveSupport::TestCase
  def test_name_with_newline_fails
    z = Zone.new(:name => "test\nzone")
    assert !z.valid?
    assert z.errors.on(:name)
  end

  def test_name_with_space_fails
    z = Zone.new(:name => "test zone")
    assert !z.valid?
    assert z.errors.on(:name)
  end
end

Imagine my surprise when testnamewithspacefails() passed, and the one I was most worried about, testnamewithnewlinefails(), did not!

Not all regular expressions are alike.

The problem is in what I thought ^ and $ actually matched. I thought these meant “match the beginning and ending of the string.” However, it turns out it means “match the beginning and ending of each line contained in the string,” where lines are divided by newlines. Ooops.

Changing ^ into \A and $ into \Z fixed this problem. Now I’m auditing all the code in this application to see if there are other problems like this.

This is just one thing to add to an ever-growing security checklist for my Rails work. It’s also a very typical security hole: programmer error.

Mmm, sharp knives...

2009-02-24T18:35:00Z

OK, so I admit it. I love sharp knives. Well, more exactly I love my knives to be sharp.

So, I bought a Wüsthof knife sharpener, fully expecting it to suck.

I am now happily using my knives to chop and cut, not saw, through things again.

Oh, and my knives are J.A.Henckels professional “S” series.

String#bitruncate

2009-02-21T19:09:00Z

And that’s ‘“bi-truncate” not “bit-uncate”.

What’s it do?

12	"This is a test".bitruncate(:length => 6) ==> "Thi...est""This is a test".bitruncate(:elength => 6) ==> "...a test"

The default options are { :length => 30 } which will produce 15 characters from the front and 15 from the end, putting … marks in the middle.

For rails, I put this in my lib/core_extensions.rb file.

#
# Copyright (c) 2009 Michael Graff.  All rights reserved.
#
# Redistribution and use in source and binary forms, with or
# without modification, are permitted provided that the following
# conditions are met:
# 1. Redistributions of source code must retain the above copyright
#    notice, this list of conditions and the following disclaimer.
# 2. Redistributions in binary form must reproduce the above
#    copyright notice, this list of conditions and the following
#    disclaimer in the documentation and/or other materials provided
#    with the distribution.
# 3. The name of Michael Graff may not be used to endorse or promote
#    products derived from this software without specific prior
#    written permission.
#
# THIS SOFTWARE IS PROVIDED BY Michael Graff ``AS IS'' AND ANY
# EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
# THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
# PARTICULAR PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL Micahel Graff
# BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
# EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED
# TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
# OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY
# OF SUCH DAMAGE.
#

class String
  #
  # Truncate from both ends of a string.  The :length parameter, which defaults
  # to 30, will return the first 15 and the last 15 characters from a string
  # if it is longer than 30 characters.  If it is shorter, the entire string
  # is returned.
  #
  # Another way to specify the front and back portions are with :flength and
  # :elength.  If you specify one of these but not the other then you will
  # not get the missing part.  e.g., :flength => 10 alone will return only
  # the first 10 charcters of the string.  This is the same as the standard
  # truncate(s, :length => 10) helper.
  #
  # If a :length parameter is provided it will override any other lengths
  # specified.
  #
  def bitruncate(options = {})
    maxlength = options[:length] || 0
    flength = options[:flength] || 0
    elength = options[:elength] || 0
    omission = options[:omission] || '...'

    if maxlength == 0 && flength == 0 && elength == 0
      maxlength = 30
    end
    if maxlength != 0
      flength = maxlength / 2
      elength = maxlength / 2
    end

    return self if length <= (flength + elength)

    front = ''
    back = ''
    if flength > 0
      front = self[0..(flength - 1)]
    end
    if elength > 0
      back = self[(length - elength)..(length)]
    end

    front + omission + back
  end
end

Vmware Server and Xen

2008-05-20T02:30:00Z

In the past, I’ve played with VMware (as a “workstation” “server”, not the bare-metal one we have access to now) but was never quite happy with it. Some of the problems I had might have been that I ran it on windows Vista, not Linux. However, from a VM point of view, the VM itself should be more or less identical.

Recently I tried out Xen on the same hardware, but using NetBSD/amd64 as the “host” OS.

Hardware

The machine is Gateway GM5446E with a dual core Intel Core 2 Duo with 3 GB of ram. The machine has three SATA hard drives, connected via an Intel AHCI controller running in native AHCI mode.

Bare Hardware Baseline

I booted a standard NetBSD-current/amd64 kernel and ran some speed tests, which gives a baseline for dom0 and guest OS disk I/O speed tests. See below.

VMware

In my VMware install, I used the machine running “Windows Vista Media Center” – it was what came pre-installed on the machine.

Host OS: Windows Vista Media Center

Guest OSs tried:

NetBSD-4.0/i386
NetBSD-4.0/amd64
NetBSD-current/i386
NetBSD-current/amd64
Linux Ubuntu (server, then-current version)
Linux Debian (then-current version)

Linux booted in 64-bit mode, but would crap out at some later point with similar issues that NetBSD had.

One machine, named “nfsd”, was dedicated to serving out home directories and source trees of NetBSD. The other host OSs mounted /home or /netbsd-src from nfsd.

Each machine had a “local” disk to store object files from pkgsrc, src, and other OS-related builds.

General VMware Problems

I could not get netbsd/amd64 (current or 4.0 release) to “self host” – build /usr/src and kernels – reliably. They would either silently lock up without reason, or they would crash with an odd CPU exception.

Timekeeping was whacky. Without running the VMware-supplied (closed source) tools on a client, time was off, and apparently in uncorrectable ways. Running ntp made things seriously whacky as the time would drift wildly as ntp tried to correct a guest.

The VMware closed-source tools are only available on a small, limited number of OS types, and then specific versions of many of them. They do supply a .so that is, in theory, linkable on many versions of Linux, but the install procedure warns loudly of warnings pertaining to compatibility.

VMware running on anything but Intel chips with synchronized cycle counters (which most OSs use for high-res timekeeping these days) was a disaster.

VMware Strengths

If the problems above are solved, VMware is a true virtual machine architecture that will run any OS without modification. VMware could run windows guests, right along with unmodified NetBSD, FreeBSD, Linux, and Solaris/x86 guests.

Xen

I tried Xen 3.1.3.

Xen is a different architecture than vmware in that it prefers to use “paravirtualization” rather than a full virtual machine. It has a host machine (called “domain 0” or “dom0”) which attaches to the physical hardware and acts as a conduit between the xen hypervisor and hardware.

The boot process is that the xen kernel is booted first, which then boots the dom0 host. Multiple domains can be created, serving different hardware, but in practice this is rarely done.

Each host OS has a config file, and is stated with “xm create /path/to/file.conf”. This boots the guest OS and connects a serial console, which can be used with “xm console ”.

Since the “dom0” is a fully functional OS in its own rights, I have it serve NFS to the guest OSs.

I created the following guest OSs:

NetBSD-current/i386
NetBSD-current/amd64
Windows XP Pro (32-bit)
Windows Server 2003 (32-bit)

Yes, I managed to install Windows XP Pro and Server 2003. They run in a “vnc” console, and for all practical purposes looks like windows. This is using Xen’s “hvm” – which is a full hardware emulated virtual machine, and allows running unmodified guest OSs. People have Vista running in a virual machine under Xen, but I do not have “real” Vista install media or licenses, just the ones that came with and is tied to my hardware.

Xen also supports both realtime and offline “migration.” As I have only one machine of the same type, I have not yet read up on how this works. The basics: A realtime copy is made of the guest’s ram, device state, and other data. It is transmitted to the new destination, and synced up until a very small switchover time can be used to swap where that guest is running. Xen claims 100 ms switchover time is possible, but there are restrictions: The disks are NOT migrated, so must reside on a shared volume. The physical network each dom0 is on is also shared, in order to avoid disruption of TCP connections. I also believe fairly identical CPU and dom0 operating systems should be used.

Offline migration involves shutting the guest down, copying the disks over, and restarting it on a new dom0. This will, of course, interrupt service.

Xen Problems

It is difficult to configure for the fist time. The documentation is… lacking. It is also only as solid as the host OS is, but vmware has the same issue in “server” or “workstation” incarnations.

Xen is also very, very “linux” specific in documentation and examples. Most of these can be translated – I certainly did so easily enough – but this is being corrected in their documentation as more OSs are able to boot as dom0.

Xen Strengths

Timekeeping in Xen, since it is paravirtualized, is almost perfect. Small drifts will occur without running ntp, but all guests (and the host) can run ntp and obtain sanity.

It also appears that all guests and the dom0 “drift” identically, so this is probably related to hardware timekeeping issues. The measured drift of an uncorrected NetBSD guest was 4 seconds in two weeks. ntp correction kept the others in perfect real-world sync.

It is as free as you want it to be. Support and commercial versions exist, but the free stuff works amazingly well.

Performance

On Xen, all tests were performed with the domu’s running but idle, and no hvm guests running (windows is just too unpredictable.) On VMware, only one VM was active at once, and the Vista host was as idle as it could be made.

I measured three main things here:

Boot speed: How fast a kernel gets from loading to the first /etc/rc message.
Disk speed: read/write speed.
CPU Performance:

Boot Speed

The dom0 boots as fast as any other kernel boots; it must probe the hardware, wait for hardware to change state, etc. No measured difference between a standard NetBSD-current/amd64 kernel and the dom0 kernel.

The domUs (guests) boot so fast it is nearly impossible to measure. This is because the devices they have access to are known – all are on a virtual bus, and are directly enumerable, so there is no need to probe for devices, wait for them to change state, or time out when not present. As best I can measure, just under 2 seconds is a fair estimate.

The hvm (windows) guest seems to be about as fast as windows is. I did not analyze this one much.

On VMware, the host OSs boot at about the same speed as a “real” machine boots unless a custom kernel is built with just “known present” devices. Even then, boot times are 15-20 seconds.

Disk Speed

In all host/dom0 tests, “iozone” version 3.263 was used, with a 1 GB file on the same disk. Each test was performed only once; for real comparison data we’d want to run it more than once, but this is just a first-pass test.

For native NetBSD/amd64, I had to increase the file size to 4 GB to avoid the cache, as the machine has 3 GB of ram.

wd0 is a 500 GB SATA 3.0Gb/sec disk.
wd1 is present but unused.
wd2 is a 320 GB SATA 1.5Gb/sec disk.

All are on different channels of an Intel AHCI controller running in native SATA mode.

For the Xen tests, all disks were mounted as files on the dom0 host. From dom0’s point of view, the file is mounted on a “vnd” virtual disk, and that virtual disk is exported to the host.

For the VMware test, all disks were mounted as files in the Windows filesystem.

OS Disk Block Size Read Write

netbsd-current/amd64 native	wd0	8192	60762	59949
16384	60545	60141
wd2	8192	78342	76342
16384	78311	75252
netbsd-current/amd64 dom0	wd0	8192	60641	60109
16384	60459	61919
wd2	8192	80258	79102
16384	80187	80295
netbsd-current/amd64 domu	wd0	8192	51205	24004
16384	51714	27971
wd2	8192	77990	23997
16384	77282	22496
netbsd-current/i386 domu	wd0	8192	41730	25012
16384	42008	24543
wd2	8192	66401	26048
16384	66201	28910
netbsd-current/i386 vmware	wd0	8192	25014	13912
16384	25417	13771
wd2	8192	38831	16100
16384	38994	16332

I also repeated one test with a raw, physical partition mounted in the netbsd-current/amd64 domU, which bypasses the “double filesystem” issue:

netbsd-current/amd64 domu	wd2	8192	79915	76992
Physical mount	wd2	16384	79744	77102

CPU Performance

Each CPU speed test was run with: Dhrystone Benchmark, Version 2.1 (Language: C) Program compiled without ‘register’ attribute.

I used an iteration count of 1,000,000,000 for each test.

Operating System Dhrystones per second

netbsd-current/amd64 native	11,013,216
netbsd-current/amd64 dom0	10,365,917
netbsd-current/amd64 domu	11,130,899
netbsd-current/i386 domu	4,935,347
netbsd-current/i386 vmware	5,012,123

Just for grins, I ran the following tests, one dhrystone on one guest and another on a different one. Since each guest is uniprocessor in my configuration, I did not run two benchmarks on the same host.

Operating Systems Speed 1 Speed 2

Running both domu/i386 and domu/amd64	4916421.0	11135857.0
Running both dom0/amd64 and domu/amd64	10298661.0	11135857.0
Running both dom0/amd64 and domu/i386	10373444.0	4921260.0

Conclusions

Xen is production ready.

When the host OS can be modified, much higher performance numbers are obtained vs. the low-end VMware server I ran.

While it might be extremely tempting to build one guest that does one very specific function, this probably does not scale: memory is pre-allocated and dedicated to a guest, and while some swapping is allowed, it will slow the guest at seemingly random times; disk can be overcommitted, but the OS sees failure to allocate a block as a hardware failure; the more hosts, the more maintenance costs are present: maintaining packages on each guest, upgrading, etc.

VMware “hmx” or whatever the name of the run-on-bare-metal product should be tested.

I’d love to install Xen on a huge machine with lots of ram and many, many CPUs as a test. Would someone like to ship me a 4 CPU quad core with 64 GB?