Ruby Best Practices

Practicing Ruby, The Newsletter

gregory.t.brown@gmail.com (Gregory Brown) — Mon, 08 Nov 2010 16:43:00 -0000

Welcome, Hacker News folks. If you find the following project worthwhile, I’d really appreciate an up vote so that I can reach a broader audience. Please read a bit about Ruby Mendicant University to see what cause this goes to support.

Since the first Ruby Mendicant University session began in August, I’ve interacted with and mentored roughly 50 students. The fascinating thing is that despite the wide gradient in skill levels we see at RMU, there are many common obstacles that trip up the majority of students attending our sessions.

As it turns out, there is a surprisingly wide gap between a foundational book such as David A. Black’s Well Grounded Rubyist which is suitable for beginners and my own Ruby Best Practices book which is designed to help intermediate developers. At least as far as I’ve seen, this isn’t a hole that has been filled in by other resources that aren’t some kind of interactive training program. My new project, the Practicing Ruby newsletter is designed to fill in that gap.

Each week, I’ll choose a topic based on things I’ve noticed tend to trip up intermediate Ruby developers. Every Tuesday, some background information on the topic along with some discussion points and open questions will be sent out to the readers. Some time will be given for open discussion on a mailing list made available to subscribers, and then a follow up article will delivered on Thursday which provides some answers to the given questions as well as some practical examples of whatever topic we are studying that week.

My hope is that this will be a great use of one of the byproducts of running RMU. Not everyone has the available free time and motivation to spend several weeks in the intense training programs we offer at RMU, and space in our sessions is severely limited. But a wide range of Ruby developers can benefit by exploring the topics we’re running into at RMU, especially if we can break them down into something that takes an hour or less a week to study. If you’re an intermediate Ruby developer who would rather do your learning in bite-size chunks, this newsletter is for you.

If this sounds interesting to you, please sign up for Practicing Ruby now. The cost is $5 a month, but you can feel good in knowing that the better off this newsletter does, the easier it will be for me to continue dedicating time to the completely free Ruby Mendicant University. I currently spend more than 30 hours a week on RMU, with about half of that going to student interaction, and would happily dedicate all my time to it if it were possible. This newsletter might be a way for me to get a little extra revenue to make up for my lost consulting hours, at least until we have a proper non-profit organization in place for supporting RMU.

While I do not have any sample newsletters ready just yet, you can download the whole RBP book for free to get a sense of my writing style. I can also tell you that the first topic we’ll be covering will be an exploration of Ruby’s method lookup path followed up by some practical examples of how to put those ideas into action. This will help answer some questions about when to use inheritance, mixins, and even per-object behavior.

Feel free to post any questions or suggestions in the comments section below. I’m really excited about this project and would be happy to hear your thoughts!

The Ruby Mendicant University Core Skills Course

gregory.t.brown@gmail.com (Gregory Brown) — Thu, 21 Oct 2010 17:35:00 -0000

For the tl;dr crowd, there are sixteen student projects and four sample exercises from the course towards the end of this document.

The Original Idea

Back when I wrote about the Ruby Mendicant University idea in June, I didn’t have a particular plan for it. I knew I wanted to teach a free online course of some kind, drawing from my life experiences as much as possible to form the learning materials. Being mentored by James Gray for the first several years of my career as a software developer really taught me the importance of having a great teacher to guide me while I worked, so I knew that mentorship would be a key part of the program. The idea behind RMU was mainly to scale that experience from a 1-1 relationship to one that could reach many students.

While it has been a wild ride so far, the results have been simply amazing. I’ll spend the next few days trying to give the Ruby community a sense of what’s going on within RMU, so they can see the awesomeness of the program for themselves. I’ve decided to start by describing the core skills course, which is something every student attending RMU needs to tackle if they wish to continue on in the program.

The RMU Core Skills Course

In order to become a permanent member of the RMU community and move on to take additional courses, each student must pass an entrance exam and then go on to successfully complete their core skills course. The entrance exam is mainly used for ensuring that the people joining the program are diligent workers who have sufficient background knowledge to do well in our core skills course. The core skills course itself though, is something I think is something genuinely unique that you can’t find anywhere outside of RMU.

Each core course starts with roughly 15 students submitting a proposal for a personal project to work on during their session. This can be anything Ruby related, but typically involves the student building an open source application or library that scratches a particular itch of theirs. This is the first learning opportunity of the course, as it filters out those who can’t think of anything to work on. There isn’t a list of suggested projects to pick from, students need to come up with an idea themselves. As each student decides what they want to work on, I work with them to figure out what they can reasonably produce in three weeks which would be useful and show measurable progress.

I’ve found that without some tangible goal to focus on, improving your skills is an abstract, disconnected experience that is only enjoyable until the going gets tough and the novelty wears off. No RMU student can even begin their session with this mindset, which I think is a key part of why the course can inspire such rapid growth. But many folks are only likely to volunteer to work on problems which they find sexy on the surface, which isn’t a good way to master a craft. That’s where assigned work comes in.

While the exercises used in each session are different, I try to include one for each of the following topics:

Integrating with other systems (typically web services)
Modeling non-trivial business logic (often a game of some sort)
Community Service (contributions to RMU and the community at large)
Classical Computer Science / Engineering Knowledge (Patterns, Algorithms, etc.)

No matter what the topic is, the end result of the exercises is some sort of real application. Even when we’re studying higher level concepts, we don’t stray too far from things that you could actually use in day to day life. The varied nature of the assignments insures that all students are challenged in one way or another, regardless of their relative level of experience entering the course.

While earlier core skills sessions we ran involved some exercises that needed to be done in isolation, the current course is set up in such a way that the exercises can be actively discussed between students without it spoiling the learning experience. This in itself is invaluable, because most of the lessons learned in RMU are not actually pre-determined by the course materials, but instead, emerge through conversations about the problems.

Rather than doing lectures, I maintain extensive office hours on IRC and the session mailing list, spending 15-20 hours per week answering student questions and discusses topics as they arise through the session. This has been extremely time consuming, so I’ve started to enlist the help of some of our alumni in mentoring session students. But this is also the area of RMU that I find most special and unique. I spend three days a week in constant conversation with my students, which means that I learn an incredible amount about each of them and can tailor the program to meet their individual needs and interests.

Students can submit their assigned work or individual projects for review at any point in time during the three week program. I approve submissions which don’t show any obvious gaps in understanding the key points, the others, I provide feedback on and ask them to resubmit when they’ve made revisions. Students who make good progress at the end of the course by getting at least three of four assigned projects approved, and by doing good work on their individual project, gain alumni status and are invited to stay involved with RMU for the long haul.

I’ll have to save the alumni responsibilities and benefits for another post. But it’s worth noting that so far, 16 of 25 students who have completed a core skills course gained alumni status. While I’d like to see that number higher, several of the students who didn’t successfully complete their session are working with me to find a way to get their status later. But the great thing is, 100% of our alumni intend to take more courses at RMU in the near future!

Student Projects from August / September

Below are the sixteen individual projects of our current RMU alumni from our August and September sessions. While many of them are still in their early stages of development, several are quite useful already. Keep in mind, most of these projects were built in just three weeks!

google_currency (Shane Emmons): Ruby Money::Bank interface for the Google Currency exchange data
cashmate (Frank Wöckener): Business proposal management system
forum_engine (Pavel Chipiga): Forum engine built on Rails 3
groupy (Justin Ramel) ActiveDirectory Group Management Utility
prism (Alex Soulim): WebSocket Server for chat
gstore (David Ruan): Ruby client library for the Google Storage API
Prawn Reference Guide / Examples Refactoring (Felipe Doria) Soon to be merged upstream
themes_for_rails (Lucas Florio): Theme support for Rails 3
unwatched (Morgan Prior): Utility for tracking watched media files
montogomery (Wojciech Piekutowski): A lightweight object mapper for MongoDB
vegtastic (Andrea Singh): Collaborative vegan/vegetarian recipe collection app
resourceful (Jaakko Vallo): Automated work schedule management.
qinfo (Ignacy Moryc): SQL Query optimization tool
minivid (Brandon Hays): 60 second video uploading and sharing service
rallot (Alberto Fernández-Capel): Secure electronic voting system backend
rmu-alarmclock (Eric Gjertsen): Client/server based reminder system

While I gave students guidance on their projects throughout the course, my focus was on education, and not on quality assurance, so please do not consider this listing of projects a personal endorsement that they’ll be suitable for a particular purpose.

Released problems from August / September sessions

If you’re curious about the sorts of problems students worked on during their sessions, you can check out the few that have been released so far. Eventually all of our materials will be released, and collected nicely in an easy to find location. For now, feel free to re-distribute these under the terms of the GNU Free Documentation License.

You can try these on your own, or look through the network graphs to see some student submissions that have been made public.

s0-e1 : Prototyping the board game Dominion
s0-e2 : IRC bot / Web Service mashup
s1-e1 : Twitter bot / Web Service mashup
s1-e2 : Prototyping a Github themed achievement system

Lots of other exercises will be released soon, so stay tuned.

Please help us with funding!

As you might imagine based on this article, running RMU is an incredibly labor intensive task. What started as something I thought I’d do as a side project after the initial setup work has gone in the opposite direction. I now spend 30-35 hours per week on RMU related responsibilities, about half of that time is spent interacting with students. My coworker Jordan Byron spends at least two days a week working on RMU, building out our web application for course management, and my wife Jia spends several hours each week dealing with administrative tasks.

I’ll write more about what goes into running RMU soon, but for now, there is a pledgie open that is meant to bridge us over until we get a proper non-profit organization with a decent revenue model set up. Your contributions would be greatly appreciated, so please donate now, and tell your friends as well.

Questions / Thoughts About RMU?

We are not accepting new students now, but you are still welcome to discuss the program with us either via irc.freenode.net in the #rmu-public channel, or via the RMU Public google group. We are happy to hear your thoughts and questions! Also, feel free to leave questions or comments here on this blog post, I’ll be sure to read them and respond quickly if there is anything on your mind.

Complexity

shortcutter@googlemail.com (Robert Klemme) — Sun, 06 Jun 2010 20:37:00 +0000

If I tell you a program has 1,000,000 lines of code you’ll immediately understand that this must be a complex piece of software. Why is that? You implicitly know that nobody copies the same 5 lines 200,000 times to create a long program which is full of redundancy. This is another term we will have to consider while trying to improve our understanding of complexity. As a first approximation we can say that a complex piece of software is free of redundancy. This is of course an oversimplification and for various reasons that we will have to talk about in a minute we will not usually reach that mark in practice.

This oversimplification however helps us to understand one important – if not the most important – task of software engineering: reduction of redundancy. Assume that someone in an application you are writing you need to output all elements contained in an Array in a special way (say, as an HTML unordered list). You would probably write something like this:

  puts "<ul>"
  
  arr.each do |x|
    puts "  <li>#{x}</li>"
  end
  
  puts "</ul>"

I know this is not the best of examples since you normally would be doing this with ERB or your favorite web framework. Picking the right tools for the job is of course another important task in software engineering but we want to focus on something different which can be nicely demonstrated with this simple toy example. (Also, you can easily follow by copying and pasting the code to IRB and play with it.)

What do you do once you discover that you also have to apply the same formatting to a Set in a different location of your program? Right, you make it a function and write something like:

def format_list(enum)
  puts "<ul>"
  
  enum.each do |x|
    puts "  <li>#{x}</li>"
  end
  
  puts "</ul>"
end

In other words, instead of increasing redundancy of your program you refactor a part of it so you can use this functionality in different locations. Now you discover that you also need to be able to write to a file. Instead of writing a second function that has a second argument for the file you want to write to you rather add an argument to this function potentially making it look like this:

def format_list(enum, out = $stdout)
  out.puts "<ul>"
  
  enum.each do |x|
    out.puts "  <li>#{x}</li>"
  end
  
  out.puts "</ul>"
end

Again you avoided increasing redundancy of the application by adding complexity to this function which now is capable of applying the formatting in an even broader range of situations. Let’s push this one step further: now we also want to be able to do custom conversions of elements in the Array instead of just always using #to_s.

TO_S = lambda {|o| o.to_s}

def format_list(enum, out = $stdout, &convert)
  convert ||= TO_S
  
  out.puts "<ul>"
  
  enum.each do |x|
    out.puts "  <li>#{convert[x]}</li>"
  end
  
  out.puts "</ul>"
end

Now we can call it for a list of Floats like this:

  floats = [1.2345, 4.567, 64.3]
  
  format_list floats do |f|
    "%010.3fcm" % f
  end

Again, complexity of this function increased but we gained versatility. In this case we are particularly lucky because all calls of the initial version of the function still work and produce the exact same result. We might have to take a bit of runtime overhead because the original string interpolation is likely faster for the general case where only #to_s is needed. (Btw, if this program runs on some form of virtual runtime environment (such as the JVM) the situation might be different because the JVM’s runtime optimization might produce better results if there is just a single function which is called more often than multiple methods that each are called infrequently.)

I’d say what we have seen above is a fairly typical evolution of a piece of software. Instead of increasing redundancy of the program we increased complexity of this function. It may seem that having different functions for various variants might be better, for example, these functions are certainly easier documented. So why did we do this and strive to stick with a single function?

Humans write software and while a piece of software might be bug free humans are not. More importantly the world keeps changing all the time and so do requirements for software. Either we did not read the spec properly or someone changed his mind and now all of a sudden we need to ouput ordered lists. That’s an easy change if we just have a single implementation:

def format_list(enum, out = $stdout, &convert)
  convert ||= TO_S
  
  out.puts "<ol>"
  
  enum.each do |x|
    out.puts "  <li>#{convert[x]}</li>"
  end
  
  out.puts "</ol>"
end

The change is so minimal one barely sees it. Yet, if we have to apply that change to multiple versions of this function potential for new bugs is much higher. We might forget one of them (remember that all these do not necessarily be located in the same file) or we might misspell in some of them etc. Also the effort is higher: for this particular change it might not be dramatic but just think about having to check in multiple files into your favourite source control system, having to update documentation for multiple functions, having to adjust unit tests or other test suites etc.

All in all I think it is fair to say that we gained more than we lost. We gained developer productivity and paid only a small runtime penalty and complexity of this function. Up to this point we can say that increased complexity has brought about a better piece of software. However you might have a premonition of degradation which will start if we add even more features to this function. Here we have the first reason why we do not have redundancy free applications in software: humans can only handle a certain level of complexity efficiently and so we must avoid too complex systems if we want to be able to maintain the code.

Another reason why we may end up with different versions of format_list is the sheer size of an application. Large applications need multiple authors and it is not too unrealistic that people independently invent similar functions in different parts. While this would increase redundancy it might actually be desirable to do it: if there is just the one implementation of the function all components that need it must depend on the component that contains it. Depending on the application and programming language used the price of an additional component dependency may actually be higher than the benefit.

Summary

Today we looked at software complexity caused by adding features to a piece of software to avoid redundancy. In our daily work we continuously have to make decisions that affect this dimension of software complexity. We have also seen how we might want to retain some level of redundancy to keep software maintainable. Next we will look at other dimensions of software complexity. Hopefully we will eventually come up with a classification of factors that lead to software complexity – and how to tame them.

Ruby Mendicant University is a Go!

gregory.t.brown@gmail.com (Gregory Brown) — Fri, 04 Jun 2010 14:50:00 -0000

Born less than 24 hours ago, my Ruby Mendicant University idea is already more than a third of the way to having its initial costs covered. In less than 12 hours, we’ve had 35 people request a copy of the entrance exam. So now that the idea is validated, it’s time to begin on the execution.

You can keep up on the latest news by following the rubymendicant tag at seacreature.posterous.com

PS: This morning, I decided that I want to consolidate my blogging and tweeting, so most of my content will be posted to my new posterous blog rather than here at RBP blog. Also, I am moving my twitter account over to @seacreature exclusively, rather than using @rubypractices. While this isn’t quite an announcement of ending the RBP Blog project, it’s a step in that direction. Thanks so much for your attention and feedback over the last year or so. Hope to see you on the other side!

IDEA: Ruby Mendicant University

gregory.t.brown@gmail.com (Gregory Brown) — Thu, 03 Jun 2010 14:50:00 -0000

UPDATE: You can donate now at Pledgie / I know this is early, but in the spirit of Ruby Mendicant, I’m looking for those trusting individuals who will give with altruistic intentions first, I’ll spend my time convincing everyone else later :)

UPDATE 2: More details to come, but those who wish to receive a copy of the entrance exam should sign up via this google form.

Back in 2008, the highlight of my year was the Ruby Mendicant project. Through some stroke of luck, I ended up getting enough funding from independent community donations to take 22 weeks off of work to focus on building Prawn, a pure ruby PDF generation library that is now nearing its 1.0 release.

Now in 2010, I’d like to do the same thing for Ruby training and education. Similar to my original post that spawned Ruby Mendicant, the details are currently fuzzy in my head, but I have enough of a rough outline to collect some comments on.

The Idea

I want to run a free online school designed to help intermediate Ruby programmers master their craft. Each session would include up to 20 or so participants and run for 3 weeks.

The basic activities would include:

an entrance exam (basic coding problem and questionnaire)
selected readings in freely available Ruby books and online resources
individual and group code review
realistic problems to work on and discuss
Q&A sessions
a final exam (which if passed, would earn the student ‘alumni’ status)
possibly some pre-recorded or live presentations
a mailing list for students to discuss questions with me
online discussions with guest speakers from our community.

The goal would be to keep these classes going on a rolling basis, so that we could reach a large number of students in the course of a year. If we did a schedule like 3 weeks on, 1 week off, we could do 13 sessions in a year, allowing me to reach up to 260 students.

Of the available slots, I would also like to reserve a certain amount of them for “hardship” situations. People who are currently out of work, those who are in tough economic climates, or have some sort of difficulty in their lives that this opportunity could really help them overcome. I have not yet figured out how to do this without introducing a ton of bias in my selections, and I have no way of verifying these details, but it seems like a good idea in theory to me at least.

At least some of the sessions would be open to the community at large, and at least some of the material would be released under an open documentation license. This way, while the core focus would be on affecting big change for small groups at a time, the whole community could benefit from the existence of this program as well.

Funding

The up front cost of building the material is large, enough where I would need to take a couple days off a week from consulting work for a period of 8 weeks or so to come up with adequate material. While the replacement cost for that work is pretty high (over 6k), I could probably still get by well enough if I had about $3000 in donations to produce the initial material.

The promising thing about the above lesson plan is that since a lot of it is interactive and tailored to individual efforts, once I build the core lesson plan, actually running the “school” will be comparatively less work. I won’t know at all until I actually run it, but I would expect that I could keep the school up and running for about $1500 a month.

But in the Mendicant spirit, I only would want to keep this thing going if it was of value to the community. So rather than looking for donations for the running of the school up front, I would only raise money to cover the initial costs of content creation.

Whether the school lives or dies, grows or shrinks, would be based on the generosity of previous students as well as that of the community at large.

Schedule

If I could raise enough money by the end of June, I could then shift my schedule around in July and August to allow me to work on content creation. That would let me start the first class around the first week of September.

Why Me?

Besides the Ruby Mendicant project, I have also taught trainings at the Lone Star Ruby Conference (2008,2009), and am also part of the trio hosting The Compleat Rubyist. I have written two books that are now completely free for people to download and read. And other stuff.

Trust me. I’d be good at this :)

If you’re wondering what’s in it for me: I believe that free education is one of the most valuable things in the world, and I also learn a ton by teaching others. If I can make enough money to offset my costs and maintain a decent house-and-wife life, I feel like this is a super good use of my time.

What do you think?

Right now, feedback is key. I only want to take this project on if I think it’ll be a sure thing. Please respond, answering these questions:

Would you want to attend this online school?
Would you be willing to donate to help with its creation?
Would you recommend this program to others?
What can I do to make it even more awesome?

Thanks in advance for your suggestions. Here’s hoping that we can make this work :)

UPDATE 2: More details to come, but those who wish to receive a copy of the entrance exam should sign up via this google form.

If you want to see the wizard, go to Oz.

gregory.t.brown@gmail.com (Gregory Brown) — Wed, 21 Apr 2010 14:50:00 -0000

Back in June of 2005, I booked a flight out to Oklahoma City, OK, but it might as well have been to Oz. I was off to see the wizard who had been teaching me black magic in both Perl and Ruby throughout my high school days.

James Edward Gray II and I had somehow convinced Ruby Central, Inc. that a web game framework was worth a Codefest grant, and we used the money as a chance to finally meet each other in person. At the time, we decided to build things from scratch, since Rails 0.12 didn’t quite have the features we needed :)

Not counting the friends I talked into studying programming here and there, this was my first time ever working side by side with another hacker. I didn’t know what to expect, but by the time I left, I couldn’t believe that what we had done was possible. Safe to say, JEG2 blew my mind, in the way that any good wizard can. Here’s what I wrote just after returning home in June 2005:

The last nine days were amazing. Exploited undocumented features of WEBRick.
Making use of circular logic flows, making code do things that it wasn’t built
to do. Pushing the limits, building truly clever tools and libraries that will
make people’s lives easier, and more fun.

I myself feel like I’m standing a little taller after all the hard work we
did. That was hacking in it’s purest nature, and though I don’t like to self
attribute that word, there is nothing else to describe it. Ideas flowed any
hour of the day, even in our dreams. No one tried to take charge, I didn’t get
mad when James replaced bits of my code with something better, and James
didn’t get mad when I questioned his logic and pushed him to raise the bar. In
the rare occasion that I proved James wrong on something, he’d just shrug his
shoulders, embrace The Right Way, and go with it.

We joked around a lot referring to certain programming practices as “The way
of life”. Though that is quite silly when it applies to 80 character linewrap
or a certain tab width, it was something that motivated us. We could look at
our code, and at a glance see if it was following “The way” or not. When it
wasn’t, we waited for something better to come along, and then we did what we
had to do to make it happen.

Now, I was just a kid at the time, but I knew something important had just happened. Most importantly I realized that the kind of interaction you can have over the internet, though amazing, pales in comparison to face-to-face interaction. Here’s what James had to say shortly after our first meeting:

“We’ve just completed nine solid days of development on Gambit. Now
that’s a “Codefest” alright! I thought I would share a little with
the curious about how I think it went…

Greg Brown flew out to my home in Oklahoma so we could work on
Gambit. It was nice to finally meet face-to-face. We’ve actually
programmed together on many projects these last couple of years, but
this was the first time we didn’t do it over the Internet. We’re
both grateful to the wonderful people at Ruby Central for giving us
that opportunity.

We’ve had practice, so we work pretty well together by now. Greg
would probably say that I’ve almost beat all his bad habits out of
him. ;) True or not, I know he’s always challenging me and pushing
me to new limits of what I can build. I think we both learned a lot
from working together on this project.

While there is a certain amount of altruism involved in mentoring a novice, it is an act that pays you back tenfold. Every concept that you can teach another person is one that you must master through and through, in ways that you’d never even scratch the surface of on your own.

Whether he realizes it or not, James revealed a secret to me during that hackfest, one that is to blame for all of my future successes in programming: Share your experiences and knowledge with others, and you win.

Fast forward to 2010, and obviously JEG2 and I have come a long way. We’ve both done talks throughout the US, and James even made it over to Japan last year. We’ve both published books on Ruby, and made more friends in the community than we ever imagined possible back in 2005. While part of me feels indebted to James still, I feel like I’ve paid it forward through my works. That made our mentorship role turn into a friendly rivalry, where we both keep pushing each other to get better at what we do, even if indirectly.

James had a big head start, so he normally beats me to the punch on things. Up until recently, I had only one thing under my belt that JEG2 didn’t: I had helped design and run a Ruby conference. Not one to be outdone, JEG2 has decided to level that playing field by running Red Dirt RubyConf.

Red Dirt RubyConf: Your ticket to Oz

If you like what I’ve been doing in Ruby, you now know that I owe it all to JEG2. Here’s the good news: He’s now inviting all of you to his neck of the woods to spend some time hacking, learning, and sharing. But you need to act fast, registration closes Friday, April 23!

Red Dirt RubyConf is a unique two day conference which blends training in with a uniquely themed single track session lineup. In it, you won’t just get to meet James, but also some other amazing Ruby people as well, including Dave Thomas and Jim Weirich.

The conference is run by James, his lovely wife Dana (who I could write a whole other post about), and some other great folks from the Oklahoma Ruby group. If you head out there, I can promise you that you’ll have an awesome time, learn a ton, and meet some amazing people.

Unfortunately, I won’t be able to attend, but I’ll sure be there in spirit. If you do make it there, tell James I said hi, and have a great time!

DISCLOSURE: I told James I’d be writing something to promote Red Dirt, but he didn’t ask me to do this, and he had no idea what I’d come up with. So I take 100% responsibility for the contents of this article.

rubyproblems.com is now live!

gregory.t.brown@gmail.com (Gregory Brown) — Wed, 07 Apr 2010 14:50:00 -0000

Back on St. Patrick’s day, I had made the following plug about rubyproblems.com at the bottom of my last post:

This site will provide realistic exercises that test your Ruby and general coding skills, along with detailed tutorials written in the style of RBP explaining possible solutions to the problems. The exercises will be free and open to everyone, the solutions we’ll be selling as nicely typeset, professionally edited, DRM-free PDF download

I am pleased to announce that this service did indeed launch yesterday, and that you can give it a try by heading over to rubyproblems.com now. If you need a bit more information, consider checking out this screencast or this announcement.

While this is still a bit of an experiment for us, we’re really excited to see how things turn out. Those who liked my book should love this site, which is why I’m announcing it here.

But now, the RBP blog should return to its non-marketing-hype self. In fact, I really want to find the time to write about why I think this is the most horrible syntax ever:

describe Array do
  its(:length) { should == 0 }
end

But I suppose that bit of flame bait will need to wait until a later day :)

Full Book Now Available For Free!

gregory.t.brown@gmail.com (Gregory Brown) — Wed, 17 Mar 2010 16:50:00 -0000

The last few weeks have been a wild ride. Starting in January, I’ve been releasing a chapter at a time here on the RBP blog, and many of those chapters were pretty well commented on. Just for the sake of completeness, here’s a link back to each of those posts:

At this point, I’d like to invite you to just grab the whole thing and read it at your leisure. This includes the final chapter “Skillful Project Maintenance”, as well as the three appendixes: “Writing Backwards Compatible Code”, “Leveraging Ruby’s Standard Library”, and “Ruby Worst Practices”.

Since the book is now open source, if you find something you don’t like about it, it’s your responsibility to help make it better! But if you do like it as it is, it might be more pleasant to read in print or on your Kindle, so you might want to pick it up from O’Reilly or Amazon.

Manuscript and Production Files

You can find the details about the manuscript and production files on github, now released under a Creative Commons license. It might be a little while before I can put some time into making this contributor friendly, but those looking to experiment or explore a bit are certainly welcome. The manuscript is written using the amazing asciidoc toolchain, which means the plaintext files you see in the git repository actually are what I used to generate something very close to the final print version of RBP.

So that’s pretty much it. While this will soon mark a new beginning, this figuratively and literally marks the last chapter in the history of RBP being “my book”. It’s yours now, do what you want with it. If you have interesting ideas that you want to run by me, I’d be happy to help however I can. Otherwise, enjoy and have fun!

But I finished the RBP book and I want more! What should I do?

Okay, you got me. I’ve been trying to take the “spirit” of RBP and apply it to two new projects, both a little more interactive in nature. For those who enjoyed the book, you may be interested in these two resources, both on the horizon.

The first is rubyproblems.com. This site will provide realistic exercises that test your Ruby and general coding skills, along with detailed tutorials written in the style of RBP explaining possible solutions to the problems. The exercises will be free and open to everyone, the solutions we’ll be selling as nicely typeset, professionally edited, DRM-free PDF downloads. We launch April 6th, 2010, but if you register now, you’ll get a free solution download when the site goes live. Whether you plan on buying content or not, the exercises themselves should provide great learning opportunities. If you liked RBP, you’ll definitely like Ruby Problems.

The second chance to bolster what you’ve learned in RBP is to attend a Compleat Rubyist training event. This is a joint venture between David A. Black, Jeremy McAnally, and myself that’s designed to provide a crash course in going from “knowing Ruby” to “being awesome at Ruby”. We did one of these in Tampa, FL in January, and it was blast. Our next one is scheduled for the Chicago area on June 18-19. We plan to do this as a regular thing, so if you can’t make it out to the Midwest, let us know where we should be going to find you! If you want a little more detail, you can check out my RubyTalk announcement about the upcoming event, and perhaps take a look at the Tampa retrospective here on the RBP blog.

Disclaimer

While I typically hate direct marketing, I am absolutely excited about the above two things, and so I felt compelled to mention them as good ‘next steps’ as you go beyond RBP. Doing these as commercial projects won’t ever make me rich, but it does allow me to take time away from billable work and really focus on producing awesome content and engaging with those who could learn something from it. So just know that whenever you send a bit of cash my way, you’re supporting a fellow hacker who will keep giving back at every opportunity.

But for those a bit weary of marketing gimmicks, the RBP blog will be going back to its pristine “no fluff, just stuff” program after today. It is a little hard to keep new content flowing, but we’ll do the best we can. Until then, happy hacking!

Ruby Tuesdays: RBP Chapter 7

gregory.t.brown@gmail.com (Gregory Brown) — Tue, 09 Mar 2010 14:00:00 -0000

If you’re reading this blog, you probably know that the Ruby Best Practices book exists. Even if you haven’t read it, you might have a sense for the sort of topics we cover based on the content you’ve seen on this blog. But now, everyone is going to get a chance to read RBP the way its meant to be read: as a conversation.

Progress To Date

We’ve already released the first five chapters, and discussion has been great. Be sure to check out these previous posts if you haven’t already:

Chapter 1: Driving Code Through Tests

Chapter 2: Designing Beautiful APIS / Chapter 3: Mastering the Dynamic Toolkit

Chapter 4: Text Processing and File Management

Chapter 5: Functional Programming Techniques

Chapter 6: When Things Go Wrong

These discussions should give you a sense of the wide range of ideas our readers have been sharing as a result of this ongoing book study.

How To Read RBP

Readers are encouraged to fight RBP as they read it, rather than just soaking up the information. Although I claim this book is about “Best Practices”, the only reason that is true is that it’s a result of countless conversations with folks who are deep in the Ruby trenches getting stuff done. The only way for RBP to remain current and relevant is to continue these discussions, using its content as a jumping off point for fresh ideas.

Today’s Topic

In 2010, programmers no longer have an excuse for not having a baseline understanding of multilingualization and localization techniques. But for those who have not yet taken a crash course, this chapter on “Reducing Cultural Barriers” provides a primer that should be good enough to get you started.

Go ahead and download Chapter 7 now. You’re encouraged to participate in the discussion here once you’ve had a chance to play with some of the ideas from the book. Don’t worry if it takes you more than a few days finish reading it, you can come back and comment any time.

Enjoy, and come back next Tuesday (2010.03.16) for the final chapter, “Skillful Project Maintenance”, which covers Ruby’s built in project management tools, and some of the techniques surrounding them.

For Your Consideration

O’Reilly is being really nice to me by letting me give away my book, especially considering that I haven’t quite cleared my advance yet (though I’m getting closer now, finally!)

If you like what you see, please buy the book. You can get it directly from the publisher or via amazon. I imagine that seeing decent sales numbers will encourage O’Reilly to keep moving towards making their content openly available, which would totally rock.

For those wishing to do interesting things with this material, note that it is released under the Creative Commons NC-SA license. This will become immensely more useful once the source documents for the manuscript are posted in late March, but if you’ve got any questions at all about this, you can ask me, and I’ll ask my publisher, and we’ll get back to you.

Ruby Tuesdays: RBP Chapter 6

gregory.t.brown@gmail.com (Gregory Brown) — Tue, 02 Mar 2010 07:00:00 -0000

Progress To Date

We’ve already released the first five chapters, and discussion has been great. Be sure to check out these previous posts if you haven’t already:

Chapter 1: Driving Code Through Tests

Chapter 2: Designing Beautiful APIS / Chapter 3: Mastering the Dynamic Toolkit

Chapter 4: Text Processing and File Management

Chapter 5: Functional Programming Techniques

These discussions should give you a sense of the wide range of ideas our readers have been sharing as a result of this ongoing book study.

How To Read RBP

Today’s Topic

This chapter on what to do “When Things Go Wrong” turned out to be one of the most enjoyable ones to write. Having a strong ability to troubleshoot and debug code is a powerful skill to have, and I do my best to share all my secrets here.

Go ahead and download Chapter 6 now. You’re encouraged to participate in the discussion here once you’ve had a chance to play with some of the ideas from the book. Don’t worry if it takes you more than a few days finish reading it, you can come back and comment any time.

Enjoy, and come back next Tuesday (2010.03.09) for Chapter 7, “Reducing Cultural Barriers”, which covers multilingualization (m17n) and localization (L10n) techniques.

For Your Consideration

O’Reilly is being really nice to me by letting me give away my book, especially considering that I haven’t quite cleared my advance yet (though I’m getting closer now, finally!)

If you like what you see, please consider buying the book. You can get it directly from the publisher or via amazon. I imagine that seeing decent sales numbers will encourage O’Reilly to keep moving towards making their content openly available, which would totally rock.

Code Massage

shortcutter@googlemail.com (Robert Klemme) — Sun, 28 Feb 2010 13:32:00 +0000

This article started out as a mental experiment and led to a surprising result. I post this mostly for the fun of it. But of course you can take something away from it. With that I do mean not only technical solutions. I believe firmly that a certain level of playfulness actually helps finding better solutions. The other ingredient you need is a certain eagerness for improvement which means to not be be content too early. OK, let’s start.

The scenario I started out was this: suppose you want to anonymize email addresses because you want to publish an email but not expose addresses to spam harvesting. Yet, you want to make sure that every address is always represented with the same replacement address in order to not change the meaning. You might immediately answer “We’ll need a Hash so we can efficiently find addresses that have been replaced already” – and so did I.

Java Style

If you are familiar with the Java standar library you know that java.util.Map has methods to check whether a key is present, to set values and to retrieve values. So after switching to Ruby you might be tempted to do it like this:

subst = {}

puts email.gsub(ADDR) {|match|
  if subst.has_key? match
    subst[match]
  else
    subst[match] = "<<MAIL #{subst.size}>>"
  end
}

Whenever we encounter an email address we must first check whether we have generated a replacement string for this address already. If not, we create a new one. No rocket science.

A little more sophisticated

You might find documumentation of method Hash#fetch when reading the library documentation which comes in handy because the block is invoked if the key is not present in the Hash. The code now looks a little shorter already:

subst = {}

puts email.gsub(ADDR) {|match|
  subst.fetch(match) {|k| subst[k] = "<<MAIL #{subst.size}>>" }
}

O||=erator

A similar thing can be achieved with the ubiquituous operator ||= which allows for conditional execution. In case you are not yet familiar with it you’ll find plenty of discussions in ruby-talk that revolve around this. The short summary is that a ||= b is equivalent to a || a = b and not a = a || b as you might be tempted to believe.

subst = {}

puts email.gsub(ADDR) {|match|
  subst[match] ||= "<<MAIL #{subst.size}>>"
}

The code has become even more shorter. But we are not finished yet!

Outsourcing

If you need that replacement in multiple places of your code you’ll likely put it into a method. However, if you want to replace different things (i.e. you need different regular expressions) which can match the same string you might want to outsource the generation of the replacement string so you can use it with different calls of gsub. You can of course do it with an additional method but there is a more elegant way to do it:

subst = Hash.new {|h,k| h[k] = "<<MAIL #{h.size}>>"}

puts email.gsub(ADDR) {|match| subst[match]}

We simply use Hash's default proc functionality for this. This is basically the same the fetch block does but now the code is attached to the Hash instance and not to the #fetch call.

You might wonder, how much further can we get? And indeed, this solution is probably the most idiomatic one and the one you see most frequent in seasoned Ruby developers’ code. It turns out though, that we can drive this further if we are prepared to use some newer Ruby features.

Getting tricky

Since Ruby 1.8.7 you can use anything as a block parameter to a method provided it implements a method to_proc which returns a Proc. Namely class Symbol implements this method in the following way: it returns a proc which needs at least one argument when called and invokes the given method with the remaining arguments on that instance. This allows for convenient operations like mapping data:

irb(main):001:0> (1..3).map &:to_s
=> ["1", "2", "3"]

One thing that bugged me was that the block handed to gsub above does nothing more than basically only forward the Hash lookup. With the new feature it should be possible to make the code a bit more concise. Luckily there are some core classes that implement to_proc already:

$ ruby -e 'ObjectSpace.each_object(Module) {|m| p m if m.instance_methods.include? "to_proc"}'
Method
Proc
Symbol

$ ruby19 -e 'ObjectSpace.each_object(Module) {|m| p m if m.instance_methods.include? :to_proc}'
Method
Proc
Symbol

We can exploit this fact and now we can write the code like this:

subst = Hash.new {|h,k| h[k] = "<<MAIL #{h.size}>>"}

puts email.gsub(ADDR, &subst.method(:[]))

Note: code changed after arthurschreiber’s comment.

Now, that looks ugly, doesn’t it? We should be able to do something about that, because after all we love Ruby for its elegance and clear syntax. Yes, we can!

Even shorter with a general solution

Since we can use any object why not provide a general mechanism for this case? Not only Hash but also Array and a lot more classes provide method [] as a general hook for lookup or exeution:

$ ruby19 -e 'ObjectSpace.each_object(Module) {|m| p m if m.instance_methods.include? :[]}'
Thread
Method
Proc
Struct::Tms
MatchData
Struct
Hash
Array
Bignum
Fixnum
Symbol
String

Now, let’s allow all these to be simply used as block parameters!

class Object
  def to_proc(m = :[])
    method(m).to_proc
  end
end

subst = Hash.new {|h,k| h[k] = "<<MAIL #{h.size}>>"}

puts email.gsub(ADDR, &subst)

Now we can just pass any Hash instance to gsub. The working logic for calculating our replacement string is now completely restricted to the Hash creation. This is a really elegant solution!

Golf

We can reduce the number of characters to type a bit more by throwing out the variable declaration and effectively turn this into a one liner:

puts email.gsub(ADDR, &Hash.new {|h,k| h[k] = "<<MAIL #{h.size}>>"})

I don’t think this is an improvement over the last variant but sometimes it helps driving things as far as possible to find out where in the process we reached the optimum.

The fun begins

Some classes do also implement method [] – we should be able to make good use of that as well. We might be tempted to create a lot of Struct instances via this method. It can be done but we have to do some tweaking because Struct.[] does not splat a single Array argument so we have to redefine it a bit:

Name = Struct.new :forename, :surname

# unfortunately this does not work with the default Struct.[]
def Name.[](a)
  new(*a)
end

p [
  ["John", "Doe"],
  ["John", "Cleese"],
  ["Mickey", "Mouse"],
].map(&Name)

# maybe a bit better:
def Name.create(a)
  new(*a)
end

p [
  ["John", "Doe"],
  ["John", "Cleese"],
  ["Mickey", "Mouse"],
].map(&Name.to_proc(:create))

I hope you had some fun reading this and more importantly playing around yourself. Trying out all things will certainly help you discover new ways and improve your skills.

As usually I have placed the code at github . If you look at it, please don’t get yourself hung up on the regular expression for matching email addresses. This is a whole topic of its own and I just hacked something together to make the code work.

Ruby Tuesdays: RBP Chapter 5

gregory.t.brown@gmail.com (Gregory Brown) — Tue, 23 Feb 2010 15:00:00 -0000

Progress To Date

We’ve already released the first four chapters, and discussion has been great. Be sure to check out these previous posts if you haven’t already:

Chapter 1: Driving Code Through Tests

Chapter 2: Designing Beautiful APIS / Chapter 3: Mastering the Dynamic Toolkit

Chapter 4: Text Processing and File Management

These discussions should give you a sense of the wide range of ideas our readers have been sharing as a result of this ongoing book study.

How To Read RBP

Today’s Topic

If you’re willing to participate in the discussion afterwards, go ahead and download Chapter 5 now. Don’t worry if it takes you more than a few days finish reading it, you can come back and comment any time.

This chapter was supposed to be my “fun” topic, but turned out to be more practical than I expected. I had originally started out writing about how to do functional programming in Ruby, but later realized that was an incredibly bad idea. Ruby isn’t a great functional programming language, but certain functional programming techniques do still pay off. This chapter is about those techniques.

Enjoy, and come back next Tuesday (2010.03.02) for Chapter 6, “When Things Go Wrong”.

For Your Consideration

O’Reilly is being really nice to me by letting me give away my book, especially considering that I haven’t quite cleared my advance yet. So if you like what you see, and want to be able to read it all now instead of waiting several more weeks for it, please consider buying the book. You can get it directly from the publisher or via amazon. While I do make a little more money when you buy from O’Reilly, I wasn’t expecting to get rich off of RBP, so don’t feel bad buying the discounted copies from Amazon. But I’m pretty sure that seeing a spike in sales would encourage them to do more open source books, so… keep that in mind.

For those wishing to do interesting things with this material, note that it is released under the Creative Commons NC-SA license. This will become immensely more useful once the source documents are posted in late March, but if you’ve got any questions at all about this, you can ask me, and I’ll ask my publisher, and we’ll get back to you.

The Complete Numeric Class

shortcutter@googlemail.com (Robert Klemme) — Sat, 20 Feb 2010 14:25:00 +0000

As announced in the previous article we will look at a complete number class today. I will use the example of a integer number which, when printed, will show up as hex number (as opposed to the decimal presentation of Fixnum and relatives). As before the main point is not sophisticated logic or usefulness of the class. Instead I will keep the logic simple so we can focus on the aspects I try to convey with today’s article:

conversions into HexNum
conversions of HexNum to other types
equivalence vs. comparability
type coercion
math and operator overloading

For completeness reasons other aspects mentioned in the previous article will be implemented as well but I won’t discuss them in detail here.

Conversions into `HexNum`

The first step into the world of HexNum is creation of a new object of course. This part is not that much interesting and so I will only gloss over the implementation. I have provided a few ways:

constructor
conversion methods (to_hex),
explicit method (similar to Integer()).

These are shown in the code snippet below:

class HexNum < Numeric

  # Create a new instance from an int or String.
  def initialize(val)
    case val
    when String
      @i = parse_string(val)
      @s = val.frozen? ? val : val.dup.freeze
    when Numeric
      @i = val.to_i
    else
      raise ArgumentError, 'Cannot convert %p' % val
    end
  end

end

# more conversions

def HexNum(i)
  HexNum.new(Integer(i))
end

class Object
  def to_hex
    HexNum.new(to_i)
  end
end

As you can see, a HexNum consists of an integer value and optionally a String. The integer value is the mandatory part which will be used in most methods while the String is just an helper intended to make conversions to String more efficient (I did not make any measurements though – I mainly wanted to make the class a tad more interesting).

Method Object.to_hex is based on the presence of method to_i. This is debatable. Basing this conversion on method to_int is as reasonable IMHO. As you can see, the set of classes which implement to_i differs from those which implement to_int:

irb(main):002:0> s1 = []; s2 = []
=> []
irb(main):003:0> ObjectSpace.each_object(Module) do |m|
irb(main):004:1* s1 << m if m.instance_methods.include? :to_i
irb(main):005:1> s2 << m if m.instance_methods.include? :to_int
irb(main):006:1> end
=> 407
irb(main):007:0> s1
=> [Complex, Rational, Process::Status, Time, File, ARGF.class, IO, Bignum, Float, Fixnum, Integer, String, NilClass]
irb(main):008:0> s2
=> [Complex, Rational, Bignum, Float, Fixnum, Integer, Numeric]

The rationale behind this is that only types which can be used as integers should implement to_int. Method to_i is merely a conversion method which turns “something” into an int. I chose to base to_hex on to_i because this increases the number of cases where you can immediately use a HexNum. If you need more strict argument checks, you can use HexNum() (the method) which is modeled similar to Integer() and in fact uses it internally in order to benefit from its argument checking.

You might have expected HexNum to inherit Integer. Actually, that’s what I would have done, too. But, it turns out, if you make a class inherit Integer you cannot create instances of it any more:

irb(main):007:0> class X < Integer
irb(main):008:1> end
=> nil
irb(main):009:0> X.new
NoMethodError: undefined method `new' for X:Class
        from (irb):9
        from /usr/local/bin/irb19:12:in `<main>'
irb(main):010:0> class X;end
=> nil
irb(main):011:0> def X.new; allocate; end
=> nil
irb(main):012:0> X.new
TypeError: allocator undefined for X
        from (irb):11:in `allocate'
        from (irb):11:in `new'
        from (irb):12
        from /usr/local/bin/irb19:12:in `<main>'
irb(main):013:0> Integer.class
=> Class
irb(main):014:0>

This is a bit unfortunate since Integer would be the proper base class for our HexNum. Inheriting Numeric is the second best we can do.

Conversions to other types

These conversions are done by the typical set of to_xyz methods:

  # conversions

  def to_s
    @s ||= (@i < 0 ? '-0x%x' % -@i : '0x%x' % @i).freeze
  end

  def to_i
    @i
  end

  alias to_int to_i

  def to_hex
    self
  end

Please note that for efficiency reasons class HexNum also contains a method to_hex. Other than that there are really no surprises here. Conversion to string may look a bit complicated but it’s really just the different treatment of negative and positive values plus caching of the converted String. The #freeze just ensures that changes of the cached value outside the class cannot backfire. If we weren’t using #freeze here, we would have to create a new String instance for every to_s invocation which would defy the whole point of caching the value.

Mutability

An important point to note is that the class is immutable. While this is not mandatory creating a mutable class does not blend well with how Ruby handles arithmetic operators. If you implement operator + as in place modification you will get all the issues typically caused by aliasing, i.e. using the same instance from different places in code. As they say, “When in Rome do as the Romans do” – so we will stick with the convention used throughout Ruby’s core library and make our HexNum immutable too. That way we can use instances of HexNum where we would otherwise have used Fixnums or Bignums. And after all this was the aim of the exercise: to demonstrate how to create a class that seemlessly blends with all the other numeric types of Ruby’s core and standard library.

Equivalence vs. Comparability

Now we slowly get to the more interesting topics. Every class has comparison for equivalence through method #eql? and operator == (see also the discussion of the topic in the previous article). For seamless blending of HexNum with other numeric types comparison with == should only look at the numerical value so that HexNum(1) and 1.0 compare true. However since we do not have influence on Fixnum's and Float's implementation of === and since equivalence is a symmetric relation (i.e. a == b must return the same as b == a) we can only establish equivalence with other HexNum instances. I think this is unfortunate. I can only speculate about Matz’s reasons to not use #coerce in this situation: I assume he did it this way for performance reasons since these types of comparisons are very frequent in a program.

So, this is how equivalence checking code looks like:

  # equivalence

  def eql?(num)
    self.class.equal?(num.class) && @i == num.to_i
  end

  alias == eql?

No big surprises here. You’ll note the type check which is necessary because of the aforementioned implementation details of core classes.

Things stand dramatically different with respect to comparison operator <=>. The definition of this operator’s semantics was nicely presented by Marc in his comments to the last article. Basically, <=> must return nil if classes of compared instances are differnt. But what is this?

irb(main):001:0> 1 <=> 1.0
=> 0
irb(main):002:0> 1 <=> 1.5
=> -1
irb(main):003:0> 0.5 <=> 1
=> -1
irb(main):004:0> 0.5 <=> 1 << 40
=> -1
irb(main):005:0>

Numeric classes allow for comparison across a wide range ot types. Can we make HexNum blend in here? It turns out, that we can. This is the time to introduce a functionality that took me a while understand initially. But this is at the heart of Ruby’s operator overloading and probably the single most important point to consider when implementing numeric types.

What’s this `#coerce` thingy?

When talking about equivalence it might have occurred to you that the behavior of == should depend not on a single class but actually on the type of both arguments. The rule whether two objects are equivalent include’s both object’s class and statements about state of both instances (in case of HexNum for example, both must have the same integer value). Now, Ruby – like many object oriented languages – has ‘single dispatch’ which means only the type of the receiver (the object to the left of the dot in a method call or self) determines which method is actually used. This works remarkably well most of the time but there are cases where you rather want to make the dispatch (and thus the decision which code is executed) depend on the receiver and one or more of the method’s arguments. Binary operators are a typical case of this.

The usual solution in other languages is to overload a method based on argument types and in Ruby you would likely employ some form of type checking similar to what I have done in HexNum's method #initialize. However, this approach has a fundamental drawback: when writing class A the author must know all classes B, C and D that are used as argument types. This hurts extensibility seriously since for every new numeric type the code of older types needs adjustment. While this would be possible due to Ruby’s dynamic nature this is on the one hand tedious and it would have performance implications on the other hand because you would likely be implementing those “enahancements” in Ruby (and not C) and methods would have to check for more and more types.

Ruby’s solution to this is very elegant (as many other aspects of the language): whenever a methods receives a type different from its own type it asks that type to do a conversion so the operation can finally be handled. It does that via method #coerce which receives the caller as argument so the other (presumably newer) type can find an adequate conversion. That way, only types written later need to know about older types (e.g. everything in the core and standard library). Let’s look at how #coerce works by using it on some standard types:

irb(main):001:0> 1.coerce 2
=> [2, 1]
irb(main):002:0> 2.0.coerce 1
=> [1.0, 2.0]
irb(main):003:0> 1.coerce 2.0
=> [2.0, 1.0]

The result of invoking #coerce has two interesting properties:

The types of returned values are identical.
The return value is really an Array with the order reversed compared to the invocation (the receiver is the second object in the array).

While we immediately see the usefulness of 1 (every class should know how to handle instances of itself) the second property irritated me at first. But it does make sense if you consider that the argument to #coerce is really the original receiver of the method call.

I don’t want to hide some inconsistencies of the standard functionality from you:

irb(main):004:0> 1.coerce 1<<40
=> [1099511627776.0, 1.0]
irb(main):005:0> (1<<40).coerce 2
=> [2, 1099511627776]
irb(main):006:0> (1<<40).coerce 3.0
TypeError: can't coerce Float to Bignum
        from (irb):6:in `coerce'
        from (irb):6
        from /usr/local/bin/irb19:12:in `<main>'
irb(main):007:0> 3.0.coerce 1<<40
=> [1099511627776.0, 3.0]
irb(main):008:0> 1<<40
=> 1099511627776
irb(main):009:0> (1<<40) + 3.0
=> 1099511627779.0

You can add Bignum and Float but #coerce fails on one direction. If anybody can provide a sensible explanation of this please let us know. For the moment I am inclined to believe that it’s due to the fact that implementations of core types are done in C and there are probably some corners cut.

Now look at a simple example which demonstrates the mechanics of #coerce:

$ ruby19 <<CODE
> a = 1
> b = Object.new
> def b.coerce(x) [x,2] end
> p b
> set_trace_func lambda {|*a| p a}
> puts a + b
> CODE
#<Object:0x10028284>
["c-return", "-", 5, :set_trace_func, #<Binding:0x10027dd0>, Kernel]
["line", "-", 6, nil, #<Binding:0x10027adc>, nil]
["c-call", "-", 6, :+, #<Binding:0x10027724>, Fixnum]
["call", "-", 3, :coerce, #<Binding:0x1002721c>, #<Object:0x10028284>]
["line", "-", 3, :coerce, #<Binding:0x10026e48>, #<Object:0x10028284>]
["return", "-", 3, :coerce, #<Binding:0x10026924>, #<Object:0x10028284>]
["c-call", "-", 6, :+, #<Binding:0x100263e4>, Fixnum]
["c-return", "-", 6, :+, #<Binding:0x10025fbc>, Fixnum]
["c-return", "-", 6, :+, #<Binding:0x10025a28>, Fixnum]
["c-call", "-", 6, :puts, #<Binding:0x10025424>, Kernel]
["c-call", "-", 6, :puts, #<Binding:0x10024f54>, IO]
["c-call", "-", 6, :to_s, #<Binding:0x10024854>, Fixnum]
["c-return", "-", 6, :to_s, #<Binding:0x10023db8>, Fixnum]
["c-call", "-", 6, :write, #<Binding:0x10023878>, IO]
3["c-return", "-", 6, :write, #<Binding:0x10022f9c>, IO]
["c-call", "-", 6, :write, #<Binding:0x10022ae8>, IO]

["c-return", "-", 6, :write, #<Binding:0x10022768>, IO]
["c-return", "-", 6, :puts, #<Binding:0x1002227c>, IO]
["c-return", "-", 6, :puts, #<Binding:0x10021b0c>, Kernel]

As you can see in line 6 Fixnum's operator + is invoked. Since Fixnum does not know how to add an Object it invokes its #coerce. For simplicity reasons we always return a fixed value of 2 for the Object instance and as you can see Fixnum's operator + is invoked again and now returns immediately. The final result is properly calculated as 3.

Now let’s look at HexNum's method #coerce given the long introduction it looks surprisingly simple:

  # coercion

  def coerce(o)
    [HexNum.new(o.to_int), self]
  end

  # comparability

  include Comparable

  def <=>(o)
    case o
    when HexNum
      @i <=> o.to_i
    when Numeric
      @i <=> o
    else
      a, b = o.coerce(self)
      a <=> b
    end rescue nil
  end

We just try to create a new HexNum if the other type can be converted to an integer and go from there. In other cases you might have to check argument types to apply specific conversions but for our HexNum this is enough to ensure functionality.

Now you can also see how comparison works for HexNum: if the type is known we rely on comparison functionality of the standard library. And in order to make HexNum also work with classes written later #coerce is invoked on all other arguments. A last detail: if #coerce fails it is supposed to throw an exception. Since the contract of <=> prohibits that we add a rescue nil so we always end up with a proper comparison result.

Math and Operator Overloading

Now that we understand which role #coerce plays in Ruby’s operator land we can implement all the operators available in a way as to ensure we always get proper results. In our case this means, we always want to have a HexNum result with proper int math.

There is one thing that we need to ensure for all operators: they must handle their own type directly. If we fail to do that, we may end up in infinite recursion:

irb(main):001:0> class X
irb(main):002:1>   def +(o)
irb(main):003:2>     case o
irb(main):004:3>     when Integer
irb(main):005:3>       o + 1
irb(main):006:3>     else
irb(main):007:3*       a, b = o.coerce(self)
irb(main):008:3>       a + b
irb(main):009:3>     end
irb(main):010:2>   end
irb(main):011:1>
irb(main):012:1*   def coerce(o)
irb(main):013:2>     [X.new, self]
irb(main):014:2>   end
irb(main):015:1> end
=> nil
irb(main):016:0> x = X.new
=> #<X:0x10145c50>
irb(main):017:0> x + 1
=> 2
irb(main):018:0> 1 + x
SystemStackError: stack level too deep
        from (irb):7:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
... 7229 levels...
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):18:in `+'
        from (irb):18
        from /usr/local/bin/irb19:12:in `<main>'irb(main):019:0> x + x
SystemStackError: stack level too deep
        from (irb):7:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
... 7229 levels...
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):19
        from /usr/local/bin/irb19:12:in `<main>'irb(main):020:0>
irb(main):021:0*

So let’s look at operator +:

  def +(o)
    case o
    when Integer
      HexNum.new(@i + o)
    when Numeric
      HexNum.new(@i + o.to_i)
    else
      a, b = o.coerce(self)
      a + b
    end
  end

We know that all Integer instances do return an Integer instance from their implementation of operator + so we can simply use that on our instance varible containing the numerical value. For all other Numeric instances (including HexNum itself) we can work with the instance value converted to int. All others are responsible for returning appropriate values for themself and the HexNum instance so the math can work properly.

Operator * works a bit differently because for proper results of multiplication we need to use the numeric value as isand convert later:

  def *(o)
    case o
    when HexNum
      HexNum.new(@i * o.to_i)
    when Numeric
      HexNum.new(@i * o)
    else
      a, b = o.coerce(self)
      a * b
    end
  end

Of course we could refactor common parts into an operator implementing method but I did not want to get too fancy here and rather demonstrate how particular math operations might need different treatment. For other operators we rely on the argument’s compatibility to an integer since other types do not really make sense here:

  # asymmetric binary operators

  def <<(o)
    HexNum.new(@i << o.to_int)
  end

  # bit operators

  def &(o)
    HexNum.new(@i & o.to_int)
  end

Summary

We have seen how a numeric class needs to be implemented differently from other classes that try to be complete. The major point is to provide #coerce with meaningful semantics and implement all operators. As always you can find the complete code at github. Have fun playing around with it!

Ruby Tuesdays: RBP Chapter 4

gregory.t.brown@gmail.com (Gregory Brown) — Tue, 16 Feb 2010 15:00:00 -0000

Progress To Date

We’ve already released the first three chapters, and discussion has been great. If you haven’t seen them already, check out the first week’s discussion on testing, and then head over and look at the double header on API design and metaprogramming. These discussions should give you a sense of the wide range of ideas our readers have been sharing as a result of this book study.

How To Read RBP

Today’s Topic

If you’re willing to participate in the discussion afterwards, go ahead and download Chapter 4 now. Don’t worry if it takes you more than a few days finish reading it, you can come back and comment any time.

While chapters 2 and 3 were about somewhat subtle features, this chapter is as concrete as they come. Those with some Perl blood in them should find it familiar, as it’s all about how to munge text and move stuff around on your filesystem. But those who struggle with the scripting side of Ruby should find a number of techniques that will make their job easier.

Enjoy, and come back next Tuesday (2010.02.23) for Chapter 5, “Functional Programming Techniques”.

For Your Consideration

For those wishing to do interesting things with this material, note that it is released under the Creative Commons NC-SA license. This will become immensely more useful once the source documents are posted in late March, but if you’ve got any questions at all about this, you can ask me, and I’ll ask my publisher, and we’ll get back to you.

RBP is useless without you

gregory.t.brown@gmail.com (Gregory Brown) — Fri, 12 Feb 2010 05:24:00 -0000

UPDATE 2010.02.13: Lots of folks have taken the initiative to comment after reading this, and others have pointed out that they indeed need more time before they can comment. But since people seem generally interested in seeing Ch4 sooner rather than later, I’ll release it on Tuesday. Coincidentally, there is a great post I found on HN that does a better job than I did summarizing how tech books (especially RBP) ought to be read. If you haven’t seen it yet, go read Kenjitsu.

Since I’ve posted the first three chapters, my amazon ratings have gone way up, and I’ve seen lots of people mention the book on twitter. This is great for marketing, but as you may know, I care much more about the actual content here.

Things started out great! We had an awesome discussion about testing two weeks ago and I expected to see the same on APIs and metaprogramming. But as it turned out, only two brave souls offered up their thoughts, even though lots of people downloaded the content. That was a big disappointment to me, and now I’m trying to guess at what went wrong.

One possibility is that I’ve oversaturated folks, so I’m giving everyone an extra week to comment on Ch2-3 before I put out the “Text Processing and File Management” chapter. Hopefully, by next Friday we’ll have a good discussion going there, and the Ch4 release will just keep the ball rolling.

But far more grave, is my fear that some folks might be too timid to express their thoughts about the material, perhaps because it’s in book form rather than written up in some generic blog post. This to me, is exactly what will prevent them from getting anything worthwhile out of my book. So if you’ve been holding back comments just because you didn’t want to challenge RBP’s “bookiness” out in the open, please don’t. I can handle all sorts of criticism, and the book will benefit as a result of it.

I am very serious about the idea that “Best Practices” really don’t have much value in a vacuum. I’m certain that the benefits that my book can offer its readers are maximized in open conversation. This is why I made sure to build an eventual open content licensing into RBP’s contract, and why I’m releasing free chapters now.

RBP is useless without you, so please, participate.

Weekend Reading: RBP Chapters 2-3

gregory.t.brown@gmail.com (Gregory Brown) — Fri, 05 Feb 2010 14:00:00 -0000

If you missed last week’s post, which introduces chapter 1, you’ll want to check that out. There was some great discussion among readers that add a lot to what I had to say in the book. I expect we’ll see the same thing happen this time around.

Today’s Topics

This week, I’m releasing two chapters, Designing Beautiful APIs and Mastering the Dynamic Toolkit. Incase the latter sounds familiar, it is indeed the book’s sample chapter. But it also happens to be my favorite chapter in the book, so I didn’t want to skip over it without a chance for discussion.

So remember, if you click those links, your responsibility is to participate in the discussion here by offering up your thoughts about what you’ve read. Best practices cannot be rules that are simply absorbed and followed if they are to be of any use… instead, they must be extracted from and influenced by those who work with the real problems that they are meant to solve.

It would be great to hear your experiences with the topics these chapters cover, even if they’re not directly related to the book content.

For Your Consideration

O’Reilly is being really nice to me by letting me give away my book, especially considering that I haven’t quite cleared my advance yet. So if you like what you see, and want to be able to read it all now instead of waiting eight weeks for it, please consider buying the book. You can get it directly from the publisher or via amazon. While I do make a little more money when you buy from O’Reilly, I wasn’t expecting to get rich off of RBP, so don’t feel bad buying the discounted copies from Amazon. But I’m pretty sure that seeing a spike in sales would encourage them to do more open source books, so… keep that in mind.

For those wishing to do interesting things with this material, note that it is released under the Creative Commons NC-SA license. This will become immensely more useful once the source documents are posted in late March, but if you’ve got any questions at all about this, you can ask me, and I’ll ask my publisher, and we’ll get back to you.

Anyway that’s all for now, enjoy the chapters and let me know what you think!

Sexp for Rubyists

judofyr@gmail.com (Magnus Holm) — Thu, 04 Feb 2010 21:00:00 +0000

Sorry to interrupt you, but you’ve totally lost me. What is this “Sexp”
you’re speaking of? I’ve heard it before, but never quite understood it…

Oh, don’t feel sorry! It’s still quite esoteric for Rubyists.

Yeah…

Okay. Let’s start at the beginning. Lisp!

With all the parenthesis?

Bingo! Have you tried it?

Not really. It seems a little too “hardcore”, if you know what I mean?

Ah, yes. It’s just a neat little language, nothing to be afraid of. Let me
show you an example:

(define (plus-five n)
  (+ n 5))

It’s a method that adds five to the argument? Like this:

def plus_five(n)
  n + 5
end

Yep, but more importantly: It’s also an example of S-expressions. Because Sexp
is really just a data format.

Data format? Like YAML?

Just like YAML. It has support for numbers, symbols, pairs, lists and nil.

Really?

Let me explain a few more things. You see, Lisp wasn’t originally supposed to
be programmed in S-expression. Nope, the idea was to have complex,
human-readable M-expressions which should manipulate data in S-expressions.

    Data  =  S-expressions
    Code  =  M-expressions

Another idea was that M-expressions could easily be compiled into
S-expressions. So the computer itself would only work with S-expressions,
while humans could focus on M-expressions.

    M-expressions  -->   S-expressions

However, the first Lisp implementations only accepted S-expressions since it
was easier to start from there and rather add M-expression support later.

So what happend with these M-expressions?

They were “receded into the indefinite future” as the creator, John McCarthy,
said. It turned out that S-expressions were readable enough.

Go on, please.

Now Lisp suddenly used S-expressions for both code and data. The interesting
thing is how code was represented in Lisp. It’s in fact very simple: The first
element in the list is the operator/function and the rest is the
operands/arguments:

    (+ 1 2)
    (plus-five 5)
    (factorial 10)

This makes Lisp a very simple language to implement. You need very few
primitives before you have a fully working Lisp system. In fact, I would
recommend watching James Coglan’s screencast. There isn’t any audio since he
used it in a talk he gave, but it’s still fairy interesting.

Scheme interpreter in 15 minutes from James Coglan on Vimeo.

Wow. I must definitely check out Lisp.

Indeed. He’s also written a “real” Lisp implementation in Ruby if you’re
interested: Heist. Oh, and I’ll also have to
recommend The Little Schemer by
Daniel P. Friedman and Matthias Felleisen, it has inspired me in several
ways…

I will, but back to Sexp?

No. Let’s talk about AST.

AST? While I certainly appreciate these digressions, (whispers_) others
might be a little bored, you know?_

Don’t worry, this is pretty essential. An Abstract Syntax Tree is a tree
representation of source code. Compilers and interpreters never work directly
on the source code, but parses it into an AST which is much easier work with.
Your Ruby example above could look something like:

[Def,
  plus_five
  [n]
  [Call, n, +, [[Lit, 5]]]]

The basic idea here is that we only deal with nodes like Def, Call and Lit,
and every node has a set of children. In this tree the nodes are built like
this:

[Def, method_name, arguments, body] # => def method_name(*arguments); body; end
[Lit, number]                       # => (integer)
[Call, receiver, method_name, arguments] # => receiver.method_name(*arguments)

I think I get your point. This is very similar to S-expressions!

(def plus_five (n)
  (call n + 5))

Yes! It turns out that S-expressions are excellent to represent abstract
syntax trees. And when compiler gurus talk about S-expressions, they don’t
talk about the file format, they talk about the way Lisp represent it’s code:
The first element in the list is the operator/function and the rest are the
operands/arguments.

So S-expressions themselves aren’t very exciting, it all depends on what you
put in them?

Exactly. The Sexp-class basically looks like this:

class Sexp < Array
  def sexp_type
    self[0]
  end
  
  def sexp_body
    self[1..-1]
  end
end

A glorified Array. That’s all there is to it.

And what do people put in these Sexps?

Let me show you an example. First, gem install ruby_parser. Then:

require 'ruby_parser'
require 'pp'

pp RubyParser.new.parse(<<-EOF)
def plus_five(n)
  n + 5
end
EOF

Interesting:

s(:defn,
 :plus_five,
 s(:args, :n),
 s(:scope,
   s(:block,
     s(:call, s(:lvar, :n), :+, s(:arglist, s(:lit, 5))))))

This is the syntatic strucure of our code. It’s not possible to reconstruct
the original snippet character-by-character; you only get what the
compiler/interpreter/virtual machine actually sees. And often that’s exactly
the only thing we want.

Aha. Who decided this structure and these names by the way?

Well, I’m not quite certain, but I believe ParseTree by Ryan Davis started it
all.

Oh, ParseTree sounds familiar.

It extracts the raw AST that Ruby 1.8 uses internally. It’s not very elegant,
so Ryan also wrote UnifiedRuby which cleans it up a bit. RubyParser returns
such a cleanup up Sexp.

And what do people use these cleaned up Sexps for?

There’s plenty of projects floating around, here’s some I’ve heard of:

Flog analyzes your code and shows how complex your methods are, and which you should refactor.
Flay analyzes your code for structural similarities, ready to be DRYed out!
Roodi warns you about design issues.
Reek detects code smells.
Saikuro is a “cyclomatic complexity analyzer”.
Ambition let’s you write SQL statements as Ruby blocks.
Parkaby compiles Markaby-like code into super-fast code.
RewriteRails adds syntactic abstractions (like Andand) to Rails projects without monkey-patching.

Cool, that’s more than I thought.

And if more people knew about them, I guess there would be even more.

Let me recap:

For Rubyists, Sexp is an Array-ish list where the first element is the
operator/function and the rest is the operands/arguments. The arguments can
also contain other Sexp. When I think about it, that’s probably the most
important thing: They are nested.

I guess it’s mostly useful when we’re actually manipulating code that
executes. Like source code or templates?

Exactly!

Ah, excellent. Why were we talking about Sexps again?

Hm.. Now that’s a good question. I simply don’t remember. Let’s talk about it
another time, shall we?

Weekend Reading: RBP Chapter 1

gregory.t.brown@gmail.com (Gregory Brown) — Fri, 29 Jan 2010 14:32:00 -0000

For the next 8 weeks, I will post a chapter from RBP every Friday. You don’t need to pay me a cent if you download it, but what I do want you to do is leave a comment here sharing your thoughts once you’ve finished reading. This way, we can use RBP’s content as a jumping off point for conversation, rather than treating it like some sort of static rulebook. Through this process, we can discover what our common practices really are. Over time, I will incorporate these ideas back into the manuscript, helping to keep the book up to date and relevant.

Today’s Topic

The first chapter I am releasing is Driving Code Through Tests, which covers a number testing philosophies and techniques. It aims to be framework agnostic, but leans on the Test::Unit API for its examples. The core idea of this chapter is to expose folks to the “spirit” of testing, rather than just the mechanics of it.

Upon re-reading this chapter in the winter of 2010 having written it in the spring of 2008, there are definitely a couple things I think differently about now. But as it turns out, I was more surprised by how much hasn’t changed, and how well this chapter seems to have stood the test of time. Of course, you may disagree, and I encourage you to tell me so if that’s the case. :)

Ready to get started? Go ahead and download the chapter now. Remember, if you click that link, I’m expecting you to post back here with your thoughts (or alternatively, write a response on your own blog and send me a link). If you find a technical error, you can report it via Github issues.

For Your Consideration

O’Reilly is being really nice to me by letting me give away my book, especially considering that I haven’t quite cleared my advance yet. So if you like what you see, and want to be able to read it all now instead of waiting eight weeks for it, please consider buying the book. You can get it directly from the publisher or via amazon. While I do make a little more money when you buy from O’Reilly, I wasn’t expecting to get rich off of RBP, so don’t feel bad buying the discounted copies from Amazon. But I’m pretty sure that seeing a spike in sales would encourage them to do more open source books, so… keep that in mind.

For those wishing to do interesting things with this material, note that it is released under the Creative Commons NC-SA license. This will become immensely more useful once the source documents are posted in late March, but if you’ve got any questions at all about this, you can ask me, and I’ll ask my publisher, and we’ll get back to you.

Anyway that’s all for now, enjoy the chapter and let me know what you think!

Reflections from "The Compleat Rubyist"

gregory.t.brown@gmail.com (Gregory Brown) — Mon, 25 Jan 2010 15:36:00 -0000

Ever since 2005, I’ve been attending and speaking at Ruby events. But the one thing I always wanted to do was run training sessions. Thankfully, over the last couple years, LSRC has allowed me to get my feet wet — but only as one of many excellent choices in a lineup of Ruby’s best and brightest. Until last weekend, I had never participated in professional training event that stood on its own.

What eventually became “The Compleat Rubyist” training, was originally an idea for “The Compleat Rubyist” book promotion. Jeremy McAnally, David A. Black, and myself have all written Ruby books, and they seem to compliment each other well. But in doing this, we realized that it would be more valuable, and more fun, to bring our show out on the open road rather than package it up in a box set.

Thus, our training event was born, and we set our sights on Tampa, FL for its first incarnation.

The Training

Instead of having a linear, specific set of bullet points to teach from, we broke our training into four half-day sessions that are representive of some of the most important topics in the Ruby ecosystem. This gave us a lot of leeway to adjust our materials and discussions based on individual student needs. The topics we picked were “Ruby Versions and Implementations”, “Metaprogramming”, “Testing”, and “Style and Substance”. Though these may seem a bit amorphous on first sight, they do a pretty good job of covering four key topics that a budding Rubyist must understand and come to appreciate.

Here are just a few thoughts from each of the modules, after seeing how things played out in class.

Ruby Versions and Implementations

One of the main things that keep people from trying out the alternative Ruby implementations and the various versions of Ruby is that they’re understandably afraid of messing up their systems. So David took a three-pronged approach to helping students overcome that fear.

The easiest stepping stone, perhaps, was to use ruby-versions.net. This great resource (maintained by David), lets you try out all sorts of ruby versions and implementations via a shell account so that you don’t need to worry about touching anything on your own machine. While this isn’t meant to be a long term solution, it’s probably the easiest way that you can try out pretty much any version of Ruby without having to go hunt down, install, and configure their packages.

In addition to this, David showed folks how to make use of multiruby (a tool that’s part of ZenTest). My favorite example of the whole day was one showing that using Ruby 1.8.7, it’s possible to write code that works on neither 1.8.6 nor 1.9.1, which lead into a whole other interesting discussion:

  $ multiruby -e "p 'hello'.map(&:upcase)"

  VERSION = 1.8.6-p388
  CMD     = ~/.multiruby/install/1.8.6-p388/bin/ruby -e p 'hello'.map(&:upcase)

  -e:1: wrong argument type Symbol (expected Proc) (TypeError)

  RESULT = pid 3988 exit 1

  VERSION = 1.8.7-p249
  CMD     = ~/.multiruby/install/1.8.7-p249/bin/ruby -e p 'hello'.map(&:upcase)

  ["HELLO"]

  RESULT = pid 3991 exit 0

  VERSION = 1.9.1-p378
  CMD     = ~/.multiruby/install/1.9.1-p378/bin/ruby -e p 'hello'.map(&:upcase)

  -e:1:in `<main>': undefined method `map' for "hello":String (NoMethodError)

  RESULT = pid 3995 exit 1

  TOTAL RESULT = 2 failures out of 3

  Passed: 1.8.7-p249
  Failed: 1.8.6-p388, 1.9.1-p378

We also took a look at rvm , which is another neat tool for managing multiple Ruby versions. While not necessarily a tool for every possible need, it is a very easy way of managing multiple rubies side by side without trashing your system-wide configurations.

Between these three tools, the students ended up with all they needed to explore the different implementations and versions for themselves. But that was really just the beginning. David then went on to discuss the various peculiarities across the board, and was sure to discuss where certain implementations or versions excel, and where they fall behind.

I’m happy to say that while very few of the students in our class had even tried Ruby 1.9 before our training, every one of them had the opportunity to do so before this session ended.

Meta-programming

At one point in time, everyone using Ruby thought meta-programming was some sort of magic. But in a post-Rails world, it is increasingly easy to help folks understand that all of this “voodoo” we do is really just making use of public method calls provided by Ruby objects that allow you to modify Ruby itself. Once a decent understanding of Ruby’s object model is established, all of the rest of the puzzle pieces just seem to fall into place.

This session had a lot of great discussions, and several exercises to encourage students to actively learn the tricks of the trade. Along the way, we stumbled across this interesting distinction between undef_method and remove_method:

  class A
    def foo
      "bar"
    end
  end

  class B < A
    def foo
      "baz"
    end

    remove_method :foo
  end

  class C
    def foo 
      "bar"
    end
  end

  class D < C
    def foo
      "baz"
    end

    undef_method :foo
  end

  p B.new.foo # "bar"

  p D.new.foo # raise NoMethodError

Testing

Fortunately, virtually all of our students had experience with test driven development of some sort or another. I think that’s a big difference between the Ruby ecosystem today and of say, 5 years ago. But similar to versions and implementations, there are so many choices out there that it’s hard to know what’s what. Jeremy kicked off this module with a state of the union address, breaking down the various different philosophies and toolchains there are out there, and what each of them bring to the table.

Once we got people thinking about the different options for testing out there, we decided to give them a glimpse behind the curtain to realize just how non-magical testing could be. To do this, we demonstrated not one, but two minimal testing frameworks that we built in less than a page or two of code to demonstrate the key concepts. While neither of these tools would be sensible to use in production, they both show how non-magical a test runner is under the hood.

Finally, David gave an open ended testing exercise that was a big hit. Pizza came, but some of our students ignored it until we reminded them to take a break. Many kept working on through lunch, all pairing together and actively discussing all sorts of language questions that went far beyond the scope of the exercise.

Style and Substance

This was my topic, and it went way better than I could have expected it to. I had a lot more content prepared than what I ended up getting to, mainly because we decided to collect some of the open questions that had come up in class and discuss those before running our scheduled content. The infamous class variable problem was discussed in great detail, and while I thought I had mastered their weaknesses, I even managed to mess up one of my assumptions about their behavior. Incase you’ve never been exposed to this fun before:

  class A
    @@foo = "from A"
  end
   
  class B < A
    @@foo = "from B"
  end
   
  # What's the value of @@foo in A now...?
   
  class A
    puts @@foo
  end
   
  # => from B
   
  # They're shared up the hierarchy?? BOOOO!!!
  # What's even worse? This ->
   
  class C
  end
   
  class D < C
    @@bar = "Magically different..."
  end
   
  class C
    @@bar = "And delicious."
  end
   
  class D
    puts @@bar
    # => Magically different...
  end
   
  class C
    puts @@bar
    # => And delicious.
  end
   
  # ZOMG THEY'RE NOT SHARED NOW.

Of course, the only sensible style advice here is to not use class variables at all, and since class instance variables can do the job just as well, that’s not really a big deal. We discussed how to do that, and then moved on to the regularly scheduled content.

I’m not sure whether we had an especially strong set of students or if people are just starting to “get” basic Ruby style idioms, but based on the first day and a half, it was evident that our students didn’t need to be told these basic things. Instead, I talked about “Ruby Style and Substance” at a more philosophical and design-oriented level, which is where I think our community tends to be weakest. Without giving too much away, a lot of my examples stemmed from discussions we’ve been having about refactoring Prawn before it’s 1.0 release, so you can check out our mailing list for a whole bunch of interesting design discussions if you want access to some of the raw materials I used for this session.

In the end, we gave everyone a chance to try their hand at basic API design for a fictional library we cooked up. I’m proud to say that every one of our students produced something that would be passable in an open source project. Ideally speaking, I’d like to say some of them took inspiration from what they learned in class, but no matter how they ended up there, I’m happy to see people writing code that looks like it comes from Ruby. This is especially encouraging considering that every one of our students came to Ruby after Rails was popularized.

The Event

We wanted “The Compleat Rubyist” to be not just a training, but an “event” as well. Because we were trying to keep registration costs low, we didn’t have much officially scheduled for this, but Tampa.rb really helped us pull through. Folks congregated at the nearby bar and restaurant Bahama Breeze from the time class ended until late into the evening Friday night. It was nice to have a mixture of students from the training and locals who just stopped by to hang out and say hi. I can honestly say that the sort of conversations I was having that night were on par with any I’ve had at the various conferences I’ve been at, but without all the associated commotion.

Pizza lunch on Saturday was a great way for us to leave a large chunk of semi-unscheduled time to discuss whatever topics students had on their mind. With a relatively small class size, I was able to get to know a number of the attendees, unlike my experience at most conferences these days.

While I expected to have a good time, I ended up having a blast. Based on my discussions with Jeremy and David, and the feedback from our students, I gather most others did as well.

The Future

While the Tampa, FL Compleat Rubyist event is now a thing of the past, we’re already thinking about how we can bring this session to other places. I was a little worried before the event that I wouldn’t want to just go somewhere else and do the same show, but now I know that every one of these is going to be different, since so much of the content was based on interaction with our attendees.

So, if the stuff I’ve said here interests you, you should follow compleatrubyist on twitter. We used our account actively during the event to share various little tips, and probably will continue to do that over time. But that’s also the place where you’ll see announcements about upcoming events, or ask us any questions you might have.

While I don’t typically like to use this blog for active marketing, I can honestly say that “The Compleat Rubyist” was a lot of fun, and that I think our students learned a ton. If folks like these summaries, I’ll try to get one out after each event. But it will be more fun, of course, if you come join us!

Learning From Bad Ideas

gregory.t.brown@gmail.com (Gregory Brown) — Sun, 22 Nov 2009 22:57:00 -0000

When we know something is ugly or “evil”, we’re quick to replace it with what we know to be a better solution. But if we don’t know why the solution is better, it makes it hard for us to investigate those things that look reasonable on the surface, yet have weaknesses just beneath the skin.

I’ll run through a quick example of what I mean, and then, it’ll be your turn to run with it and report back here. While I don’t use this approach all the time, I find it’s a neat tool to have handy when you’re trying to push your understanding of your code just a little bit farther.

Continuations are Evil?

It’s pretty common knowledge that continuations in Ruby are evil. But let’s pretend we didn’t know that. We might end up catching ourselves writing code like this:

  table.each do |row|
    call_cc do |next_row|
      row.each do |field|
        if field.valid?
          do_something_with(field)
        else
          STDERR.puts "skipping a bad row"
          next_row[]
        end 
      end
    end
  end

The continuation approach actually doesn’t look so bad. All we’re doing is iterating over a two-dimensional structure and skipping ahead to the next row if there is a problem with any of the fields. But having to pass along the next_row callback seems a bit heavy handed, and also makes one wonder what sort of magic it might be concealing. Preserving the same basic flow, we could be using catch / throw instead, lowering the conceptual baggage:

  table.each do |row|
    catch(:next_row) do
      row.each do |field|
        if field.valid?
          do_something_with(field)
        else
          STDERR.puts "skipping a bad row"
          throw :next_row
        end 
      end
    end
  end

So, now, we have something that looks more-or-less like a transactional block, which gets aborted when you throw :next_row. Not too bad, but if you’re like me, you probably get annoyed about having your eyes bounce up and down while you trace the flow, in both of these examples. We can fix that, of course:

  def process(row)
    row.each do |field|
      if field.valid?
        do_something_with(field)
      else
        STDERR.puts "skipping a bad row"
        return
      end
    end
  end
  
  table.each { |row| process(row) }

Now this is more like it! This code is probably the same as what a beginner might write, but it’s also the clearest out of what we’ve seen here. But without it as a point of reference, neither the catch / throw or callcc solutions would look terrible.

Through this quick little set of examples, I found a good reason not to use continuations for this particular use case. Arriving back at a simple solution by starting with a “clever” one and reducing it down to the fundamentals was a refreshing experience for me.

Your Homework

Now it’s your turn. I’m encouraging you to ignore “Best Practices” and the common idioms that we often accept as gospel.

Pick one technique that’s typically considered a no-no and do the best you can to use it in a somewhat reasonable way. You don’t want to waste your time with something that not even the most muddy-eyed coder would touch with a ten-foot pole, as you won’t learn much that way. Instead, work with something that looks fine or even clever at a first glance.

Then, try to fix up what you built by using more idiomatic, simpler code. Your goal is to create something that still looks good, but isn’t as much as a liability (conceptually) as what you started with. If you fail to do so, the worst thing that could happen is that you may have come across a new idiom worth sharing with others. But more likely, with a little effort you’ll have no trouble uncovering the wisdom behind whatever idiom you were putting to the test.

What’s the difference? Now, something that used to be a somewhat arbitrary rule to you is now common sense. As contexts shift, you may need to conduct new experiments, but repeat this exercise often enough and you’ll start to see a powerful intuition develop that allows you to internalize your design decisions rather than running through a punch list of patterns. And at least for me, I find that experience a lot more fun. I hope you will, too.

If you try out this exercise, please share what you’ve done, either via a link to your own blog, or a gist. I’m interested in how many bad ideas we can rack up here. :)

The Complete Class

shortcutter@googlemail.com (Robert Klemme) — Sat, 24 Oct 2009 18:05:00 +0000

A remark: we enabled comment moderation because the blog was recently target of spam. You probably have not seen much of it because we were pretty quick in removing it manually. So if your comment does not show up please be patient.

There are some basic concepts (often called “aspects”) that need to be implemented for many classes although not all classes need all (or even any) of them:

initialization
conversion to a printable string
equivalence
hash code calculation
comparability
cloning (clone and dup)
freezing
customized persistence (Marshal and Yaml)
matching
math and operator overloading

Which of these is needed for a particular class depends of course completely on the circumstances. Classes which are never written to disk or used in a DRb context will not need customized persistence handling. For other classes there might not be a reasoable ordering.

We will look at these concepts individually in subsequent sections. For the sake of this presentation I will create a slightly artificial class which will have particular properties in order to be able to show all the concepts:

mutable fields
redundant fields, i.e. fields which carry cached values that can be derived from other fields
at least two fields for ordering priorities

Class Album implements a music album with a title, interpret, sequence of tracks and a fixed pause duration between tracks. I picked a slightly different approach than Eric in his article in two ways:

Eric has a stronger focus on collaboration with standard library classes while my guiding question was “what does a class need to be complete and consistent?”,
I present all the features in a single class to show how aspects play together.

You will likely never implement all these aspects in a single class. Even for those aspects you do use, you might not use the same implementations I will present. That’s OK. The implementations presented in this article are intended to cover aspects thoroughly even though you will not always have to do that in practice. For example, certain code is there in order to make the class work properly as part of an inheritance hierarchy. If you write a one off class for a script which is never intended for inheritance you can simplify many of the presented methods.

I left out the topic of math and operator overloading since that does not mix well with the concept of music album. Instead I will cover that in the next article in the blog which will present a class that shows how to override operators in Ruby and that plays well with Ruby’s built in numeric classes.

Initialization

Implementing method initialize is typically one of the first things I do when implementing a new class unless I can use the default implementation of Struct. There are a few things though that are worth considering.

First of all, who owns arguments to initialize? It is important to make clear what happens to arguments that are provided to the call to new. There are three cases

the value is immutable (like Fixnum) or
the caller retains ownership or
ownership is transferred to the new instance.

Case 1 is the simple one: basically you do not need care thinking about who owns the object as there are no bad effects which can be caused by aliasing. If the instance is mutable these effects can show up. If you want to make your code as robuts as possible you must ensure you got your own copy of the instance (typically via copying it, for example by using dup). The downside of this is of course that this costs performance as you’ll likely copy too many objects. In practice you will probably most of the time do nothing special and keep the code as simple as an assignment.

class Album

  def initialize(title, interpret, pause = 0)
    super()

    # main properties
    self.title = title
    self.interpret = interpret
    @tracks = []

    # additional properties
    self.pause = pause

    # redundant properties
    # left out
  end

  def title=(t)
    @title = t.dup.freeze
  end

  def interpret=(i)
    @interpret = i.dup.freeze
  end

  def pause=(time)
    @pause = time
    @duration = nil
  end

end

The other important aspect is inheritance. Most classes that are written probably do not have a super in their initialize method. If there are chances that you reopen the class and add mixin modules you should include super() right from the start because even if you only implicitly inherit Object and initialize in Object does nothing you may later reopen the class and add a mixin module which has an initialize method itself. If you inherit another class than Object you should explicitly mention arguments with super and not rely on having the same argument list as the super class initializer. That way you are making the call more explicit and are robust against changes in your initialize method.

class CdAlbum < Album
  attr_accessor :bar_code

  def initialize(t, i, code)
    super(t, i, 2)

    self.bar_code = code
  end
end

While we’re at it: if you write a module which is intended as mixin and needs initialization itself the initializer should simply pass on all arguments in order to be compatible with arbitrary inheritance chains:

module AnotherMixin
  # Prepare internal state and pass on all
  # arguments to the super class
  def initialize(*a, &b)
    super
    @list = []
  end
end

Conversion to a printable String

Often it is desirable to be able to convert an instance to a human readable string. Which representation is most appropriate depends on the uses of the class. One policy is to create a string representation so the instance can be later reconstructed given this string as is the case for all the numeric types from the standard library. In the case of our sample class we will provide all interesting information:

  def to_s
    "Album '#{title}' by '#{interpret}' (#{tracks.size} tracks)"
  end

If you want to reuse a string field as external representation you could copy it in order to avoid bad effects from aliasing. However, since typically to_s is invoked for printing and no references are held I would say most of the time it is safe to not explicitly copy the field in these cases.

Equivalence

There are two methods that deal with object equivalence eql? and ==. (Method equal? tests for object identity and should not be overridden.) Some core classes do have different equivalence relations implemented:

irb(main):003:0> 2 == 2.0
=> true
irb(main):004:0> 2.eql? 2.0
=> false

But most of the time both methods will implement the same equivalence relation. This also helps avoid confusion. Note that eql? is special because it is used by Hash and Set to test for instance equivalence. We will come to that in a minute when we look at hash code calculation.

Equivalence of instances must be tested against significant fields and should ignore redundant fields. Looking at derived field values does not add anything to equivalence and makes the process slower at best.

  def eql?(album)
    self.class.equal?(album.class) &&
      title == album.title &&
      interpret == album.interpret &&
      tracks == album.tracks
  end

  alias == eql?

Now here you see a test for class identity which you likely have not seen in other classes. Why is it there? Mathematical equivalence is a symmetric relation which means that if a.eql? b returns true so should b.eql? a. Implementing eql? and == that way also helps prevent strange effects when working with Hash instances. So if you are going to compare two instances of a class and a subclass then you might get true when called on the super class instance and false on the subclass instance. The same happens for two completely unrelated classes where the set of fields of one instance is a subset of those of the other instance. You may actually be tricked into thinking they are equivalent because the common fields are equivalent while the two instances represent completely different concepts. The only way to remedy this is to check for identity of the class.

The reason that omitting the class identity check does not cause issues most of the time is simple: usually you stuff uniform instances into a Hash or Set and even if you mix different classes most of the time they will have different fields and different field values. Still it is good to remember the point in case you experience unexpected effects with Hash keys.

Note that I omitted the test for self identity which you might be used to from Java classes. I believe this is a premature optimization because most of the time you are going to test different instances for equivalence so most of the time you pay the penalty of the failing identity check and win only in rare circumstances.

Hash Code Calculation

This topic is closely related to instance equivalence: classes Hash, Array, Set and other core classes rely on the fact that equivalent instances also return the same hash code. Note that this is not symmetric: instances wich have the same hash code may actually not be equivalent. But if the hash code differs they must not be equivalent.

An instance’s hash code should be based on the same fields that are used for determining equivalence. Our class Album has more fields and it is advisable to do some bit operations (often involving XOR) to combine all member hash codes into a single value in order to ensure better diversity of these values. If you base the hash code only on a single field you increase the likelyhood that non equivalent instances fall into the same bucket of the Hash which makes additional equivalence checks via eql? necessary. You can find more about how hash tables work on Wikipedia.

  def hash
    title.hash ^
      interpret.hash ^
      tracks.hash
  end

Comparability

Many classes have a natural ordering such as integers which are ordered by their numeric value. If your class does also have a natural order you can implement operator <=> and include module Comparable to get implementations of <, <= etc. for free.

  include Comparable

  def <=>(o)
    self.class == o.class ?
      (interpret <=> o.interpret).nonzero? ||
      (title <=> o.title).nonzero? ||
      (tracks <=> o.tracks).nonzero? ||
      (pause <=> o.pause) || 
      0 :
      nil
  end

Cloning

When cloning or duping an instance the default mechanism sets all fields of the new instance to refer to the same objects as the cloned instance. While this is not an issue with immutable instances (e.g. Fixnum or instances which are frozen) bad things can happen if you have a mutable instance (for example Array or String) which is suddenly referenced by two instances which both believe they are the sole owner of it. All bad effects can happen including violation of class invariants and the instance state likely becomes inconsistent. Do deal with such cases even for shallow copies such as #clone and #dup you need to copy a bit more. Fortunately Ruby provides a hook (#initialize_copy) which is invoked after the instance has been copied and which can make appropriate adjustments. In our case we only need top copy the Array of Track instances because all other fields are immutable:

  def initialize_copy(source)
    super
    @tracks = @tracks.dup
  end

There is a recent discussion on comp.lang.ruby that started out with the subject of deep cloning but uncovered some general aspects of cloning.

Freezing

For freezing similar reasoning applies as for cloning: immutable fields need no additional attention as you cannot change those objects anyway. For others you need to decide how deep you want the freeze to go. In case of class Album we certainly want to prevent addition of more tracks after an album has been frozen. We also explicitly trigger calculation of duration so to avoid errors when accessing the duration on the frozen instance; the value cannot change any more anyway. This also makes it important to invoke super as last method.

  def freeze
    # unless frozen?
    @tracks.freeze 
    duration
    # end

    super
  end

Note that the code is robust against multiple invocations of #freeze because duration is calculated only once (on first transition from unfrozen to frozen). If you have more complex calculations going on that involve state changes of the instance you should place unless frozen? and end around the custom freeze code.

Custom Persistence

If you want to serialize the complete state of your instance there is nothing more to do: you can simply use Marshal and YAML from scratch. However, if you have redundant data (such as field duration in our case) which you want to omit from the serialization you have to adjust the process yourself.

For Marshal the proper approach is to use the newer approach which invoves writing methods marshal_dump and marshal_load. The former is supposed to return something which is serialized instead of the current instance and the latter is invoked on a new empty object and handed the deserialized object so fields can be initialized properly.

  def marshal_dump
    a = (super rescue [])
    a.push(@title, @interpret, @pause, @tracks)
  end

  def marshal_load(dumped)
    super rescue nil
    @title, @interpret, @pause, @tracks = *dumped.shift(4)
  end

For YAML it is even simpler: you basically just need to ensure to_yaml_properties returns a list of symbols containing only those members that you want serialized.

  def to_yaml_properties
    a = super
    a.delete :@duration
    a
  end

Both approaches do have their own strengths: with Marshal it is easier to completely replace an object with something else. While that can be done with YAML as well it is not as simple and elegant as in the case of Marshal. YAML on the other hand shines because you need to override only a single method.

One word about the implementations: while these methods could look simpler I picked an approach which also works when inheritance comes into play. You can repeat the pattern of marshal_dump, marshal_load and to_yaml_properties throughout a class hierarchy and still have each method only deal with fields of the class in which it is defined. That makes it easier to deal with later additions or removals of fields in some class in the hierarchy. This is even more so important when dealing with mixin modules, which can happen on a per instance basis (via #extend).

Matching

I use the term “matching” for the functionality of the three equals operator (===) and for the matching operator (=~). The first is used in case statements and with Array#grep while the latter is usually used only explicitly.

The semantic is completely up to the class at hand and there are no general guidelines that could be given. Implement it when it is reasonable for your class. If you want to elegantly use instances of your class in case expressions you have to implement === doing something meaningful.

  # Check whether we have that track or track title by title.
  def ===(track_or_title)
    t = (track_or_title.title rescue track_or_title)
    tracks.find {|tr| t == tr.title}
  end

  # Match title against regular expression.
  def =~(rx)
    rx =~ title
  end

Summary

Today we looked at a class which implements various common aspects that you will meet over and over again when coding Ruby. Many of these are in fact present in other programming languages as well: Java has serialization, equals and hash code calculation. C++ also has an equivalence operator and other operators that can be overloaded to implement ordering etc. You can find the full code of the class at github.

Structs inside out

shortcutter@googlemail.com (Robert Klemme) — Mon, 21 Sep 2009 21:48:00 +0000

Today we’re back to normal blog mode, where each article stands for itself. Muppet Labs are closed and we will be continuing our journey across the Ruby universe starting with an indepth look at Ruby’s Struct class — Ruby’s Swiss army knife for structured data.

Struct can be used without any additional require statement — it’s just there. This means it comes with zero additional overhead during initial interpreter startup — one of the many advantage of using Struct. But first let’s look at the basics.

Data Container

Class Struct is a strange beast if you are confronted with it the first time: it has a method new like other classes. But that method does not return an instance of Struct but rather a class which is a subclass of class Struct:

irb(main):001:0> pnt = Struct.new :x, :y
=> #<Class:0x101c3d68>
irb(main):002:0> pnt.class
=> Class
irb(main):003:0> pnt.ancestors
=> [#<Class:0x101c3d68>, Struct, Enumerable, Object, Kernel, BasicObject]

Arguments to Struct.new are names of fields that the generated class will have. These names must be provided as Symbols which makes sense, because these field names are basically identifier you hardcode into the program. (I will explain later why this is an accurate assessment when I look at Struct vs. OpenStruct vs. Hash.) You must provide them as Symbols because of another peculiarity of Struct:

irb(main):001:0> Struct.constants
=> [:Tms]
irb(main):002:0> s = Struct.new "Point", :x, :y
=> Struct::Point
irb(main):003:0> Struct.constants
=> [:Tms, :Point]
irb(main):004:0> t = Struct.new "doesNotWork", :x, :y
NameError: identifier doesNotWork needs to be constant
        from (irb):5:in `new'
        from (irb):5
        from /usr/local/bin/irb19:12:in `<main>'

I have to say I never used this feature of Struct because

I don’t want all my custom classes in a single namespace because that leads to potential issues,
I prefer explicit assignment to constants which makes clearer what’s happening.

If someone knows advantages of this auto constantification which I overlooked please let us know via the comment function. Otherwise I suggest to rely on standard constant magic if you want to name your Struct:

irb(main):053:0> pnt.name
=> nil
irb(main):054:0> Point = pnt
=> Point
irb(main):055:0> pnt.name
=> "Point"

Behavior

A Struct generated Ruby class is more similar to a C++ struct than to a C struct because it does not only carry data but also functionality. Some of that is already predefined when Struct.new returns:

irb(main):001:0> pnt = Struct.new :x, :y
=> #<Class:0x101c2fa0>
irb(main):002:0> puts pnt.instance_methods(false)
x
x=
y
y=
=> nil
irb(main):003:0> puts Struct.instance_methods(false)
==
eql?
hash
to_s
inspect
to_a
values
size
length
each
each_pair
[]
[]=
select
values_at
members
=> nil

These can be devided into two categories:

methods which originally deal with struct members.
methods that allow an instance to mimic behavior of other classes.

Struct Member Behavior

First of all there are attribute accessors (those defined in the Struct generated class, irb listing position 2). Then there are those methods that make a Struct instance usable as Hash key:

eql?
hash

These are defined in such a way that two Struct instances are equivalent if all of their fields are:

irb(main):014:0> a = pnt.new 1,2
=> #<struct x=1, y=2>
irb(main):015:0> b = pnt.new 1,2
=> #<struct x=1, y=2>
irb(main):016:0> a.eql? b
=> true
irb(main):017:0> a.hash
=> -1066353017
irb(main):018:0> b.hash
=> -1066353017

Method == also implements a similar equivalence relation but with a twist:

irb(main):019:0> b = pnt.new 1.0,2
=> #<struct x=1.0, y=2>
irb(main):020:0> a.eql? b
=> false
irb(main):021:0> a == b
=> true

It uses method == of field values internally in the same way as eql? uses eql? of field values. By providing these methods Struct can save you a lot of tedious typing of pretty dull code that you would hack manually otherwise. As we all know, the less code we have to write the less errors we can do…

Mimicry

A Struct can — within certain limits — mimic an Array as well as a Hash:

# Array alike
irb(main):027:0> a.to_a
=> [1, 2]
irb(main):028:0> a[0]
=> 1
irb(main):029:0> a[1]
=> 2
irb(main):030:0> a.values_at 1,0
=> [2, 1]
irb(main):031:0> a.length
=> 2
irb(main):032:0> a.size
=> 2
irb(main):033:0> a.each {|e| p e}
1
2
=> #<struct x=1, y=2>
irb(main):034:0> a.select {|e| e % 2 == 0}
=> [2]
irb(main):035:0> a << "more" # of course not
NoMethodError: undefined method `<<' for #<struct x=1, y=2>
        from (irb):41
        from /usr/local/bin/irb19:12:in `<main>'

# Hash alike
irb(main):036:0> a[:x]
=> 1
irb(main):037:0> a["x"]
=> 1
irb(main):038:0> a.each_pair {|k,v| printf "%p => %p\n",k,v}
:x => 1
:y => 2
=> #<struct x=1, y=2>
irb(main):039:0> a.keys # not quite
NoMethodError: undefined method `keys' for #<struct x=1, y=2>
        from (irb):38
        from /usr/local/bin/irb19:12:in `<main>'
irb(main):040:0> a.members
=> [:x, :y]
irb(main):041:0> a[:y]=123
=> 123
irb(main):042:0> a
=> #<struct x=1, y=123>

Btw. the generated class also exposes some interesting methods, which do not really need additional explanation:

irb(main):049:0> pnt.members
=> [:x, :y]
irb(main):050:0> pnt[10,20] # for lazy typers
=> #<struct x=10, y=20>

Custom Behavior

From time to time you will want additional methods in your Struct class. Well, there is a “hidden” feature — the documentation does not mention it but you can pass a block to Struct.new which will be used as class body:

irb(main):001:0> Point = Struct.new :x, :y do
irb(main):002:1*   def distance(point)
irb(main):003:2>     Math.sqrt((point.x - self.x) ** 2 +
irb(main):004:3*       (point.y - self.y) ** 2)
irb(main):005:2>   end
irb(main):006:1> end
=> Point
irb(main):007:0> Point[3,4].distance Point[0,0]
=> 5.0

This obsoletes the frequently seen idiom which uses inheritance from a Struct generated class to add more methods:

irb(main):001:0> class Point < Struct.new(:x, :y)
irb(main):002:1>   def distance(point)
irb(main):003:2>     Math.sqrt((point.x - self.x) ** 2 +
irb(main):004:3*       (point.y - self.y) ** 2)
irb(main):005:2>   end
irb(main):006:1> end
=> nil
irb(main):007:0> Point[3,4].distance Point[0,0]
=> 5.0

When using inheritance like this you are wasting resources (a class in this case) and so far I see only one advantage: changing the constructor is a bit easier because the old implementation is available via super without having to resort to alias or reimplementing member initialization. If you want to have multiple subclasses of a Struct class then inheritance is OK of course. But in that case I would rather make the super class a named class by assigning it to a constant and use that later when subclassing.

Struct vs. OpenStruct vs. Hash

All three classes offer functionality with some similarities and overlap. Sometimes you can use them interchangeably. Still there are some guidelines you can use to decide when to use which of these. As always — take these recommendations with a grain of salt. If you have your own different rules of thumb let us know.

Use Struct if

you need a data container and fields are fixed and known beforehand (i.e. at time of writing of the code),
you need a structured Hash key,
you want to quickly define a class with a few fields,
you need to detect errors caused by misspelled field names.

Use OpenStruct if

the number of fields is fixed at runtime but varies frequently during development time (this is often the case for objects that should hold the result of command line option parsing),
you need a mock or a want to quickly have objects at hand which can be filled via usual attribute setters and getters. You might want to replaced with a proper class (or Struct) later — with explicit attribute declarations via attr_accessor and relatives.

Use Hash if

the number of fields is unknown at coding time,
there is a potentially unlimited number of fields (e.g. when reading key values from a file as is often the case for script based text processing).

Now it also becomes apparent why I indicated that Symbols are appropriate for naming Struct fields: when defining a Struct you are actually declaring attributes much the same way as with an ordinary class. Still, Struct is a bit inconsistent here when it allows Symbols and Strings as keys for the Hash like access via [] and []=. The convenience of being able to use both probably outweighs this inconsistency. Overall that Hash likelyness is probably one of the less important features of Struct. You are likely going to use it when refactoring from / to real Hashes of if you have an application that has to deal with several Structs in a uniform way and needs to use metadata (obtained via #members) to access fields.

Deficiencies, any?

My whishlist for Struct is pretty short. The only thing that I am missing is an equally quick and elegant way to add Comparable functionality to a generated Struct class. That could be achieved by having something like this:

class Struct
  def self.comparable
    define_method :<=> do |o|
      members.each do |m|
        c = self[m] <=> o[m]
        return c unless c == 0
      end
      0
    end
    include Comparable
  end
end

Now we can do

irb(main):014:0> Point = Struct.new(:x, :y).comparable
irb(main):015:0> puts points = (1..5).map {Point[rand(4), rand(4)]}
#<struct Point x=3, y=3>
#<struct Point x=1, y=3>
#<struct Point x=2, y=3>
#<struct Point x=1, y=0>
#<struct Point x=0, y=0>
=> nil
irb(main):016:0> puts points.sort
#<struct Point x=0, y=0>
#<struct Point x=1, y=0>
#<struct Point x=1, y=3>
#<struct Point x=2, y=3>
#<struct Point x=3, y=3>
=> nil

Summary

Today we looked at various aspects of Ruby’s built in class Struct. Struct allows you to create classes quickly (typically in a one liner) while providing a lot of useful functionality out of the box — most notably it is the fastest way to get a class suitable as a Hash key (apart from using one of the other built in classes like String, Symbol or even Array). If you haven’t been using Struct so far you might want to try it out in your next hack. Enjoy less typing with more Structs!

Muppet Labs closing for now

shortcutter@googlemail.com (Robert Klemme) — Mon, 07 Sep 2009 19:03:00 +0000

Looking back

It is time to look back at the Muppet Laboratory series. Here’s my list of noteworthy items — I will keep it rather short as interest in the series seems to have dwindled anyway:

The first version with LRU caching integrated was way too complex and had ridiculous performance. Continuous checking of interactions against the search criteria made the code complex and slow. In this case the premature optimization was worse than ineffective.
I am test lazy (I confess) but apparently this works remarkably well. Note, that this does not mean I don’t test. But for my scripts I usually do not write unit tests or RSpecs. (file under: Ruby Bad Practices — fortunately the acronym is the same)
Overall the LRU approach works remarkably well at limiting memory usage while processing huge amounts of input data so I would say the plan succeeded mostly.
I found the idea to use block passed to a method as a class body especially interesting. I will probably recycle this for other cases where I want to provide a framework (including command line parsing) which needs to be filled with user code.
Summertime is a bad time for a blog series — author and readers are occupied with other things, mainly enjoying sunny outsides.

What did you consider most valuable for you? What would you have done differently (software as well as blog series wise)? I would be curious to learn what your thoughts are.

What’s ahead?

The next article will be about the versatility of Struct. This is another swiss army knife of the lazy Ruby coder.

James will be right back after these interruptions

james.britt@gmail.com (James Britt) — Sun, 30 Aug 2009 13:00:00 +0000

James will be right back after these interruptions

How many times has this happened to you?

[WAY-TOO-LOUD TV VOICE]: You’re working on some code, making good progress, solving a particular issue, when Oh No!,
you’ve run into a new bug. Now what?

You could open a browser, go to your issue tracker, and open a ticket. But that could take several minutes!

Your train of thought would be derailed, code momentum lost, and time wasted! But you can’t just let that bug slide away! Don’t you wish there was an easier way to handle this?

As far as I know there have been no late-night TV commercials selling solutions to software developers, but there are times we could use them. The problem of staying focused on one task, while not ignoring others, is a real one, at least for me, and I suspect many others.

Here’s what happens: You pick a bug to fix or feature to add, and set out to complete it. Almost inevitably, though, you encounter other issues or interests. If it looks to be minor, it’s tempting to fix it right then. Sometimes this is trivial; a typo in the UI text in the file you’re working on. These are the cases where even adding a FIXME or TODO to the code would be about as much work as just fixing the problem.

Other times, though, it’s obvious that this new concern will take a bit of time. Or, maybe worse, there’s no simple way to quickly judge how much work would be needed. Dropping what you’re doing to fix something else isn’t a recommended practice. Given that each task is prone to uncover a new task, you’re doomed from the start to never finish anything.

That leaves two (admittedly broad) options. One is to ignore everything that isn’t immediately related to the immediate task. The other is to stop what you’re doing and make a note of the problem. But how long is that going to take?

Now watch this!

Lately I’ve been interested in focus and attention. Like many people my ability to focus often varies. For tasks that don’t have an intrinsic lure (e.g. bug fixing) I work best if I stick to well-defined tasks over specific durations. (This is an approach sometimes referred to as time boxing. Some formal examples are the Pomodoro Technique and the Autofocus System).

An important aspect is to really spend that time doing that specific task. If it leads to assorted sub-tasks, it matters how much time those things take, since time is limited. If necessary digressions take too long, the whole “do one thing” plan falls apart, as does one’s focus.

So now we have two problems. One, the code distraction that just appeared, and two, the need for a way to quickly and effectively record it. Here’s what I’ve been doing about that second one, which leads to solving the first one.

I first started looking into the idea of rapid notation when I was working a contract job that needed some detailed time tracking. After spending two hours adding what seemed like a simple feature to a Java Web app I wanted to be able to look a back a see just where the time went.

I started by using a plain text file. Every so often, when I found myself doing something that didn’t seem to be directly adding to a solution (for example, rebooting the app server to fix JVM out-of-memory issues) I made a note.

It worked OK, but it was a bit slow. There was a lot of friction on opening the file, adding text, and saving it. Even leaving the file open all the time was not helpful enough. As I usually work at the command line I wrote a small Ruby script to let me write my notes right from the terminal.


    #!/usr/bin/env ruby

    @data_file = "/home/james/data/_at-today-please.txt" 

    def ts; Time.now.to_s; end

    def edit
      IO.popen "gvim #{@data_file} & "
    end

    def add
      msg = ARGV.join( ' ' ).to_s.strip
      File.open(@data_file, "a") do |f|
        f.print "--- #{ts}:\n\t#{msg}\n\n\n"
      end
    end

    ARGV.empty? ? edit : add

I’m not making any claims for this code other than follows the principal of WFJ: Works for James. I have it in a file at ~/bin/@; adding a short note is about as simple as it gets. If I need to add something complex (or something that just doesn’t play well at the command line), I pass no arguments and edit the file directly.

It’s flawed, of course. Depending on the shell, different chracters (quote marks, for example) can throw things off. But it works good enough for quick time-stamped notes.

But wait, there’s more!

Some time after this hack I found myself working with Trac. It does a good job of coordinating source code, issues, project planning, and assorted other goodies. Entering bugs, though, was a drag. The Web UI meant entering text in multiple fields and selecting values from a few dropdown lists. Once again, to keep from getting too distracted while developing I wrote some code to post tickets from the command line. The project was called Tracula (because the key to any project is a clever punny name) and helped my development process by removing some of the friction of tracking issues uncovered during development. Usage hit some snags when different versions of Trac required somewhat more gnarly means of athenticating HTTP posts, while my use of Trac dropped in favor of other project management tools.

Nonetheless, the general idea stayed with me, and when picking any tool I look to see if there’s a simple HTTP interface that allows for automation and custom command-line driven clients.

I now have several projects on GitHub.com, which has become much like a hosted Trac, but with more ass-kickery. Each repo can have an issue tracker, and I’ve just started using it for my Rhesus project.

Rhesus is yet another WFJ tool that has been evolving as I use it. I had been making notes about the code in various places when I realized there was an actual bug tracker available. I added a few issues, and immediately wondered if there was an API for this. And there is (thanks, Engine Yard!).

With Tracula there were three key options: fetch a list of open tickets; post a single new ticket; do a batch post of mulitple tickets. How to do these with the GitHub API? Well, fetching tickets is really easy, since there’s no need for any authentication. If you have curl or wget installed, it’s a just a well-crafted URL:

    http://github.com/api/v2/json/issues/list/Neurogami/rhesus/open

will give you a list of open Rhesus issues. Very slick, but since the results are a JSON string, maybe not the easiest thing to read. You can change that to XML or YAML if you like, but for better control, how about using some Ruby to slurp it in and present it in a more refined manner?

And if you call RIGHT NOW …

A basic approach to showing the current set of open issues might be: make a call to the GitHub API; parse the results into some useful Ruby structure; selectively show the details to avoid overcrowding the screen. One thing you need to consider, though, is just how often will the list of issues change relative to the number of times you want to see it. In my case, I’ll likely want to see the list way more often than it will change. To make life easier both for me and for GitHub, I want my code to cache the list of issues and render that cache by default. I’m also going to make this a Rake task, and I’ll allow for a parameter to indicate that the cache should be updated.


  namespace :issues do
    desc 'Open issues'
    task :open do
      paged_issues ENV['GH_NO_CACHE']
    end
  end

The page_issues method will grab the set of issues and render them one by one:


  def ___
    warn "\n" + '_' * 30
  end

  def paged_issues no_cache = false
    ARGV.clear # Otherwise the call to gets will immediately find input
    warn "Hit <enter> after each issue to see the next one."
    issues(no_cache).each do |issue|
      ___
      warn "Title: #{issue['title']}"
      warn "Details: \n#{issue['body']}"
      ___
      gets
    end
    warn "\nThat's it!"
  end

The issues method will either pull from the cache or fetch anew from GitHub


    def issues no_cache = false
      issues_cache = 'issues.yaml'
      if no_cache || !File.exist?(issues_cache)
         warn "Reloading issues from GitHub ..."
         File.open( issues_cache, 'w'){ |f| f.puts get_issues }
      end
      YAML.load(IO.read(issues_cache))['issues']
    end

This allows for

rake issues:open

and

rake issues:open  GH_NO_CHACHE=true

Operators are standing by!

What about adding a new issue? The docs say that an authorized POST is needed. And what’s an authorized POST? It’s an HTPP POST with parameters for your user name (login) and your authorization token (token). The latter may be found on your GitHub account page.


     namespace :issues do
      # ...
      desc 'Create a new issue'
      task :create do
        issue_details = ENV['DETAILS'].to_s.strip
        if issue_details.empty?
          warn "No issue details!"
        else
          title, body =  issue_details.split(' -- ', 2)
          create_issue  title, body 
        end
      end
    
     # ...
     end

Rake is not the most friendly when it comes to passing arguments, so to (somewhat) simplify things I decided to use one parameter that employed a delimiter to set off the issue title and body. The code that does the actual ticket creation is oblivious to this, and could easily be used in a regular Ruby app. The choice of making something a Rake task or a stand-alone executable depends on how you expect to use it. With Rake, it’s easy to get a reminder of what tasks are available using rake -T. The downside is that argument handling is not so nice.

(Were this more than demo code, though, I would ultimately make the core code a library so that having both a Rake task and a standalone app would be equally easy. And, in fact, some folks have been working on GitHub API Ruby libraries.)

Doing a POST is a bit more complex than just reading from a URL, but no big deal:


      def create_issue title, body
        require 'net/http'
        require 'uri'

        token = ENV['GH_AUTH_TOKEN']
        url = 'http://github.com/api/v2/yaml/issues/open/Neurogami/rhesus'
        res = Net::HTTP.post_form(URI.parse(url),
                                  {'title'=> title, 
                                      'body' => body,
                                      'login' => 'Neurogami',
                                       'token'=> token }
                                 )
        warn res.body

      end

Note that the code pulls the authentication token from an environment variable. That’s done to avoid having to embed it in the code or some config file that may inadvertently reach the wrong eyes.

As for bulk posting of issues, well, better add a ticket for that.

But that’s not all; you’ll also get this!

I must confess that I’ve been writing the code as I wrote this post. It’s an amazingly useful way to get some code written.

I thought about the whole “Oh, if you were really doing this right, you’d make this a library” thing, and, well, I made this a library thing.

You can get the code here: http://github.com/Neurogami/ghissues/tree/master

It’s basically the above code, with some provisions for reading a local config file that defines the repo, repo owner, and a file for saving a cache of the repo issues. (A down-side of coding-for-content is that certain niceties, like docs and tests, may fall by the wayside. The code, however, is pretty simple.)

Re-doing the Rake tasks using this gem is left as an exercise for the reader.

Don’t wait; act now!

I’ve written a few other apps along these lines. One lets me add items to my list at Todoist.com. I can even set reminders to pester me to do something later. Very handy; the improved speed in setting a reminder or adding a new bug ticket is well worth the time spent coding up such tools. There are numerous existing apps out there for things like this as well. (Check out this thread on Hacker News.). See if one doesn’t help you do more with less friction.

But if you can’t find one that really works for you, consider writing your own. It’s not hard, and there’s a real payoff in having something that fits your style and keeps you moving.

Completing the Animal

shortcutter@googlemail.com (Robert Klemme) — Mon, 24 Aug 2009 21:09:00 +0000

Leftovers

Summertime was a bit tough — but the end is near! Today I will cover two major parts that were needed to complete the “animal”. After that I will present lessons that I learned in this public blog software project. Muppet Laboratories will then close their doors but the code will still be there.

For a fully functioning Animal mainly two things were missing:

command line parsing,
creation of a proper filter mechanism based on command line arguments.

We will first look at command line parsing.

Command Line Parsing

There is actually no rocket since involved whatsoever. The most noteworthy piece of information is probably that you need to include ‘optparse/time’ in order to be able to allow OptionParser to parse time stamps passed as option arguments. (Funny thing is, OptionParser will also happily accept strings like “now” and parse them as timestamps.)

require 'ostruct'
require 'optparse'
require 'optparse/time'

  def self.parse_command_line(argv = ::ARGV) 
    o = OpenStruct.new(:output_dir => '.')

    # parse
    OptionParser.new do |opts|
      opts.on '-d', '--dir=DIRECTORY', 'Output directory ' do |v|
        o.output_dir = v
      end

      opts.on '-r', '--rx=REGEXP', ::Regexp, 'Regular expression matched',
      'against log line text' do |v|
        o.rx = v
      end

      opts.on '-t', '--time=TIME', ::Time, 'timestamp' do |v|
        o.ts = v
      end

      opts.on '-s', '--start=TIME', ::Time, 'start timestamp' do |v|
        o.start_ts = v
      end

      opts.on '-e', '--end=TIME', ::Time, 'end timestamp' do |v|
        o.end_ts = v
      end

      opts.on '--ids=ID_LIST', ::Array, 'Comma separated list of interaction ids' do |v|
        (o.ids ||= Set.new).merge v
      end

      opts.on '--id-file=FILE', 'File with ids one per line',
       '(empty lines are ignored)' do |v|
        s = o.ids ||= Set.new

        File.foreach v do |line|
          line.strip!
          s << line unless line == ''
        end
       end

      opts.on '--buffer=INTERACTIONS', ::Integer,
        'Max no. of interactions to keep in memory' do |v|
        o.max_size = v
        end

      opts.on_tail '-h', '--help', 'Print this help' do
        puts opts
        exit 0
      end
    end.parse! argv

    raise 'Only one of time or (start, end) allowed' if o.ts && (o.start_ts || o.end_ts)
    raise 'Missing end timestamp' if o.start_ts && !o.end_ts
    raise 'Missing start timestamp' if !o.start_ts && o.end_ts

    o
  end

I personally find OptionParser very elegant and complete but other seem to prefer other command line parsing packages, like GetoptLong and there are others around as well. Main advantages of OptionParser in my opinion:

Options, their documentation and their processing are all placed close to each other.
Built in support for conversion of argument strings to most basic types and even lists of values.
It is a part of the standard Ruby distribution, i.e you can safely assume that it is available.

Filter Creation

There are several approaches that can be taken on creating a bit of filter code from options taken from the command line — all have different strengths and weaknesses:

Interpreting the option set on each filter invocation (slow but easy to implement).
Combination of written filter code with filter criteria values stuffed in a closure (faster than the first option but equally easy to implement).
Generation of filter code (usually fastest, can be a bit complex to implement).

I picked option 2 because of the speed advantage and the avoidance of safety issues that come with using eval. I also find it a bit inelegant to generate code but that’s a purely aesthetic argument which I don’t claim any substance for. (But since we’re into Ruby at least partly for the fun, that argument is not too far off the mark.)

As interface I choose lambda's square brackets. So, here is the filter creation code:

    def create_filter(opts)
      # store in local variables to ensure
      # ensure values do not get changed
      # and a small speed improvement
      o_ids = opts.ids
      o_ts = opts.ts
      o_start_ts = opts.start_ts
      o_end_ts = opts.end_ts
      o_rx = opts.rx

      f = case
          when o_ids
            warn 'WARNING: Ignoring time filters with ids given' if
            o_ts || o_start_ts || o_end_ts
            lambda {|ip| o_ids.include? ip.id}
          when o_ts
            lambda {|ip|
              ip.entries.first.time_stamp <= o_ts &&
                ip.entries.last.time_stamp >= o_ts
            }
          when o_start_ts && o_end_ts
            lambda { |ip|
              ip.entries.last.time_stamp >= o_start_ts && 
                ip.entries.first.time_stamp <= o_end_ts
            }
          end

      case
      when f && o_rx
        lambda {|ip| f[ip] && ip.entries.any? {|e| o_rx =~ e.line}}
      when o_rx
        lambda {|ip| ip.entries.any? {|e| o_rx =~ e.line}}
      when f
        f
      else
        YES
      end
    end

As you can see there is a bit of prioritization going on: I choose to ignore time range options if also interaction ids are used as filter criterion. Reason is that interactions are usually short and additional time based filters would either have to add more interactions to the result or remove interactions whose id was selected (depending on whether “and” or “or” combination was chosen).

The text filter (regular expression really) on the other hand is “and” connected with the other filter, i.e. if the list of interactions selected by time or ids is large then it will be narrowed down through the text filter.

One word about time filters: when given a single point in time, the test is pretty straightforward — the first timestamp must be lower and the last timestamp must be larger than the test time, i.e. the interaction was active at the given time. The test for a given time interval (start and end time given separately) may look a bit odd at first. However, there is no typo involved. The test ensures that the test interval and the interaction’s time interval overlap. Or put differently: the test ensures that all interactions are included that were active during the test interval.

Things left to do

For now I am pretty satisfied with the results so I do not feel a lot pressure to continue working on it. However, there are a few things that could be done:

Cleanup of require and deletion of source files that were not used.
Adding of statistics output like lines read, lines written, interactions found etc.
Profiling and optimization.

Anything else?

Be Nice and Have Fun

gregory.t.brown@gmail.com (Gregory Brown) — Fri, 21 Aug 2009 11:25:00 -0000

Yet again, we’re experiencing a firestorm that will shake things up and change the Ruby community in a big way. But like anything else, the best thing we can do is remember that the sky is not falling, and that all internet drama is ephemeral at best.

Sure, we’ll all dearly miss a valued community member who inspired many of us. But thanks to distributed revision control systems, we’ll be able to commemorate _why through his works, currently being mirrored on github .

So what do we do in this post _why world? The answer to that is fairly simple: the same thing we’ve been doing. The true spirit of the Ruby community (and even the free software hacker community at large) is to work on fun stuff, and try to be nice to people. If we can keep that up, the sky will not fall, and may even make room for new stars to light up the present darkness.

I’ll let this brief statement stand on its own here, as RBP has no intentions of becoming a gossip blog, but feel free to leave your thoughts in the comments.

Expect a new Ruby 1.9 series post some time before or just after LSRC. Until then, keep your heads up folks.

NOTE: I have Aaron Patterson to thank for expressing what I was thinking in words ‘do fun stuff and try to be nice’

A bit of Optimization

shortcutter@googlemail.com (Robert Klemme) — Sun, 09 Aug 2009 20:47:00 +0000

As suggested earlier I did a bit of benchmarking and found these timings of the first version which kept everything in memory and the next version with two LRU storages (for InteractionProcessor and IO objects):

v0.1

robert@fussel ~/Eigene Dateien/Projects/muppet-laboratories
$ time ruby19 bin/sample-animal.rb sample.log

real    11m53.226s
user    6m34.561s
sys     4m33.921s


v0.2.1

robert@fussel ~/Eigene Dateien/Projects/muppet-laboratories
$ time ruby19 bin/sample-animal.rb sample.log

real    25m40.842s
user    14m40.655s
sys     8m29.701s

I then went on and refactored the code in these ways:

removed LRU storage of IO objects
removed state pattern and replaced it with a single filter test again

The solution now resembles the first version quite a bit with the difference that the storage of InteractionProcessor instances is now an LRUHash. The InteractionProcessor now looks very simple — again:

  class InteractionProcessor
    
    # mode for writing files
    OPEN_MODE = IO::WRONLY | IO::CREAT | IO::TRUNC

    attr_reader :id, :coord, :entries

    def initialize(id, coordinator)
      @id = id
      @coord = coordinator
      @entries = []
    end

    # Process the first line
    def process_initial(time_stamp, line)
      process(time_stamp, line)
    end

    # Process an initial line
    def process(time_stamp, line)
      @entries << Entry.new(time_stamp, line)
    end

    # Append a continuation line to the last entry
    def append_line(line)
      @entries.last.line << line
    end

    def finish
      if ! @entries.empty? && @coord.filter[self]
        fn = file_name
        FileUtils.mkdir_p(File.dirname(fn))

        File.open(fn, OPEN_MODE) do |io|
          @entries.each {|e| io.puts(e.line)}
        end
      end
    end

    private

    # calculate the file name, this fails if
    # there are no entries!
    def file_name
      ts = @entries.first.time_stamp
      File.join(@coord.options.output_dir,
                ts.strftime('%Y-%m-%d'),
                ts.strftime('%H-%M'),
                ts.strftime('%S.%3N-') + id)
    end
  end

And now this is the timing we get:

v0.3

robert@fussel ~/Eigene Dateien/Projects/muppet-laboratories
$ time ruby19 bin/sample-animal.rb sample.log

real    7m29.546s
user    2m16.702s
sys     4m8.296s

Now, this is even faster than the first version! The difference seems to mainly be caused by a reduction in user time. I would have expected at most a small change in system time for less paging since we limit the memory usage. Maybe the difference is caused by the fact that the main Hash's size does not increase beyond a fixed limit.

I guess the LRU based storage of file handles qualified as premature optimization: apparently it wasn’t necessary to get rid of individual log lines as fast as possible. Instead, it makes more sense to treat an interaction as entity and wait with the processing until the interaction is complete (well, not really: until the LRU storage decides, that the interaction must be purged from memory). That’s a nice example how granularity of processing directly influences performance and application design. In fact, treating an interaction as entity is probably also the most straightforward approach. And it turns out to be more efficient than the much more complex state pattern logic.

Should I Tap that Hash? (Ruby 1.9 Style)

gregory.t.brown@gmail.com (Gregory Brown) — Wed, 05 Aug 2009 12:00:00 -0000

UPDATE: Be sure to read the comments. I’ve already learned a lot from them, and you might, too.

Though the Ruby Best Practices book covers Ruby 1.9, it is decidedly not a “Ruby 1.9 Best Practices” book. The reason for this, of course, is that common idioms for Ruby 1.9 haven’t evolved yet.

Let’s work together to change that. Every week until Ruby 1.9.2 comes out, I’ll be coming up with short, manageable bits of Ruby 1.9 code for discussion, as part of a “Ruby 1.9 Style” series. These should not be taken to be authoritative in any way, but instead work as conversation starters so that we can move forward as a community and separate the good from the bad when hacking code on Ruby 1.9.

Rather than describing in detail how this series will work, I’ll just begin with some real content and you can get a sense of the format from that. The only key thing is to remember to not be lazy, and to actually comment on these posts. Ruby 1.9 is basically the wild west right now, and its going to take quite a few sound voices to tame it. So be sure to share your thoughts after you take a look at the proposed idiom below.

Object#tap, a seemingly benign addition.

Ruby 1.9 gives us tap. And tap is sort of neat because it can be useful as a debugging tool, especially in chained code.

  >>  [1,2,3].map { |x| x + 1 }.tap { |y| p y }.inject(:+)
  [2, 3, 4]
  => 9

There isn’t much conceptual weight here, either. Essentially, you’re looking at something like this:

  def tap
    yield(self)
    self
  end

But hmm, doesn’t that look familiar?

Object#tap as an alternative to ActiveSupport’s returning

Those who work with Rails or with ActiveSupport in some other capacity will have seen returning. The problem it is meant to solve is seen in the code below.

  def ugly
    results = {}

    [:x, :y, :z].each do |letter|
      results[letter] = rand(100)
    end

    results
  end

Using returning, this can be cleaned up a bit:

def sexy
  returning({}) do |results|
    [:x, :y, :z].each do |letter|
      results[letter] = rand(100)
    end
  end
end

This code ~~is definitely more DRY, and~~ looks pretty clean. The only problem with it is that since it relies on ActiveSupport, you’re essentially sifting through DHH’s junk drawer for a piece of non-standard functionality for a fairly basic purpose. While it may not be a “Rails Best Practice”, I tend to shy away from this sort of thing to keep from being too dependent on a framework that I only use part of the time.

But if we think for a second, how would returning be implemented? Something like this, right?

  def returning(obj)
    yield(obj)
    obj
  end

Go check out the end of the previous section. What we can see now is that returning is just Object#tap inside out! So what happens when we use tap that way? Let’s take a look.

Tapping that Hash.

I’ve started on a few Ruby 1.9 production apps, and that’s where I first ran into attempting to use tap as if it were returning. Take a look at this bit of code:

  def times_for_period(date)
    days = work_days_for_period(date)
    regular_hours_by_week = Hash.new(0)

    { :regular_hours => 0, :other_hours => Hash.new(0) }.tap do |results|
      days.each do |d|
        times = d.times

        results[:regular_hours] += times[:regular_hours]
        regular_hours_by_week[d.date.cweek] += times[:regular_hours]

        times[:other_hours].each { |k,v| results[:other_hours][k] += v }
      end

      results[:weekly_hours] = regular_hours_by_week.keys.sort.map do |k| 
        regular_hours_by_week[k]
      end
    end
  end

What I’m doing here is calling tap on a default hash and then populating it via the code within the block. If I was simply iterating over some values, I might use inject or each_with_object, but my needs were a little more complicated here.

This approach adds a little complexity, but in my experience, having to explicitly return a collection at the end of a method can be cumbersome, and I frequently forget about it during refactoring and cause my tests to go red on me.

What do you think? Is Object#tap the new returning? Or should we put this knife carefully back into DHH’s junk drawer for safekeeping? I’m pretty sure Rails didn’t invent this functionality, but that’s where a lot of us might have encountered it first.

Help me decide whether this is elegant or nasty in the comments below.

A Gentle Reminder

I have to warn you, some of my ideas might be bad. I’ll be posting code that I find interesting or stuff that I don’t know exactly how I feel about. If we hit a dud from time to time, it’s only to be expected. But if you want to help me out, consider contributing content to this series. If you have an idea for a guest post, go ahead and email me with the details, and I’ll get back to you.

Looking forward to your thoughts on this one and many more in the coming weeks!

LRU Integration explained

shortcutter@googlemail.com (Robert Klemme) — Fri, 31 Jul 2009 16:33:00 +0000

Today I will present my reasoning which lead me to the implementation of the LRUHash as well as how I integrated it into the main code. We will look at how LRUHash works and how it is integrated into the project. Then I will answer some questions that were actually asked — or only thought.

How `LRUHash` works

The interface should resemble a Hash as much as possible so that LRUHash can be easily substituted for a Hash instance without alteration of the code that uses it. As long as the instance remains below the maximum size it behaves much the same as a regular Hash. Only when the size limit defined via max_size is reached any subsequent store operations will remove the oldest entry.

Internally a LRUHash maintains a linked list of LRUHash::Node instances and a Hash. Nodes contain key and value of a hash entry as well as links to their predecessor and successor. The Hash is used for fast access of individual nodes while the linked list is used to maintain information about access order: every node which is accessed is moved to the head of the list so the least recently used element is always at the tail.

There are two nodes referenced as head and tail which are never changed. This makes it easier to move nodes around in the list because extraction and insertion are always operations on inner nodes and do not need to account for the first and last element on the list which is tedious because then you would also have to change the first and last pointer.

Additionally to the default_proc which works exactly the same way as in Hash there is a release_proc which is invoked whenever an item is removed from the LRUHash — either via explicit delete operations like delete, delete_if and clear or via the automated expiry which kicks in as soon as the LRUHash reaches its maximum size.

How `LRUHash` is used in the project

There are two LRUHash instances in class Coordinator:

A storage for interaction processors,
A storage for File objects.

The first instance is used to limit the overall number of interactions which are kept in memory. Once the LRUHash is filled and a new interaction needs to be stored the least recently used one is expired and removed.

The second one is needed because InteractionProcessor instances keep their File object open and the limit for open file handles per process is usually much lower than the reasonable limit for interactions kept in memory. So File objects are stored in the LRUHash and closed whenever necessary and reopened as well.

Why do you use `equal?`?

There are no “equivalent” nodes in a single LRUHash instance because they all have different keys. Actually the concept of equivalence is not needed here; rather I just needed to check for identity which is exactly what equal? does.

Why did you not use PUPA’s Ruby/Cache?

ged suggested in his comment the use of PUPA’s Ruby/Cache. I had looked at it before cooking my own little version and decided against using it for several reasons:

It did not include the default_proc mechanism of Ruby’s Hash although you can use fetch with a block the same way as in Hash.
It uses the notion of “object size” (explained here) as one of the limits of cache size. Besides the problems with defining “object size” which I pointed to here and here the more crucial reason was the problem with updating the cache’s idea of the current size of objects (either the overhead is significant, because every time the overall size limit is checked all objects need to be looked at or the cache cannot be aware of size changes if it caches object sizes).
It uses wall clock expiration time for cached objects — something I did not have use for in my implementation.
The fun of cooking my own.

While items 2 and 3 could be remedied by using arbitrary large limits the overhead of calculating values would remain (at least the documentation does not state that the overhead is saved when any of these limits are left out).

Performance Observations

While I did not actually measure timings the current version is significantly slower than the first draft version which kept everything in memory before writing out interactions to individual files. I suspect that this may be caused by these factors:

Criterion evaluation is done multiple times per interaction (although it’s still a dummy currently).
Frequent opening and closing of files caused by LRU handling of IO objects which is necessary because of the much lower limit of open file handles compared with interaction instances.

Both were introduced to get rid of log file lines as soon as possible. This also lead to a complex design (especially in the area of evaluation of the criterion). The effect may change though if real criteria are used. This is certainly something that I need to analyze.

An improved version would probably get rid of the immediate writing of lines as soon as possible and defer that to the point in time when the interaction is removed from the LRU cache of interactions. That way the criterion needs to be executed only once, can be made much simpler — especially for complex criteria which need to look at multiple entries such as the time range criterion. Also, we do not need to keep multiple files open.

Note: This shows how important it is to keep an open mind and to think again about the code you have written from time to time. I have discovered quite a bit of quirks I put into my code by this. Of course you can say that if you do it right the first time, this is never necessary — but who would claim to write optimal code in the first attempt?

Finally, a Best Practice

During my vacation I read Zen and the Art of Motorcycle Maintenance (you can browse the text but I’d rather read a real book). I had read it some 15 years ago and wanted to go back to this fascinating text which provides so many perspectives and crystallization points for thought.

Now, what does this have to do with Ruby? The best practice here is: once in a while do something completely different. It helps keep your mind flexible and will open it up to new approaches to old matters as well as inspire your creativity. You will also notice, that while you’re away from your everyday business your mind will actually continue to work on it which typically shows by suddenly having an idea that turns up when you least expect it.

LRU Integration

shortcutter@googlemail.com (Robert Klemme) — Fri, 10 Jul 2009 21:07:00 +0000

This will be just a short article since I am a bit tight on time right now.

For those of you who might not have heard the term “LRU” means “least recently used”. It refers to a replacement strategy used in caches. For every cache with an upper limit on the number of elements cached it must be decided which element to remove from the cache once the cache gets full and another element should be put in the cache. The decision can have dramatic impacts on the efficiency of the cache (its hit ratio).

LRU is easy to implement and works pretty well in many cases so I picked that one. “LRU” means, remove the element which has not been used for the longest time. This is typically implemented using a doubly linked list because with that elements can be moved to the head very quickly (O(1)). The algorithm works by moving every accessed element to the front of the list and deleting the last one when space is needed.

I created a class with a Hash like interface and an additional feature, a release_proc which gets called with the removed entry’s key and value. That way we can automate the cleanup process easily.

Internally the class has a Hash for fast access which has list nodes as values. These are the main methods for LRU mechanism during read access.

  def fetch(key, &b)
    n = @h[key]

    if n
      # hit -> move to front
      front(n).value
    else
      (b || FETCH)[key]
    end
  end

  # move node to front
  def front(node)
    node.insert_after(@head)
  end

Individual entries are of class Node:

  # A single node in the doubly linked LRU list of nodes
  Node = Struct.new :key, :value, :pred, :succ do
    def unlink
      pred.succ = succ if pred
      succ.pred = pred if succ
      self.succ = self.pred = nil
      self
    end

    def insert_after(node)
      raise 'Cannot insert after self' if equal? node
      return self if node.succ.equal? self

      unlink

      self.succ = node.succ
      self.pred = node

      node.succ.pred = self if node.succ
      node.succ = self

      self
    end
  end

And here’s the code for the removal of old entries:

  def store(key, value)
    # same optimization as in Hash
    key = key.dup.freeze if String === key && !key.frozen?

    n = @h[key]

    unless n
      if size == max_size
        # reuse node to optimize memory usage
        n = delete_oldest
        n.key = key
        n.value = value
      else
        n = Node.new key, value
      end

      @h[key] = n
    end

    front(n).value = value
  end

  # remove the node and invoke the cleanup proc
  # if set
  def remove_node(node)
    n = @h.delete(node.key)
    n.unlink
    release_proc and release_proc[n.key, n.value]
    n
  end

  # remove the oldest node returning the node
  def delete_oldest
    n = @tail.pred
    raise "Cannot delete from empty hash" if @head.equal? n
    remove_node n
  end

You can see the whole story in the git repo where the LRUHash is also integrated into the main animal code. Class InteractionProcessor has changed a bit as well as Coordinator. Some places are still a bit inelegant but that will have to wait until I have a bit more time.

25th Post!

gregory.t.brown@gmail.com (Gregory Brown) — Tue, 07 Jul 2009 05:00:00 -0000

This is the 25th post to RBP blog! Hooray!

We’ve been really happy with how things have gone so far. Conversations have been for the most part civil and lively, we haven’t run out of content, and the blog has seen new content nearly every week.

I know that we aren’t seeing as many voices on a week to week basis as we hoped for, but let’s face it, people are bound to get busy. That having been said, it’d be great to know what readers are thinking about our content so far.

So, what do you think? Ranging from technical issues to ones of personal taste, it’d be nice to know what keeps RBP in your feed reader, and what we can do to keep it there. All feedback is welcome, so go ahead and let us know what’s on your mind.

If you’ve got a great idea for a post, a series, or just some minor nitpick you want to voice, go for it. So far we’ve gained a lot from user responses, so I’d like to keep with that tradition and keep the feedback loop tight.

Now that you’ve seen 25 posts, how should the next 25 look?

Code Blocks: Ruby's Swiss Army Knife

gregory.t.brown@gmail.com (Gregory Brown) — Tue, 07 Jul 2009 04:00:00 -0000

The following blog post is a direct excerpt from the Ruby Best Practices book. If you’ve been enjoying this blog, you’d probably love the book, so I’ve decided to release some content here to give you a sense of what to expect. Enjoy!

UPDATE: For those coming from other languages, Ruby’s code blocks are inherently closures , and provide syntactic sugar for methods that accept Proc objects (Ruby’s anonymous functions). While not strictly necessary for understanding this article, a solid grasp on what closures are and how they work will take you far.

In Ruby, code blocks are everywhere. If you’ve ever used Enumerable, you’ve worked with blocks. But what are they? Are they simply iterators, working to abstract away our need for the for loop? They certainly do a good job of that:

  >> ["Blocks","are","really","neat"].map { |e| e.upcase } 
  => ["BLOCKS", "ARE", "REALLY", "NEAT"]

But other blocks don’t really iterate over collections—they just do helpful things for us. For example, they allow us to write something like:

  File.open("foo.txt","w") { |f| f << "This is sexy" }

instead of forcing us to write this:

  file = File.open("foo.txt","w") 
  file << "This is tedious" 
  file.close

So blocks are useful for iteration, and also useful for injecting some code between pre-processing and post-processing operations in methods. But is that all they’re good for? Sticking with Ruby built-ins, we find that isn’t the case. Blocks can also shift our scope temporarily, giving us easier access to places we want to be:

 "This is a string".instance_eval do 
    "O hai, can has reverse? #{reverse}. kthxbye" 
  end 
  #=> "O hai, can has reverse? gnirts a si sihT. kthxbye"

But blocks aren’t necessarily limited to code that gets run right away and then disappears. They can also form templates for what to do down the line, springing to action when called for:

  >> foo = Hash.new { |h,k| h[k] = [] } 
  => {} 
  >> foo[:bar] 
  => [] 
  >> foo[:bar] << 1 << 2 << 3 
  => [1, 2, 3] 
  >> foo[:baz] 
  => []

So even if we label all methods that accept a block as iterators, we know the story runs deeper than that. With this in mind, we can leverage some basic techniques to utilize any of the approaches shown here, as well as some more advanced tricks. By doing things in a way that is consistent with Ruby itself, we can make life easier for our users. Rather than piling on new concepts, we can allow them to reuse their previous knowledge. Let’s take a look at a few examples of how to do that now.

Working with Enumerable

The most common use of blocks in Ruby might be the most trivial. The following class implements a basic sorted list, and then mixes in the Enumerable module. The block magic happens in each():

  class SortedList 
    include Enumerable 
  
    def initialize 
      @data = [] 
    end 
  
    def <<(element) 
      (@data << element).sort! 
    end 
  
    def each 
      @data.each { |e| yield(e) } 
    end 
  end

Our each() method simply walks over each element in our @data array and passes it through the block provided to the method by calling yield. The resulting iterator works exactly the same as Array#each and Hash#each and all the Ruby built-ins, and indeed simply wraps Array#each in this case:

  >> a = SortedList.new 
  => #<SortedList:0x5f0e74 @data=[]> 
  >> a << 4 
  => [4] 
  >> a << 5 
  => [4, 5] 
  >> a << 1 
  => [1, 4, 5] 
  >> a << 7 
  => [1, 4, 5, 7] 
  >> a << 3 
  => [1, 3, 4, 5, 7] 
  >> x = 0 
  => 0 
  >> a.each { |e| x += e } 
  => [1, 3, 4, 5, 7] 
  >> x 
  => 20

This shouldn’t be surprising. What is really the interesting bit is that by including the module Enumerable, we gain access to most of the other features we’re used to working with when processing Ruby’s built-in collections. Here are just a few examples:

  >> a.map { |e| "Element #{e}" } 
  => ["Element 1", "Element 3", "Element 4", "Element 5", "Element 7"] 
  >> a.inject(0) { |s,e| s + e } 
  => 20 
  >> a.to_a 
  => [1, 3, 4, 5, 7] 
  >> a.select { |e| e > 3 } 
  => [4, 5, 7]

In a lot of cases, the features provided by Enumerable will be more than enough for traversing your data. However, it’s often useful to add additional features that build on top of the the Enumerable methods. We can show this by adding a simple reporting method to SortedList:

  class SortedList 
    def report(head) 
      header = "#{head}\n#{'-'*head.length}" 
      body = map{|e| yield(e)}.join("\n") + "\n" 
      footer = "This report was generated at #{Time.now}\n" 
      [header, body, footer].join("\n") 
    end 
  end

which, when run, produces output like this:

>> puts a.report("So many fish") { |e| "#{e} fish" } 
  So many fish 
  ------------ 
  1 fish 
  3 fish 
  4 fish 
  5 fish 
  7 fish 
  This report was generated at 2008-07-22 22:47:20 -0400

Building custom iterators is really that simple. This provides a great deal of flexibility, given that the code block can execute arbitrary expressions and do manipulations of its own as it walks across the elements. But as mentioned before, blocks can be used for more than just iteration.

Using Blocks to Abstract Pre- and Postprocessing

We looked at the block form of File.open() as an example of how blocks can provide an elegant way to avoid repeating tedious setup and teardown steps. However, files are surely not the only resources that need to be properly managed. Network I/O via sockets is another place where this technique can come in handy.

On the client side, we’d like to be able to create a method that allows us to send a message to a server, return its response, then cleanly close the connection. The first thing that comes to mind is something simple like this:

  require "socket" 

  class Client 

    def initialize(ip="127.0.0.1",port=3333) 
      @ip, @port = ip, port 
    end 

    def send_message(msg) 
      socket = TCPSocket.new(@ip,@port) 
      socket.puts(msg) 
      response = socket.gets 
    ensure 
      socket.close 
    end 
  
  end

This is reasonably straightforward, but what happens when we want to add another method that waits to receive a message back from the server?

  require "socket" 
  
  class Client 
    
    def initialize(ip="127.0.0.1",port=3333) 
      @ip, @port = ip, port 
    end 
    
    def send_message(msg) 
      socket = TCPSocket.new(@ip,@port) 
      socket.puts(msg) 
      response = socket.gets 
    ensure 
      socket.close 
    end 
    
    def receive_message 
      socket = TCPSocket.new(@ip,@port) 
      response = socket.gets 
    ensure 
      socket.close 
    end 
  end

This is starting to look messy, as we have repeated most of the code between send_message and receive_message. Ordinarily, we’d break off the shared code into a private method that the two could share, but the trouble here is that the difference between these two methods is in the middle, not in a single extractable chunk. This is where blocks come to the rescue:

  require "socket" 

  class Client 
    def initialize(ip="127.0.0.1",port=3333) 
      @ip, @port = ip, port 
    end 
  
    def send_message(msg) 
      connection do |socket| 
        socket.puts(msg) 
        socket.gets 
      end 
    end 

    def receive_message 
      connection { |socket| socket.gets } 
    end
     
    private 
    
    def connection 
      socket = TCPSocket.new(@ip,@port) 
      yield(socket) 
    ensure 
      socket.close 
    end 
  
  end

As you can see, the resulting code is a lot cleaner. As long as we use our connection() method with a block, we won’t need to worry about opening and closing the TCPSocket; it’ll handle that for us. This means we’ve captured that logic in one place, and can reuse it however we’d like.

To make things a bit more interesting, let’s take a look at a simple server with which this code can interact, which gives us a chance to look at yet another way that blocks can be useful in interface design.

Blocks as Dynamic Callbacks

There is a lot of power in being able to pass around code blocks just like they were any other object. This allows for the capability of creating and storing dynamic callbacks, which can later be looked up and executed as needed.

In order to play with our Client code from the previous example, we’re going to create a trivial TCPServer that attempts to match incoming messages against patterns to determine how it should respond. Rather than hardcoding behaviors into the server itself or relying on inheritance to handle responses, we will instead allow responses to be defined through ordinary method calls accompanied by a block. Our goal is to get an interface that looks like this:

  server = Server.new 

  server.handle(/hello/i) { "Hello from server at #{Time.now}" } 
  server.handle(/goodbye/i) { "Goodbye from server at #{Time.now}" } 
  server.handle(/name is (\w+)/) { |m| "Nice to meet you #{m[1]}!" } 

  server.run

The first two examples are fairly simple, matching a single word and then responding with a generic message and timestamp. The third example is a bit more interesting, repeating the client’s name back in the response message. This will be accomplished by querying a simple MatchData object, which is yielded to the block.

Though making this work might seem like black magic to the uninitiated, a look at its implementation reveals that it is actually a fairly pedestrian task:

  class Server 
    
    def initialize(port=3333) 
      @server   = TCPServer.new('127.0.0.1',port) 
      @handlers = {} 
    end 
    
    def handle(pattern, &block) 
      @handlers[pattern] = block 
    end 
    
    def run 
      while session = @server.accept 
        msg = session.gets 
        match = nil 
        @handlers.each do |pattern,block| 
          if match = msg.match(pattern) 
            break session.puts(block.call(match)) 
          end 
        end 
        unless match 
          session.puts "Server received unknown message: #{msg}" 
        end 
      end 
    end
     
  end

The handle() method slurps up the provided block using the &block syntax, and stores it in a hash keyed by the given pattern. When Server#run is called, an endless loop is started that waits for and handles client connections. Each time a message is received, the hash of handlers is iterated over. If a pattern is found that matches the message, the associated block is called, providing the match data object so that the callback can respond accordingly.

If you’d like to try this out, use the following code to spin up a server:

  server = Server.new 
  server.handle(/hello/i) { "Hello from server at #{Time.now}" } 
  server.handle(/goodbye/i) { "Goodbye from server at #{Time.now}" } 
  server.handle(/name is (\w+)/) { |m| "Nice to meet you #{m[1]}!" } 
  server.run

Once you have that running and listening for connections, execute the following client code:

  client = Client.new 
  
  ["Hello", "My name is Greg", "Goodbye"].each do |msg| 
    response = client.send_message(msg) 
    puts response 
  end

You will get back something like this:

  Hello from server at Wed Jul 23 16:15:37 -0400 2008 
  Nice to meet you Greg! 
  Goodbye from server at Wed Jul 23 16:15:37 -0400 2008

It would be easy to extend both the client and server to do more interesting things that build on this very simple foundation. Feel free to take a few minutes to play around with that, and then we’ll look at one more block trick that’s fairly common in Ruby.

Blocks for Interface Simplification

Does it feel like the word “server” is written too many times in this code?

  server = Server.new 
  server.handle(/hello/i) { "Hello from server at #{Time.now}" } 
  server.handle(/goodbye/i) { "Goodbye from server at #{Time.now}" } 
  server.handle(/name is (\w+)/) { |m| "Nice to meet you #{m[1]}!" } 
  server.run

When you see code like this, it might be a sign that you could do better. Although there are merits to this somewhat standard approach, we can cheat a little bit with blocks (of course) and make things prettier. It would be nice to be able to write this instead:

  Server.run do 
    handle(/hello/i) { "Hello from server at #{Time.now}" } 
    handle(/goodbye/i) { "Goodbye from server at #{Time.now}" } 
    handle(/name is (\w+)/) { |m| "Nice to meet you #{m[1]}!" } 
  end

As you may recall from an earlier example, it is possible to execute a block within the scope of an instantiated object in Ruby. Using this knowledge, we can implement this handy shortcut interface as a simple class method:

  class Server 
    # other methods same as before 
    def self.run(port=3333,&block) 
      server = Server.new(port) 
      server.instance_eval(&block) 
      server.run 
    end 
  end

This is all you need to get the new interface running, and rounds off our quick exploration of the different ways that you can use blocks to improve your API design while simplifying your method implementations. Let’s recap with a few tips before we wrap up here.

Things to Remember

Keep the following things in mind when using blocks as part of your interface:

If you create a collection class that you need to traverse, build on top of Enumerable rather than reinventing the wheel.

If you have shared code that differs only in the middle, create a helper method that yields a block in between the pre/postprocessing code to avoid duplication of effort.

If you use the &block syntax, you can capture the code block provided to a method inside a variable. You can then store this and use it later, which is very useful for creating dynamic callbacks.

Using a combination of &block and instance_eval, you can execute blocks within the context of arbitrary objects, which opens up a lot of doors for highly customized interfaces.

The return value of yield (and block.call) is the same as the return value of the provided block.

Did you enjoy this post?

If you had fun reading this, definitely consider picking up a copy of Ruby Best Practices . If you need more convincing, there is a whole sample chapter for you to check out in addition to this excerpt. If you’ve already picked up the book and are enjoying it, tell your friends, and consider leaving a review on the Amazon product page . If you don’t have enough time for that, go help me win the popularity contest over at RubyTrends, as there are several books that I admire that still need to be overcome :)

As always, I welcome your thoughts and feedback. Let me know what you think of the techniques shown here, or share your own personal favorite code block trick.

Animal Interaction Processing

shortcutter@googlemail.com (Robert Klemme) — Tue, 30 Jun 2009 16:30:00 +0000

As stated I intend to exploit locality of interaction log messages by using LRU cache algorithm and handle log lines as efficiently as possible. The general concept looks like this:

Whenever a new interaction is seen in the log file, create a new InteractionProcessor for it and put it in some storage which allows for fast access.
Any further processing of log lines read for a particular interaction is dealt with by that InteractionProcessor instance.
The storage should have an upper limit on the number of entries and expire entries via a cache algorithm in order to avoid overusing memory.
To further reduce memory usage, write log lines of an interaction to the corresponding output file as soon as possible.

It should be immediately clear that this confronts us with a number of challenges we will now have to look at.

Challenge: Early Expiry

Considering that we do not have a means to detect the end of an interaction we can only rely on the expiry algorithmof our InteractionProcessor storage. This means, there is always the chance that an InteractionProcessor is evicted from the storage although there are more log records to come for this interaction. This situations becomes more likely if

the size limit of the storage is reduced,
the number of interactions which are active at one point in time increases.

The latter can be caused by high activity on the original system (many interactions are started in a time interval) or interactions having huge gaps (making them live longer).

Now, what does it mean for our processing? Since we can only detect the initial line of an interaction through the fact that there is not yet an interaction processor for this interaction, we may be in a situation that we believe this is a new interaction while in fact we have seen it already. This may lead to wrong filtering results (for example, if the beginning timestamp of this interaction needs to be evaluated for filtering and the difference between the real first timestamp and the second “first” timestamp makes a difference).

There are a few things that we can do:

Check the filesystem for this interaction and read previous log records into memory before continuing processing this interaction.
Remember all interaction ids along with their initial timestamps or with the file name to know whether an interaction has been seen already and to find it efficiently in the filesystem to read previous records (see 1).
Nothing.

Option 1 is very inefficient, because we do not know the initial timestamp and so must potentially search multiple directories. Option 2 is dangerous because it does not impose any limits on the storage needed for interaction ids with their timestamps. That leaves us with option 3 – a seemingly bad choice on first sight.

To comfort you let’s try to find out how dangerous it actually is to do nothing about it. Assuming for the moment that test-gen.rb has a realistic model of log files we will see in practice. Now let’s also hope I get the math right… Looking into the source code, you will see that the average time interval between two lines of a single interaction is 5 seconds:

Timer = Struct.new :t do
  def tic
    self.t += rand(10_000) / 1000.0
  end
end

Furthermore we see that there are 1000 new interactions per minute (variable commands) or, put differently, a new interaction starts every 60ms. Furthermore there are on average 5.5 (2 + 3.5) log records per interaction (note, I fixed the time distribution of start times which wasn’t uniform in the first version of the generator):

while t < end_time
  # create new entries
  commands.times do
    id = generate_id
    tx.t = t + rand(60_000) / 1000.0

    entries.add(Entry.create(tx.t, id, 'START'))

    (2 + rand(8)).times do
      pick = rand(10)
      msg = pick == 0 ? MESSAGES.last : MESSAGES[pick % 2]
      entries.add(Entry.create(tx.tic, id, msg))
    end

    entries.add(Entry.create(tx.tic, id, 'END'))
  end

  # write entries of this minute
  entries.print t
  t += 60
end

This means, the average life span of an interaction is 22.5 seconds (5 sec * (5.5 – 1)). So now, continuing with our average calculation, this means that there are roughly 375 new interactions in a 22.5 second interval (22.5 sec / 60ms). So, for the average case keeping 400 InteractionProcessors in memory is sufficient. Of course we need to take into accound that this is just an average calculation and also does not cover temporary load surges but if we apply factor 100 (i.e. 40,000 storage limit) this still doesn’t look like it will blast system memory and we should be safe for most situations (unfortunately my stochastic skills are totally rusty today, so if someone cares to calculate the likelyhood of failure that would be interesting to see). Assuming that we need 0.5 KB per InteractionProcessor because of log lines that need to be kept in memory we’re at about 20 MB which isn’t a lot of memory these days.

To sum it up: with the formula (avg entries - 1) * avg interval * new IA per minute / 60s we get the average number of interactions active at a time. It turns out that for our sample log generator this is 375 which leaves a lot of headroom for the interaction processor storage. Without diving into stochastics too much it is obvious that we are pretty safe if we do not implement a solution for dealing with interactions that were expired too early.

Challenge: How to keep as few log lines in memory as possible?

Since our concept includes getting rid of log lines from memory as soon as possible, we will have to look at how this might be achieved. Remember, we want to only output interactions which are included in our filter criteria, which can be quite different and need to look at

interaction id,
initial timestamp,
last timestamp,
a string matched somewhere in any of the log lines.

It should be immediately clear that not all of these criteria can be evaluated when the initial line of an interaction is seen. A simple matches / does not match logic won’t help here. We need a filter that can tell us “matches”, “does not match” and “maybe matches” where the first two answers must only be given if there is no new information (log lines) which can make it invalid. So we will use a filter that returns any of :yes, :no or :maybe. This is our dummy filter, which also has a changed interface:

    YES = Class.new do
      def first(iap, line, ts) :yes end
      def initial(iap, line, ts) :yes end
      def followup(iap, line) :yes end
    end.new

We do not use the complete InteractionProcessor as argument but add the current line and also the timestamp if needed. This also leads to three processing states in our InteractionProcessor: undecided, including, excluding. We handle this using the state pattern:

  class InteractionProcessor

    UNDECIDED = Class.new do
      def process_initial(iap, line, time_stamp)
        case iap.coord.filter.first(iap, line, time_stamp)
        when :yes
          # we'll improve this once LRU is there
          iap.entries << Entry.new(time_stamp, line)
          INCLUDE
        when :no
          iap.entries.clear
          EXCLUDE
        when :maybe
          iap.entries << Entry.new(time_stamp, line)
          self
        else
          raise 'Illegal return'
        end
      end

      def process(iap, line, time_stamp)
        case iap.coord.filter.initial(iap, line, time_stamp)
        when :yes
          # we'll improve this once LRU is there
          iap.entries << Entry.new(time_stamp, line)
          INCLUDE
        when :no
          iap.entries.clear
          EXCLUDE
        when :maybe
          iap.entries << Entry.new(time_stamp, line)
          self
        else
          raise 'Illegal return'
        end
      end

      def append_line(iap, line)
        case iap.coord.filter.followup(iap, line)
        when :yes
          # we'll improve this once LRU is there
          l = iap.entries.last and l.line << line
          INCLUDE
        when :no
          iap.entries.clear
          EXCLUDE
        when :maybe
          l = iap.entries.last and l.line << line
          self
        else
          raise 'Illegal return'
        end
      end
    end.new

    INCLUDE = Class.new do
      def process_initial(iap, line, time_stamp)
        iap.entries << Entry.new(time_stamp, line)
        self
      end

      def process(iap, line, time_stamp)
        iap.entries << Entry.new(time_stamp, line)
        self
      end

      def append_line(iap, line)
        l = iap.entries.last and l.line << line
        self
      end
    end.new

    EXCLUDE = Class.new do
      def process_initial(iap, line, ts)
        self
      end

      def process(iap, line, ts)
        self
      end

      def append_line(iap, line)
        self
      end
    end.new

  # ...

    # Process the first line
    def process_initial(time_stamp, line)
      @state = @state.process_initial(self, line, time_stamp)
    end

    # Process an initial line
    def process(time_stamp, line)
      @state = @state.process(self, line, time_stamp)
    end

    # Append a continuation line to the last entry
    def append_line(line)
      @state = @state.append_line(self, line)
    end

  # ...

  end

Challenge: Number of Open File Descriptors

If we store 40,000 InteractionProcessors in memory then the worst case with regard to file descriptors is that all have one open. That’s more than usually allowed for user processes. We’ll remedy this by using a LRU strategy here as well: we will have a set of open files and close the least recently used if we hit a fixed limit. I will cover this in one of the next versions, when we’ll have LRU handling integrated. (Maybe there is even a gem we can reuse.)

Summary and Outlook

This has grown into a rather largish article but I wanted to cover InteractionProcessor with reasonable completion. Note, that we still do not have the LRU mechanics and a couple of other things so this class will change again. Next articles will have to deal with LRU implementation if I do not find anything suitable and integration of that into the rest of the application. Furthermore, I expect the implementation of the filtering to be one more complex and thus interesting piece. Stay tuned.

The Decorator Delegator Disco

gregory.t.brown@gmail.com (Gregory Brown) — Sat, 27 Jun 2009 16:00:00 -0000

As promised, we’re back with more design discussion and hopefully some interesting ideas. Let’s start with a quick recap of what has happened so far.

First, Sandi Metz posted on her blog about type specific coupling that a case statement can introduce. Her proposed solution encouraged us to pass the responsibilities down to the individual objects, which is certainly desirable.

Aaron Patterson got in the mix, pointing out the issues that can happen when you are too aggressive about adding methods to core Ruby. When Sandi suggested that Double Dispatch might be a way around this problem, Aaron responded with an implementation using the visitor pattern to accomplish this sort of thing.

I had not seen Aaron’s initial work before setting out to solve the problem on my own, and ended up taking an entirely different path to get there. I’m not much of a patterns guy, but after I had a working implementation, it seems like the approach I took mostly represents a combination of Delegation and Decoration techniques.

A quick demo

I think this code will be easier to understand if I show how it is used right up front. So here’s a tiny slice to get started with:

  module ActionView
    module Helpers 
       class Extensions
          extend Decoration
        
          decorator_for(Array) do
            def as_array
              self
            end
          end
        
          decorator_for(Object) do
            def as_array
              [self]
            end
          end
       end
     
       # totally incomplete, just returns decorated object
       def form_for(record_or_name_or_array, *args, &block)
         Extensions.decorate(record_or_name_or_array)
       end
    end
  end

  include ActionView::Helpers

  p form_for(:foo).as_array #=> [:foo]
  p form_for(Object.new).as_array #=> [#<Object:0x3f47ec>]
  p form_for([1,2,3]).as_array #=> [1,2,3]

In the code above, I’ve implemented Sandi’s as_array helper. I left the actual form_for implementation very incomplete, causing it just to return the decorated object. This is for example purposes only, so you could see that we can indeed call as_array successfully.

Before we check out how this all works, here are two extra hints about what this approach is affording us.

  my_obj = [1,2,3]
  decorated = form_for(my_obj)
  
  p decorated.as_array #=> [1,2,3]
  
  # Raises NoMethodError
  [1,2,3].as_array
  
  # Also raises NoMethodError
  p my_obj.as_array

As you can see, if core extensions are like a meat cleaver, this approach is more like using a scalpel, no changes bleed out into unexpected places. Now that we see how it is used, let’s take a look under the hood of Decorate to see how it is built.

The `Decorate` module

Though fairly densely packed, the whole implementation of Decorate is pretty tiny.

require 'delegate'

# with minor tweaks from tjstankus
module Decoration
  def decorator_for(*types, &block)
    types.each do |type|
      decorators[type] = Module.new(&block)
    end
  end
 
  def decorators
    @decorators ||= {}
  end
 
  def decorate(target)
    obj = SimpleDelegator.new(target)
    
    # walk in reverse order so most specific patches get applied LAST
    target.class.ancestors.reverse.each do |a|
      if decorators[a]
        obj.extend(decorators[a])
      end
    end
    
    return obj
  end
end

Here, we are taking advantage of SimpleDelegator, a class provided from Ruby’s delegate standard library. Here’s a quick glimpse of how it works:

>> require "delegate"
=> true

>> a = [1,2,3]
=> [1, 2, 3]

>> b = SimpleDelegator.new(a)
=> [1, 2, 3]
>> b << 4
=> [1, 2, 3, 4]
>> b.delete_at(1)
=> 2

>> b
=> [1, 3, 4]
>> a
=> [1, 3, 4]

>> b.class
=> SimpleDelegator
>> a.class
=> Array

Essentially, SimpleDelegator provides an easy way to wrap an object within a proxy. This gives us a convenient place to hang per-object behavior without mutating the original object.

  >> def b.kitties
  >>   "MEEEOW"
  >> end
  >> b.kitties
  => "MEEEOW"
  >> a.kitties
  NoMethodError: undefined method `kitties' for [1, 3, 4]:Array
  	from (irb):15

You probably see where this is going now, so let’s get back to Decorate module. If we look at decorator_for, we can get a good sense of how all this magic comes together:

def decorator_for(*types, &block)
  types.each do |type|
    decorators[type] = Module.new(&block)
  end
end

def decorators
  @decorators ||= {}
end

Here, we see that decorator_for basically takes a type or list of types, and a block. The block is then used to define the body of an anonymous module that gets stashed away in a hash for later use. So going back to our example code, we see that the following definitions are actually module definitions. They could have even been written this way:

class Extensions
   extend Decoration
 
   module ArrayDecorator
     def as_array
       self
     end
   end
 
  module ObjectDecorator
     def as_array
       [self]
     end
  end
  
  decorators[Array]  = ArrayDecorator
  decorators[Object] = ObjectDecorator
end

Of course, this lacks the necessary sugar shock, so I show it only to make the implementation clearer. But now that we have a way to stack up modules with per-object behavior in them, and a place to hang that per-object behavior off of, we only need a mechanism to apply it. For this functionality, we turn to the decorate method:

  def decorate(target)
    obj = SimpleDelegator.new(target)
  
    # walk in reverse order so most specific patches get applied LAST
    target.class.ancestors.reverse.each do |a|
      if decorators[a]
        obj.extend(decorators[a])
      end
    end
  
    return obj
  end

We want to be able to preserve the ability to use super in our decorated method calls, and we definitely wanted to pick up extensions to objects higher in the inheritance chain without any duplication. So this meant we needed to do something a little more complicated than key the target object’s type directly to a decorator module.

Rather than just picking the most recent ancestor, this code walks the chain in reverse order so that the modules can neatly stack on top of one another. In short, this means that the decorator for Object gets applied before the one for Array so that you can emulate an inheritance relationship in your extensions.

This pretty much sums up the whole implementation, so let’s just take a look at one more example before evaluating the pros and cons of this approach.

An alternative to core extensions

Here is a bit of code that completely covers the core extensions that Sandi needed for her Rails patch, without the core extensions:

   
  class Extensions

     extend Decoration
     
     decorator_for Symbol, String do
       def form_object_name
         self
       end

       def as_fields_for_form_object(args)
         args.first
       end

       def acts_like_model?
         false
       end
     end

     decorator_for Array do
       def as_form_object
         last
       end

       def as_array
         self
       end
     end

     decorator_for Object do
       def as_form_object
         self
       end

       def as_fields_for_form_object(args)
         as_form_object
       end

       def form_object_name
         ActionController::RecordIdentifier.singular_class_name(self)
       end

       def as_array
         [self]
       end

       def acts_like_model?
         true
       end
     end
     
  end

You would use the same approach I suggested before to wire this up, something like Extensions.decorate(my_obj)

Pros for this solution

There are definitely things about this technique that I like. The obvious benefit is that it prevents core hackery, and even goes a step beyond something like my_object.extend(SomethingSpecial), since it only extends behavior into a throwaway proxy object.

Another clear benefit is that defining extensions is about as easy as it is to do raw core extensions. Because the blocks are used as module definitions, you can do pretty much anything you’d ordinarily do when defining methods. This lowers the amount of ‘special stuff’ you need to remember.

Perhaps the most interesting thing however, is that this approach can be used to apply customized behavior to several types of objects at once, and can also be used to apply behaviors based on what mixins are included in an object. This means there is nothing stopping you from doing something like decorator_for(Enumerable) or decorator_for(Comparable), which provides you a little extra flexibility beyond the traditional inheritance hierarchy.

Another interesting aspect of this is that decorators can be implemented at a per-module (or per-class) level, which means that different libraries can have different extensions without interfering with each other.

Keep in mind, it’s not all roses. Let’s take a look at some of the pitfalls.

Cons for this solution

A major drawback here is that we’re still relying on types. Whether its a class or module, we still expect an object to be a certain thing, which sort of goes against traditional duck typing in which we’d rather make assumptions based on what an object can do rather than what it is. So it might be nicer to apply these decorators based on what the target object responds to, not what ancestors it has.

There is also the performance overhead factor. I have not benchmarked yet, but I suspect that the cost of both building up a SimpleDelegator and doing proxy calls is non-negligible. While this may not be an issue in many cases, for code that needs to run in tight loop or be scaled heavily, this approach may be too heavy for its benefits to outweigh its costs. But of course, this is a huge YMMV situation, and without hard numbers it’s tough to say to what extent this would be a huge issue.

Perhaps the most fatal flaw with the current setup is that respond_to? gets delegated to the target object, so with the current implementation, unless you define respond_to? yourself, it will not pick up any of your new methods. I imagine that this is something that can be fixed, I just have not looked into it yet.

Finally, there is the legitimacy problem. As Sandi said in the comments on Aaron’s article, sometimes a case statement is just a case statement. If we go around filling all our code with high-architecture bazookas when all that was needed was a pen-knife, there are bound to be consequences. Is this overkill or awesome? I don’t really know, and I imagine it depends on the situation.

Conclusions

When Aaron and I compared our solutions to Sandi’s problem, we basically concluded that both of our approaches might be useful for different things. Aaron’s shines when dealing with providing type-specific implementations of a particular behavior, which is mostly just the visitor pattern shining through. My approach is probably better for situations in which you need to collect several behaviors in one place.

Ultimately, this may be a matter of preference, depending on whether you want to abstract things based on the method level or the object level. Of course, sometimes neither is necessary, and that may be an important fact to remember.

I am very much interested in what readers have to say. Please check out Sandi’s article , then Aaron’s , then let me know here which approaches you might take along with their context. If you have your own ideas that are totally different than ours, I want to know about that too.

We need more conversations like these in our community, so let’s start here and see where it takes us.

The Animal raises its head

shortcutter@googlemail.com (Robert Klemme) — Mon, 22 Jun 2009 22:36:00 +0000

You can find the first sample-animal.rb over there at github. Deficits of this version:

No command line parsing,
No filtering,
Works only for moderately sized files,
No particular optimizations yet (well, apart from freezing of id Strings).

One remark about method main: this is the result of asking myself “What is the minimal interface I can provide for users of the Animal and who basically only need to define the parser class?” At first I found it a bit odd, but in terms of lines of code this is likely one of the leanest solutions.

I’ll have to call it a day now, but I invite you to have a look and comment. Do you consider method main a good idea? I’ll chime in with some more explanations later.

[Update:_] If you are looking for the entry point of the implementation for the sample log you need to look into file sample-animal.rb:

#!/usr/local/bin/ruby19 -w

# Implementation of the parser for the sample
# generator script test-gen.rb.
$: << File.join(File.dirname(File.dirname($0)), "lib")

require 'time'
require 'animal'

# main defines the custom parser class!
Animal.main do

  TIME_FORMAT = '%Y-%m-%d %H:%M:%S.%N'.freeze

  attr_accessor :year
  attr_reader :interaction_id, :time_stamp

  def parse(line)
    if %r{
       ^
       ( \d{4}-\d{2}-\d{2} \s \d{2}:\d{2}:\d{2}(?:\.\d+)? )
       \s+
       (\S+) # interaction_id
       \s+
       }x =~ line
      @time_stamp = Time.strptime $1, TIME_FORMAT
      @interaction_id = $2
    else
      @time_stamp = nil
      @interaction_id = nil
    end
  end

  def initial_line?
    time_stamp
  end
end

# EOF

The intersting thing is: this is complete already! As long as the file format does not change we won’t have to touch this file any more. Basically we only provide information about the log file format through the parser whose class is defined with the code block passed to main. The rest – command line argument parsing, creating of all necessary objects etc. is done in main which is not much longer either:

require 'ostruct'

# Namespace for all the Animal related classes
# Animal is the project on Ruby Best Practices
# blog which demonstrates the thought process
# of writing an application
module Animal

  # Parse the given command line and return an option instance.
  # Options are removed from the argument so what is left in
  # there must be file names.
  def self.parse_command_line(argv = ::ARGV) 
    o = OpenStruct.new(:output_dir => ".")
    # parse
    o
  end

  # This metho allows to write extremely short applications
  # because it accepts a block which is used to define the
  # parser class.  Alternatively users can provide a parser
  # instance.  The default command line is added implicitly
  # and will be option parsed and the whole processing will
  # start automatically.
  def self.main(parser = nil, argv = ::ARGV, &class_body)
    $stderr.puts 'WARNING: ignoring class body' if parser && class_body
    parser ||= Class.new(&class_body).new
    options = parse_command_line(argv)
    coord = Coordinator.new
    coord.parser = parser
    coord.options = options
    coord.process_files argv
  end

  # autoload init
  %w{
    Coordinator
    ProcessingStorage
    FileStatistics
    InteractionProcessor
  }.each do |cl|
    autoload cl, "animal/#{cl.gsub(/([a-z])([A-Z])/, '\\1_\\2').downcase}"
  end

end

There is really not much magic in main apart from (ab)using the method block as class body. (Btw, this is something you can do with Struct as well. I’ll write about that in a future post.) I can’t think of a leaner interface to get the job done but maybe someone out there has another idea.

I had thought of defining a single regular expression with capturing group indexes for timestamp and key. But then we might also need a way to specify timestamp conversion. And then there are also file formats which have timestamps separated in time and date and even binary formats… This looks like an area we should revisit after v1.0 – maybe there is some room for improvement. For now we’ll stick with the line oriented formats which should cover a pretty wide range – including the requirements.

I am also not convinced that the autoload part is really that great. Initially it looked like a good idea but now I am not sure any more: writing out class and file names is probably better readable and as easy as the automated class name to file name conversion hack.

Now, what do you guys say?

Shadow of the Animal

shortcutter@googlemail.com (Robert Klemme) — Sun, 21 Jun 2009 19:36:00 +0000

Let’s recapitulate where we have been for readers who are new to the blog and for the convenience of others. I started this a while ago as an experiment trying to give some insights into my way of reasoning. I choose processing of large log files as subject which should help me analyze problems on production systems. I do believe that this might also help others as log file analysis is a fairly common task. The key point here is to be able to efficiently sift through many large logfiles and give access to the relevant information (see also the list of requirements and the initial description of the scope).

I picked “Animal” as name of the logfile analyzer because that must sift through tons of log data in a similar way as the like named character of a TV show treated his instrument – ferociously.

In today’s article I will present some major design decisions. Some core components of the architecture have been mentioned in the previous article already so I won’t repeat them here.

Design Decisions

As I want to give insights into my way of reasoning about software and design in particular I will list some of the major design decisions along with my rationale.

No class for individual log entries

The reasoning has been presented in a previous article: I want to avoid the overhead of allocating short lived objects. You can play around with a toy Java project (Eclipse as well as ant) to see the effect I was talking about. I had up to 9% time difference between the straight version and the version which puts data in a special record class.

No meta data

I had pondered the option to use a two step approach with an initial analysis step which will generate meta data which then is used by another program to efficiently extract relevant information. The advantage clearly would have been that we can apply a lot of different filtering and selection criteria without having to go through the initial analysis phase. I have decided against for these reasons:

Easier propagation of relevant data: Others do not need the extraction program and the original log files to look at interactions,
Simpler software structure: Only one program is needed, no writing and reading of meta data needed,
More robust: A change in the location of the original logs does not affect analysis.

Efficient Processing

This is the part I personally find most interesting: how do we manage to stream process log data while still limiting memory usage? We cannot simply aggregate all interactions in a huge Hash and dump the interesting bits at the end because this will burn too much memory (which costs speed through paging) or even make the whole process fail completely. That would be especially bad since in a case of lacking memory we would loose all the work that has been done up to the point.

My idea for solving this involves two components:

A hash based storage with LRU semantics,
An interaction processor.

Basically the LRU storage is responsible for keeping only a limited number of interaction processors in memory while the interaction processor is responsible for dealing as efficiently as possible with a single interaction. This means, the interaction processor will have to efficiently decide whether an interaction is included in the output and write out log lines as soon as possible (!) to a file in a given location.

Output Storage

Every interaction will be stored in its own file. To avoid potential issues with file systems becoming inefficient with thousands of files we will use an output directory tree with these levels:

date, formatted YYYY-MM-DD (example ‘2006-06-21’),
hour and minute, two digit 24 hour clock and minute (example ‘19-11’).

Interaction file names will include the timestamp of the first record. The naming will be like this: “SS.SSS-” where “SS” are the seconds of the timestamp and “SSS” are milliseconds". A valid name might then look like this 2009-06-21/19-11/00.003-sdmyrsmmlbxodfd.

The advantage is that we will not face any filesystem issues because there will be only a few thousand entries per minute. Plus, because of the hierarchical naming of directories and files we can easily use textual sorting of names and get chronological output. If we discover that there are too many entries per minute we can still extend the schema pretty easily to include a directory level for seconds.

Structure

The structure of the application will look like this: there is an option parsing method which outputs options which contain a combination of all processing options. These options will be hand off to the Coordinator which is the main driver of the processing. The Coordinator also needs a Parser instance which is capable of parsing log lines and providing interaction_id as well as a time_stamp of the last parsed entry. The coordinator will create InteractionProcessors as needed and hold them in a structure keyed by interaction_id. Each InteractionProcessor will handle a single interaction only. It knows the coordinator through which it gains access to options and a filter evaluator which determines whether an interaction is included; this filter works on the InteractionProcessor.

Thought Process

Now you might wonder how I arrived at these decisions. Well, I wanted the resulting bit of software to get the job done simply and efficiently. Generating a whole lot of meta data which helps in finding interactions efficiently would have created additional complexity as I have pointed out. So it was logical to rather choose a grepish approach, i.e. read the input once and spit out everything that we are interested in.

Since volume of input data is large we cannot expect to hold all the input in memory at some point in time. So we would rather have to pick a streaming approach. This means, we can only hold as many entities in memory at a time. Luckily, log entries of a single interaction have high locality, i.e. whenever the first occurs follow up lines are not too far away and the interaction end is usually near as well.

Now, I considered two approaches for aging out interactions from the processor’s storage: timestamp based and access based. I picked access based because in reality there are some interactions which take a bit longer than the usual interaction which would make picking a high latency necessary. This in turn would keep short interactions longer in memory than necessary. Also, with this approach you cannot put a hard numeric limit on the number of interactions in memory which could lead to failures for some logs while other logs are processed ok. With a LRU based storage we can set a size limit and avoid this problem. Of course, long interactions which also have many log entries could still cause memory issues but since these are comparatively rare the likelyhood of issues is small.

I will now wander off hacking together a rough application which does not yet do much but should give you an idea of how all the pieces are supposed to work together.

The Double Dispatch Dance

aaron.patterson@gmail.com (Aaron Patterson) — Fri, 19 Jun 2009 20:30:00 -0000

EDITORIAL NOTE: This is the first of hopefully many guest posts to the RBP blog. If you are interested in writing an article for us, contact any of the RBP bloggers for details on how to submit content.

Sandi Metz recently blogged about ruby case statements being similar to calling is_a? on a particular object, and how we should be using duck typing rather than inspecting the type of the class. Her solution was to add methods to some core classes so that they all “quacked” the same way.

I completely agree with Sandi’s sentiments, but at the same time, I don’t like reopening core classes, so I wanted to see if I could accomplish the same task, but minimize my impact on any core classes. The way I chose to accomplish this task is with the double dispatch and visitor pattern .

Double Dispatch

The way the double dispatch pattern works is by adding a method to each object that should know how to double dispatch. That method takes one argument, and calls a method on the argument passing itself in. In my solution, I called the method “accept”, and I want all objects to do double dispatch, so I’ve added that method to Object:

  ##
  # The visitable module makes a class "visitable". It follows the double 
  # dispatch pattern and simply calls "visit" on the argument with itself.
  module Visitable
    def accept visitor
      visitor.visit self
    end
  end

  ##
  # For this example, we want to make *all* objects visitable, so we'll mix 
  # the Visitable module in to Object.
  class Object
    include Visitable
  end

Excellent, but what does this buy me? It gives me an entry point and common interface for dispatching a method based on the object I am visiting. I just implement an object that responds to “visit” and that methods knows how to dispatch.

Our Dispatching Visitor

I’ve implemented a class that responds to “visit”, and based on the type of the object, dispatches to a particular method. If a string is passed in to the “visit” method, it will try to dispatch the object to “visit_String”. If “visit_String” is not defined, it will try look at the string’s ancestor list trying to find a place to dispatch. If it can’t dispatch, an exception is raised. My Visitor base class also has a helper method “visitor_for” to help define handlers.

##
# The Visitor class is our base class for all visitors.  It implements the
# "visit" method which Object#accept will call.
class Visitor
  ###
  # Dynamically create a visitor method for each class in +klasses+
  def self.visitor_for *klasses, &block
    klasses.each do |klass|
      define_method(:"visit_#{klass.name}", block)
    end
  end

  ##
  # This method will examine the class and ancestors of +thing+. For each
  # class in the "ancestors" list, it will check to see if the visitor knows
  # how to handle that particular class. If it can't find a handler for the
  # +thing+ it will raise an exception.
  def visit thing
    thing.class.ancestors.each do |ancestor|
      method_name = :"visit_#{ancestor.name}"
      next unless respond_to? method_name
      return send method_name, thing
    end

    raise "Can't handle #{thing.class}"
  end
end

My base Visitor class is not really of much use at this point. All it knows how to do is accept an object and try to send a method based on the type. Let’s take a look at a visitor that will convert any object to an array.

Array Conversion Visitor

One of the tasks is to convert any object in to an array. I can write a Visitor class that has one responsibility; turning any object in to an Array. The rules for this conversion are easy. If the object is an array, return the array, if it’s something else, return that thing inside an array:

  require 'visitor'
 
  ##
  # This Visitor knows how to turn any object in to an Array
  class AsArray < Visitor
    visitor_for Array do |array|
      array
    end
 
    visitor_for Object do |o|
      [o]
    end
  end

That was pretty easy. If the target of the “accept” method is an array, the visit_Array method will be called an the target object will be returned. If it’s something else, visit_Object will be called and a new object will be created.

I can use this style pattern for converting any object to pretty much any other.

Cleaning up our API

So far my API is kind of strange. I have to call “accept” on the target object and pass in new visitor of some sort. To clean this API up a bit, I’ve written a proxy object that takes any object for it’s constructor, then provides nice method names to abstract the visitor classes:

  require 'visitable'
  require 'visitor'
  require 'as_form_object'
  require 'as_array'
  require 'as_fields_for_form_object'
 
 
  ###
  # This class sets up a proxy to +thing+. The purpose of this class is to let
  # us convert an object to something else without knowing what visitor is
  # required to do that conversion:
  #
  # p form_for([1, :two, [3]]).as_form_object
  #
  class ConversionProxy < Struct.new(:thing)
    def as_form_object
      thing.accept AsFormObject.new
    end
 
    def as_array
      thing.accept AsArray.new
    end
 
    def as_fields_for_form_object args
      thing.accept AsFieldsForFormObject.new(args)
    end
  end
 
  def form_for thing
    ConversionProxy.new(thing)
  end
 
  p form_for(:two).as_form_object
 
  p form_for(:two).as_array
  p form_for([:two]).as_array
 
  p form_for(:two).as_fields_for_form_object [:foo]
  p form_for(['hello world']).as_fields_for_form_object [:foo]

Pros for this solution

With one monkey patch to Object, I now have the ability to convert any object in to any other object without needing to add new methods to core classes.

Each visitor has one specific responsibility. My visitor classes are just normal classes, so if I find any repetition between them, I can refactor and inherit, or refactor and mix in.

Cons for this solution

I still had to monkey patch Object and add a method. The method I added was very generic and simple, but I still had to add it.

Visitation rules may need to change. My current behavior of looking up the ancestor tree may not be desired. Fortunately if I need to change that, my visitor subclass can just override visit.

Double dispatch might be confusing. The code is very simple, but it took me a while to understand and harness it’s power (I have the pooowwwweeerrr!).

Is this any better than case/when statements? I’m not sure. It still works based on object type, but it does nicely encapusulate the conversion logic.

Follow up notes

When I’m implementing this pattern in “the real world” I’ve never done it for core classes. I typically use this for converting my own objects in to some other object. Since I’m performing these actions on my own objects, there is no monkey pathing involved (yay!).

Also, its worth keeping in mind that babbies can form in many ways, and this is only one of them. Tune in next week to watch Greg tackle the same problem using Decorators, to see the pros and cons from yet another angle.

First Design Considerations

shortcutter@googlemail.com (Robert Klemme) — Fri, 12 Jun 2009 20:36:00 +0000

Now the interesting part begins! In an early phase like this I like to look at the problem I want to solve from different angles to get a feeling for implementation options. The output will look like an unsorted collection of notes – and that’s what it is! I don’t claim that this is the best approach around but this helps me to get a better grip of the problem.

Side note: I believe that despite all the methodologies around the way you approach a new problem is highly influenced by your personal way of thinking and working. There is no one size fits all approach and “chaos” at the beginning is nothing to be afraid of. In fact it may well be that “chaos” gives birth to the best ideas.

Obtaining Information about individual Records

Requirement M3 makes it necessary that in some place we have a plug where we can insert something that does log file format specific things. Basically for parsing a single line a block would be sufficient. If the format parses ok, the interaction identifier is returned, if not, we have a continuation line. However, it seems this solution is too simplistic: for time filtering (M6) and time gap detection (S3) we certainly also need the timestamp of the record. M7, which I had forgotten initially, also makes it necessary that the year is somehow synthesized – or inserted into the line parser from surrounding code. So, we will likely end up with something that would be an interface in other programming languages, i.e. we need a parser class which speaks a certain protocol / has a certain API.

Here’s an alternative approach: we create a record class and write our parser in a way that it spits out instances of this record class all the time:

Record = Struct.new :time_stamp, :interaction_id, :message

parser = ...

parser.each do |rec|
  interaction = find_interaction(rec.interaction_id)
  time_gap = rec.time_stamp - interaction.last_time_stamp
  ...
end

From an object oriented point of view there’s much which speaks in favour of this:

A record is an entity in our description of the problem domain and as such it is naturally to have it represented in the application,
We can add record specific functionality to this class (e.g. get all potential keywords used for searching / filtering),
We do not have to care about multiline detection, this is completely encapsulated in the parser,
We can enforce class invariants (e.g. all fields must be non nil and time_stamp must be a Time instance).

However, there’s a price to pay in object allocations and collections. Remember, we are talking about files with tons of records; if all these records are short lived (i.e. not used for storing them somewhere) we’re generating a lot of overhead which the garbage collector has to get rid of again. So I rather lean towards the “parser interface” approach mentioned above:

parser = ...

parser.line = line

if parser.continuation_line?
  last_record << "\n" << line
else
  ...
  last_record = line
  diff = parser.time_stamp - last_time
  ...
end

Now, it’s debatable whether this falls into the category of premature optimisation, but since we know that we are going to crunch a lot of data we should probably avoid this kind of overhead.

Filtering

From the requirements it is clear that we would like to have quite flexible filtering (M6, S3, S4). We have two options:

We apply the filtering during processing.
Processing gathers meta data about the data and we can efficiently query later with whatever filters we like.

Hybrid approaches are also possible, e.g. time filtering during processing but searching for keywords later. From the user’s point of view option 2 is certainly more desirable because it gives greater flexibility. On the other hand this must be balanced with processing overhead and disk space. As usual, if we want great flexibility for queries after analysis we need more disk space for meta data (indexes). In another hybrid approach the types of desired filters are provided as input into the processor so that only indexes for criteria are created which are actually queried later. The “flexible query” approach certainly reminds me of relational database.

When we do online filtering (option 1 above) we need to keep some aspects in mind. For example, if we want to filter by text found in an interaction log message (the part after the interaction id in the sample generator output), that text might not appear in the first record of a particular interaction yet we would want to see all preceeding records as well (M1). Similar for time range filtering: an interaction might have started before the range start but end after it. It should be clear that if we choose this approach we need a way to keep some history of past records during processing. Since we can’t store everything we’ve seen so far we need to decide what we need to keep and what we can discard.

I need to ponder this a bit more but at the moment I would rather choose the online filtering approach for these reasons:

Less disk space used on output,
Potentially less write IO during processing, which would help performance since we are almost certainly IO bound,
Easier distribution to other staff – if we need meta data to get at information efficiently this also means that someone needs to have the software to extract the information he wants (plus the index data and the original data set which is large).

Statistics

Collecting statistics like no. of records and no. of lines per file isn’t too expensive so we probably create a statistics class which records just that.

Application Structure

If you do not know CRC Cards yet this is a good opportunity to look into this simple concept. It does not have the power of an UML collaboration diagram but it helps identify classes and how they might play together.

We can roughly identify some pieces already:

Parser: parse individual lines of the input, identify relevant information (timestamp, interaction id).
FileStatistics: collect various data points about a processed file (lines, time taken…).
OptionProcessor: process command line options.
Coordinator: This is the MCP which binds everything together and coordinates the processing.
ProcessingStorage: this is the foggiest entity so far; it will be responsible for storing information about processed interactions. Since I haven’t decided on the strategy yet, I’ll leave it at this for now. We’ll have to make this more concrete later. One thing is for sure: this will likely the most complex class – or rather set of classes.

So much for initial design considerations. I’ll stop here, to give us a break and allow for some discussion, which probably will turn up new aspects and new ideas.

"And your Mom, too." Ruby-talk best practices.

james.britt@gmail.com (James Britt) — Wed, 10 Jun 2009 21:59:00 +0000

Mailing lists (and bulletin boards and forums and so on) are important parts of becoming a better developers. Even the best programmers need help, and a mailing list such as ruby-talk (or the Usenet version, comp.lang.ruby) is the traditional place to go.

Mailing lists are also a good place to get better at your craft; watch the list, and try to answer questions. Think of it as pop quizzes at Ruby U.

Mailing lists are also good place to toss out ideas, conjecture about the future of Ruby, and have general Ruby-related discussions.

There are, however, right and wrong ways to go about this. The canonical source for what is termed “Netiquette” is Internet Engineering Task Force RFC 1855.

You should read it. However, it’s somewhat lengthy, and some aspects dated. Here, I’d like to focus on a few key items as they relate to Ruby and ruby-talk.

RFC 1855 in a Nutshell

For the tl;dr crowd: Don’t be a complete jerk.

I can’t suggest not being any kind of jerk at all in anyway, because it’s impossible. The Web is a big place. No matter what you say or do, there is every chance that someone, somewhere, will take offence, or think you stupid, or in some way take you the wrong way.

There’s not much you can do about that, but you can be mindful of some general decency. For Rubyists, the key is MINASWAN: Matz is nice, and so we are nice.

This doesn’t mean not being critical, or not pointing out mistakes, or not offering bold opinions. It means directing your comments to ideas, not personalities. It means understanding that being wrong doesn’t make a person bad or stupid, just wrong. And being wrong on one thing does make make a person wrong on everything.

It also means not jumping down someone’s throat because their sense of humor doesn’t jibe with yours.

Discussions on ruby-talk can get heated, and the temptation can be strong to let loose with snark and bile. Typically it ends up counter-productive, but it’s understandable.

Threads would likely be much less fun if everything was sanitized. However, the line needs to be drawn at personal attacks and comments.

Ideas can be foolish; people are not fools simply for expressing a foolish idea. Code can be bad; people are not bad for writing such code.

It’s About Code and Ideas

Few people post to ruby-talk to make personal attacks (barring trolls, who should never ever be fed), but it happens. Before letting loose that little bit of one-upmanship, consider what you are really trying to accomplish. If the goal is hurt someone: Please remember MINASWAN. If you cannot, then please go elsewhere.

If the goal is to see Ruby improve, or to express an idea, or to get help with some code, or to offer help, then stop and think.

That brief moment of satisfaction will fade, and you will have failed in your larger intent.

Ruby-talk has a tradition of being very welcoming to newcomers, with many people willing to help answer even the goofiest of questions. I’ve asked a fair number of those goofy questions, and the help I received encouraged me both to stick with Ruby and to try to return the favor. Please support that.

A Random Set of Guidelines

Assorted do’s and don’ts. If you can think of more, please add them in the comments.

Don’t top-post

See here.

Before retorting with how this or that mail reader or client makes top-posting the default, consider this: the idea is to make things easy for the readers, not for you, the poster.

If the goal is have people read, and understand, what you are posting, then place you reply in context, and trim away irrelevant text.

If you are unpersuaded by arguments against top-posting (and many are), know that many people will simply not bother with what you post.

Do your homework

Ruby is a great language, fairly intuitive to many people, but sometimes you get stuck trying to figure something out. It may seem easy to dash off a note to ruby-talk asking, “How do I do foo?”, but first apply some due diligence. Use Google. Use ruby-doc.org (Handy tip: Ruby-doc.org uses a custom Google search focused on Ruby sites. There’s a special search URL as well: http://www.ruby-doc.org/q/your+search+terms+here.)

Search the ruby-talk archives:

http://groups.google.com/group/ruby-talk-google
http://www.nabble.com/ruby-talk-f13890.html
http://blade.nagaokaut.ac.jp/ruby/ruby-talk/index.shtml.

When you do post, explain what steps you’ve taken to solve the matter on your own. This serves two purposes. First, it shows people you are diligent, making more people more likely to want to help you. It also avoids wasting time with people suggesting things you’ve already done or know about.

Whenever possible, show your code. If you have more than a dozen lines or so, use a pastie site, such as Gist.

Be specific

If you have a problem, give details about your environment. What operating system? What version of Ruby?

Know, too, that there are specialized mailing lists for many Ruby tools and applications. For example, if you have a problem with Rails or Ramaze, please ask on the Rails or Ramaze mailing lists. You are more likely to get a faster, more accurate response on the proper list.

Ask concrete questions. Explain what you are trying to do, what results you get, and how those results differ from what you want or expected.

Don’t threadjack.

That is, do not reply to a post with starting a new topic. If a thread starts to drift off-topic, please consider starting a new thread with a more appropriate topic.

Repetition is not an argument

Don’t expect anyone to change their opinion because you’ve repeated your same points over and over. Accept that sometimes people will never, ever, agree with you. I know, it’s weird, but true.

Watch for perma-threads

There are some topics that come up at regular intervals. Adding static typing to Ruby. Best GUI toolkit. Best IDE. Adding significant indentation.

Before posting, see if you are not simply repeating what has been said. Check the archives. If you really feel you have something new to add, post, but please show that you are familiar with prior discussion.

Conversely, if you see a post that looks to be a perma-thread, don’t assume it has been discussed to death (although that is possible). Certain topics, such as GUI tools or IDEs, see new additions and advancements over time. It doesn’t help to direct people to a thread from last year as the final word on evolving technology.

Walk the walk

People often ask for one or another change to the Ruby language or standard library. This is good, this is how languages evolve.

Well-considered change requests should be posted to the ruby-core mailing list.

Before proposing any change, see if someone hasn’t already implemented it as a library. There are a few libraries, such as Facets, that provide many extensions.

If there is no existing code, see if you can first implement it yourself.

For example, you may think it be would super cool awesome if the Array class could tell you what elements were of a certain size:

  a = %w{ foo bar bar thisoneisdiffernt }
  short_words = a.length_is 3

Before arguing in the abstract just how magical this would make life, code it yourself:

  class Array
    def length_is length
      self.select{ |i| i.size == length }
    end
  end

Make it a gem, or a gist pastie, or otherwise available for others to try. Then explain why it needs to be added to the language rather than used as a gem.

If, for whatever reason, you cannot implement it (even a weak version), explain why.

The bottom line is this: Many, many suggestions have been made, many with the presumption that someone else is expected to do the heavy lifting, swayed only by rhetoric. That’s unfair, and unproductive..

When first attempting to code it yourself, certain results are likely:

You discover that there is no need to build it into the language, since it is trivial to add (e.g. using a gem)
You discover that it is not so easy to add as-needed, but you now have the code for it, so adding it to the language would be that much easier
You discover that, now that you have a working version, you don’t use it as you expected anyway
You find out that some things are hard to do

Best of all, the effort should make you a better Ruby programmer.

Quack Attack: Making Your Code More Rubyish

gregory.t.brown@gmail.com (Gregory Brown) — Tue, 09 Jun 2009 13:00:00 -0000

I’ve been doing some FFI work recently, which means that I’ve needed to wrap underlying C libraries so that I can call them from Ruby. While I won’t get into how the low level wrappers work, I can show you what the raw API calls look like for just a few functions:

  trip = API.PCMSNewTrip(server_id)
  API.PCMSNumStops(trip)
  API.PCMSAddStop(trip, string)
  API.PCMSGetStop(trip, index, string_ptr, buffer_size)
  API.PCMSDeleteStop(trip, index)
  API.PCMSClearStop(trip)

All things considered, this looks pretty good for a direct wrapper on top of a C library. In fact, it’s relatively simple to mirror this to a more normalized Ruby layout. We can start by noticing that these calls are basically object oriented, focusing on the Trip object. While a Trip has other responsibilities, among them is managing a list of stops. With this in mind, we can flesh out a basic Trip object:

class Trip
  def initialize(server_id)
    @server_id = server_id
    @pointer = API.PCMSNewTrip(@server_id)
    @stops = StopList.new(self)
  end
  
  attr_reader :stops
  
  def call(*args)
    API.send("PCMS#{args.first}",  @pointer, *args[1..-1])
  end
end

The Trip#call helper removes some of the duplication for us, but it’ll be easier to see how it works in just a moment. For now, it’s worth pondering what a StopList should be.

If you look at the functions listed for dealing with stops, you’ll notice they map nicely to one of Ruby’s structures. We’re dealing with an ordered list of objects that can be added to and removed from. It can also be queried for its length, and deleted entirely. These features sure sound a lot like Ruby’s Array object, don’t they?

With this in mind, let’s do a quick experiment in interface design:

class StopList

  include Enumerable

  def initialize(trip)
    @trip = trip
  end

  def length
    @trip.call :NumStops
  end

  def <<(loc)
    @trip.call :AddStop, loc
  end

  def [](index)
    ptr = FFI::MemoryPointer.new(:char, 100)
    @trip.call :GetStop, index, ptr, 101
    ptr.read_string
  end

  def delete_at(index)
    @trip.call :DeleteStop, index
  end

  def each
    length.times do |i|
      yield self[i]
    end
  end

  def clear
    @trip.call :ClearStops
  end

end

Without paying too much attention to the implementation details, let’s take a look at what behaviors this new object supports:

  t = Trip.new(server_id)
  
  t.stops << "New Haven, CT"
  t.stops << "Naugatuck, CT"
  t.stops << "Boston, MA"
  
  p t.stops.length #=> 3
  
  p t.stops[2] #=> "02205 Boston, MA, Suffolk"
  
  t.stops.delete_at(1)
  p t.stops[1] #=> "02205 Boston, MA, Suffolk"
  
  p t.stops.map { |e| e.upcase } #=> [ "06511 NEW HAVEN, CT, NEW HAVEN",
                                 #     "02205 BOSTON, MA, SUFFOLK" ]
  
  t.stops.clear
  p t.stops.length #=> 0

If this sort of interaction looks familiar to you, it’s because you’ve likely already done things like this thousands of times. But to make it blindingly obvious, let’s replace Trip#stops with an Array.

stops = []

stops << "New Haven, CT"
stops << "Naugatuck, CT"
stops << "Boston, MA"

p stops.length #=> 3

p stops[2] #=> "Boston, MA"

stops.delete_at(1)
p stops[1] #=> "Boston, MA"

p stops.map { |e| e.upcase } #=> ["NEW HAVEN, CT", "BOSTON, MA"]

stops.clear
p stops.length #=> 0

You’ll notice that aside from lacking the location name normalization that our real code does implicitly, the key points we’ve highlighted have exactly the same behavior, using exactly the same interface. One benefit is immediately obvious after seeing this; the API for StopList doesn’t require you to learn anything new.

A more subtle gain that comes with this approach is that so long as it is restricted to the subset which StopList supports, code which expects an Array-like thing does not need to change, either.

For example, the following code will work just fine with either an Array or a StopList:

def humanized(list)
  list.each_with_index do |e,i|
    puts "#{e} (#{i+1} / #{list.length})"
  end
end

This makes things easier to test, and easier to be re-used for different purposes. Both are solid reasons for using this technique.

Of course, I’ve been glossing over a semi-major issue in the original code here, which I am sure has frustrated our more pedantic readers. The current StopList code, while quite useful, does not quack perfectly with Array. We needn’t look far for signs of divergence.

p t.stops << "Chicago, IL" #=> 1
p t.stops.delete_at(2)     #=> 1
p t.stops.clear            #=> nil

These side-effect bearing functions are returning their C based values, which are different than what you’d expect from a Ruby Array. Luckily, each of these are easy to remedy.

In the case of the append operator (<<), we should return the StopList itself, to permit chaining:

def <<(loc)
  @trip.call :AddStop, loc
  return self
end

To play nice with Array, our delete_at method should return the deleted object:

  def delete_at(index)
    obj = self[index]
    @trip.call :DeleteStop, index
    return obj
  end

Finally, since clear may also be chained, it should return the original object as well.

def clear
  @trip.call :ClearStops
  return self
end

With these fixes in place, we can re-visit our previous example:

p t.stops << "Chicago, IL" #=> #<Trip::StopList:0x0fcac ...>
p t.stops.delete_at(2)     #=> "60607 Chicago, IL, Cook"
p t.stops.clear            #=> #<Trip::StopList:0x0fcac ...>

There are probably some other minor details to catch, but now that our Array-ish StopList is “Good Enough For Government Work”, we have a nice stopping point. Let’s wrap things up with a little summary of things to remember.

Guidelines For Making Your Code More “Rubyish”

This is just one technique among many for improving your code, but I’d argue its a fairly important one. If you have a structure that mimics a subset of a core Ruby object’s capabilities, you can gain a lot by standardizing on a compatible interface. While sometimes the similarities end at the Enumerable and Comparable mixins, it’s reasonable to stretch things farther when it makes sense to do so. If you go that route (as we did here), there are just a few things to keep in mind:

You don’t need to implement every last feature of a core Ruby object in order to use this technique. So many functions rely on just a handful of available methods, that it makes sense to use this technique even when your problem domain is very small.

For the features you do implement, take care to maintain the same interface both on input and output. It’s fine to not support certain use cases, or to add extensions for new ones, but you should not diverge in the behaviors you do implement unless you have a good reason to.

Pay close attention to the return values of side-effect inducing functions, especially the ones mentioned in this article. Many Ruby methods are designed to be chainable, and breaking that feature can create a mess.

While this technique opens the door for using primitive objects for testing higher level functions, do not forget to adequately test the functionality of the actual objects you are implementing. Basically, make sure your code really quacks like a duck before substituting it with a duck somewhere else.

To our readers: Where have you seen good uses of this technique in action? Have you tried it yourself and run into any problems? Please give your answers to these questions, as well as any other discussion points in the comments.

Requirements Summary of the Laboratory Project

shortcutter@googlemail.com (Robert Klemme) — Mon, 08 Jun 2009 21:59:00 +0000

I will now take on the role of project secretary and put all requirements into a form suitable for easier reference. There will be the following groups:

must (M): requirement must be fulfilled
should (S): requirement should be fulfilled but is not a show stopper
nice to have (N): this one is purely optional
future (F): this might be a good idea for a later release but should not influence the current version

I will later reference requirements via letter and number, e.g. “S4” refers to “filter by text” requirement (and not a fast car from a German manufacturer).

Until now there was not much reasoning about software although there were some ideas (and even prototypes) mentioned in the discussion thread of the last article. With the next article I will start to flesh out the architecture (or even some alternatives) along with the reasoning that led to different decisions.

Must have

number	description
1	Provide efficient access to all entries of selected interactions
2	Retain ordering of input lines per interaction
3	Parsing needs to be able to identify multi line entries
4	Parsing of lines needs to be easily adjustable to different line formats
5	Analysis must not rely on any particular text to identify first and last lines of an interaction
6	Filter interactions by time range (between a and b, before a, after a)
7	If timestamps are parsed formats without year must be properly parsed

Should have

number	description
1	Input file names should be provided via command line arguments
2	Analysis should deal gracefully with partial interactions (at the beginning and end of a logfile)
3	Idenfity pauses in interactions with a configurable length and report them
4	Filter interactions by some text contained in messages, ideally via regexp matching
5	Make interactions available in their original formatting

Nice to have

number	description
1	Identification of the proper chronological order of files
2	Read from stdin
3	HTML output
4	Do not use more than 20% on top of a `cat >/dev/null`
5	Statistics about the analysis such as time range, files read, lines read, interactions found etc.

Future

number	description
1	Analysis data stored in a relational database for later analysis with SQL queries
2	Correlate data from different components with different log file naming schemes and formats

Please let me know if I missed something or there are other mistakes (such as contradictory requirements).

Name that method: Symmetry

judofyr@gmail.com (Magnus Holm) — Fri, 05 Jun 2009 19:39:00 +0000

class Users < Application
  provides :xml, :json

  def index
    @users = User.all
    display @users
  end
end

Take a look at the snippet above. It’s from Merb, and the provides/display feature is something that will be implemented in Rails 3 after it was decided to merge the two projects. Don’t think too much about what it does, the point is to see how it will look in Rails 3:

class UsersController < ApplicationController
  respond_to :html, :xml, :json

  def index
    @users = User.find(:all)
    respond_with(@users)
  end
end

A subtle difference. Let’s have a look at another example, this time from Camping:

module Blog::Controllers
  # Define routes using R()
  class Index < R '/'
  end
  
  class Posts < R '/posts'
  end
end   
  
module Blog::Views
  def index
    # Generate URLs using R()
    a :href => R(Posts) { 'Show me the posts!' }
  end
end

Notice how you use the class method R to define routes in the controllers, and the instance method R to build URLs to those routes again.

What’s the deal?

I’m not a very good designer, but I have managed to get a hold of some of the principles. One of them is called the principle of proximity, and says that related items should be grouped together. We programmers already do this all the time: We group our code into classes and modules and separate them into different files and folders. However, this same principle can be applied when it comes to method names: related methods should have related names.

“Provide” and “display” are neither synonyms nor antonyms, nor do they share a common prefix/suffix. They are however closely related and obviously symmetric. That symmetry should be reflected in their names.

When you quickly glance over some code, your brain starts creating connections between different pieces and how they interact with each other. A well named method makes it far easier to understand the flow, while a poorly named one means you might have to start looking a completely different place while still be able to quickly jump back and continue interpreting the rest. By naming related methods alike, you make the connections more obvious and easier to spot.

It may not seem like a big deal, but it certainly makes your code look cleaner and easier to understand.

RubySpec as a Community Resource (Interview w. Brian Ford)

gregory.t.brown@gmail.com (Gregory Brown) — Thu, 04 Jun 2009 07:52:00 -0000

Though I still need to collect the details, I’m about to start on another major open source project, called Unity . It will be a community driven effort to provide a nice web resource that builds on top of the RubySpec project with reporting and search capabilities.

But readers might wonder what RubySpec has to do with them, if they are not language implementers. Here’s what Brian Ford had to say.

In an ideal world, what would the RubySpec project do for the Ruby community?

1. Provide a reasonably precise definition of the Ruby programming language so that there can be a certain degree of uniformity among the implementations. The goal is for the Ruby user to not have to think about which implementation her code runs on until she needs to, rather than writing code that works on one implementation and porting it if necessary. All software has a life cycle and the needs of the software are not always constant. A Ruby programmer should be able to develop using MRI on their laptop and deploy on Rubinius or JRuby without missing a beat, for example. Note that I use “Ruby user” and “Ruby programmer” with a distinction. A Ruby user might be anyone who relies on or provides software written in Ruby, not just someone who writes software in Ruby. So I envision a uniform definition of Ruby benefiting every strata of the Ruby community.

2. Provide quality assurance and regression notification for MRI and all implementations. People put a huge amount of trust in an implementation when they decide to invest large amounts of money in developing a software product. We’d like them to get some assurance for the trust they put in. Programmers are human and make mistakes. Automating the detection of mistakes is a no-brainer.

3. Improving the quality of Ruby as a language by discovering dusty corners and inconsistencies. Recently, a number of inconsistencies in bang methods (e.g. String#lstrip!) were discovered by one of the RubySpec contributors. Matz quickly fixed up the issues. (You can read the gory details here: http://redmine.ruby-lang.org/issues/show/1550). Ruby as a programming language certainly exists in Matz’s brain. But Ruby as an implementation (MRI) exists in many developers’ brains. RubySpec is an attempt to expose all that knowledge and those assumptions in an objective manner.

4. Experimentation with implementations and the Ruby language itself. I have recently implemented nearly a dozen variations of Hash for Rubinius to determine which algorithms perform best. In each case, I start by writing code that will pass the RubySpec Hash specs. The specs have been generalized to use a helper method rather than referencing the Hash class constant directly. This has been a huge benefit, especially in Rubinius where the core library Hash class is written entirely in Ruby. I can inherit from Hash and override any methods to write a completely new implementation that does not interfere with the existing Hash class. The RubySpecs also provide a huge safety net and testing ground for trying out new language features and seeing how they interact with existing features.

5. A learning tool that provides Ruby programmers with concrete examples of code that demonstrates every single Ruby language feature. The Pickaxe book may have better prose, but when it comes to comprehensiveness, I want RubySpec to be number one. And while we have a lot of work to do to make the spec description strings better, you can already learn a lot about Ruby by reading the specdoc output.

With everything changing so fast, I imagine keeping the specs up to date is a constant struggle. How do implementers keep up with what’s going on?

Some things are changing fast but when you look at how many distinct behaviors RubySpec covers, a lot isn’t changing. It’s hard keeping up with the 1.9 changes. So far in Rubinius, we haven’t targeted 1.9 specifically, but we have a contributor, Marc-Andre Lafortune, who has added a ton of 1.8.7 and 1.9 features to the core libraries. As with anything, you do it one thing at a time. Having RubySpec available to tell you if you are doing it right is a big help.

I would really love to make RubySpec a household name in the general Ruby community. The project I’m breaking ground on now (called Unity), will hopefully help do this. What sort of information do you think is most valuable to pull out from RubySpec, and who do you see benefiting from it?

Again, the primary goal of RubySpec is to provide a definition of the Ruby programming language along with a way to test that an implementation conforms to the definition. So primary pieces of information are which specs pass and which fail on a given implementation. Evan and I have discussed adding a scoring mechanism to MSpec that provides a single percentage number as a measure of an implementation’s conformance to the standard set by RubySpec. This is something that can give users better information about an implementation’s completeness when it goes bandying about benchmarking numbers. I’ll be implementing this in MSpec shortly.

RubySpec spans a ton of functionality. It’s amazing to see how much work has already been done. However, there must be some dark spots still left. What areas of Ruby are still not adequately specified?

That is the 10 million dollar question. We can infer something about completeness based on implementations like JRuby that run applications the same as MRI and pass the vast majority of the RubySpec cases. When implementations discover bugs in their own code and enhance the RubySpec coverage, we get a nice closed feedback loop.

A lot of the standard library still needs to be covered. There are also some little used methods in the core library that lack specs. The procedural-style methods in Kernel come to mind (e.g. using methods without receivers like #gets, #puts, #sub, etc. that operate on $_). We have a utility that creates the spec files for methods and leaves marker specs that we tag as incomplete. In Rubinius, for example, you can run ‘bin/mspec tag —list incomplete’ and get a printout of specs that need to be reviewed for completeness.

Ultimately, we need a uniform methodology for auditing the completeness and accuracy of the specs. This is still a work in progress.

Charles Oliver Nutter (of JRuby) chimes in with some additional feedback on this question:

I would inject another small answer here:

In my experience with RubySpec, the areas least covered are unfortunately often the areas least understood and therefore those most in need of coverage. For example, OpenSSL specs are practically nonexistent, and yet it’s such a crucial bit of functionality for any app doing SSL (as a client or a server) or working with any PKI or certificate stuff. But it’s also very hard to test…most people have zero knowledge of crypto, and so finding people that can write those specs is about as easy as finding people that can write OpenSSL wrappers for new implementations. So there’s definitely specific areas of RubySpec that need someone with domain knowledge to dive in.

As you can see, RubySpec is a vibrant, exciting project with a lot of challenges it needs to face. My plans are to do my part by building up the Unity app to make this awesome body of work more accessible to the Ruby community at large. But definitely don’t wait on it before getting involved, check out RubySpec today and see if there is a way you can pitch in, especially if you think you can help with some of the projects mentioned here.

To our readers: What sorts of things can you see RubySpec being used for? What kind of data are you interested in seeing reports on? Do you think Brian’s goals for RubySpec are attainable? Share your thoughts in the comments below.

The Laboratory Project

shortcutter@googlemail.com (Robert Klemme) — Wed, 03 Jun 2009 21:56:00 +0000

The company I am with creates billing systems for telecom companies. More precisely we call them “convergent billing systems” because billing is not limited to phone calls or SMS. You can also bill multimedia content, internet traffic and whatnot. Basically everything that runs through your telephone or IT infrastructure. In our office we even have a solution where the billing system is used to charge money via our corporate id cards at the local vending machine.

These billing systems have a tremendous amount of interesting features. Listing them all would need an article of its own. For the sake of our project it is important to remember that these systems are quite complex and have a high throughput. We have a central component implemented as a JEE application which front ends to customer care and other systems that may be present at a customer site. (The term “customer” is really ambiguous: there are our customers, which are telco companies and their customers, i.e. everybody using their phone lines. I will talk about our customers only.)

Now, this central application writes out logfiles using usual mechanisms (you can look at log4j in case you do not know Java logging and are interested in more details). As you can imagine these log files grow from large to huge because they will contain several lines of per business interaction (basically this is a remote call from a client system). Sometimes these interactions fail and our support staff need to look at these log files to find out what’s wrong. When it’s a tricky issue we people from the development department need to do this as well. Simplyfying this task is the goal of this toy project.

Some things you need to know about these log files:

all log entries belonging to the same interaction share a common identifier,
they are huge,
multiple interactions are interspersed,
some log entries span multiple consecutive lines,
log files are rotated, i.e. there are some incomplete interactions at the beginning and end.

I have set up a git repository where I created a generator script which you can use to generate sample logs and get a rough idea how they look like. Of course, there is no real data but the basic properties from above are present. Try to run the script with test-gen.rb -h to see supported parameters. If you don’t the script will throw a lot of text at you. I included “START” and “END” messages per interaction only to simplify testing – in practice we cannot rely on the presence of lines which unambiguously identify the first and last line of an interaction.

What I like to have at the end of the project is a Ruby application which efficiently separates individual interactions thus they can easily be analyzed individually. In order to get there, we will start collecting requirements. Let’s assume for the moment that I am your customer and you try to get a list of requirements. I would like to invite you to query me via the comment system. I am sure, together we will come up with a better list than if I would write it down alone. In the next article of the series I will then present the summary of our efforts.

Enter the Muppet Laboratories

shortcutter@googlemail.com (Robert Klemme) — Fri, 29 May 2009 17:54:00 +0000

I have decided I will start an experiement. I haven’t done this before and I have no idea whether it will work as indended but I am sure we can pull this off together. Here’s the deal: I will try to let you participate in my thought process which hopefully will turn up some useful if not best practices. I don’t claim to be Don Knuth but hopefully you’ll find it interesting and worthwhile nevertheless. The tricky part will be to actually catch all the reasonings and record them in a format suitable for blogging.

In this article I will give a brief overview of how I intend to proceed. Of course a thing like this needs some form of project. We will come to that in a minute. The whole series of articles will roughly look like this:

Initial presentation (this article)
Description of the project goal
Summary of collected requirements
Several postings about actual code and decisions
Presentation of the final result

After I present the goal you are invited to help distill requirements via the blog’s comment system. I will be the customer for the moment and I am sure with your active support we will get a list together with requirements that the project can meet.

The whole thing is loosely related to my job but I think you can extract at least parts of it for your own purposes. Of course we’ll be doing everything nicely modularized so reuse won’t be an issue. :-) I any case the main benefit for us all should be the thought process and insight into decision making during software development.

The project will be about analysis of large log files which contain high numbers of small business transactions. In parallel with coming up with a more detailed description I will try to come up with a generator which we can use to create files for testing purposes. This should help you get a better idea of what I am talking about.

Tune in next week when you’ll hear Dr. Bob say…

Reading Ruby's Standard Library for Fun and Profit

gregory.t.brown@gmail.com (Gregory Brown) — Sun, 24 May 2009 18:25:00 -0000

Note: This post is inspired by JEG2’s excellent code reading talk at MWRC 2009, called LittleBIGRuby Go watch it if you have time, then come back and read this. You might also want to check out the Question 5 Ways interview at Pat Eyler’s “On Ruby” blog for more code reading goodness.

We all need to hunt bugs and we all need to integrate our code with other systems. Some of us need to make use of undocumented libraries, and others need to examine code to perform security audits. Why is it then, that so many developers suck at reading code?

Almost a decade ago, Joel Spolsky would have told you it’s harder to read code than it is to write it , and that was probably true at one point. But times have changed, and in a language like Ruby, this definitely does not have to be the case. As an avid code reader, I’ll let you in on a secret: All you need to do is practice!

Beginners and even intermediate developers may need a lot of hand holding at first, that’s only to be expected. However, if you want to progress to higher levels of understanding, you must turn to the source. The merits of higher level idioms and design strategies won’t be very clear until you need to understand someone else’s code. Once you start digging around in someone else’s code base, you’ll learn a lot about your own strengths and weaknesses.

But where should you start? It’s tempting to suggest that you should just follow your favorite projects on Github and keep an eye on the patches as they fly. However, this can be a bit overwhelming at first, and it may be hard to keep up with the velocity and transient nature of active open source projects. For this reason, I suggest picking a slower moving target that is still useful to you. Ruby’s Standard Library fits this description perfectly.

How I Read Code

I’m going to walk through the process I generally follow when I’m reading code for the purpose of understanding its implementation. I sometimes change things up when I have a particular goal in mind, but most of what I’m about to cover is generally applicable to digging into any new code base. Keep in mind though, our goal here is mainly just to explore a bit and maybe learn a thing or to along the way.

I’ve chosen the Observable library here because it is small, self-contained, and easy to understand. There are many libraries like this in the stdlib, so if you feel motivated after reading this post, you might want to go searching for a few other low hanging fruit to test your skills on.

I’ve done what I can to boil the process down to a few repeatable steps, so you’re welcome to follow along at home if you’d like.

Start with a simple example.

I don’t like diving into code without at least a little bit of context. So this means that before I go source diving, I’ll typically use the API documentation (if it exists), to code up a simple example. If I can’t find documentation, I’ll search the web for examples. It’s relatively rare to come up completely empty, and a bit of a warning sign if you do. But since Observable is well documented , it’s easy enough to come up with something.

We’re dealing with the Observer design pattern (also known as Pub-Sub), and the first idea that came to mind here for me was a simple chat model. Users could subscribe to a room and the room would publish the messages that were processed to them. I wanted the transcripts to be personalized, so that they looked something like this:

  = Greg's Transcript =
  Me: Hi
  Jia: Hello
  Me: Fun, huh?
  Jia: Yeah!

  = Jia's Transcript =
  Gregory: Hi
  Me: Hello
  Gregory: Fun, huh?
  Me: Yeah!

Thinking from the outside in, we can imagine the Ruby code used to generate this output:

  room = Room.new

  greg   = Person.new("Gregory", room)
  jia    = Person.new("Jia", room)

  greg.says "Hi"
  jia.says "Hello"

  greg.says "Fun, huh?"
  jia.says "Yeah!"

  puts "= Greg's Transcript ="
  puts greg.transcript + "\n"

  puts "= Jia's Transcript ="
  puts jia.transcript

Ultimately, what the Observable module buys us is the ability to track subscribers implicitly rather than explicitly. While it’d be pretty easy to roll your own code here, you can see in the following definition that Observable provides a few shortcuts for us:

  require "observer"

  class Room 
    include Observable

    def receive(message, sender)
      changed
      notify_observers(message, sender)
    end
  end

  class Person
    def initialize(name, room)
      @name, @room = name, room
      @transcript  = ""
      @room.add_observer(self)
    end

    attr_reader :transcript, :name

    def update(message, sender)
      if sender == self
        message = "Me: #{message}\n"
      else
        message = "#{sender.name}: #{message}\n"
      end
    
      @transcript << message
    end

    def says(message)
      @room.receive(message, self)
    end
  end

This is all the code you need to reproduce the desired output, so if you’re
interested in tinkering with it a bit, go ahead and do that now. The real purpose of doing this however was not to implement something useful. Instead, it was to identify some entry points into Observable that can be our first areas of focus.

If you look over the code carefully, you find that changed, notify_observers, and add_observer are the keys to what makes our code tick (And incidentally, the only parts that we didn’t implement ourselves.) Let’s dig a little deeper with those targets in mind.

Hunt for tests, if they exist.

Many of Ruby’s standard libraries have tests distributed with the source, some do not. Unfortunately in the case of Observable, I couldn’t find any tests for it in the Ruby source distribution. However, that didn’t mean that it was time to give up. In cases where MRI/YARV do not have tests, you can try to turn to the great RubySpec project, which is attempting to codify the behavior of Ruby with executable RSpec-like tests.

As it turns out, RubySpec has “several tests”: for Observable. They are split up by each individual function. As an example, here are the specs for notify_observers:

  describe "Observer#notify_observers" do
 
    before(:each) do
      @observable = ObservableSpecs.new
      @observer = ObserverCallbackSpecs.new
      @observable.add_observer(@observer)
    end
 
    it "must call changed before notifying observers" do
      @observer.value.should == nil
      @observable.notify_observers("test")
      @observer.value.should == nil
    end
 
    it "verifies observer responds to update" do
      lambda {
        @observable.add_observer(@observable)
      }.should raise_error(NoMethodError)
    end
 
    it "receives the callback" do
      @observer.value.should == nil
      @observable.changed
      @observable.notify_observers("test")
      @observer.value.should == "test"
    end
 
  end

This code uses some helper objects, which are defined as follows:

  require 'observer'
 
  class ObserverCallbackSpecs
    attr_reader :value
 
    def initialize
      @value = nil
    end
 
    def update(value)
      @value = value
    end
  end
 
  class ObservableSpecs
    include Observable
  end

You can see how the structure of these mirror our earlier example. When you combined them with the specs, you learn quite a bit about this method.

First of all, notify_observers doesn’t do anything if you don’t first call the changed method:

  it "must call changed before notifying observers" do
    @observer.value.should == nil
    @observable.notify_observers("test")
    @observer.value.should == nil
  end

Also, we see that any of the observers used need to implement the update callback, matching the signature of notify_observers:

  it "verifies observer responds to update" do
    lambda {
      @observable.add_observer(@observable)
    }.should raise_error(NoMethodError)
  end

Of course, when the callback exists and the observable object has been marked as changed, we see a successful result:

  it "receives the callback" do
    @observer.value.should == nil
    @observable.changed
    @observable.notify_observers("test")
    @observer.value.should == "test"
  end

While these specs aren’t necessarily giving us every last detail, they provide a general outline of what to expect when we look directly at the source. This will help us avoid getting overwhelmed when working through the actual implementation. I’ll leave it to you to check out the rest of the Observable specs if you’d like. Doing so may help you understand the code in the following section.

Dig into the source with kindness and curiosity

The main purpose of reading over the tests first is that they give you a bit of insight into what behaviors the implementation code needs to satisfy. Assuming you didn’t skip the last section, it should be pretty clear what’s going on in the following method definition:

  def notify_observers(*arg)
    if defined? @observer_state and @observer_state
      if defined? @observer_peers
        @observer_peers.each { |k, v| k.send v, *arg }
      end
      @observer_state = false
    end
  end

Of course, the ‘why’ is as important as the ‘how’ when you’re exploring someone else’s code. Without looking any farther, one thing that tripped me up was this line:

  if defined? @observer_state and @observer_state

It was immediately clear to me what it did, but not why it was needed. Upon further investigation, I remembered one of my bad habits. The following code will issue a warning as written:

  module A
    def foo
      @foo
    end
  end

  class B
    include A
  end

  B.new.foo

However, we can avoid this warning using the exact same approach used in Observable, simply by checking whether the instance variable is defined before accessing it:

  module A
    def foo
      @foo if defined?(@foo)
    end
  end

Yes, this is more verbose, and seems a bit redundant. But writing code this way does have a real benefit. If you follow this convention of checking whether an instance variable is defined whenever you aren’t sure it will be, this warning can be used to easily catch typos. The following example, when run with warnings enabled, will illustrate how:

  class A
    def foo
      @foo = 10
      bar
    end

    def bar
      @bar = @fooo
    end
  end

  A.new.foo

At this point, I already found a helpful reminder by walking through this code. While the rest of the source was an interesting read, this was my takeaway. Please go ahead and read the full source and let me know in the comments if you found anything of interest.

In fact, I have a question for our readers. Can you find any reason why the following code is written as it is?

  # Query the changed state of this object.
  #
  def changed?
    if defined? @observer_state and @observer_state
      true
    else
      false
    end
  end

This looks a bit like a wart to me, and if anything, seems like a long way to go in order to avoid !!obj. But maybe someone else sees it in a different light? Actually, that reminds me…

Caveats about the Standard Library

While Ruby’s stdlib is a great place to learn new things, not everything in it is downright beautiful. Some things are older than others, some reflecting the rust that comes when code is not updated to reflect the latest techniques.

Furthermore, standard library code tends to be less clever than the sort of stuff we might write for our scripts and gems, by design. You’ll find that in many places, explicit and simple code is favored over tricky stuff. I imagine this is for maintainability purposes.

Like anything else, you’ll need to do a bit of hunting to find the right code to learn from. This practice in and of itself will prove to be useful, so don’t be afraid if you run across some cruft now and again.

Of course, another good thing about reading code is that it often inspires us to write code. While looking through the standard library, consider putting together documentation patches, adding tests, and fixing things you don’t like. I’m sure the folks on ruby-core would be happy to look at your contributions, and it’s another great way to learn.

Key Things To Remember

Code reading is just as important as code writing. The standard library has about 100 places for you to look for inspiration, and its more likely than not that you’ll find something of interest if you pick one at random. If you repeat this process, you’ll get a better understanding of how Ruby itself works, and be able to carry this experience into your own work.

Start by playing with some examples, then look for tests, either in the source or in the RubySpec project. Once you feel like you understand how to use a feature, investigate how it works, and then ask yourself why it is written the way it is. Don’t expect to understand the whole thing all at once, and don’t worry if it takes some time to adjust to someone else’s style.

If you do this often enough, you’ll gain a valuable skill that’ll pay off like crazy, especially if you work on open source software. You’ll notice both patterns and anti-patterns, and this will influence the code you write. In the end, your fellow code readers will thank you :)

Control flow features and readability

shortcutter@googlemail.com (Robert Klemme) — Tue, 19 May 2009 20:46:00 -0000

First of all I would like to thank our readers who participate in discussions so actively! These discussions provide interesting food for thought as well as inspirations for new blog entries. Today’s article was partly inspired by the question that surfaced recently: “Why is catch .. throw seen so infrequently?”

In this article I will explore how the choice of control flow constructs and their usage affects readability and maintainability of code. For the purpose of this investigation I will provide a definition, partly because the Wikipedia article seems a bit inconsistent and partly because it may change.

A control flow construct is a language feature which disrupts the normal progression to the next statement and conditionally or unconditionally branches to another location in source code.

This definition includes the usual if ... then but also method invocation and return.

Classes of control flow statements

Following a sequence of regular (i.e. non control flow) statements is easy. Things get only slightly more complicated for the reader in case of if ... then ... else ... end as long as proper indentation is used and we do not have to scroll multiple pages to see the other branch(es).

But it is a different story altogether if the stack frame changes and we as readers have to jump to a completely different location — maybe even in a different file! Hence I will classify control flow constructs depending on the distance measured in stack frames:

level-0: There is no change in stack frame at all.
level-1: Code moves one stack frame up or down.
level-n: An arbitrary amount of stack frames can be added or removed.
level-x: The stack is completely exchanged.
level-1n: This is a hybrid where “technical” and “visual” jumps differ.

I am sure those first three categories do not bear any surprises for you. The last two probably need a bit of explanation. Category level-x contains Ruby’s continuations. I won’t cover them here because I am not too familiar with them, they are rarely used in “ordinary” code and they are so complex that they probably deserve an article of their own. It should be obvious that this complexity does not really help understanding code.

level-0

This category includes

if ... then ... elsif ... else ... end as well as unless ...,
ternary operator ... ? ... : ...,
for, while and until loops,
both forms of case statements,
and and or,
all statement modifiers (if etc. at the end of a statement).

All these provide for easy following of code and can — when chosen wisely — even form Ruby code which almost reads like English. There are really only two things that can make them hard to follow

Not indenting code properly.
Putting too much lines of code between different keywords belonging to the same construct.

As long as you follow these basic rules, the reader of your code will be able to follow the flow easily.

level-1

In this category we have

method invocation,
method return.

Note, that I did not place return in this category — we will see it in level-1n below.

The readability of a method invocation depends mostly on how good it conveys what the method does. A Ruby programmer who reads x.to_s immediately knows that this expression returns a string representation of x. (Strictly speaking there is of course no guarantee that this method actually returns a String instance but for all practical purposes we can safely assume this.)

For unknown methods the name is crucial for our understanding — or at least rough idea — of what happens during the method call. This is important because only if we have this understanding we can continue reading and understand what the current method does. This shows how maintainability not only depends on proper modularization but also on well chosen method names. In fact it might be more important to get a method’s name right than to have documentation which covers all aspects of the method’s semantics. Don’t get me wrong, I do not want you to neglect your documentation! But a proper chosen name goes a long way in telling the reader of the code that uses this method what it does.

level-n

Again, there are two contenders:

raise ... rescue,
catch ... throw.

Although these two may seem to only differ in syntax on first sight, there are fundamental differences. I believe that these are ultimately the reason why we see exceptions quite frequently while we rarely discover a catch statement in code. Let’s first look at exceptions:

The power of exceptions comes from decoupling the signalling of an error and its handling. The meaning of the error condition is engraved in the exception class (via inheritance and documentation). We know that Errno::ENOENT denotes a non existing file and we can write an exception handler for this. When we raise this exception we do not know how many stack frames upwards there will be a handler for it — and we do not need to. If there is none ultimately the interpreter will exit with an error message and an exit code != 0.

Contrast this with catch ... throw — everything seems to be the opposite here:

This combination is for regular control flow and not for dealing with error situations.
You’ll first see the catch and then throw.
There is strong coupling between the catch location and the throw location in code via the symbol used; and both statements can be in different methods altogether.

Now, if you indeed place catch and throw in different methods you have established a strong link between the two: in order for the program to work properly both need to use the same symbol and both must be aware of the returned object (if any) so the result of catch can be processed in any meaningful way. You also need to take care about the last statement in the block attached to the catch in order to not accidentally return something which will be interpreted as thrown return value.

It’s fairly easy to complicate things even more by placing catch and throw methods in different classes or by nesting two or more catch constructs. It’s fairly safe to say that these obfuscating effects are best avoided by using catch ... throw in a single method only — and in that case there are other control flow constructs that we can usually use. In fact I am still searching for more convincing examples of catch ... throw usage; so far the best contender has been a jump out of multiple nested loops. Although, to me the “multiple nested loops” item has a slight code smell of its own. But, read on…

Addition: Actually, you can find a good application of catch ... throw in Ruby’s standard library. In this case it is elegant and catch ... throw has the advantage of not interfering with exception handling: assume more complex code in the block which is passed to Find.find which has a rescue internally then using exceptions behind the scene might have surprising effects.

level-1n

You might be curious why I came up with this category. Let’s first look at an example: assume there is a method that yields all Ruby files which are found below any number of directories and we use that to write another method which returns the first of those files that we own:

def find_my_ruby_file(*dirs)
  find_ruby_files *dirs do |f|
    return f if File.stat(f).owned?
  end
  nil
end

Now, if return is executed find_my_ruby_file exits and returns f, we go up one stack frame. Visually this is true, but in reality, when return is executed, the stack is several levels deeper than it appears to be. You can easily check that by inserting something like puts caller(0) before the return – here’s a complete version that you can use to experiment:

require 'find'

def find_ruby_files(*dirs)
  Find.find *dirs do |file|
    yield file if test ?f, file and /\.rb\z/i =~ file
  end
ensure
  puts 'exit find_ruby_files'
end

def find_my_ruby_file(*dirs)
  find_ruby_files *dirs do |f|
    if File.stat(f).owned?
      puts 'found!', 'stack <<<', caller(0), '>>> stack'
      return f
    end
  end
  nil
ensure
  puts 'exit find_my_ruby_file'
end

p find_my_ruby_file(*ARGV)

You’ll find some surprising entries there and as a side effect I now know one more example where catch ... throw does seem like the best tool for the job…

This property of being able to exit more stack frames than visible on first sight is shared by break which can be used to exit multiple stack levels in a similar way. It won’t leave the current method (i.e. the method in which it is lexically scoped) but via invoked methods with blocks the unnesting can include any number of stack frames. This reinforces what we have said before since we now clearly see that there are more ways that a block of code can be left “not normally” besides raise and throw.

What do we learn?

Looking at control flow constructs from an execution stack perspective has provided some interesting insights (at least to me). While general rules for readability (proper indentation, limit the amount someone has to read etc.) do apply here as well, some of those constructs can have surprising effects on program behavior and should be used carefully. This is especially true for the “far reaching” constructs which can actually make the stack unwind multiple levels — or even change completely as in the case of callcc.

Hello World (Literally)

gregory.t.brown@gmail.com (Gregory Brown) — Tue, 05 May 2009 06:42:00 -0000

If Google Analytics is right, we’ve reached 68 countries in the last three days. Although I guess this doesn’t mean much at all on the internet, I think it still makes for a pretty map:

From RBP

Of course, I’d like to back up these statistics with some direct data, so I’ve got three questions for our readers:

Where are you from?
What is the Ruby scene like where you live?
What would you like to see on the RBP blog?

Please join in the conversation and let us know who you are. So far, the discussions on this blog have been great and I want to get to know more of our readers. We’ll resume regularly scheduled deep tech content soon, but hopefully this small diversion will prove to be interesting.

The Universe between begin and end

shortcutter@googlemail.com (Robert Klemme) — Fri, 01 May 2009 10:05:00 -0000

This time we’ll explore the space between begin and end. Today’s article won’t be as much about individual best practices but rather I will try to explore various aspects of begin ... end blocks so you can decide how to make best use of this tool. In fact, there are so many aspects that this construct sometimes reminds me of a swiss army knife – not so much because you can do everything with it but rather because it has so many features that you can use. When it comes to control structuring elements begin ... end is probably the most complex thing found in Ruby.

One final introductory remark: although I am pretty sure that Ruby hasn’t changed between 1.8 and 1.9 with regard to begin ... end I did my tests with 1.9.1 only. So if you find something to be wrong, please comment!

Starting the Investigation

For easier reference here’s a block which contains all the options:

begin
  # do our work
rescue
  # standard oops!
rescue SomeException => e
  # oops!
rescue Exception => e
  # deal with other errors
else
  # good, no exception surfaced!
ensure
  # good or bad, this needs to be done
end

Note that you can replace begin with def meth... to define a method with “integrated” exception handling. That way you can avoid one level of nesting.

There are various interesting aspects to each section:

control flow (most particularly, how is the section left?),
result (if any),
documentation (what does it tell me that code is in a particular section?).

We’ll keep these in the back of our heads when exploring section after section.

`begin`

This is the main section where you place the code that does the work you want to get done. If evaluation reaches the end of this section normally the result of the last expression evaluated is also the result of this section. In the absence of rescue or else that value is propagated to the surrounding context.

If we look at control flow things start to get a bit more involved. First, you should note that there are these additional ways this section (actually any section of code) can be terminated:

return is executed,
break is executed in a do block invoked,
an exception is triggered via raise,
throw is invoked.

How many of those did you think of? (Did I miss another one?) All these have one thing in common: the end of the section is not reached normally and the code produces a different result.

`rescue`

This can have an optional ExceptionType => variable in which case exceptions of this class and subclasses (if there are no preceeding rescue clauses with them) are caught. If that optional part is missing only StandardError and subclasses are caught.

If there are multiple rescue clauses, order matters. You must rescue most specific errors first and less specificerrors later because otherwise a super class rescue clause will shadow a sub class clause which comes later. Only one rescue clause of a group is ever executed.

In case a rescue clause is executed the result of the whole block is that of the code in the rescue clause. In other words: when catching exceptions the exception code replaces the block’s result. Unless, that is, you invoke raise or retry.

The tricky part about rescue is which exceptions to catch. If you cannot handle an exception, you should not catch it because otherwise you will prevent other code that is capable of handling it to work. Actually, as long as you do not raise again, nobody outside will notice that there was an error in the first place. A special case is raise without arguments: sometimes it is reasonable to catch all exceptions, log the event and rethrow:

def f
  do_work
rescue Exception => e
  log.error "There was an error: #{e.message}"
  raise
end

`else`

This section does make sense only if there is a rescue as well. (You’ll get a warning if you use it without it.) If the main section completes regularly and an else section is present it is executed.

Now, some of the discussion revolved around the utility of this section and whether code placed here is equivalent to code placed in other places. While at first sight it may seem that you can just place it at the end of the main section a closer look reveals some subtleties, some of which have been mentioned already in the discussion. I’ll list them here anyway for completeness reasons:

If the code raises an exception, it might be rescued when placed in the main section but it won’t be rescued when placed in else section. This might also lead to unnecessary retries if there is a retry in the rescue clause.
Placing the code after end generally has a similar effect as putting it in an else section but if there is an ensure section order of execution between the two code bits is reversed. This can make a serious difference if the else code uses a resource which is cleaned up in ensure.
When using the version of begin ... end in a method definition there is no place “after end” which is part of the method invocation:

def do_the_work
  some_code_which_may_throw
rescue ArgumentError
  $stderr.puts "Ooops! Passed the wrong argument."
else
  puts "Job done."
end

`ensure`

This section is the last one before the final end. Code in this section will be executed regardless of how the main section is left, i.e. even in case of raise, throw, return and break! The result of the section is ignored unless you choose to explicitly return it or raise an exception. While raising exceptions might be reasonable in some cases, it should generally be avoided because those exceptions will shadow errors coming from the main section or from rescue clauses:

irb(main):007:0> def f
irb(main):008:1> raise "Foo"
irb(main):009:1> ensure
irb(main):010:1* raise "Bar"
irb(main):011:1> end
=> nil
irb(main):012:0> f
RuntimeError: Bar
        from (irb):10:in `ensure in f'
        from (irb):10:in `f'
        from (irb):12
        from /usr/local/bin/irb19:12:in `<main>'
irb(main):013:0>

Another thing you should definitively avoid is a return statement in ensure because this will shadow the result from the “business logic” code in the main section:

irb(main):013:0> def f
irb(main):014:1> 1
irb(main):015:1> ensure
irb(main):016:1* 2
irb(main):017:1> end
=> nil
irb(main):018:0> f
=> 1
irb(main):019:0> def g
irb(main):020:1> 1
irb(main):021:1> ensure
irb(main):022:1* return 2
irb(main):023:1> end
=> nil
irb(main):024:0> g
=> 2
irb(main):025:0>

There’s a reason why the result of executing ensure section is ignored: this code has nothing to do with the “business logic” which should go into the main section but is solely for cleaning up. Assume you open a file with io = File.open (which I hope you won’t do after reading a previous post), have search code in the main section and place io.close in an ensure section. Then you would not want to shadow the result of the search by the result from the close operation.

Although ehsanul questioned the utility of ensure I certainly use it more often than else. While else section code can be put somewhere else in some cases, there is no other place of code which can easily replace an ensure clause and guarantee the same robustness of the code at the same time. This is nicely demonstrated by his suggested alternative which uses only rescue without Exception — this does not catch all exceptions! “Robustness” in this case refers not only to runtime robustness but also robustness of the code against maintenance (i.e. changes).

Let’s assume you have caught all exceptions via rescue clauses which do not raise and placed your cleanup code after end as it was suggested. Code works as expected and everything is fine. You might have to use a local variable to make sure the proper value is returned from your method but this is just a nuisance. Now all these changes will put execution of your cleanup code at danger:

A rescue clause is removed.
A previously uncaught exception type is thrown from the main section (the change need not be in your piece of code).
A rescue clause is changed to raise an exception.
An else section is introduced and code might throw.
Code in else section is changed to raise an exception.

And lastly, do not underestimate readability! Placing code in an ensure section tells the reader immediately that he can ignore it when trying to understand what the main purpose of the code is. This is “only” cleanup which makes sure some resources that were used for the calculation are not kept longer than needed. Whereas, if you place that code after end it could belong to the normal flow of the core business logic.

Guidelines

So after all the detail here are some rules which I hope will guide you in making the most appropriate use of begin ... end blocks.

If you have cleanup code that must be executed under all circumstances, put it in ensure.
Do not place return or break in ensure sections and try to avoid throwing exceptions from them.
Place rescue clauses for more specific exceptions (sub classes) before those for less specific ones (super classes).
rescue the most specific exception you can handle.
Do not rescue exceptions that you cannot or do not want to handle.

Keep in mind that these are just guidelines and you have to check on a case by case basis which is the most appropriate solution. As Buddha says: verify the teaching with your own mind.

Fun with Class.new

gregory.t.brown@gmail.com (Gregory Brown) — Thu, 30 Apr 2009 17:55:00 +0000

Update: I came across an inlink to this page that was layering in advertisements in a frame. Luckily, there are technical solutions to at least some social problems, and you may now browse again in peace :)

If you’ve worked with Ruby for at least a little while, you might already know that classes in Ruby are objects themselves, in particular, instances of Class. Before I get into the fun stuff, let’s quickly recap what that means.

Here’s the ordinary way we define classes, as you all have seen.

  class Point
    def initialize(x,y)
      @x, @y = x,y
    end
    
    attr_reader :x, :y
    
    def distance(point)
      Math.hypot(point.x - x, point.y - y)
    end
  end

The interesting thing is that the previous definition is essentially functionally equivalent to the following:

  Point = Class.new do
    def initialize(x,y)
      @x, @y = x,y
    end
  
    attr_reader :x, :y
  
    def distance(point)
      Math.hypot(point.x - x, point.y - y)
    end
  end

This always struck me as a beautiful design decision in Ruby, because it reflects the inherent simplicity of the object model while exposing useful low level hooks for us to use. Building on this neat concept of anonymously defined classes, I’ll share a few tricks that have been useful to me in real projects.

Cleaner Exception Definitions

This sort of code has always bugged me:

  module Prawn
    module Errors
     
       class FailedObjectConversion < StandardError; end
     
       class InvalidPageLayout < StandardError; end        
     
       class NotOnPage < StandardError; end

       class UnknownFont < StandardError; end   

       class IncompatibleStringEncoding < StandardError; end     

       class UnknownOption < StandardError; end

    end
  end

Although there are valid reasons for subclassing a StandardError, typically the only reason I am doing it is to get a named exception to rescue. I don’t plan to ever add any functionality to its subclass beyond a named constant. However, if we notice that Class.new can be used to create subclasses, you can write something that more clearly reflects this intention:

  module Prawn
    module Errors
       FailedObjectConversion = Class.new(StandardError)
            
       InvalidPageLayout = Class.new(StandardError)     
     
       NotOnPage = Class.new(StandardError)

       UnknownFont = Class.new(StandardError)

       IncompatibleStringEncoding = Class.new(StandardError)     

       UnknownOption = Class.new(StandardError) 

    end
  end

This feels a bit nicer to me, because although it’s somewhat clear that these are still subclasses, I don’t have an ugly empty class definition that will never be filled. Of course, we could make this more DRY if we mix in a little const_set hackery:

module Prawn
  module Errors
  
    exceptions = %w[ FailedObjectConversion InvalidPageLayout NotOnPage
                    UnknownFont IncompatibleStringEncoding UnknownOption ]

    exceptions.each { |e| const_set(e, Class.new(StandardError)) }

  end
end

Even with this concise definition, you’ll still get the functionality you’d expect:

>> Prawn::Errors::IncompatibleStringEncoding.ancestors
=> [Prawn::Errors::IncompatibleStringEncoding, StandardError, Exception, Object, Kernel]
>> raise Prawn::Errors::IncompatibleStringEncoding, "Bad string encoding"
Prawn::Errors::IncompatibleStringEncoding: Bad string encoding
	from (irb):33
	from :0

You can even take it farther than this, dynamically building up error classes by a naming convention. If that sounds interesting to you and you don’t mind the fuzziness of a const_missing hook, you’ll want to check out James Gray’s blog post “Summoning Error Classes As Needed”. Though I tend to be a bit more conservative, there are definitely certain places where a hack like this can come in handy.

The drawback of using any of these methods discussed here is that they don’t really play well with RDoc. However, there are easy ways to work around this, and James details some of them in his post. In general, it’s a small price to pay for greater clarity and less work in your code.

Continuing on the theme of dynamically building up subclasses, we can move on to the next trick.

Safer Class Level Unit Testing

In my experience, it’s a little bit tricky to test code that maintains state at the class level. I’ve tried everything from mocking out calls, to creating a bunch of explicit subclasses, and even have done something like klass = SomeClass.dup in my unit tests. While all of these solutions may do the trick depending on the context, there is a simple and elegant way that involves (you guessed it) Class.new.

Here’s a quick example from a patch I submitted to the builder library that verifies a fix I made to the BlankSlate class:

  def test_reveal_should_not_bind_to_an_instance
    with_object_id = Class.new(BlankSlate) do
      reveal(:object_id)
    end

    obj1 = with_object_id.new
    obj2 = with_object_id.new

    assert obj1.object_id != obj2.object_id,
       "Revealed methods should not be bound to a particular instance"
  end

Notice here that although we are doing class level logic, this test is still atomic and does not leave a mess behind in its wake. We get to use the real reveal() method, which means we don’t need to think up the mocking interface. Because we only store this anonymous subclass in a local variable, we know that our other tests won’t be adversely affected by it, it’ll just disappear when the code falls out of scope. The code is clear and expressive, which is a key factor in tests.

If you’re looking for more examples like this, you’ll want to look over the rest of the BlankSlate test case . I just followed Jim Weirich’s lead here, so you’ll see a few other examples that use a similar trick.

For those interested, the thing that sparked my interest in writing this article today was a tiny bit of domain specific code I had cooked up for my day to day work, which needed to implement a nice interface for filtering search queries at the class level before they were executed.

  class FreightOffer::Rail < FreightOffer
    hard_constraint do |query|
      query[:delivery] - query[:pickup] > 5.days
    end

    soft_constraint do |query|
      query[:asking_price] < 5000.to_money
    end
  end

While it’s easy to test these constraints once they’re set up for a particular model, I wanted to be able to test drive the constraint interface itself at the abstract level. The best way I could come up with was to write tests that used Class.new to generate subclasses of FreightOffer to test against, which worked well. While I won’t post those tests here, it’s a good exercise if you want to get the feel for this technique.

The next trick is a little bit different than the first two, but can really come in handy when you want to inject a little syntactic diabetes into the mix.

Parameterized Subclassing

You might recognize the following example, since it is part of the sample chapter of Ruby Best Practices . More than just a shameless plug though, I think this particular example shows off an interesting technique for dynamically building up subclasses in a low level system.

Let’s start with some vanilla code and see how we can clean it up, then we’ll finally take a look under the hood. What follows is a Fatty::Formatter, which abstracts the task of producing output in a number of different formats.

  class MyReport < Fatty::Formatter 
  
    module Helpers 
      def full_name 
        "#{params[:first_name]} #{params[:last_name]}" 
      end 
    end 
    
    class Txt < Fatty::Format 
      include MyReport::Helpers 
       def render 
        "Hello #{full_name} from plain text" 
       end 
    end 
    
    # use a custom Fatty::Format subclass for extra features 
    class PDF < Prawn::FattyFormat 
      include MyReport::Helpers 
      def render 
        doc.text "Hello #{full_name} from PDF" 
        doc.render 
      end 
    end 
    
    formats.update(:txt => Txt, :pdf => PDF) 
  end

The MyReport class in the previous code sample is little more than a glorified “Hello World” example that outputs text and PDF. It is pretty easy to follow and doesn’t really do anything fancy. However, we can certainly clean it up and make it look better:

  class MyReport < Fatty::Formatter 
    
    helpers do 
      def full_name 
        "#{params[:first_name]} #{params[:last_name]}" 
      end 
    end 
    
    format :txt do 
      def render 
        "Hello #{full_name} from plain text" 
      end 
    end 
    
    format :pdf, :base => Prawn::FattyFormat do 
      def render 
        doc.text "Hello #{full_name} from PDF" 
        doc.render 
      end 
    end 
    
  end

I think you’ll agree that this second sample induces the kind of familiar sugar shock that Ruby coders know and love. It accomplishes the same goals and actually wraps the lower level code rather than replacing it. But is there some sort of dark magic behind this? Let’s peek behind the curtain:

  def format(name, options={}, &block) 
    formats[name] = Class.new(options[:base] || Fatty::Format, &block) 
  end

As you can see, it is nothing more than a hash of formats maintained at the class level, combined with a call to Class.new to generate the subclass. No smoke and mirrors, just the same old Ruby tricks we’ve been using throughout this article. The curious may wonder how helpers() works here, and though it’s slightly tangential, you’ll see it is similarly simple.

  def helpers(helper_module=nil, &block) 
    @helpers = helper_module || Module.new(&block) 
  end

Though I won’t get into it here, fun tricks can be done with anonymous modules as well. Can you think of any that are particularly interesting? If so, let me know in the comments.

Although I only showed a part of the picture, you might want to check out the rest of Fatty’s source. I call it a ‘67 line replacement for Ruport’, which is a major exaggeration, but it is really surprising how much you can get out of a few dynamic Ruby tricks when you combine them together properly. A lot of these ideas were actually inspired by the Sinatra web framework, so that’s another project to add to your code reading list if you’re looking to learn new tricks.

Anyway, I’m getting off topic now, and it’s about time to wrap up anyway. I’ll go on one more tiny tangent, and then send you on your way.

Shortcutting with Struct

With all this talk about using anonymous sub-classes, I can’t help but recap one of the oldest tricks in the book. The method Struct.new can be handy for shortcutting class creation. Using it, the very first example in this post could be simplified down to:

class Point < Struct.new(:x, :y)
  def distance(point)
    Math.hypot(point.x - x, point.y - y)
  end
end

The constructor and accessors are provided for free by Struct here. Of course, the real reason I mentioned this was to get you thinking. If Struct.new can work this way, and we know that Class.new can accept a block definition that provides a closure, what other cool uses of parameterized subclasses can we cook up?

Please Share Your Thoughts

I wrote this article in hopes that it’ll be a jumping off point for discussion on more cool tricks I haven’t seen before, or failing that, ones that other readers haven’t seen before. While there certainly is a lot of nasty, scary looking “meta-programming” out there that makes people feel like this stuff is hard and esoteric, there are many techniques that lead to simple, elegant, and beautiful dynamic code. Please let me know what you think of these techniques, and definitely share one of your own if you’d like.

Writing Block Methods with automatic Resource Cleanup

shortcutter@googlemail.com (Robert Klemme) — Mon, 20 Apr 2009 19:47:00 -0000

After we have seen how File.open with a block is safer than without we will look into how such methods are created today.

Ingredients

We need two ingredients:

yield
A begin … end block with ensure

yield is a keyword and invokes the block which is passed to a method. If the caller did not provide a block he’ll earn a LocalJumpError. Result of evaluating yield is the value of the block (remember, we also called them “anonymous functions”).

As you will probably know begin … end blocks can be used to catch exceptions and handle them properly. But this construct has two more features apart from rescue; a full blown block might look like this:

begin
  # do our work
rescue SomeException => e
  # oops!
rescue Exception => e
  # deal with other errors
else
  # good, no exception surfaced!
ensure
  # good or bad, this needs to be done
end

Code in an else clause after rescue is executed when the block is left normally, i.e. without an exception being thrown. Note that in case of an exception it is irrelevant whether it is caught in this begin … end block or not — else will not be executed. Code after ensure is executed under all circumstances — regardless whether an exception is thrown or not. This is the feature that we’ll exploit for our cleanup.

An important thing to know is that the result of the “ensure” code does not affect the block’s result which is normally the value of the last expression evaluated between begin and the first rescue. So anything the cleanup code returns is invisible to the caller (which makes perfect sense if you think about it). Results of rescue and else clauses are retained when they are executed.

Cooking

Assume we have a class that distributes stdin to a set of files much the same as the ‘tee’ command line utility does. This class features a method open which opens files and another method close which closes all file descriptors. Then we can create a class method similar to File.open() which does the automatic cleanup via an ensure section:

class Tee
  def self.open(file_names, mode = "w")
    tee = new(file_names, mode)
    tee.open

    if block_given?
      begin
        yield tee
      ensure
        tee.close
      end
    else
      tee
    end
  end
end

This code does actually work similar to File.open because it acts differently depending on the presence of a block:

If there is no block, then the “tee” is created and opened. It is the return value of the method.
If there is a block, then that is called with the opened “tee” instance and after termination this is closed.

That’s really all there is to it. A final remark: in order to ensure that this pattern works properly you need to make sure the initialization is done before the begin. Otherwise any exceptions thrown during initialization will trigger the cleanup code which then will act on an incomplete or completely different object altogether (chances are that you will get NoMethodError from nil in this case).

You can also look at the full code if you want to see the complete story.

Rails Modularity for Lazy Bastards

gregory.t.brown@gmail.com (Gregory Brown) — Thu, 16 Apr 2009 04:31:00 -0000

When we develop standalone systems or work on libraries and frameworks, modularity seems to come naturally. When something seems to gain a life of its own, we pull it off into its own package or subsystem to keep things clean. This is a natural extension of the sorts of design decisions we make at the object level in our software, and helps us work on complicated problems, often working alongside other hackers, without losing our wits.

It seems a bit surprising that these helpful guiding principals can often evaporate when we bring our software to the web. Sure, we may make use of plugins and gems for foundational support, but if you take a good look at a Rails application that has more than 50 models or so, you’ll be hard pressed to find a unified purpose behind all that logic.

However, if you try to break things down into core sets of functionality, you may find that certain vertical slices can be made that allow you to break out bits of functionality into their own separate focus areas. For example, you may discover that part of your application is actually a mini CRM system. Or maybe you’ve snuck in a small reporting system without noticing it consciously. The list goes on, but the underlying idea here is that the larger a system gets, the harder it is to define its core purpose.

While splitting out these subsystems may seem like the obvious choice in a standalone application, there seems to be a certain amount of FUD about this strategy when it comes to the web. This probably originates from a time before REST, in which interaction between web applications was overly complex,making the costs of fragmentation higher than the benefits of a modular architecture. These days, we live in better times and work with better frameworks, and should reap the benefits that come along with it.

But actions speak louder than words, so let’s take a look at some of the underlying tech and how to use it. Building on the CRM scenario, I’ll start by showing how to access a Customer model from another application using ActiveResource.

Sharing Model Data via ActiveResource

Suppose that we’ve got a CRM system that has a Customer model that has a schema that looks something like this:

  create_table :customers do |t|
    t.string :first_name
    t.string :last_name
    t.string :email
    t.string :daytime_phone
    t.string :evening_phone
    t.timestamps
  end

With this, we can do all the ordinary CRUD operations within our application, so I won’t bore you with the details. What we’re interested in is how to accomplish these same goals from an external application. So within our CRM system, this essentially boils down to simply providing a RESTful interface to our Customer resource. After adding map.resources :customers to our config/routes.rb file, we code up a CustomersController that looks something like this:

  class CustomersController < ApplicationController
  
    def index
      @customers = Customer.find(:all)
      respond_to do |format|
        format.xml { render :xml => @customers }
        format.html
      end
    end
 
    def show
      customer = Customer.find(params[:id])
      respond_to do |format|
        format.xml { render :xml => customer.to_xml }
        format.html
      end
    end
 
    def create
      customer = Customer.create(params[:customer])
      respond_to do |format|
        format.html { redirect_to entry }
        format.xml { render :xml => customer, :status   => :created, 
                                              :location => customer }
      end
    end
 
    def update
      customer = Customer.update(params[:id], params[:customer])
      respond_to do |format|
        format.xml { render :xml => customer.to_xml }
        format.html
      end
    end
 
    def destroy
      Customer.destroy(params[:id])
      respond_to do |format|
        format.xml { render :xml => "", :status => 200 }
        format.html
      end
    end
 
  end

This may look familiar even if you haven’t worked with ActiveResource previously, as it’s basically the same boiler plate you’ll find in a lot of Rails documentation. In the respond_to block, format.xml is what matters here, as it is what connects our resource to the services which consume it. The good news is we won’t have to actually work with the XML data, as you’ll see in a moment.

While there are a few things left to do to make this code usable in a real application, we can already test basic interactions with a secondary application. Using any other rails app we’d like, we can add an ActiveResource model by creating a file called app/models/customer.rb and setting it up like this:

  class Customer < ActiveResource::Base
    self.site = "http://localhost:3000/"
  end

Now here comes the interesting part. If you fire up script/console on the client side application that is interfacing with the CRM system, you can see the same familiar CRUD operations, but taking place from a completely separate application:

  >> Customer.create(:first_name => "Gregory", :last_name => "Brown")
  => #<Customer:0x20d2120 @prefix_options={}, @attributes={"evening_phone"=>nil, "updated_at"=>Thu Apr 16 03:53:59 UTC 2009, "daytime_phone"=>nil, "id"=>1, "first_name"=>"Gregory", "last_name"=>"Brown", "created_at"=>Thu Apr 16 03:53:59 UTC 2009, "email"=>nil}>
 
  >> Customer.find(:all)
  => [#<Customer:0x20a939c @prefix_options={}, @attributes={"evening_phone"=>nil, "updated_at"=>Thu Apr 16 03:53:59 UTC 2009, "daytime_phone"=>nil, "id"=>1, "first_name"=>"Gregory", "last_name"=>"Brown", "created_at"=>Thu Apr 16 03:53:59 UTC 2009, "email"=>nil}>]
 
  >> Customer.find(1).first_name
  => "Gregory"
 
  >> Customer.delete(1)
  => #<Net::HTTPOK 200 OK readbody=true>
 
  >> Customer.find(:all)
  => []

While the interface and behavior isn’t downright identical to ActiveRecord, it bears a striking resemblance and allows you to retain much of the functionality that is needed for basic data manipulation.

Now that we can see the basic functionality in action, let’s go back and fix a few key issues. We definitely want to add some sort of authentication to this system, as it is currently allowing any third party application to modify and destroy records. We also will most likely want a more flexible option for locating services than just hard coding a server address in each model file. Once these two things are in place, we’ll have the beginnings of a decentralized Rails based application.

API Keys with HTTP Basic Authentication

I want to preface this section by saying I’m typically not the one responsible for any sort of security hardening in the applications I work on. That means that I’m by no means an expert in how to make your applications safe from the malignant forces of the interweb. That having been said, what follows is a simple technique that seems to work for me when it comes to slapping a simple authentication model in place.

First, in the app that is providing the service, in this case, our fictional CRM system, you’ll want something like this in your ApplicationController:

  def basic_http_auth
    authenticated = false
    authenticate_with_http_basic do |login, password|
      if login == "api" && password == API_KEY
        authenticated = true
      end
    end
 
    raise "Authentication failed" unless authenticated
  end

Here, API_KEY is some shared secret that is known by both your service providing application, and any client that wishes to use your service. In this blog post, I’ll be using the string “kittens”, but you’ll obviously want to pick something longer, and with significantly more entropy.

After dropping a before_filter in your CustomersController that points to basic_http_auth, you’ll need to update your ActiveResource model definition.

  class Customer < ActiveResource::Base
    self.site = "http://localhost:3000/"
    self.user = "api"
    self.password = "kittens"
  end

If you forget to do this, you won’t be able to retrieve or modify any of the customer data. This means that any application that does not know the shared secret may not use the resource. Although this is hardly a fancy solution, it gets the job done. Now, let’s take a look at how to make integration even easier and get rid of some of these hard coded values at the per-model level.

Simplifying Integration

So far, the work has been pretty simple, but it’s important to keep in mind that if we really want to break up our applications into small, manageable subsystems, we might need to deal with a lot of remote resources.

Pulled directly from some commercial work I’ve been doing with Brad Ediger of Madriska Media Group (and of Advanced Rails fame), what follows is a helper file that provides two useful features for working with remote resources via ActiveResource:

  require 'yaml'
  require 'activeresource'
 
  class ServiceLocator
 
    API_KEY = "kittens"
 
    def self.services
      return @services if @services
      config_file = File.join(RAILS_ROOT, %w[config services.yml])
      config = YAML.load_file(config_file)
      @services = config[RAILS_ENV]
    end
 
    def self.[](name)
      services[name.to_s]
    end
  end
 
  def Service(name)
    Class.new(ActiveResource::Base) do
      self.site = "http://#{ServiceLocator[name]}"
      self.user = "api"
      self.password = ServiceLocator::API_KEY
    end
  end

The ServiceLocator part was Brad’s idea, and it represents a simple way to map the URLs of different services to a label based on what environment you are currently running in. A basic config/services.yml file might look something like this:

  development:
    crm: localhost:3000
    reports: localhost:3001

  production:
    crm: crm.example.com
    reports: reports.example.com

This is nice, because it allows us to configure the locations of our various services all in one place. The interface is very simple and straightforward:

  >> ServiceLocator[:crm]
  => "localhost:3000"

However, upon seeing this feature, I decided to take it a step farther. Though it might sacrifice a bit of purity, the Service() method is actually a parameterized class constructor that builds up a subclass filling out the API key and service address for you. What that means is that you can replace your initial Customer definition with this:

  class Customer < Service(:crm)
    # my custom methods here.
  end

Since Rails handles the mapping of resource names to class names for you, you can easily support as many remote classes from a single service as you’d like this way. When I read this aloud in my head, I tend to think of SomeClass < Service(:some_service) as “SomeClass is a resource provided by some_service”. Feel free to forego the magic here if this concerns you, but I personally find it pleasing to the eyes.

Just Starting a Conversation

I didn’t go into great detail about how to use the various technologies I’ve touched on here, but I’ve hopefully provided a glimpse into what is possible to those who are uninitiated, as well as provided some food for thought to those who already have some experience in building decentralized Rails apps.

To provide some extra insight into the approach I’ve been using on my latest project, we basically keep everything in one big git repository, with separate folders for each application. At the root, there is a shared/ folder in which we keep some shared library files, including some support infrastructure for things like a simple SSO mechanism and a database backed cross-application logging system. We also vendor one copy of Rails there and simply symlink vendor/rails in our individual apps, except for when we need a specific version for a particular service.

The overarching idea is that there is a foundational support library that our individual apps sit on top of, and that they communicate with each other only through the service interfaces we expose. We’ve obviously got more complicated needs, and thus finer grained controls than what I’ve covered in this article, but the basic ActiveResource functionality seems to be serving us well.

What I’d like to know is what other folks have been doing to help manage the complexity of their larger Rails apps. Do you think the rough sketch of ideas I’ve laid out here sounds promising? Do you foresee potential pitfalls that I haven’t considered? Leave a comment and let me know.

Solve the problem, not just a problem

james.britt@gmail.com (James Britt) — Mon, 13 Apr 2009 20:11:00 -0000

So you decide you want a way for a number of people to post short articles to a Web site, and maybe allow for other people to leave comments. What do you do? That’s easy: jump to the white board and start sketching out a vast array of boxes, blobs, and arrows that define a sophisticated content management system with multi-level admin roles, content versioning, threaded discussion boards, syntax highlighting, the works.

Right?

Don’t be too quick to judge. It’s easy to fall into the trap of defining the wrong requirements for a project.

Part of the reason is that building software is (or should be) fun, and often bigger projects are more fun. There is also the tendency to think about “what if”, imagining all the things that maybe one day who knows you never can tell might be needed.

People also tend to think in terms of what’s familiar, of how things have been done in the past or by others.

There are many ways to satisfy the needs described in the first paragraph. Some don’t require writing any software at all.

For the Ruby Best Practices blog, the general goals were modest. Allow a core set of people to easily add new content. Allow other people to contribute content as well, but only with the OK from someone in that core group. (BTW, there are eight of us in the RBP blog team. See here for my take on the RBP team logo)

We wanted to allow the use of Textile, and not make people use a browser for editing. Basically, turn simple flat files into a Web site with minimal fuss.

Korma takes an interesting approach to building Web sites. The whole app is ~230 lines of Ruby. Its key function is to take content from a Git repository, run it through some templating, and write out the site files.

Relying on Git for that back end is stunningly perfect. Git provides versioning, access control, and distributed contributions.

It becomes the database layer common to most blogging tools. For free. Right off the bat, no need to write any admin tools or content versioning code .

At the heart of Korma is the grit gem. As the project blurb says, “Grit gives you object oriented read/write access to Git repositories via Ruby.” Very sweet.

The korma.rb file takes two arguments. The first is required, and is the path to a git repository. The second argument is optional; it tells Korma what directory to use for the generated content, and defaults to ‘www’, relative to where you invoke the program.

The app uses grit to reach into the current contents of the repo and parse the needed files. Files have to be committed to be accessible to Korma.

There is a configuration file that describes some basic site metadata, such as the base URL, the site title, and the author list. When called, korma.rb grabs this config file, sets some Korma::Blog properties, and then writes out the static files.

An early version of Korma used Sinatra; basically, Korma was a lightweight Web app to serve the blog posts. But as simple as it was, it was overkill, since there was no real dynamic content. It made no sense to have the Web app regenerate the HTML on each request, since it changed so infrequently.

A next version replaced the Web app part with static files, making it a straightforward command-line program. This solved another problem: how to automate the regeneration of files. The answer: use Git’s post-commit hook to invoke the program.

For example:

   #!/bin/sh
   # File .git/hooks/post-commit   
   /usr/local/bin/ruby /home/james/data/vendor/korma/korma.rb /home/james/data/vendor/rbp-blog /home/james/data/vendor/korma/www

Early versions also used Haml for site-wide templates. Not being a fan of Haml, I added in a configurable option to use Erb. It was nice and all, but it was a feature without a requirement. No one was asking for configurable templating, so that option was dropped and Erb replaced Haml as the default.

If you are wondering why working code was removed, consider that any code you have is something you have to maintain. As bug-free and robust as you may like to think it, no code is easier to maintain than no code. Configurable templating was simply not a problem we were needed to solve, and a smaller, simpler code base is more valuable than a maybe “nice to have.”

There was some discussion about the need or value of allowing comments. In the end they were deemed good, but there was no good argument for hosting them as part of the blog site itself. That meant a 3rd-party solution (in this case, Disqus) was perfectly acceptable. Again, a goal was to have comments, not to write a commenting system (as entertaining as that may be).

Using Git allows for yet another feature for free: easy one-off contributions. In many systems, allowing anyone to contribute means creating an account, granting some sort of access, managing all the new user annoyances. Or, an existing user has to play proxy, accepting content and entering it into the system on behalf of the real author. That’s work! With git, one can clone the repo, add new content, and issue a pull request back to the master branch. Anyone with commit rights to the master can then merge it in (or politely decline). No-code features FTW.

None of the blog design requirements are written in stone, and they may change tomorrow, but by identifying the real needs, addressing what was deemed essential, offloading what we could, and skipping the feature bling, we have a system that is easy to understand, easy to maintain, and easy to change.

Using Blocks for Robustness

shortcutter@googlemail.com (Robert Klemme) — Thu, 09 Apr 2009 14:25:00 -0000

Ruby’s blocks can be used for many purposes — in fact, they might well be the most used feature of the language. Today we will start looking at a frequently-used idiom, analyze its implications for robustness and demonstrate how blocks can greatly improve it.

Robustness

When talking about robustness of software¹ we mean the degree to which it is able to function properly under changed circumstances. Those changes could be internal (i.e. introduced by a code change) or external (e.g. by changed input data). Sometimes achieving robustness is a hard task especially for complex software like operating systems. In this article we’ll look at robustness at a smaller scale.

A common file handling idiom

When you read through ruby-talk you will find code that looks like this (I just did some minimal adjustments to formatting and corrected obvious errors):

  # read file into an array
  f = File.open("mylogfile.log")
  saved = File.open("archive.log", "a")
  results = f.readlines
  # Is there a way to remove everything that's been read here?
  f.close

  results.each_line do |sending|
    conn_to_db.print("data #{sending}")
    # Make sure we keep a backup
    saved.puts("#{sending}")
  end

  # Close the archive file
  saved.close

The logic is straightforward: a file is opened, all its lines are read and then it is closed again. Similarly, the second file is opened for appending, then written to and finally closed. Sure, the second file could be opened later, but this is not where I want to draw your attention to.

Rather, please notice that File.open is used together with an explicit close. Closing the file IO object ensures that all pending output is written to the file and operating system resources associated with the underlying file handle are freed. While not closing a read only file usually does not have dramatic consequences, not closing a file opened for writing likely has dramatic consequences. You can see it by running this bit of code through Ruby 1.8.*:

  NAME = "foo"

  f1 = File.open NAME, "w"
  f1.puts "text 1"
  # forgot to close f1
  f2 = File.open NAME, "a"
  f2.puts "text 2"
  # forgot to close f2

On my system (CentOS 5.3, Ruby 1.8.5) I see only the first line in the file but the second puts has no effect. Even if you add a f2.close at the end of the script you won’t see any difference. You need to add the f1.close before you open f2 to actually see two lines in the output file.

File.close is the solution, or is it?

We just remember to always add a f.close whenever we have opened a file and be done. Everything written to the IO object will reach the file and we do not need to worry about anything any more. Or do we? Unfortunately, this is not the case. Especially in situations like the one shown above where exceptions can happen (we do not know what conn_to_db.print does, but it looks like another IO operation which could fail) we need to take additional measures to ensure the file is really closed.

We can use begin … rescue … end to do proper exception handling and make sure file descriptors are closed properly. But this tends to get verbose. Fortunately there is another option: File.open accepts a block and ensures that the file descriptor is closed regardless how the block is left.

Large files

When dealing with files as shown in the code example above there is another potential source of trouble: if files can grow arbitrarily large they will burn a lot of memory when read as a whole or even terminate the program because there is not enough memory available.

For reading files there is an even better alternative: File.foreach will read a file line by line (or using a different separator given as second argument) and hand each string to the block that is provided. This saves an additional io.each {|line| …} inside the File.open block — less typing and one level of indentation less. Also, as an added benefit we can efficiently process large files because we do not have to hold the complete file in memory when using iterative solutions.

Example rewritten

With these tools equipped we are now ready to rewrite the original bit to a more robust version:

  # read file and write to archive
  File.open("archive.log", "a") do |saved|
    File.foreach("mylogfile.log") do |sending|
      conn_to_db.print("data #{sending}")
      # Make sure we keep a backup
      saved.puts(sending) # removed superfluous string interpolation
    end
  end

Note, that if the timing of the original script is critical we might want to use File.readlines in order to read in all lines of “mylogfile.log” before starting to write to the “archive.log” as the original code does.

Conclusion

Using Ruby’s blocks goes a long way to making programs and scripts that deal with file IO more robust with regard to

exceptions occurring somewhere along the way,
large files.

So, whenever you do file IO in a Ruby program, remember you can use

the block form of File.open or
File.foreach

to make your code more robust without paying a significant price.

You can even resort to File.readlines if you have to read the whole file into memory and at least save a bit of typing. If you make it a habit to always use these idioms chances are that you’ll save a lot file IO related bug hunting in the future.

Next time I will look into how to create a method like File.open which uses a block to ensure automatic resource deallocation.

¹ Here is a definition of robust and a Wikipedia Article about Robustness Principle.

Welcome to the RBP blog

gregory.t.brown@gmail.com (Gregory Brown) — Tue, 07 Apr 2009 17:02:00 -0000

Hello, and welcome to the Ruby Best Practices blog. I am happy to announce this new collaborative writing project, along with the great team that’ll be running it. A couple weeks ago I put out a call for volunteers to help me provide a new community resource for discovering and discussing best practices in Ruby, and now, we’re ready to get rolling.

In addition to myself: James Britt, Kirk Haines, Robert Klemme, Jeremy McAnally, Sean O’Halpin, Magnus Holm and Lakshan Perera round out the team. I am absolutely thrilled to have such a diverse and talented group to work with, and I can’t wait to see the content that we’ll produce together.

The general theme we’ve agreed upon for this blog is that each post should exhibit Ruby code that we can be proud of. We want to expose best practices through practical examples, so you will surely see a lot of open source code being discussed here. We want to give you less of “Here’s six great Rails plugins” and more of “Examples of Connascense in ActiveRecord”, or “Common Pitfalls in DSL Design, and How to Avoid Them”. Not every single post will be focused directly on best practices and idioms, but all will contain them.

This will be a code and high-level design heavy blog. We don’t plan to bore you with war stories or skimp on the technical value of our content to promote our latest cute but useless hack. That having been said, we plan to have fun while we write, and the occasional diversion will happen as a result. However, it’s safe to say that our content will be geared more towards the motivated developer looking to improve her craft rather than the general “plzgivemethecodes” crowd. If that sounds like your kind of thing, I think you’ll really enjoy this blog.

The blog engine we’ve decided to use is Korma which I’m sure you’ll hear more about in the near future. It was designed specifically to run this blog, is git based, and gives us a lot of cool features that we wanted to make sure to have in place before launch. Because we want to provide both the authors and the readers freedom to choose which content they are interested in, it is possible to subscribe to individual author feeds on this blog. This means that if you like 6 of us, the two you don’t like won’t bother you. Of course, we hope you like us all.

Over time, we’ll eventually begin accepting outside contributions to this blog, likely through something as simple as a fork and pull request on github. This means if you have an idea for an article that you’d like to see here, we may be accepting new content from third party contributors soon. However, before we can do that, we need to wait for the dust to settle, so look for an announcement in a few weeks.

It may take us a little while to work out the kinks in our blogging software. We want to keep things simple, but there still might be a few bugs here or there. If you run into problems using this blog, please email gregory.t.brown at gmail.com and let me know.

I think that pretty much wraps up the welcome letter. New posts should be cycling in within the next couple days, and we hope to remain active moving forward. Thanks for checking out this new blog, and happy hacking!

Ruby Best Practices

Practicing Ruby, The Newsletter

The Ruby Mendicant University Core Skills Course

The Original Idea

The RMU Core Skills Course

Student Projects from August / September

Released problems from August / September sessions

Please help us with funding!

Questions / Thoughts About RMU?

Complexity

Summary

Ruby Mendicant University is a Go!

IDEA: Ruby Mendicant University

The Idea

Funding

Schedule

Why Me?

What do you think?

If you want to see the wizard, go to Oz.

Red Dirt RubyConf: Your ticket to Oz

rubyproblems.com is now live!

Full Book Now Available For Free!

Manuscript and Production Files

But I finished the RBP book and I want more! What should I do?

Disclaimer

Ruby Tuesdays: RBP Chapter 7

Progress To Date

How To Read RBP

Today’s Topic

For Your Consideration

Ruby Tuesdays: RBP Chapter 6

Progress To Date

How To Read RBP

Today’s Topic

For Your Consideration

Code Massage

Java Style

A little more sophisticated

O||=erator

Outsourcing

Getting tricky

Even shorter with a general solution

Golf

The fun begins

Ruby Tuesdays: RBP Chapter 5

Progress To Date

How To Read RBP

Today’s Topic

For Your Consideration

The Complete Numeric Class

Conversions into HexNum

Conversions to other types

Mutability

Equivalence vs. Comparability

What’s this #coerce thingy?

Math and Operator Overloading

Summary

Ruby Tuesdays: RBP Chapter 4

Progress To Date

How To Read RBP

Today’s Topic

For Your Consideration

RBP is useless without you

Weekend Reading: RBP Chapters 2-3

Today’s Topics

For Your Consideration

Sexp for Rubyists

Weekend Reading: RBP Chapter 1

Today’s Topic

For Your Consideration

Reflections from "The Compleat Rubyist"

The Training

Ruby Versions and Implementations

Meta-programming

Testing

Style and Substance

The Event

The Future

Learning From Bad Ideas

Continuations are Evil?

Conversions into `HexNum`

What’s this `#coerce` thingy?

How `LRUHash` works

How `LRUHash` is used in the project

Why do you use `equal?`?

The `Decorate` module