Saturn Flyer

Cache key for collections in ActiveRecord

Jim Gay — Tue, 27 Oct 2009 16:01:30 GMT

I’ve been working on the performance of a Rails application and much of my recent work involves simply caching. It was an application that was an excellent proof-of-concept which quickly turned into a production application.

We’ve got memcached setup and running and now it’s a speedy little interface.

But one thing I knew I needed but wasn’t sure how to do was cache a group of records. Rails fragment caching is super simple for one record:

<% cache [object] do %>
  ... this will be cached based upon your object.cache_key ...
<% end >

That’s fine and dandy until you want to cache the view for not just one object in your loop, but the entire rendered view of @objects. As much as I looked around, I found that nobody has written about this problem. I still wonder if I’m being naive and there is some obvious way to do this that has simply slipped by me.

When I first tackled this, I had just a handful of objects in one particular view and would never have more, so I just concatenated the cache_keys of all of them and I was done. In another instance (with a larger number of objects) I tried something different by prepending another object which could be changing and appending a string: cache [user_role, @project_application,'task_lists'] do. This was painful because it required that I ensure that all the objects down the relationship tree call :touch and update their belongs_to parent until finally my @project_application was updated.

The task lists in this application don’t just have tasks, they have attachments and notes too and this just felt like the invalidation of the cache was far too intertwined among the related objects for my liking.

Finally I realized that it was much simpler than that. What I need here is pretty simple: a cache_key for a collection.

A group_key method on a collection would be so much simpler than any convoluted process I attempted. Here’s what I did:

def self.group_key
  count = ActiveRecord::Base.connection.select_value("select count(*) from widgets").to_s
  timestamp = ActiveRecord::Base.connection.select_value("select max(updated_at) from widgets").to_s.parameterize.wrapped_string
  count+'-'timestamp
end

That makes 2 simple calls to the database to get a parameterized representation of the most recent count and updated_at value for my Widget. If any of my objects change for any reason or if I add a new record, my group_key will be updated.

I took this and put together a simple gem called group_cache_key which you may find on gemcutter.org. The source is a bit different in that it includes a max updated_at and max created_at and it doesn’t go back to the database for the information:

def cache_key
  if self.empty?
    'empty'
  else
    update_timestamp = max {|a,b| a.updated_at <=> b.updated_at }.updated_at.to_i.to_s
    create_timestamp = max {|a,b| a.created_at <=> b.created_at }.created_at.to_i.to_s
    self.first.class.to_s.underscore+'/'+length.to_s+'-'+create_timestamp+'-'+update_timestamp
  end
end

I hope you find it useful. It’ll give you a cache key value of something like widget/2-1253224342-1253311589.

To use it, grab the gem (gem install group_cache_key) and add this to your config/environment.rb:

config.gem `group_cache_key`

Let me know what you think.

Update

I’ve pushed a newer version of the gem out there which creates a hash of the given ids. The hash is created in the order given, so if you’re doing a sort in your interface, you’ll get an entirely different cache_key.

Changes in Radiant AdminUI for 0.8.1+

Jim Gay — Sat, 19 Sep 2009 19:00:53 GMT

There’s a lot of focus on the visual changes to the Radiant interface, but there are some functional changes that should be discussed too. I’ve started playing with the new AdminUI as I try to understand the changes (since I haven’t had time to take part in the development). The first thing I tackled was figuring out how to rearrange the navigation items and it’s not too easy to understand what’s going on.

First, here’s the new structure of the tabs:

NavTab #=> [NavSubItem, NavSubItem, ...]

NavTab is a subclass of Array and it’s accessed via admin.nav although admin.tabs will still be available (with a warning of deprecation) for some time.

I’ll be posting details like this on the Radiant blog once the code is more developed.

There’s debate about where Snippets ought to go: content or design? Do site editors use them, or do designer/developers use them? So that’s a good place to start.

In order to do this, I had to generate an extension. Then, in my extension, I added this to the activate method:

admin.nav['content'].delete_if{ |t| t.name.to_s == 'snippets' }
admin.nav['design'] << admin.nav_item(:snippets, "Snippets", "/admin/snippets")

That’s complex, but it moves the “Snippets” link from the “Content” tab to the “Design” tab. It should be simpler; something like this would be nice:

admin.nav['content']['snippets'].nav_tab = admin.nav['design']

but better yet, I’d rather have:

admin.nav_items['snippets'].move_to('design', :after => 'pages')

It doesn’t seem too valuable to implement that just to move one thing, but with the new tab/nav structure I would think that there will be a lot more customization by users/developers.

1) Does that syntax appeal to you? (Why or why not?)

To make it even simpler, it might be nice to be able to define structure in an initializer or a yaml file somewhere so that Radiant would load it’s default setup, but then read your changes and rearrange the navigation accordingly. I wouldn’t want to tackle that until other questions are answered, but it’s an idea for the future.

Next, the recent changes only allow you to specify visibility on the main NavTab and not on the NavSubItem.

2) Do you want to be able to specify visibility on the NavSubItem?

Using the above example:

admin.nav_items['snippets'].move_to('design', :after => :pages, :visibility => :admin)

Personally, I think something like this is a must. I see tabs as a way to organize links and not as much of an authorization mechanism. By that I mean that in order to hide a section to all but a specific role, you’d need to put it in it’s own tab even thought it might already make sense in a place like “Content”.

These recent changes are going to cause a bit of pain for developers to upgrade the numerous extensions out there, but it’ll be well worth it in the long run; especially if it’s easy to move things around.

Updates in Radiant have been causing some pain do to the major changes being made recently and if you’re not well-versed in the core application it can be difficult to keep up. Please post your questions to the mailing list where people are generally very helpful. Although we’d all like to avoid upgrade problems, they sometimes comes from changes in Radiant or changes in Rails, and we are doing what we can to create problem free upgrade paths. But this doesn’t mean that the creators or maintainers of the particular extension that you use are yet aware of your problem. Hence, the mailing list is your best option for help.

Thanks very much to John and Sean for hacking away and getting this into the master branch.

UPDATE: There’s plenty of discussion going on about this on the development email list, and I’ve got some related changes that I hope to add to the core here http://github.com/saturnflyer/radiant/commits/tab

Fat Free CRM on Heroku

Jim Gay — Tue, 08 Sep 2009 11:46:24 GMT

Heroku is an excellent way to easily deploy Rails applications. But there’s a trick to doing it: it’s a read-only file system.

To some, this may be old news, but to others who have come across a project using SASS, you might feel like your out of luck. Fat Free CRM, for example is a great new project, and it uses SASS. Try running it on Heroku and you’ll start getting frustrated. If you’re not familiar with SASS, it automatically generates static CSS files for you in production, meaning it needs write access to the file system.

So, what do you do? The good thing is Heroku has a plugin to help with SASS. But, in this instance, it doesn’t seem to work. I’ve installed it with Fat Free CRM and have pushed up to Heroku, but I get errors so rather than debug the issue you can try another option: http://github.com/mooktakim/herokusassand_cache/tree/master.

Here’s a step-by-step. First, if you don’t have a Heroku account, signup and install the gem:

$ gem install heroku

Next, get the Fat Free CRM code:

$ git clone git://github.com/michaeldv/fat_free_crm.git

And turn it into a Heroku project

$ cd fat_free_crm
$ heroku create

You’ll need to install a plugin to manage the location for SASS files.

$ script/plugin install git://github.com/mooktakim/heroku_sass_and_cache.git

Read the documentation for the plugin, but you’ll need to add this to config/routes.rb:

map.heroku_sass_and_cache

Just drop that on the first line of your routes.rb file and commit all of your changes.

$ git add .
$ git commit -m "feeling sassy"
$ git push heroku master

That last bit will deploy your application to Heroku. Fat Free CRM requires that you run the rake crm:setup task.

$ heroku rake crm:setup USERNAME=myusername PASSWORD=mypass EMAIL=my@email.com

After that, you should have a working Fat Free CRM on Heroku. Try heroku open and you’ll land at your new login screen.

This will get you going, but you’ll find that minor things like storing avatars won’t work because they expect to be stored on the file system. Missing avatars aren’t an application deal-breaker and I imagine that there might be some options in the future for separate storage options, or even just using Gravatar.

If you have any luck with the official Sass support on Heroku, leave a comment and I’ll update the post.

Why you should use Devver

Jim Gay — Tue, 28 Jul 2009 19:21:48 GMT

For good reason, developers share concern over not only a well-tested application, but also a regularly tested application. If you are managing a group of developers and are responsible for the outcome, wouldn’t you first find a solution to allow your developers to run the test suite whenever they want? If your team members are running tests whenever they want on your massive application it’ll probably slow them down, so of course you’ll implement a continuous integration server. But then you’re left with the requirement that they checkin their code. That could be a painful situation if they checkin some feature breaking hack. Enter Devver.

I’m participating in the Devver beta. When I got my invitation I immediately starting testing out the RadiantCMS source. While the code base is relatively simple, Radiant has some great test coverage including Cucumber features.

Radiant is a bit of a strange bird when it comes to Rails apps since it has it’s own Radiant::Initializer rather than a Rails::Initializer. Devver wasn’t quite expecting this, but Dan, Ben, and Avdi were extremely quick to respond to my requests.

There’s been a good deal of development around the problem of running a massive test suite for an application. There’s CruiseControl or CruiseControl.rb, and Integrity or Inotegration, or RunCodeRun and plenty of others that I’m missing. It seems to me that the problem with those approaches is that you need to checkin your code. In some respects, it’s too late to run your tests when you’ve already checked in the code.

Of course you can use things like ZenTest with autotest. And there’s spork for speeding up your tests as well. But a massive test suite that takes a while to run is something you might like to avoid running locally. Offload that to a server! Run rake devver:spec or rake devver:test. It’s a very simple process.

You could, of course use testjour if it’s a viable option in your development environment, but Devver is still probably much simpler to setup.

Installation is simple.

Download and install the Devver gem
Download and install the Devver Rakefile
Configure Devver to use your API key
Declare your gem dependencies
Run it!

I’ve not been compensated in any way by Devver. They’ve taken a great idea (run tests before a code checkin) and made it easy and fast; and I thought you should know about it.

Textpattern on Github

Jim Gay — Tue, 28 Jul 2009 17:34:11 GMT

I’m a former user of Textpattern and I’m glad to see it more easily accessible on github. I hope it’ll be easier for other users out there to contribute to the project.

My path to abandonment began when I built a real estate management interface on top of Textpattern. It was an interesting experience to say the least. The system is built to provide clear methods like graf() which I’m sure you would guess is a way to display an HTML paragraph element…

And of course, for your developing delight there are other methods like sLink(), eLink(), wLink(), dLink(), and aLink(). I’ll leave it to you to guess what features they implement. My favorite is listed in there too.

I am glad, however, that in the very unfortunate event where I might need to use Textpattern again it’ll be much more accessible to track other forks and branches. Cheers to the Textpattern community, from a thankful but recovering user.

External website resources

Jim Gay — Fri, 03 Jul 2009 01:15:36 GMT

I wrote a recent post about managing ALL of your content where I discuss the idea that your content is not just the content on your site.

In my continuing research for better ways to manage content I came across a real gem. Net::DNS which is a helpful tool for resolving DNS and gives you a different perspective than Ruby’s Resolv.

If you want to check if a domain is valid then something as simple as this might do the trick:

Net::DNS::Resolver.start("google.com").answer.size > 0

But of course, there’s much more to it than that. Check it out!

Ruby FuzzyHash

Jim Gay — Thu, 02 Jul 2009 12:44:22 GMT

I came across an interesting way to use a Hash at http://github.com/joshbuddy/fuzzyhash/tree/master.

I may look into this further for Vapor since this is basically what that RadiantCMS extension needs to do.

I’ve altered the sample code but it does all the explaining:

hash = FuzzyHash.new
hash[/^\d+$/] = ‘number’
hash[/.*/] = ‘something’
hash[‘chunky’] = ‘bacon’
hash[‘foo’] = ‘vader’

hash[‘foo’] #=> ‘vader’
hash[‘food’] #=> ‘something’
hash[‘123’] #=> ‘number’

Rails Metal in RadiantCMS

Jim Gay — Thu, 25 Jun 2009 12:27:49 GMT

Radiant edge now supports loading Rails Metal from extensions!

I’m really excited to announce that. I had been working on it before the release of 0.8.0, but hadn’t had the time to hammer it out before the release. Admittedly, I push some sloppy commits into the main repo, and I should have rebased them and cleaned them up. This was the first opportunity I’d had to look around at the way Rack middlewares are loaded in Rails, so a lot of my effort was just poking around. Gregg Pollack’s screencast on Rack & Metal had some helpful tips in it. Check it out if you want to try out the new features in Radiant.

This will make things like checking a login status (like we’re doing with Practice Greenheath and the Header Authorize extension) much faster.

More importantly, Vapor(the extension to allow users to write their own redirect rules which also caches all the rules so its nice and speedy) will be moving to metal. I’ve created a separate branch for this, but I’m considering backward compatibility so that if you’ve got an older version of Radiant, it’ll still operate the same old way (by catching the requests in the SiteController). I’ll need to re-evaluate the code before it goes into the master branch, but it works!

Managing ALL of Your Content

Jim Gay — Thu, 18 Jun 2009 14:47:19 GMT

We’ve been working hard to provide great features to our customers that are using Radiant and we’ve been pushing a lot of that work out into the community. We’ve still got plenty more in store for our extension development and of course a lot more to contribute to Radiant.

One of the problems any organization might face is the task of keeping track of your content. Of course, you’d look to a content management system like Radiant, but even beyond your content is the content from others. Many organizations have relationships with others and their content is often linked back and forth. As the content changes, some things may get stale and we’re working on a way to keep track of that.

Your content isn’t just your content. Your site depends on its environment and you’ll need to react to any changes in it.

When you’re writing content for your site, you shouldn’t just be worried about what you’ve got. You also need to make sure that your target sites (or sites on which you might comment or to which you might send your visitors) are up and running too. Site Watcher is a great way to keep track of what’s happening on your own site and we’ll be adding more features there, but there will be more to come for tracking the rest of the world too.

I’ll be posting more about this in the future as we get ready to release our upcoming projects.

Pretending to Be Different

Jim Gay — Wed, 17 Jun 2009 00:17:12 GMT

It seems to be very common to say in many ways that you are different, but when it comes to showing it, many companies must be too afraid to actually *be* different.

I’ve seen a TV commercial for the Audi Q5 which claims that it is unmistakable. From the behavior of the actors in the ad I would guess that it’s supposed to be unmistakably different, but when it comes time to show how different it is you are shown that it is… black. The other cars, you see, are beige. Never mind that they look the same in every other way.

This ad just plain confuses the point:

Are you doing this? Is your company claiming to be different but not showing your differences?