The iPhone app wasn’t CATA’s first foray into real-time bus tracking. About a year ago, CATA released a live Google map of their bus routes. While useful when you’re a near a computer, my biggest gripe was that the map wouldn’t load on most (if any) phones.

So one weekend last Spring I built Catafind.

It’s not as flashy or polished as the iPhone app, but it works on virtually any phone with Internet access.

Eager loading in ActiveRecord is a great feature. It prevents what’s called the “n+1″ problem. From the docs (emphasis mine):

Eager loading is a way to find objects of a certain class and a number of named associations. This is one of the easiest ways of to prevent the dreaded 1+N problem in which fetching 100 posts that each need to display their author triggers 101 database queries. Through the use of eager loading, the 101 queries can be reduced to 2.

The docs then give this example:

This can scare beginners, as it did to me when I started learning about eager loading. This query doesn’t magically work by itself. You must have the proper associations set up in your models. Once I started to think of that include clause in terms of the associations, everything made sense:

Before:

#controller:
@posts = Post.find(:all)
 
#view:
< % @posts.each do |post| %>
   < %= post.title %>
   < %= post.body %>
   < %= post.author %>
   < %= post.date %>
< % end %>

After:

#controller:
@posts = ActiveRecord::Base.connection.execute("SELECT * FROM posts")
# @posts.class #=> Hash
# @posts[0].title will return a method missing error

For the view you have two choices.
1)Rewrite the method calls:

< % @posts.each do |post| %>
    < %= post['title'] %>
    < %= post['body'] %>
   < %= post['author'] %>
    < %= post['date'] %>
&lt; % end %&gt;
#this is tedious

or
2) Hack the hash to make keys act as methods. There are several ways to do this. From a stackoverflow question I asked, someone showed me a module in the Ruby standard library called OpenStruct:

#controller
require 'ostruct'
@posts = OpenStruct.new(ActiveRecord::Base.connection.execute("SELECT * FROM posts"))
 
#view
< % @posts.each do |post| %>
   < %= post.title %>
   < %= post.body %>
   < %= post.author %>
   < %= post.date %>
< % end %>

Other ways to do this (from stackoverflow answerers):

• define the method_missing function for the Hash class
• define the method_missing function for just the hash instance you’re dealing with

You can try these out with a really cool pastebin that actually runs the code:

• OpenStruct
• modify Hash class
• modify specific hash instance

It’s interesting that all of these solutions used about 5mb of memory according to the ideone.com.

I always wanted the app to be able to continually check the xml feeds, which are updated every minute, so I could store the data persistently and then use memcache to handle large spikes in traffic (hopefully my site will have this good problem). In order to do this, I needed to set up a cron job. Well I was in luck as Google recently added support for cron jobs, a feature not previously built in to App Engine. 

The setup is really simple and intuitive. Here’s the relevant file structure:
/app.yaml
/cron.yaml
/main.py
/models.py
/my_cron.py

app.yaml:

1
2
3
4
5
6
7
8
9
10
11
12

application: myapp
version: 1
runtime: python
api_version: 1
 
handlers:
- url: /my_cron
  script: my_cron.py
  login: admin
 
- url: .*
  script: main.py

Whenever a request is sent to your app, its url is checked against the list of handlers. If the url is myapp.appspot.com/my_cron, the file my_cron.py is executed. All other urls go to the main guts of the site, main.py. Notice the login: admin part. This makes sure only the admin can access this url.

my_cron.py:

1
2
3
4
5
6
7

from models import *
from tools import *
 
#pseudocode:
info = get_data(args)
i = MyModel(info=info)
i.save()

Another benefit from setting up this cron job was that it forced me to separate my models from main.py (the code is looking more Django-y by the day).

Now let’s tell App Engine how often we want my_cron.py to be executed.

cron.yaml:

1
2
3
4

cron:
-description: grabs some data
  url: /my_cron
  schedule: every 20 minutes

The schedule parameter is the most important part. It can be simple, as I have shown, or very complex, following these conventions.

That was easy.

Certainly there’s an AdWords-like business there, but, as [Todd] Chaffee [an Institutional Venture Partners general partner and a new Twitter backer] told us, Twitter has another “wild card.”

In the future, searches won’t only query what’s being said at the moment, but will go out to the Twitter audience in the form of a question, like a faster and less-filtered Yahoo Answers or Wiki Answers. Users would be able to tap the collective knowledge of the 6 million or so members of the Twitterverse.

“You put a question out to the global mind, and it comes back,” Mr. Chaffee explained. “Millions of people are contributing to the knowledge base. The engine is alive. You get feedback in real time from people, not just documents.”

This is essentially what Aardvark does. And a really good job of it, I might add. Around the same time that Twitter’s biggest goal was to stop crashing, the Aardvark team received $6 million in investment to refine social search. Right now, Aardvark routes its questions and answers between several channels: Google Chat, AIM, Windows Live Messenger, as well as good old e-mail. I posed this question to “assistant curator of birds” of Aardvark, Rob Spiro:

adelevie: @robsp can Vark work through Twitter the same way it works through GChat and e-mail?

12:55 PM Mar 4th from web

His reply:

robsp: @adelevie yup, we’re definitely planning an aardvark-twitter integration, using Twitter as another communication channel… coming soon…

4:24 PM Mar 4th from web in reply to adelevie

Would their two services compete? Could Twitter acquire Aardvark? We’ll just have to wait and see.

Aardvark has recently been ramping up activity, changing the name of their company from Mechanical Zoo to Aardvark, preparing to offer unlimited friend invites to current users, and launching a blog.

I always had to fix the indents, add some basic import statements, and include my favorite web scraping library, Beautiful Soup. So I created a template of my own and named it “better_project_template” (creative? I know). The folder has the Beautiful Soup module, base.html and index.html templates, as well as some useful shortcuts added to the main.py file. Anyone interested can download this folder here for their own use.

The most significant new feature is profile pages. Just as Delicious accounts are in the form of delicious.com/username, Bit.ly accounts are in the form of username.bit.ly. Instead of linking to a Delicious account to show the world what you are sharing, you can just link to your Bit.ly account. However some features are missing from the profile pages, most notably, share/click counts. Bit.ly could easily list next to each link on adelevie.bit.ly the number of times someone has shortened the same url as well as total number of times people have clicked on that link. The data is all there, it’s only a matter of incorporating it into the user interface. Nevertheless, the people at Bit.ly do seem to understand the power of using their own click data to find what is relevant on the web. They’re testing the waters very slowly with this concept.

Back in October of 2008, the Bit.ly blog had a post titled Yesterday’s Top Bits where they simply listed the highest-clicked Bit.ly links. Later, in February of 2009, Bit.ly created a Twitter account called bitlynow which tweets the highest-clicked Bit.ly links for a certain time period. This gives Bit.ly the ability to keep a pulse as to what’s being shared on the web in a similar manner as Twitturly. Twitturly’s advantage is that its data include just about every link that goes through Twitter. Bit.ly’s advantage is that its click/referral tracking feature allows it to look beyond just Twitter. I’d like to see some amalgamation of Twitter, Twitturly and Bit.ly.

While Bit.ly’s new features are encouraging, they still don’t address the 800 pound gorilla in the room: its growing database of semantic data about each Bit.ly link (from Open Calais‘ api). Solely tracking the flow of links only gives part of the picture. Tracking the flow of the meaning of those links is a lot more valuable. It would be nice to see Bit.ly make sense of all this data.

1
2
3
4
5
6
7
8

    template_values = {
      'greetings': greetings,
      'url': url,
      'url_linktext': url_linktext,
      }
 
    path = os.path.join(os.path.dirname(__file__), 'index.html')
    self.response.out.write(template.render(path, template_values))

To avoid this morass of boilerplate code, and to better keep track of your code, simply add the following snippet to the beginning of every instance of the RequestHandler class (each “page,” if you will):

1
2
3
4

disp = self.response.out.write
def render(template_name, template_values):
          path = os.path.join(os.path.dirname(__file__), '%s' % template_name)
          disp(template.render(path, template_values))

Now to render a template, simply use return render('index.html', {'form' : someForm()}).
Think of this as App Engine’s version of Django’s Render_to_Response function ;).
Similarly, if all you want to do is spit out data that won’t go through a template just use return disp("some string").