Why do you need to do this?  For any number of reasons.  Maybe you have software that needs to dump diagnostic info up somewhere central, or you are trying to automate a build process by dropping artifacts into a central repository.

Anyway, here’s all you need to do:

server {
    listen 80;
    server_name put.domain.com;
    charset utf-8;

    location / {
	 root /var/www/dump;
	 client_body_temp_path /tmp;
	 dav_methods PUT DELETE;
	 create_full_put_path   on;
	 dav_access user:rw group:rw all:r;
	 client_max_body_size 100M;
	 autoindex on;
   }
}

You could protect this by enabling HTTPS and/or authentication as additional nginx configs.

Anyway, once this is done, you could put a file like so:

curl -T /path/to/file http://put.domain.com

And open up http://put.domain.com in your browser to view what’s been put in there, and download anything of interest.

Let’s do gems.

Say you have a nginx installation. And so you have a build server churning out gems. Just do an HTTP PUT against a server configured similar to the following:

server {
listen localhost;
client_body_temp_path /tmp;
root /var/www/data/gems;
server_name gems.example.com;
dav_methods PUT DELETE;
create_full_put_path on;
dav_access user:rw group:rw all:r;
client_max_body_size 25M;
}

This allows you to do something like:

curl -T /path/to/gem http://gems.example.com

Then, using incron, cron, or manually, rebuild your gem server’s index:
gem generate_index -d /var/www/html/gems
(but this could have been yum’s createrepo or apt’s equivalent.

Finally, use your gem server in a gem file!

source 'http://gems.example.com'

Gist for configuring the nginx server:

https://gist.github.com/4108841

• You don’t want to have any gems or code being installed on your production Rails server from web sources, but you can ensure that are running the script below on a machine with the same architecture and distro as the target server (particularly important if you are using a gem with C extensions, such as the pg gem)
• You want a separate package for your Rails app, separate from the assets of that Rails app

Packaging Your Rails App

bundle install # acquire all dependencies as declared in the Gemfile
bundle exec rspec # run tests--don't build a broken app, right?!
bundle package #places gems in /vendor/cache
bundle install --path vendor/bundle --local
# fake function to your 'packaging' routine.
# You could be creating a tar, deb, rpm, whatever.
# Anyway, you would package up the entire directory at a minimum.
package_app

Packaging Your Rails Assets

# create asset bundles
RAILS_ENV=production  bundle exec rake assets:clean
RAILS_ENV=production bundle exec rake assets:precompile
# package up the folder public/assets and deploy to your web asset server
package_assets

Hope that helps!

If you are new to Ruby and Rails development in general, it can be a bit difficult to know where Rails magic ends and standard Ruby begins.  Here is a quick check-list that one can follow in order to use a model, from another gem, in your Rails 3 application.  The following steps assume your gem is called your_gem (which means within the gem .rb files, your module name is  YourGem)

In Your Rails App

1. Optional: In application.rb in your Rails app, before the declaration of your module, add an include statement to make your model names available as non-namespaced symbols.

   include YourGem

2. In routes.rb of our Rails app, be sure to use scope,  if you want to use resource style routes:

   scope as: "your_gem" do

       resource :some_model

   end

In Your Gem

1. In your gem, if you are using FactoryGirl to test your models, you will need to specify :class=> “YourGem::SomeModel”, because I believe FactoryGirl needs ‘help’ for models contained within namespaces.

   FactoryGirl.define do

       factory :your_model, :class => 'YourGem::SomeModel" do

       # ...

       end

   end

2. In your gem, make sure ActiveRecord has a database connection before your tests start.  So, in spec_helper.rb, do the following:
   1. ActiveRecord::Base.establish_connection(dbconfig) # where dbconfig could be loaded from the test context of config/database.yml
3. Be sure your gem follows Rails conventions.  A gem with a name of your_gem should have a module name of YourGem.  Anything else and you may have problems with the way rails looks for models to load.

In the end, I did not have to modify config.autoload_paths in my application… I certainly thought I would.

The closest thing I can think of that’s type safe and Javaish is pasted below–but also saves time typing mundane data structures.

With this code, one can do:

Pair pair = Tuple.createPair(“oh hai”, 12345)

By the way–I hate this solution. It’s still too much tip-tapping on the keyboard.

.NET’s Tuple is cleaner.


package copy.paste;

@Override public FIVE get5() {
return five;
}
};
}
}

I’ve been passing request-reply messages around with JMS for quite some time in Camel, and it’s always bothered me that threads have been tied up for the duration of the message.  Asynchronous Processing fixes that, if you choose to use it.

I don’t know what it is about Camel, though, but I have a hard time figuring out anything in terms of initially bootstrapping my routes and how that corresponds to actual Java code.  So since I got JMS + Asynchronous Processing to work, I thought I’d capture it here to help out. Here goes:

Route and Inline Consumer

// whenever  message is sent to a.test, our AsyncProcessor defined here will fire
from("jms:queue:a.test").process(new AsyncProcessor() {
    @Override public boolean process(final Exchange ex, final AsyncCallback cb) {
           String body = ex.getIn().getBody(String.class);
           ex.getOut().setBody("Response: " + body);
           cb.done(false);
           return false;
}

@Override public void process(final Exchange exchange) throws Exception {}
});

Sending message as client:

 // get endpoint for JMS queue defined in route above
 Endpoint endpoint = camelContext.getEndpoint("jms:queue:a.test");
 // get producer so that we can fire messages
 Producer producer = endpoint.createProducer();
 producer.start();

 // create test message
 final Exchange exchange = producer.createExchange();
 exchange.setPattern(ExchangePattern.InOut);
 DefaultMessage message = new DefaultMessage();
 message.setExchange(exchange);
 message.setBody("test");
 exchange.setIn(message);

 // this cast is safe because this must be a JmsProducer
 // .process() completes immediately; AsyncCallback instance passed in has done() called
 // when the consumer above has sent it's response.

 ((AsyncProcessor)producer).process(exchange, new AsyncCallback() {
      @Override public void done(final boolean b) {

          // our async callback. access original exchange for response body
          // should always check exchange.isFailed()
          System.out.println("Got response: " + exchange.getOut().getBody());
      }
 });

That’s it! It’s pretty straightforward once you know what to do.

Original Post:

Using the Facebook API is the one of the worst experiences as a developer I have ever had.  This past week has been a real trial, and a true challenge to keep my energy and spirits up so that I remain efficient.

Whew, that’s off my chest.  Let me stop with the bashing, and try to digest down what they are doing wrong in an effort to try and be constructive rather than purely inflammatory.

POOR DOCUMENTATION

When you document your API, you have to at a minimum do the following:

1. Define each method
2. For each method, define inputs
3. For each method, define outputs
4. Describe the possible values for every input
5. Detail possible error responses
6. Detail how configurations outside the scope of the API call may affect your code.
7. Performance limitations (rate limiting in the case of Facebook)

Facebook fails to come through on all of these points, even #1

Example:

I was trying to find some part of the documentation that would describe all Facebook-created dialogs that I can use, because I wanted to show a ‘friend finder’ equivalent to what is natively available in Facebook, but on my own website.  I find  FB.ui (a ‘Core Method’), look over the documentation for that method, which I see takes an argument called … ‘method’. Ok, fine, so FB.ui must actually be a class or package and not a method, but whatever.  I’m already confused a little but no biggie.

So once I take a look to see what’s available in FB.ui…there are only two methods stream.publish and stream.share! That’s not much.

I back up to the parent of FB.ui (‘Core Methods’), click around every link I find there, do google searches, and look through the developer forum, but find no more UI-related methods.

I know that can’t be right. Fb.ui looks suspiciously under-documented.  I find nothing but a link to github for the Javascript SDK.  Sighing, I download the code, find the file that corresponds to FB.ui, and  see:

• friends.add
• stream.publish
• stream.share
• fbml.dialog
• bookmark.add
• profile.addtab

I freaking knew it.

How do these extra methods work? What are there limitations? It’s just javascript so even reading through the code won’t help much because it interacts with Facebook servers and therefore I can get only a partial picture at best.

Anyway, I then zero in on fbml.dialog, because I vaguely know FBML has a lot of predefined tags made by Facebook that I could use to make my own UI.  So I turn to the FBML documentation.  The main page says this along the top now:

Note: We do not recommend FBML for new developers. If you aren’t already using FBML, you should instead implement your application within an iframe, using the JavaScript SDK and social plugins for client-side integration with Facebook services.

Wait a second… I shouldn’t use FBML because I should use the Javascript SDK?  But the Javascript SDK has a method called fbml.dialog, which led me here!  So now I’m wondering if by using FBML, am I opening myself up to have the floor ripped out from under me when Facebook gets rid of FBML, because on their roadmap I see:

We will stop allowing new FBML applications, but will continue to support existing FBML tabs and applications. Instead, we recommend using IFrames.

But this is just a roadmap, and moreover, it’s the dubious Facebook roadmap.  I decide I won’t worry about it, and try to figure out how to use fbml.dialog on my own. I know the only other option left for me is to post on the developer forums, but just the thought of doing so leaves me despondent.

In closing on documentation, searchability is terrible, navigation is terrible, the content is extremely thin and often wrong, and the developer forums are a wasteland of frustration.

POOR TESTABILITY

If I’m going to use an API, I have to test it first, right?   This is even more important with the Facebook API(s), since the documentation is so poor and the amount of issues you encounter with almost any API call ensures the only way to really know what the Facebook API will do is to simply use it.

Well, what if there were an API call that only worked with production code?  That’s pretty limiting, especially in the face of all the other Facebook issues.  It’s a bit chicken-and-egg to have to have production code act as your testbed, wouldn’t you say?

Well, unfortunately, any API call that has to do with the dashboard is subject to this limitation.

Side rant: what is the dashboard?  Good luck finding a generic description in the API documentation as to what it is.  You have to sift though forum posts and blog posts by Facebook to infer that the dashboard is an end-user’s left navigation.  That’s the ‘dashboard’.  I’d call it a left nav but hey, Dashboard sounds better, right?

Anyway, you can see all dashboard API methods here if you scroll down.  Oh but by the way, don’t use publishActivity.  From scraping the forums for hours for anything related to the dashboard, I can tell you it doesn’t work because other users have already been burned by using just the documentation.

Back to testability.  I tried out dashboard.addNews, which should make a news item show up in my stream … according to the documentation (are you starting to feel a description in the docs don’t count for a lot?). Here is the relevant description of dashboard.addNews:

Calling this method appends the news items included in this call to the news stream. Facebook displays up to two news items (either personal, global, or both) at a time for a given application, starting from the two most recently added.

Great! Just what I needed.  So I take the time to create a test application, create test users, and use dashboard.addNews successfully–at least, as far as the response of the API call is concerned.  But when I check the test user’s account, there is strangely no news showing for the application.  So back to the developer forums to look for similiar problems.

I find this post, which seems to imply the application must be bookmarked and submitted to the directory–or doesn’t require being in the directory depending on which user you believe in the post. Also, this bug referenced in the forums simple says that ‘sandboxed’ applications can not be used with the dashboard, so I made sure that my application was not configured as sandboxed (which makes no sense for a test application but you do what you gotta do).

I found a few other forum posts talking about bookmarking, so I tried that out first when debugging. But apparently the way you bookmark applications recently changed, confusing me a while how I was supposed to do this.  Long story short is, I got my application bookmarked, and if you can imagine, addNews still didn’t work.

So I posted on their developer forum.  After 24 hours, no responses yet.  I could wait, but since most posts don’t get responses on the forums, I have to assume I won’t get one.

So moving on, I thought, well, I never tried submitting my application to the directory.  So I found the link for that (which is not in the configuration of the app where I spent a few minutes looking first), clicked it, and was told that I have to have at least 5 users of the application.

…

Wow.  So just to test the idea if being submitted in the directory is important, I have to somehow get 5 users associated with my application?   I had actually signed up four test users with the application, but the application reported that I only had a grand total of 0 users.

I eyeball another application I made for a different company, which had more than 5 users but was never published because it didn’t need to be found via the Facebook directory.  But if I could just try out dashboard.addNews with this application, maybe I’d have an answer!

Since I’m  luckily a co-founder of this other company, I said the heck with it, I’ll publish it.  And so I did. After 4 hours waiting for it to be approved, I then tried dashboard.addNews with this application instead of my test application.

Nothing happened.

And, this time, I can’t seem to successfully bookmark it.

If I make progess on this, I’ll update my developer forum post at least with my solution.

UPDATE:  I can’t play with the configuration of my application so that I can continue debugging because of a rather major bug affecting all applications for the past 24 hours or so.

In closing on testability, there are other reasons that it is difficult to test Facebook applications, but I think the example here is good enough.

NO RESPONSE TO SERIOUS ISSUES

I’ll zero in on two.

ORDER BY AND LIMIT in FQL

FQL allows developers to make queries to fetch results in a SQL-like way.   A while back (I think June 2009), Facebook added ORDER BY and LIMIT to the language.  But it is horribly broken.  If you like, you can try to discern what’s wrong with ORDER BY and LIMIT by navigating through the NEW and UNCOMFIRMED bugs opened here: Note that almost all bugs are NEW and UNCONFIRMED, so don’t just assume that a new bug is actually new in terms of age.

Please, don’t bother looking through their bug tracker.  I’ll tell you what’s wrong with ORDER BY AND LIMIT.  Basically, it seems Facebook has a sharded database backend, and simply proxies your ORDER BY and LIMIT to it’s back end data stores, meaning each sharded data store, which only has a partial set of the overall data, can’t possibly do this correctly since LIMIT will cause the sharded data store to paginate into it’s own data, limiting the data you get back artificially and incorrectly.

Protip: If you use ORDER BY/LIMIT, you should set the LIMIT to the max of 1000, and just get as much as you can.  Don’t ever try to paginate with FQL.

UPDATED_ON/CREATED_ON CORRUPTED BY USING ORDER BY AND LIMIT

I found that, for some facebook fan pages, that if I specified ORDER BY, LIMIT, then the created_on and updated_on fields returned are completely incorrect.  I opened a bug on it in February 2010, and nearly 7 months later, the state of my bug was marked as duplicate of another bug.

Oh boy, 7 months and I can fix my application!  Yaaaah!

Hold on, wait, what’s this?  The other bug is marked as unconfirmed still. Oh. It was opened in August 2009.  So it’s been over a year. I guess you just have to be patient.

Hey, but surely they’d warn in their documentation that ORDER BY and LIMIT are broken?

Not here.

Not here either.

No.

CONSTANTLY CHANGING API

If you are going to constantly change your API, deprecate things, remove methods, etc, you had better be fixing major issues along the way.

But everything indicates that the API(s) only get more confusing and more broken, and migrations are done usually rather brutally.  You can search through the developer forum if you want to corroborate.

A side-effect of a constantly changing API is that you aren’t sure what you should be using versus what’s fallen out of favor.

Consider the the reasonably new Open Graph API.  It lets you read and modify a great deal of data found within Facebook.  You could already do this with the Old REST API, but this API is easier!! and better!!

As a developer who has already been through the onerous task of learning the REST API, this new API was only bad news.  I don’t care to relearn, and I don’t want to rewrite existing code.  But I feel compelled to. After all, the REST API is now old.

But I can’t do certain things with the shiny new Open Graph API.  For instance, the dashboard API methods in the Old REST API (I can’t get over the fact they call it OLD.  It drives me crazy every time I read it) are not in the Open Graph API.  So now, I have to use the Open Graph API if I want to be modern and in-line with Facebook, but I absolutely have to use the REST API if I want to use the dashboard (you might be laughing to yourself if you’ve noticed I’m worried about the Dashboard API when I can’t even get it to work).

The removal of application notifications was a good example of this too. The dashboard API is actually a replacement for the old notification API.  I’m not sure why they couldn’t leave the old notification API in place, and just change the actual behavior, but apparently they had to build a newer and more broken API.

I’m not alone by the way in my frustration.

Bottom Line

Facebook, get it together. It’s not hard to have good documentation or good customer service, but it does require internal prioritization.  Stop not caring about developers.

The easiest way to alter Hadoop’s classpath is to configure hadoop-env.sh.  By default, hadoop-env.sh has a commented-out line near the top, that looks like:

# Extra Java CLASSPATH elements.  Optional.

#export HADOOP_CLASSPATH=

Using this information, I updated hadoop-env.sh to:

export HBASE_HOME=/path/to/apache-hbase
export HADOOP_CLASSPATH=$HBASE_HOME/hbase-0.20.3.jar:$HBASE_HOME:$HBASE_HOME/lib/zookeeper-3.2.2.jar:$HBASE_HOME/conf

Note: once you update hadoop-env.sh, you will need to restart tasktracker and jobtracker, which is most easily done by running stop-mapred.sh then start-mapred.sh located in /path/to/apache-hadoop/bin.

Note: don’t try to set HBASE_HOME or HADOOP_CLASSPATH in your local terminal profile if this is the first time trying this–stick to configuring them in hadoop-env.sh.  The reason is that hadoop runs jobs by creating an SSH terminal, and depending on your system’s settings, your local terminal settings may or may not take effect in this hadoop-created SSH terminal.

Ok, now you are ready to export your table.  To see all of the export options, you can run:

bin/hadoop jar /path/to/hbase-0.20.3.jar export

Usage: Export <tablename> <outputdir> [<versions> [<starttime> [<endtime>]]]

To export the current version of the entire table, just supply tablename and outputdir.  Note that outputdir will actually export to HDFS, not your local filesystem.  In my case, the hbase cluster that I was exporting the table from had no direct IP access to my destination hbase cluster, so I had to get the files out of hdfs and into a local directory so that I could move them manually.

So, let’s export:

bin/hadoop jar /path/to/hbase-0.20.3.jar export your_table /export/your_table

Our hbase table is in HDFS under /export/your_table. Let’s get the entire table into your local filesystem (hopefully your hbase table isn’t huge!), by running:

bin/hadoop -copyToLocal /export/your_table /somewhere/local

I then scp -r /somewhere/local to my local dev station (where my other hbase ‘cluster’ is).  With the file on my dev system after the copy, we need to get the file back into HDFS, and then run the import MR job.

Copy from your system’s harddrive to HDFS:

bin/hadoop -copyFromLocal /somewhere/local /import/your_table

bin/hadoop jar /path/to/hbase-0.20.3.jar import your_table  /import/your_table

Note: If you were copying large amounts of data between clusters, SCP copying probably isn’t a real good solution.   Hadoop has a feature called distcp, which can be used to leverage the power of the two clusters to copy large amounts of data.  You can read more about that here.

Finally, I want to give a huge amount of thanks to jdcryans, who literally bootstrapped me through the entire process.  Without him and his constant help in IRC, I don’t know what I’d do (and many others, I’m sure).

However, using it can be a real pain when building a desktop application.  It’s easiest to describe the problem with an example.

Say you want to build a desktop application that integrates with Twitter.  To do so, you have to signup for an API account at Twitter (true for any OAuth service provider), and obtain a secret key which you must ensure never falls into the hands of anyone but you and your application.  The reason this key must stay secret is that it is used to create a hash (OAuth refers to it as signing)  of every API message sent from your application to the OAuth service provider; lose the key, and anyone can impersonate your application.

So the secret key must stay secret.  And what do you, the hapless  developer, do?  You put it on a server, because it’s the only place you can ensure that it stays safe.  So, now, for your desktop application to do anything with the API, it has to send messages up to your server to have the API message signed.

Now here’s the real sticking point.  You have to know the identity of the user before signing the request coming to your server.  And how do you know that?  The user has to somehow log in to this server you were using previously just to perform signing.

Great. Now the user has to create a username & password to your site, just so you can use OAuth with Twitter.

So, in the end, you users still have to get involved with a username & password.  Sure, your Twitter username & password stays locked in Twitter… which is good, no question. But the user experience is still cumbersome, and the amount of work the developer to do is quite high, especially in the case that you don’t already have a web site and login process.

So, in summary, to make a desktop application secure with OAuth, you are forced to make a website with a login if you don’t already have one.  Ouch.

I’m admittedly new to OAuth but as far as I can tell, this is the state of affairs with OAuth and desktop applications.

EDIT!  The SWF above is just an animation I made of two screenshots; one of Word, one of Flash text engine results.  It is 100% to make the text selectable, but this particular animation does not showcase that.

A Flash generated bit of text overlaid on top of Microsoft Word.  Every word of the paragraph matches pixel-perfect.  It’s actually rather hard to notice–when the red squiggly lines disappear, that’s when you are looking at the flash text.  When they reappear, that’s when you are looking at Word text.

I never thought this day would come, but go ahead and consider me a convert… 

The new flash.text.engine namespace within Flash 10 is the typographically correct way to render text for the web.  Anyone who has attempted to render a simple paragraph across multiple browsers quickly realizes how little control we as developers have over the rendering of text on the web.  Just take a look at Font Rendering Differences: Firefox vs. IE vs. Safari if this problem doesn’t resonate with you.

With flash.text.engine, persistent problems such as lack of support for custom fonts and kerning go away entirely. Pixel-perfect rending across browsers is achievable–the only question left is whether flash is acceptable to you and your Web 2.0 philosophies.

If you are still with me, then let’s explore flash.text.engine a little more deeply…

Because the new Flash text engine provides such accurate text rendering, a web application can now exert minute control over the layout of text, therefore having the capability to render formats that assume access to print-quality text rendering. For example, say we wanted to render an uploaded Microsoft Word file in a web page, with faithful reproduction of the pagination and positioning of every element in the document, as rendered by the Word application itself. If one were to inspect a Microsoft Word document, they would see that it contains information about font size, family, page margins, and the actual text contained in the document, but does not explicitly indicate on which page any element within the document falls (such as paragraphs, tables, and images). Word instead figures out pagination as it loads and renders a given document . So, your web application would have to be able to mimic the Word rendering engine to pull this off! Good luck trying to do that with only HTML, CSS, and Javascript. (to be fair, good luck trying that with any technology, but we are going to try our luck with Flex for the sake of this article).

Let’s back up a second and look at basic usage of the Flash text engine. At the heart of the text engine is the TextBlock class. It makes rendering paragraphs of a set width very straightforward.

To use it, set up the TextBlock with font information and the text to render:


var text:String = "Lorem ipsum dolor sit amet, consectetur adipiscing elit. Aliquam ut odio. Curabitur faucibus, orci ut adipiscing lacinia, ipsum arcu suscipit nulla, vitae mollis sapien lectus ac nulla. Suspendisse eget massa. Fusce auctor nunc. Sed in nisl. In at dui. Nulla elementum gravida quam. Nullam scelerisque lacus ac mi."
// For our example, we'll use Times New Roman, Size 16, kerning on
var fontDescription:FontDescription = new FontDescription("Times New Roman");
var format:ElementFormat = new ElementFormat(fontDescription, 16.0); 
format.kerning = flash.text.engine.Kerning.ON;
// create a text element, which TextBlock will consume in order to begin rendering lines
var textElement:TextElement = new TextElement(text, format);
var block:TextBlock = new TextBlock(textElement, null, null, "rotate0", "roman", 0, true, null, 16.0);
// render a paragraph of the text using a width of 400px
var previousLine:TextLine = null;
while(true)
{
    var textLine:TextLine = block.createTextLine(previousLine, 400.0);
    if(textLine != null && block.textLineCreationResult == TextLineCreationResult.SUCCESS)
    {
        trace("new line of text:" + this.text.substr(textLine.textBlockBeginIndex, textLine.rawTextLength));
        // align text 100px from the left, and offset each line from the last by a factor of 1.15 * font size
        textLine.x += 100px;
        textLine.y += linePosition;
        this.addChild(textLine);
        // TextBlock uses the previous line to know where it should start in the supplied TextElement,
        // for the next call to createTextLine
        previousLine = textLine; 
    }
    else if (textLine == null && this.block.textLineCreationResult == TextLineCreationResult.COMPLETE) 
    {   
        // no more text left to render as TextLine                      
        break;
    }
}       


 
This code results in the following:

With that small bit of effort, a paragraph is now created which represents the string of text and accompanying font information.

 
Taking this code a bit further to account for page margins, one can, as near as I can tell, recreate perfectly a paragraph found within Word by using the flash.text.engine, as shown at the top of this post.

By specifying the same paragraph width as parsed from the Word document, same font size and family, line breaks naturally occurred in our test application exactly where they occurred within Word, creating a very promising avenue to someday displaying a Word document in a web page, without the need for any special client software.

A very trivial example project is available with build script (assumes the Flex SDK bin directory is in your path).