developers.hover.in

hoverlets and a case study : Zemanta API

Bosky — Sat, 20 Jun 2009 18:46:27 +0000

In this post we see :

What is a hoverlet?
Creation of a hoverlet
Usage of a hoverlet
Case Study : A Zemanta Hoverlet

What is a hoverlet?

When we launched the concept of a hoverlet, we wanted it to be as extensible as possible with respect to what was possible. We decided not to go for proprietary packaging and app creation standards for creating a hoverlet since the web is already an open enough standard. We simply decided to augment the same simple markup, javascript, and CSS that everyone is familiar with - with a few objects extra exposed.

Currently you need to register with us to create a hoverlet, But if you are a hosting environment/company would like to provide hoverlet hosting - we’d love to talk to you - so that our users and the community as a whole can benefit from defragmented user-generated content. And we intend to open source whatever it takes to see a wave of adoption that transcends a company and has a chance to benefit the user-experience of the web as it is today.

A hoverlet is just a regular html document that exposes a javascript object that contains details like which is the word hovered, from which page, which site ,and several other parameters to let developers build contextual mashups. This can easily be done by including a CGI script or other server-side scripting methods irrespective of the server and language used.

Creation of a hoverlet

So essentially we’re just hosting webpages that are interpreted in realtime to understand certain macros and have a javascript object that contains context. In our beta product - a developer can create a hoverlet that shows contextual information based on whichever keyword is hovered.
A publisher needs to say - “i like this map hoverlet, now let me associate a keyword to this hoverlet”, or “i like this stock quotes hoverlet, let me associate a company to it”.

Here’s a screencast of how to create a hoverlet, by registering for hover.in beta .

Using a hoverlet

Using a hoverlet was made a much more easier and straightforward. A hoverlet can be embedded into any webpage by including the start.hover.in script and without any registration or credentials. Following that - since the URI is the API , all you need to to to embed or render a hoverlet into a webpage is just give the href or title attribute of an anchor tag to point to a valid hoverlet resource. That’s about it! Here is a structure of how the URI will look like.

http://.hover.in//////...

We have made 10-20 hoverlets during our internal hackdays using the method specified in the video above.

crunchbase
blinkxvideos
relatedyoutube
pubinfo
relatedflickr
relatedpicasa
relatedwordpress
twittersearch
googlemaps
priceinfo
twitterprofile
zemantarelated
safaribooks
bookmarks
nytimes
cricinfo
aolvideos

These are all made by our nick ‘hover.in’. However, we will open up the community in January 2010, where anyone can create hoverlets (like creating facebook apps) . If you are a content provider or a developer, you might want to register well before that so that you can be well-prepared to understand through the platform and the possibilities of a hoverlet - be it a registration form , a dictionary lookup or a paid transaction to buy a product.

Note: If you’re into RDF and semantic web vocabularies, we might be able to dwell deeper into the implication of having a hoverlet vocab?

Case Study : A Zemanta Hoverlet

We loved the idea of showing developers and publishers how they can use mashups and API’s such as Zemanta API and hover.in’s hoverlets , as posted on the Zemanta blog [cheers guys : ) ]to bring context and increased user-engagement for their users.

Here’s a simple example of the start.hover.in library in action using the zemanta related hoverlet created by hover.in for getting related articles to the Iran election in just few lines of markup:

 
 

  iran election

So here just to clarify :

http://onclick.hover.in/hoverlet/hover.in/zemantarelated/election
translates to
http://.//////...
Hence
    event -> onclick
    host-> hover.in
    type -> hoverlet
    nick -> hover.in
    hoverlet  -> zemantarelated
    keyword -> election

You can see a demo of the zemanta hoverlet at http://start.hover.in/#demo4 , and other demo’s of the hoverlets along with more details mentioned above at the homepage of the start.hover.in library.

if you want to pull in relevent content and use the Zemanta API - you can get your own API key, and start building truly contextual applications and hoverlets for the web. And of course, If you’re a content provider, an application developer on an existing platform or a hacker who wants to build hoverlets - do sign up for beta, and get in touch with me via kode at hover dot in, or ping hover.in on twitter.

Keep Clicking,
~B

intern challenge - 1

interns — Sun, 24 May 2009 15:30:45 +0000

Problem:

Given a URL find out the location of the RSS feed and show the corresponding posts.
Bonus: Provide context by finding related posts to a term / given query

Steps:

1. Find the various feeds from the URL using a parser or a similar web service
2. Taking the feed and show the posts
3. Finally host the application
4. Bonus: pass in the context using the HOVER APIs

Solutions:

· Using server side scripting (Perl/PHP/Erlang) to discover RSS / Atom feeds.
· Using Yahoo Pipes to auto discover RSS / Atom feeds.
· Using YQL to fetch the links to RSS.
· Using Google Ajax APIs to find Feeds on Zembly or Appjet or a Hoverlet.

Using PHP to discover RSS/Atom feeds

We have been using PHP for developing our different projects like the Wordpress plugin named wp-hover. So we decided to start with PHP.

Firstly this is how typical RSS links appears in a page.

We tried to parse using Zend_Feed::findFeeds() but it was not giving RSS links as expected. At times it was just printing whole content of the body, and other times gave DOM Element objects which when var_dumped contained just integers and some metadata. If we could get the result using the above function, then we could have easily parsed that RSS URL using the function Zend_Feed::import().

The pseudocode for the solution in PHP:

//finding the rss feed
$feedArray= Zend_Feed::findFeeds('http://developers.hover.in’);
// get the RSS feeds
Foreach($feedArray as $item){
// get the rss feed
$rss = Zend_Feed::import($item);
// Loop through the items in the feed
// and print it out

Since the above didn’t work as expected, we looked at other options.

Using Perl to discover RSS/Atom feeds

Similarly in perl which is known for string and regex handling, we found Feed::Find for this purpose which worked much better. So we setup perl (and its long list of dependency) as follows:

#Installing Perl with CPAN modules (Ubuntu) using apt-get to install :
sudo apt-get install build-essential  libssl-dev   libc6-dev    perl    yaml-mode
#followed by :
perl -MCPAN -e ‘install Feed::Find’
#or running cpan terminal and use command ‘install Feed::Find’

Perl Script:

#!/usr/local/bin/perl
use Feed::Find;
@feeds = Feed::Find->find('http://trak.in/');
print "@feeds";

Using the above script we were able to find all the RSS / ATOM links for the given URL. By default it finds all kinds of feeds. It would have been better if the find function took second argument of currently you need to set this in the source itself.

Since we interested in web application the following Perl code can be run from a web server as a CGI script.

Using Erlang to discover RSS/Atom feeds

Since erlang is predominantly use at hover.in, we explored the same in erlang. MochiWeb is an Erlang library for building lightweight HTTP servers. Since it is an open source project - it has been used in big projects like Couchdb, erlyweb etc and we were encouraged to test out it’s parsing capabilities.

XPath is a language for addressing parts of an XML document, which is based on element nodes, attribute nodes and text nodes.

Reading through a post on pplov’s blog we came to know that he had contributed an xpath parser for mochiweb which can be downloaded at the mochiweb google group

Code:

module(find).
-export([feeds/1]).
feeds(Url)->
    {ok,{_,_Headers,Body}} = http:request(Url),
    Tree = mochiweb_html:parse(Body),
    Xpath = "//link[@type='application/rss+xml']",
    [ {_Tag1,Attributes1,_Content1}|_Rest] = mochiweb_xpath:execute(Xpath,Tree),
    BinUrl = lists:foldl(
         fun({<<"href">>,Href},_Prev) -> Href;
             (_Else,Prev)-> Prev
         end,error,Attributes1),
    binary_to_list(BinUrl).

To run above code

1>application:start(inets).
ok
2> find:feeds("http://developers.hover.in").
"http://developers.hover.in/blog/feed/"

Conclusion:
Using either PHP or Perl or Erlang or basically similar modules from other languages are defiantly viable options, but we decided to also check out solutions that have come up more recently such as Pipes, YQL , Google AJAX APIs which could be hosted on environments like Appjet (AppJet is a website that allows users to create web based applications in their web browser), Zembly or Hoverlets — hover.in’s own hovering widget hosting environment. But more on the application hosting later. First let’s try other methods to discover feeds from a webpage.

Using Yahoo Pipes to auto discover RSS/Atom feeds

Yahoo! Pipes is a web application from Yahoo! that provides a graphical user interface for building data mashups that aggregate web feeds, web pages, and other services. hover.in has always been a big fan of pipes and in showcasing it.

The above snapshot shows that how simple it is to find all RSS and ATOMs for given URL. You can run the findFeeds pipe at http://pipes.yahoo.com/pipes/pipe.info?_id=00ff6bb493d2785b7594eea76e55c988.

The above snapshot is a clone of the first pipe extended to show the post of an RSS Feed for any given URL. You can run the showRelated pipe at http://pipes.yahoo.com/pipes/pipe.info?_id=_iMHbOhG3hGjsadmgQSecQ as well as view its source.

To see how far we could differ in traditional implementations - the final pipes was hosted on Appjet and hence aptly called pipes-on-a-jet ; ) This is how it looks like in the appjet IDE

examples of using pipesOnAJet:

search for kolkata knight riders on trak.in
search for demo on techcrunch

Using Yahoo! Query Language to discover RSS/Atom feeds

YQL (Yahoo! Query Language) is an expressive SQL-like language that lets you query, filter, and join data across Web services.

Running “select * from data where URL=’http://developers.hover.in’” gave the content of the body tag in that page. One feedback we have is that we were not able to find the content of head element tag. We look forward for this feature in upcoming builds. So we tried another query with the help of Open Data Tables (Open Data Tables enable developers to add tables for any data on the Web to our stable of API-specific tables), which returned me the entire link tags in given URL. But this was not sufficient since we had to find the links to all the RSS feeds. So playing around with it we found that a clever hack was to provide a minimal CSS selector and we got this!

Final statement:

use 'http://yqlblog.net/samples/data.html.cssselect.xml' as data.html.cssselect;
select * from data.html.cssselect where url="" and css="link"

You can execute this here:

Using Google AJAX APIs to discover RSS/Atom feeds

Google’s AJAX APIs let you implement rich, dynamic web sites entirely in JavaScript and HTML. You can add a map to your site, a dynamic search box, or download feeds with just a few lines of JavaScript. Unlike most javascript libraries out there - this one focuses more on data and less on the typical UI capabilities.

Google Ajax libraries snippet to discover RSS and show the post

google.load("feeds", "1");

function OnLoad() {
   var query;
   // Query for finding posts related to erlang on the dev blog
   query = 'site:http://developers.hover.in/ erlang';

   // OR Query to find related posts to the hovered word within a hoverlet
   // query = 'site:' + HOVER.site +' '+ HOVER.kw;

   google.feeds.findFeeds(query, findDone);
}

function findDone(result) {
   //traverse and print out
}

google.setOnLoadCallback(OnLoad);

You can run find the above code live here, and has been used for a related posts from your blog hoverlet and is being used on sites like trak.in to show related posts to Kolkata Knight Riders , and basically any word that the hover.in user specifies in his dashboard.

And finally…..here’s the result of a couple of days of hacking around with perl, php, erlang+yaws, cgi, y!pipes, y!ql, appjet, google apis . To top it all– I think it’s safe to say that it took longer to edit this post though! ; )

You can look forward to more posts that deal with the web apps, API’s and hosting environments – apart from the official HOVER API documentation that will be announced soon that will enable you to build your own contextual applications. Signup, get in touch with us for more or follow hover.in on twitter.

~
for hover.in

Kanchan, Ravi, Zeeshan
( o8-o9 hover.in developer interns from Symbiosis, Pune)

hover.in presenting at Erlang Factory London 2009

Bosky — Tue, 21 Apr 2009 10:21:51 +0000

We are extremely thrilled to announce that hover.in will be presenting at Erlang Factory’s London Conference to be held between June 25-26th 2009.

List of speakers
http://www.erlang-factory.com/conference/SFBayAreaErlangFactory2009/speakers

List of talks
http://www.erlang-factory.com/conference/London2009/talks

Abstract of the talk : Erlang at hover.in

From our experiences at ‘hover.in‘ I’d like to present how we first chose erlang late 2007, and got about using it as our bridge across our multi-node cluster. In particular the architectural decisions that went into making our distributed python crawler backend running off mnesia with its sharding & fragmentation strategies for tables that span several millions of rows, load-balancing to our 3-node yaws web servers, tweaks to serve more requests, trade-offs in efficiency vs cost, experiments in DHT’s, our cache worker implementations & messaging queues, cron’s & dispatching jobs while throwing light on design choices that can fit in distributed and heterogeneous environments. We have also recently built and intend to opensource our own in-memory cache workers, persistent stats & logging system, and in the process of now deploying an A/B testing framework, that we’d love to talk about. The talk will also bring in some interesting metaphors from bacteria and its group dynamics, as well as the human brain in handling concurrency & memory.

link

All in all - we’re very excited to push the realms of both erlang, as well as the functional programming scene in India, and looking forward to share and learn form other erlang legends, developers, enthusiasts, and startups at the Erlang Factory - so do ping me if you are in the neighborhood.

Keep Clicking,
~B

hover.in at devcamp bangalore

Bosky — Sun, 12 Apr 2009 15:23:34 +0000

A quick shoutout to Sidu Ponappa, had a great time at devcamp edition 2 held on Apr 11, in Bangalore.

some of the interesting talks were…

sahi, by Narayan Raman

While the morning at devcamp was by a committer on the sahi test suite talk on some pretty impressive javascript trickery, where he demonstrated how to automate tests to login send mails , then go into the sent mail, and delete it …all automated. considering that it works on one the most beefed up apps on we web right now - gmail , extremely imprssive ( even got to detect , and write into gmail’s rich text editor though some smart DOM traversal. Even works on IE , which is commendable . It works by having our browser point to a proxy server which inserts scripts to simulate various actions. Another thing to notice was that alerts,and other blocking calls were overwritten,and handled differently. One question that still lingered was how they simluated mouse cursors, and how many of it’s event properties would be passed on the event handlers, considering the entire actions were simulated.

function login($username, $password){
    _setValue(_textbox("Email"), $username);
    _setValue(_password("Passwd"), $password);
    _click(_submit("Sign in"));
}

login("sahi.abcde", "tough123");
_click(_spandiv("Compose Mail"));
_setValue(_textarea("to"), ", ");
_setValue(_textbox("subject"), "important subject");
_rteWrite(_rte(0, _near(_textbox("subject"))), "lots of content");
_click(_spandiv("Send[9]"));
_assertExists(_cell("Your message has been sent. View message"));
_click(_link("Sent Mail"));
_assertExists(_spandiv("To: dummy.email"));
_click(_checkbox(0, _near(_spandiv("To: dummy.email"))));
_expectConfirm("You are about to move the entire conversation to the Trash.
  Are you sure you want to trash the entire conversation containing your sent message?", true)
_click(_spandiv("Delete[14]"));
_assertExists(_cell("The conversation has been moved to the Trash. Learn more Undo"));
_assertExists(_cell("No sent messages! Send one now!"));
_click(_link("Sign out"));

Visual metrics for code, by neil ford

A thoughtworker showcasing the various evolution of visual code metrics, most of which are open source and fun projects that give your projects/codebase a completely different dimention ,literally speaking. one of the most impressive was the codecity view

a codecity example of modules as districts, buildings as functions wrt LOC,function calls,etc

Software for a concurrent world, by chromewatir dev’s

The afternoon saw plenty of talks on concurrency, including one by couple of developers of chromewatir ( who forgot to introduce themselves btw!, they’re from thoughtworks as well ) on software transactions, locking mechanisms, and compared the atomicty of an ecommerce payment implemented in java, haskell as well as erlang. very impressive.
UPDATE: Sai is the ChromeWatir guy, and Hari works on Rapa (ActiveResource for Java)

CouchDB, by Anand Chitipothu

I was fortunate to bump into Anand Chitipothu, early dev at Infogami & now the Chief Web Programmer at Internet Archive . He’s had the pleasure to work Aaron Shwatrz (of RSS, reddit,infogami,webkit,ycombinator, and openlibrary fame…phew!) . So i got down talking with Anand over lunch, finding out how he got to meet Aaron, (started when he starting contributing to web.py ) , moving over to his involvement with infogami, and now the IA,which gets over 2 million hits a day, talked about erlang ,python a while, and his experiments with couchdb. His talk debunked some lingering questions i still had wrt this REST- based db. There were lot of relevent questions from couple of mysql developers. Backing up a document is as easy as copying it to another system/locations, and the document can now be accessed from a couchdb instance on that system/location. Nice for debugging between home & office he says. View javascript functions are used to create view index, you can also create another reduce view . there are no edits, only appends to file ( or recreate it yourself) . you need to handle conflict resoltuion yourself, views are recreated lazily during a recent read .

He also mentioned that it for 30 million records, it took:

nearly 3-4 hours for inserting 30million entries
1 day to create the view index

This is slightly starting. Also, any slight change in the view requires that all documents are re-updated. he also talked about why he’s skeptical about the concept of having presentation data inside couchdb as well. (the concept of an entire blog running off just couchdb ). So if it’s just quickly setting up some key ,value pairs - couchdb is for you. He also talked about when his site db/backups went kapoot - his replacement until things were brought back into control - was couchdb instance of key,value pairs and its various views accessed from python bindings.

couchdb internals

Other interesting talks

ruby for the curious hacker, by Sidu thoughtworker who i first met as the co-founder of inactive - the micro-messaging twitterish utility for mobile phones alone
on open source licences by the folks at twinglyfoundation
augmented reality demos, by uber flash/flex guru Mrinal Wadhwa where he coded up those now popular “animating some thing on a piece of paper,taking input from a web cam”. really cool!

erlang at hover.in

and finally …..[drum roll ] ; ) , here’s my talk ( hosted on slideshare ) on erlang at hover.in, and to make things interesting i brought in context about how bacteria, and the brain handle concurrency and memory issues.

hover.in founded in late 2007
the web ~ 10-20 years
humans been around for hundreds of thousands of years…
but bacteria…. been around for millions of years, so this talk will talk about what we can learn from bacteria, the brain , and memory in a concurrent world.

erlang at hover.in , Devcamp Blr 09

View more presentations from bosky101.

All in all, a great unconference, and looking forward to meet and work with more like-minded hackers in the near future.

Keep Clicking,
~B

YUI’sing Tsung to choose your CDN

Bosky — Tue, 31 Mar 2009 07:34:18 +0000

An introduction to Tsung

Tsung is an open-source multi-protocol distributed load testing tool, written in …you guessed it erlang, by the folks at process-one ( they also contribute heavily to multiple erlang projects, and infact maintain ejabberd, etc ).

I built Tsung 1.3.0 from source on erlang R13A ( built from source, catch the earlier post on first impressions ) on an AMD 64-bit , 4 GB Ram machine running ubuntu. Beebole has a nice intro on how to setup Tsung. The only key hints I have - you can avoid tsung recorder , and straight away copy sample tsung config files from /usr/local/share/tsung/examples/http-simple.xml to $HOME/.tsung/tsung.xml , and edit/tweak them. Once set , running tsung start, should show the log directory that it references.

We needed to estimate and gauge how much load the quad-code servers at hover.in could take . To start off I ‘m simulating static file requests, and you can have a lot of control over what kind of requests you can to simulate .So I could have different set of user visit frequencies - like one set of requests at x frequency from M browser asking for url A, while another set of faster frequency y , form I browser asking for url B.

While it may take some time to figure out where you need to tweak you app/server ,since there are several areas that your LYME app can be optimized :

kernel-polling , and other erlang VM optimizations
size of log files , whether gzip is enabled in yaws.conf (through deflate = true )
yaws with & without gzip ( deflate flag in yaws.conf against each server tag)
max process’s set for the VM
mnesia specific - the dump_log_time_threshold, table loading timeouts, etc
yaws even has some other flags that you can insert into yaws.conf such as max_num_cached_files , max_size_cached_file, etc

So…. let’s go server some static files ! And what better way to test your load testing tool than to test it against various different content delivery networks out there. Now it’s a pity that I’m only running this on one node - will definitely work on using Tsung on a cluster soon.

Load-testing the YUI library across various networks

I decided to simulate that for a period of 10 minutes, a request would be made to a static file every 0.1.seconds!

You can even give a combination of different sessions, of varying probability ( their summation being 100% ). But since i wanted to test only the single yahoo minified javascript file, a added a single session of 100% probability.

The experiment

Our favourite javascript library has been the YUI hands down for the foresight, documentation and great developer community for the past many years. So we tried serving the same YUI javascript base file from 5 different networks, test their response times, effects of of hundreds of simultaneous users , and then to analyze and compare their performances, pro’s and con’s of :

Yaws server 1.77 ,running on erlang 12-B
Amazon S3
Amazon CloudFront
Yahoo’s YUI hosting
Google’s javascript library hosting ( ~little difference since google hosts only from 2.6.0 while the rest were 2.4.1 )

Results / Main statistics

The statistics that comes with tsung coupled with perl graphing magic, gives you some comprehensive statistics. In order not to be biased , I even took two trials of load -testing the yahoo yui library against four networks / or content delivery networks if you will, which we at hover have tested at some point. While firebug definitely gives you the end-user perspective , time to load, etc - you can have a look at the latency and network effects , and couple of stat’s that i’ve compiled ( tsung gives a much more detailed view btw) , and make your own judgements for yourself.

network	highest 10sec mean	lowest 10sec mean	Highest Rate	Mean
hover yaws 1	9.167 sec	0.642 sec	11.3 / sec	0.975 sec
hover yaws 2	1.093 sec	0.170 sec	8 / sec	0.634 sec
S3 1	5.329 sec	0.285 sec	10.8 / sec	0.750 sec
S3 2	1.878 sec	0.256 sec	11.2 / sec	0.478 sec
Cloudfront 1	4.121 sec	0.259 sec	11.8 / sec	0.693 sec
Cloudfront 2	4.420 sec	0.256 sec	11.2 / sec	0.478 sec
Yahoo 1	45.5 sec	73.67 msec	11.2 / sec	0.738 sec
Yahoo 2	29.9 sec	78.71 msec	16.1 / sec	2.657 sec
Google 1	2.188 sec	0.178 sec	14.6 / sec	1.033 sec
Google 2	4.420 sec	0.170 sec	15.1 / sec	0.808 sec

Simultaneous Users per second

hover yaws

Amd 64-bit quad core machine ,4GB RAM ( extra un-used node that we’re removing as part of our see-how-much-you-can-get-out-of-one-machine scaling attempts ). We felt that for site-specific js/static files ,etc - we need’nt look to a CDN. The only thing you need to tweak around, ( and we seem to be getting loads of errors on yaws is that of possibly insufficinet sockets , when the load moves beyong 2-3 requests per second from our experiences. It infact increased when enabling gzip, but there may be other reasons. Anyhow tsung should help crash test our local dev servers long enough , and slow enough for us to figure out where the bottlenecks really are.

hover yaws 1

hover yaws 2
Amazon s3

Reading through the dynamo paper gives some nice insights into their architecture, and we never had any issues with them. Cheap & effective . Only qualm is that they don’t gzip files ( but I hear that you can push gzipped content into S3 through few hacks )

S3 1

S3 2
Amazon CloudFront

Our move to amazon cloudfront has had drastically good results with static file serving. One reason I might quit CloudFront though is that the files when pushed to S3, seem to be inconsistent across data centers. Although the lame way would be to keep giving new filenames, ( which by the way deleting from S3, doesnt delete from cloudfront) Simply not acceptable. ( version 1 from asia, version 2 from the US ,etc for the same file at the same time ) . We’ve beein doing upwards of 25 million GET requests from Cloudfront last month. Overall, Cloudfront steals the thunder - since it’s most likely the only edge network serving the static file among the other networks.

Cloudfront 1

CloudFront 2
yahoo’s free static content hosting

They’ve been offering free hosting for something for various releases, and the free hosting is definitely worth it - be it for quick prototyping, or spanking new web apps.

yahoo apis 1

yahoo 2
google free static content hosting

YUI was added recently to part of Google’s ajax api’s iniatives. The javascript needs to be lazy loaded using javascript though. Doubt that they’re edge networks.

googleapis.com 1

google 2

And another compilation of number of HTTP status responses per second , again two trials per network.

HTTP Status Response Times

hover yaws 1	11 / sec
hover yaws 2	8.3 / sec
S3 1	12.2 / sec
S3 1	10.7 / sec
Cloudfront 1	8.4 / sec
Cloudfront 1	9.1 / sec
Yahoo 1	11.1 / sec
Yahoo 1	12.6 / sec
Google 1	10.9 / sec
Google 1	14.1 / sec

Conclusion

What’s interesting is that the figures indicate that the Amazon servers score less on handling concurrent users , and more on combating latency. But that’s quite understandable considering that every S3 object needs to be fetched from atleast 2 nodes while reading , and a conflict resolution is done during the reads, not the writes. ( they try to avoid neglecting writes from any subsystem at any cost ). But the increased response times from google , yahoo and not to forget - our very own hover server running on yet another web server - could be justified by the lesser number of requests being served by the respective CDN’s when compared to Amazon. What also interesting to see was that the yahoo api servers seem to consistently give higher throughput from their YTS/1.17.8 server.

That said, static files are just half the story, I’d love to benchmark more libraries when we get the time, but next up might be benchmarking images , css , and more dynamic content - possibly a comparison of distributed hash stores, cache’s and the like.

Somethings to rejoice about

Bosky — Fri, 27 Mar 2009 02:48:25 +0000

We’re really excited to check out the new erlang R13A release , here’s a summary of what caught our attention.

Highlight #1

OTP-7500  The runtime system with SMP support now uses multiple,
	      scheduler specific run queues, instead of one globally shared
	      run queue.

Not inducing concurrency into your programs can hinder the effectiveness and optimal utilization of your machine . That said - just because erlang can spawn a million process’s doesn’t quite warrant making every call concurrent. Most experiments don’t take into consideration that the same system may well be yet another web server, building your own crons using flowcontrol through message queues , loading data into or out of the database - at the same instant of time when your benchmarking algorithm wants you to start a multi-node concurrent version of every function that you come across. Infact it’s quite a misnomer. Sure you can spawn a mighty - concurrent sorting algorithm across 100 nodes. But what if each of those 100 nodes tried to do the same thing. Would it be more efficient to spawn more process’s on the same node versus RPC‘ing and handling latencies from different nodes. Now I have nothing against location transparency, and mnesia’ ability to decide which tables need to be on RAM, completely transparent from any node, or only accessible form a third node - that granularity is from another planet. Same goes with gen_server’s being in memory, being able to configure to use trees, vs lists vs queue’s vs sets. it rocks. period. But here’s where improvements in SMP and infact the very emphasis of your code relying on being able to add jobs to nodes, but internally making sure that X node run’s process’s at 2X courtesy of concurrency helps, as oppose to spawning process’s for every element in the list on every node in the cluster. It’s simply not controlled enough , and sometimes good old serially running jobs , but making sure they run faster - makes a hell of a difference. Well thats what we’ve been trying to do, and has held us in amazing stead . Something like replacing every lists:map with a parallel version is left to when you have way too much money to spend buying extra nodes that you can replicate your data on. With that in mind that said, even with the R13A we intend to continue mnesia reads even being made synchronous for things like crons, while the handling of that data immediately after - that massages the data into whatever form it wants - is encourages to be made more concurrent. This works out well when we want to compare datasets stored in DB. One way we’ve tried to optimize the cores is to go through our code to see if We can typically avoid unnecessary list fold’s , in favour of list map’s which are much more concurrent friendly since the input to each map function doesn’t depend on the previous or next values ( as opposed to foldl / foldr ). But wouldn’t it be nice to know when the last map function ( by making a lists:map concurrent , each concurrent process may take different time and complete in any order, which sort of resonates with what i meant by asynchronous.

I’ve compared some techniques from many pmap implementations out there, and here are some notes…

While it’s common practice to send the origin pid or to make a reference to the callee pid, most could easily reuse a variable .

lists:map fun(Msg)->
                  Pid ! { self() ,Msg }
                end , Data ).

%% Neater and should be faster , since reducing recomputing self N times
Self = self(),
lists:map fun(Msg)->
                  Pid ! { Self ,Msg }
                end , Data ).

Now an interesting part wrt receives is where it is placed, and there are a fair share of different ways of placing it, while the most commonly used method is :

Pids = lists:map fun(Msg)->
                  Pid ! { Self ,Msg }
                end , Data ),
lists:map ( fun(Pid) ->
                     receive
                            Pid -> ok
                     end
                 end, Pids).

Pids = lists:map fun(Msg)->
                  Pid ! { Self ,Msg }
                end , Data ),
[ receive   Pid -> ok  end ||  Pid <-  Pids].

I’ve always been content on avoiding the extra lists:map, and receiving results back form the concurrent process’s - within the same scope / lists:map

lists:map fun(Msg)->
                  Pid ! { Self ,Msg },
                  receive
                     Pid -> ok
                 end
               end, List ) .

And to testing the above on R12B on a 32bit, R12B on a 64bit & the spanking new R13A on a 64-bit node, I asked hacked up a simple module that took a list of elements, and attempted :

UPDATE To reiterate what the module is for …“How would you get a counter of how many times some success() happened for each element of a list , and also store the results of each as well . Typically this would be a lists:foldl . Could there be someway faster than a lists:fold ( approach serial_something1) ?”

So, my explorations were that if you were to use a lists:map where one spawned process is used to maintain state, I found that results were much better. ( approach something1, something2 ) .

something1 concurrently get the number of attempts and status of each attempt, using receive inside the first map
something2 concurrently get the number of attempts and status of each attempt, using receive in a separate lists:map
serial_something normal lists:map that synchronously get the number of attempts and status of each attempt ( no concurrency )

I then attempted to pass in elements of various sizes , 10, 100, 1000, and 10000.

erl , on R12B-1 on a 32bit node

Erlang (BEAM) emulator version 5.6.1 [source] [smp:2] [async-threads:0] [kernel-poll:false]
Eshell V5.6.1  (abort with ^G)
1> something:multi_bench().
[{[{161,something1},      {29,something2},        {7,serial_something}]},      %% 10    elements
 {[{543,something1},     {347,something2},       {80,serial_something}]},      %% 100   elements
 {[{5672,something1},  {12840,something2},     {5227,serial_something}]},      %% 1000  elements
 {[{50898,something1}, {64534,something2},   {617422,serial_something}]}]      %% 10000 elements

erl , on R12B on a 64bit node

Erlang (BEAM) emulator version 5.6.1 [source] [64-bit] [smp:4] [async-threads:0] [hipe] [kernel-poll:false]
Eshell V5.6.1  (abort with ^G)
1> something:multi_bench_something().
[{[{40,something1},       {15,something2},         {5,serial_something}]},     %% 10    elements
 {[{260,something1},     {243,something2},        {83,serial_something}]},     %% 100   elements
 {[{2633,something1},   {2600,something2},      {5794,serial_something}]},     %% 1000  elements
 {[{23505,something1},  28484,something2},    {622124,serial_something}]}]     %% 10000 elements

erl , on R13A on a 64bit node

Erlang R13A (erts-5.7) [smp:4:4][rq:4] [async-threads:0]
Eshell V5.7  (abort with ^G)
1> something:multi_bench().
[{[{1,something1},           {1,something2},         {1,serial_something}]},   %%  10     elements
 {[{1,something1},           {1,something2},         {1,serial_something}]},   %%  100    elements
 {[{1,something1},           {1,something2},         {1,serial_something}]},   %%  1000   elements
 {[{15000,something1},   {30999,something2},    {531999,serial_something}]}]   %%  10000  elements
 {[{234000,something1}, {265000,something2},  {68109999,serial_something}]}]   %%  100000 elements [BONUS]

What’s surprising is that with R13A, the first approach of avoiding an extra lists:map fn for receive’s seems to working to consistently work out better across all platforms for large datasets. Here could be some key takeaways:

if the list is < 1000 elements then it’s better to infact use another lists:map ( doesn’t make a difference on R13A though )
if the list > 10000 elements, expecting the receive in another lists:map could turn out twice as slower ( across all releases )

The code used to run the tests , including the two different approaches to the receives is something.erl on github.

Highlight #2

 OTP-7648  Support for Unicode is implemented as described in EEP10.
	      Formatting and reading of unicode data both from terminals
	      and files is supported by the io and io_lib modules. Files
	      can be opened in modes with automatic translation to and from
	      different unicode formats. The module 'unicode' contains
	      functions for conversion between external and internal
	      unicode formats and the re module has support for unicode
	      data. There is also language syntax for specifying string and
	      character data beyond the ISO-latin-1 range.

	      The interactive shell will support input and output of
	      unicode characters when the terminal and operating system
	      supports it.

And with the awesomeness of binary pattern matching, there’s nothing stopping us now . That said we’ve already had a good run with unicode ( despite the myths ) . Our prayers were answered through a common glue between erlang,python as well as erlang. It’s dreadfully simple, yet a life-saver.

%% how we handled unicode pretty printing prior to R13A ( 3 cheers for Base64 encoding ! )
out(Arg)->
          B64 = base64:encode( SomeBinaryKeyword () ),
	  Kw = {script, [],  [
                           "document.write(Base64.decode('",
                             B64,
                            "'));"
                     ]},
          {ehtml, Kw }.

Since Binary can be spitted out by the yaws ehtml structure , there is no need to convert from binary back to string . Wrt unicode, I hope this also means that we will be able to actually work with unicode in the erlang shell, inside code, and so on. Erlang’s nature of storing strings as integers turned out to be a boon for us, because it now does’nt matter if out python crawler crawls english or swahili - it;s all integers stored as binary - which makes it cheaper than strings, and more parser friendly. Having a look at leex/yecc for building unicode compilers might be a hack for another day.

Highlight #3

Message passing has been further optimized for parallel
	      execution. Serial message passing is slightly more expensive
	      than before, but parallel send to a common receiver is much
	      cheaper.

I’ll be looking forward to also comparing gen_server2 which has been doing the rounds in some projects.

Highlight #4

OTP-7804

The BIFs atom_to_binary/2, binary_to_atom/2, and
	      binary_to_existing_atom/2 have been added.

Highlight #5

OTP-7826
Nodes belonging to different independent clusters can now
	      co-exist on the same host with the help of a new environment
	      variable setting ERL_EPMD_PORT.

Naming nodes ( hostname vs fully qualified ip ) tends to get little tricky if you don’t have static IP ’s. Especially when different hostnames try to interconnect. For those who are’nt familiar - creating a distributed cluster is as simply as

net_adm:ping( 'name@ip').
or
net_adm:ping('justhostname') .

and if the nodes set the same cookie ( just a security identifier ) and if all goes well - it should return pong. If there are problems, pang. If the nodes were all connected on the same Lan and sharing the same cookie - there would ‘nt be too many hassles. And until now , I don’t think nodes from different clusters could communicate. I wonder if this means that we could have nodes talking to each other from different lan networks. This is big.

Highlight #6

OTP-7641  When chunk reading a disk log opened in read_only mode, bad
	      terms could crash the disk log process.

Some time back, we noticed that the nodes were crashing, mostly “out of memory” errors which would bring down the VM, sometimes even the node , which isn’t pretty at at all. We followed it by more logging, to no avail. I don’t know how we missed it -but it turned out that the “error_logger” module will registered one and only one process to receive and handle all log messages. ( at least till 1.77 ) had a bottleneck ,so infact logging to find out what was wrong kept forcing more crash’s . Since then we’re simply cut down logging to almost nil - and we haven’t faced the same issue since then. It was quite embarrassing when I did try it out locally. Simply running [ error_logger:info_msg( "writing ", [ X ] ) || X<- lists:seq(1, AHighEnoughNumber) ] brought yaws down. I hope this is fixed either will the latest yaws release or R12B, if that’s what they meant by this fix. Pity we did’nt debug this earlier.

Highlight #7 (mnesia)

    OTP-7753  With bad timing several api functions could return or exit
	      with a bad error message when mnesia was shutting down.

    OTP-7835  mnesia:clear_table/1 cleared all nodes table content even if
	      the table was local_content only type.

We’ve also noticed inconsistencies in R12B, wrt clearing fragments of mnesia tables. What our module which we’ll talk about later - frag_mnesia does is spawn process’s that will address each fragment explicitly and clear it. We’ve got similiar functions for querying disk-only tables as well. Here are some results

%% wrt time taken
an mnesia dirty_match_object  on a fragmented table ( eg: 8 fragments + the base table )
%%is slower than
 spawning as many process's as there are fragments and dirty matching over them
%%is slower than
mnesia reads

When you have data that changes over time, - perhaps it could change from more that one point of entry, or you’re perhaps accepting a whole bunch of tags. Now the smart thing to do for editing is infact making the mnesia table of type set, and simply overwriting. Infact the create & edit records are often the same function at some point . But when you have data that is one is to many - eg: meta tags of a webpage . There maybe multiple meta tags for the same webapge. We decided to have the key as

%Approach1 : Table is type set, Key as tuple , no conflict resolution, but slower
{ Url , Kw }
%Approach2 : Table is type bag, Key is either Kw or Url ,  conflict resolution, but much quicker
Kw

%sample entries for a mnesia set table
{ <<"unitednations.com/">> , <<"india">> } , MetaData, TimeT1
{ <<"unitednations.com/">> , <<"india">> } , NewMetaData, TimeT2  %% overwrites TimeT1 which is great

%sample entries for a mnesia bag table
 <<"unitednations.com/">> , <<"india">>  , MetaData, TimeT1
 <<"unitednations.com/">> , <<"india">>  , NewMetaData, TimeT2  %% two verions of data

For a crawler, recrawls of webpages are something you need to cater to. If the table above was of tpye set, then the record TimeT2 would simply overwrite the first row with new data which is ideal. if the table above was of type bag, then there would have been two rows for the same url and keyword combination. We did’nt want to have to deal with the inconsistent rows , so we kept the table as sets. Which drastically reduced query speeds on large fragmented tables. There are two approaches that we might look at from here on. Keeping another table, where the index infact is the keyword or the other approach could be storing the tables as bags and dealing with conflict resolution of different versions of data. Amazon’s paper on dynamo talks about reading multiple versions of data (infact an S3 object is typically replicated on 3 nodes, reads are only allowed if two nodes respond, writes are always accpted. But they say this ratio changes from webapp to webapp within Amazon itself. But the interesting part was that of the conflict resolution was done during the reads. not during the writes. infact vector clocks are used to merge two different versions of data.

Another design decision we took early was how we sharded the data. I guess we broke every rule in the book, when we decided that every user would have their own set of tables. Wont there be too many tables ? Yes, and that many lesser table locks to deal with , letting us have more workers accessing multiple tables that have absolutely no transaction locks between them or drastically reduce the number of process’s trying to access the same data. Tell me more ! Backing up is easier, having user specific rule are easier, analysing and awarding more workers on the fly to high-priority publishers becomes a charm ( much like spawning ec2 instances i assume ) . Wait did you say you stored crawler data in mnesia ? That’’s right , sharded from the word go, disc-only fragmented from the word go ( allowing us to spawn process’s to even search each fragment which turns out to be several magnitude times faster that non-concurrent match’s. What’ll be interesting is how far we can go with mnesia. But how do you get in so much data so fast ? It is’nt lighting speed, but it takes around 4 seconds for everything from crawling/creating a whole bunch of index’s , and loading into mnesia from heat seeking ( will crawl based on popularity of pages ) hybridqueue . We also have another singleton crawler that’s more of a traditional techniques. User specific Metadata is often replicated across all nodes , and are disc_copies when are required in most calls. Stat tables are typically disc-only tables, and we’re currently have awstats configured over yaws logs to spruce us some http analytics juice. We’re also working on an rrdstats implementation that uses a gen_server to fill in data within the step frequency , still need to spend some more time before it’s production ready. And the rest of the table are mainly disc_copies or in RAM? Where does that leave us ? Denormalization - works great you need to render amazing amounts of data to some of India’s biggest sites in under 200-300 ms .Loads of inverted index’s being generated through extremely well-controlled and predictive flowcontrols. But we’re still exploring our own custom cache worker’s that are typically ring buffers of fixed size , but we’d love to spruce things up with perhaps squid servers, memcached , merle , or a distributed key-value store ?

Highlight #8 ( stdlib )

    OTP-7230  The functions lists:seq/1,2 return the empty list in a few
	      cases when they used to generate an exception, for example
	      lists:seq(1, 0). See lists(3) for details. (Thanks to Richard
	      O'Keefe.)

    OTP-7740  When a process started with proc_lib, gen_server, or gen_fsm
	      exits with reason {shutdown,Term}, a crash report will no
	      longer be generated (to allow a clean shutdown, but still
	      provide additional information to process that are linked to
	      the terminating process).

All in all, it’s interesting times at hover.in with R13A coming up, and the buzz is definitely in the air wrt erlang. Too bad I could’nt make the CFP for the erlang factory’s Bay Area 2009 conference, it looks like it’s going to be the biggest and most widely anticipated erlang conference yet! Can’t wait for the videos of the talks to roll out. There are also few talks by companies like Facebook, and Mochiweb. and ofcourse the amazing guys at ProcessOne.

And there are some more things to rejoice about at hover.in too :

Peaking at 50,000 hovers a single day, where a hover is everytime a viewer moves their cursor over a word that the publisher / blogger chooses to see content . We’re now happy to see around 15,000 - 20,000 hovers a day from the few publishers that have signed up with us.
Interesting our scaling experiences with erlang & yaws has been …..reducing one node from our 4 node cluster : ) We ‘re more excited about pushing how much we can do with how less. ( 2-3 quad- core machines serving up as production servers ) with anywhere between 3 and 30 requests per second, we’re just touched quarter of a million hovers in under 2 months, with zilch marketing or PR, and established large-size Indian portals ( Sify Finance , Oneindia and more to be announced )
full support for unicode languages, which means we’re the first intext company to hoverize regional languages. Go have a look. http://thatstamil.oneindia.in . A big shoutout to the folks at OneIndia for making it happen, and believing in the potential of the product. We however are looking for an NLP hacker to help us move especially in parts of speech recognition, classification and clustering for not just english, but pioneer with regional languages as well.
Building the next generation of contextual , and in-text engaging content , be it a crunchbase hoverlet , a map hoverlet, music from last.fm or videos from youtube, a deck from slideshare, graphs from compete, conversations from twitter, and virtually any widget or 3rd party gadget out there. We typically work on erlang, python , javascript ( based on the excellent YUI ), and will be launching our facebook / wordpress / firefox plugins soon to be launched.
the 26th alliance, was announced in our typical low-key affair, on January 26th and attempts to bring together a growing set of Indian consumer- web companies, trying to get together on common ground to pursue a singular cause - to develop and distribute widgets , and help expose and utilize upcoming API’s .

It’s been a fantastic ride through the past 18months , and hopefully we’ll be able to keep in touch within the eco-system of bloggers, content-providers and developers in the near future. take a look at the product demo’s on the the main hover.in blog, keep a track of hover.in on twiitter .

Thank’s for dropping in, and a big shoutout to everyone behind erlang, everyone at #erlang and github and the blogosphere promoting erlang - on behalf of our team of 5 developers, including myself : )
~
Keep Clicking,
Bhasker V Kode , aka bosky
Co-Founder & CTO

developers.hover.in

Bosky — Mon, 02 Feb 2009 04:58:22 +0000

This is the developer blog at hover.in , where posts are contributed by the developers (and wannabe-cartoonists until we get another blog) . For the most part - this group blog will touch upon the following areas :

erlang, python , system utils , and other stuff mostly on the LYME stack ( linux, yaws, mnesia, erlang )
javascript, user interfaces , css , graphics discussions
theoretical computer science, caching, CDN’s ,and scaling experiences
tech announcements updates on features, open-source work, API talk, and even hiring

Do get in touch with us , follow us on twitter,or shoot in an email to contact at hover dot in.

~
FTW
Team hover.in

developers.hover.in

hoverlets and a case study : Zemanta API

What is a hoverlet?

Creation of a hoverlet

Using a hoverlet

Case Study : A Zemanta Hoverlet

intern challenge - 1

Problem:

Steps:

Solutions:

Using PHP to discover RSS/Atom feeds

Using Perl to discover RSS/Atom feeds

Using Erlang to discover RSS/Atom feeds

Using Yahoo Pipes to auto discover RSS/Atom feeds

Using Yahoo! Query Language to discover RSS/Atom feeds

Using Google AJAX APIs to discover RSS/Atom feeds

hover.in presenting at Erlang Factory London 2009

hover.in at devcamp bangalore

sahi, by Narayan Raman

Visual metrics for code, by neil ford

Software for a concurrent world, by chromewatir dev’s

CouchDB, by Anand Chitipothu

Other interesting talks

erlang at hover.in

YUI’sing Tsung to choose your CDN

An introduction to Tsung

Load-testing the YUI library across various networks

The experiment

Results / Main statistics

Simultaneous Users per second

hover yaws

Amazon s3

Amazon CloudFront

yahoo’s free static content hosting

google free static content hosting

HTTP Status Response Times

Conclusion

Somethings to rejoice about

Highlight #1

Highlight #2

Highlight #3

Highlight #4

Highlight #5

Highlight #6

Highlight #7 (mnesia)

Highlight #8 ( stdlib )

developers.hover.in