NasheBlog.com

Some of the books I am reading

noreply@blogger.com (Gideon) — Fri, 21 Oct 2011 23:37:00 +0000

Infinite Reality: Avatars, Eternal Life, New Worlds, and the Dawn of the Virtual Revolution

Amazon.com description

How do The Matrix, Avatar, and Tron reveal the future of existence? can our brains recognize where "reality" ends and "virtual" begins? What would it mean to live eternally in a digital universe? Where will technology lead us in five, fifty, and five hundred years? Two innovative scientists explore the mystery and reality of the virtual and examine the profound potential of emerging digital technologies Welcome to the future . . . The coming explosion of immersive digital technology, combined with recent progress in unlocking how the mind works, will soon revolutionize our lives in ways only science fiction has imagined. In Infinite Reality, Jeremy Bailenson (Stanford University) and Jim Blascovich (University of California, Santa Barbara)—two of virtual reality's pioneering authorities whose pathbreaking research has mapped how our brain behaves in digital worlds—take us on a mind-bending journey through the virtual universe. Infinite Reality explores what emerging computer technologies and their radical applications will mean for the future of human life and society. Along the way, Bailenson and Blascovich examine the timeless philosophical questions of the self and "reality" that arise through the digital experience; explain how virtual reality's latest and future forms—including immersive video games and social-networking sites—will soon be seamlessly integrated into our lives; show the many surprising practical applications of virtual reality, from education and medicine to sex and warfare; and probe further-off possibilities like "total personality downloads" that would allow your great-great-grandchildren to have a conversation with "you" a century or more after your death. Equally fascinating, farsighted, and profound, Infinite Reality is an essential guide to our virtual future, where the experience of being human will be deeply transformed.

How the Mind Works

Amazon.com Review

Why do fools fall in love? Why does a man's annual salary, on average, increase $600 with each inch of his height? When a crack dealer guns down a rival, how is he just like Alexander Hamilton, whose face is on the ten-dollar bill? How do optical illusions function as windows on the human soul? Cheerful, cheeky, occasionally outrageous MIT psychologist Steven Pinker answers all of the above and more in his marvelously fun, awesomely informative survey of modern brain science. Pinker argues that Darwin plus canny computer programs are the key to understanding ourselves--but he also throws in apt references to Star Trek, Star Wars, The Far Side, history, literature, W. C. Fields, Mozart, Marilyn Monroe, surrealism, experimental psychology, and Moulay Ismail the Bloodthirsty and his 888 children. If How the Mind Works were a rock show, tickets would be scalped for $100. This book deserved its spot as Number One on bestseller lists. It belongs on a short shelf alongside such classics as Darwin's Dangerous Idea: Evolution and the Meanings of Life, by Daniel C. Dennett, and The Moral Animal: Why We Are the Way We Are: The New Science of Evolutionary Psychology, by Robert Wright. Pinker's startling ideas pop out as dramatically as those hidden pictures in a Magic Eye 3D stereogram poster, which he also explains in brilliantly lucid prose. --This text refers to an out of print or unavailable edition of this title.

Taking the Amazon Cloud Drive for a spin

noreply@blogger.com (Gideon) — Tue, 29 Mar 2011 17:32:00 +0000

I set out wanting to write about the Amazon.com Cloud drive. This is mostly because a while back I had a discussion with a friend about a service like this (though the web player component is something we did not talk about back then.) I remember this discussion came up again when Yahoo! closed their file service.

I have signed up for the Cloud drive service and have uploaded all my music which surprisingly did not fill up the 5GB drive quota. As I write, I am listening to some songs I uploaded from my desktop at home.

Gigaom has a write up on one of the writer's experience using the product. I had very similar opinions about the service (minus the Android experience because I do not have an Android device).

A Rundown of Server Virtualization Technology

noreply@blogger.com (Gideon) — Fri, 11 Mar 2011 20:07:00 +0000

Computers have come a long way over the years, but this isn’t always so obvious when it comes to servers. Servers are computers that host information for other computers on a network. They have a lot of processing potential. A lot of power. And yet most physical servers–also called hosts–fail to tap into that power. That’s because each physical server needs its own operating system and is usually assigned a single task. The server’s processing potential is wasted. But there is a solution to this: server virtualization technology.

Read the rest of the article at blowide.com

The Hadoop Ecosystem

noreply@blogger.com (Gideon) — Mon, 04 Oct 2010 03:21:00 +0000

One of the problems with describing Hadoop (which I am sure is true with any young technology) is that it is not yet clear what the components are from a business point of view. As someone who adopted this new technology early on, I am also susceptible to the many mischaracterizations of some of the business aspects. I never really looked into the many business aspects of Hadoop mostly because my interest was mostly from a scientific point of view, especially how the technology made it very easy to process massive data sets.

Here is a diagram that attempts to categorize the many aspects of Hadoop, courtesy of gigaom.com:

All three layers are necessary to handle the unprecedented volumes of data that need to be turned into meaningful results. The results are extremely varied, whether looking for a friend, getting a movie, book or professional colleague recommendation, understanding the spread or triggers of disease, detecting fraud, comprehending the behavior and buying pattern of a customer; the opportunity and competitive advantage inherent in burgeoning data is vast. The Big Data solution ecosystem already handles these and many other analytics today.

For the ecosystem to function, it first needs to address the different layers of Hadoop. As a collection of Apache Foundation projects, Hadoop includes the core MapReduce and distributed file system projects. It also includes Chukwa, HBase, Hive, Pig and ZooKeeper. Such a mesh of projects forms the essential cylinders of Hadoop-related power. To ultimately deliver value to today’s enterprise, these projects each need to work well within private data center or public cloud infrastructures (or both simultaneously). Cooperation with existing data repositories is key, and tools are required that turn collections of open-source projects into highly valuable, prime-time manipulators of vast amounts of data.

You should read the rest of the article at gigaom as it does a very good job at explaining the different layers of Hadoop from a business vantage point.

Hadoop: The Definitive Guide
Algorithms of the Intelligent Web
Cassandra: The Definitive Guide

Cloud Computing: Where really does your data get stored?

noreply@blogger.com (Gideon) — Fri, 16 Jul 2010 00:29:00 +0000

Here is an interesting article from techradar about the data centers that enable cloud computing. Some of these places are very interesting.

I would love to work for the Swedish ISP, Bahnhof's which converted a nuclear bunker into a data center which looks almost like you are on a scifi movie set.

From techradar:

Cloud computing doesn't really mean storing your data in a cloud: it means your stuff's been stuck on a server in an enormous temperature-controlled room somewhere. That doesn't mean it has to be in a boring building, though: it could be down a mine, in something that looks like Dr Evil's control centre or even off the coast of Suffolk.

The article is here.

Login failed for user NT AUTHORITY\ANONYMOUS LOGON

noreply@blogger.com (Gideon) — Fri, 21 May 2010 17:13:00 +0000

Last night a friend of mine called me with an issue on a project he was working on. He had setup a webserver using XAMPP (mostly using Apache and Filezilla) and his website was connecting to a SQL Server 2005 database hosted on another machine in the same domain. He used a Data Source Name connection to the SQL Server database and in the PHP connection he would establish an ODBC:

$conn = odbc_connect($hostname,'','');

The username and password field would be empty because the DSN was set up to use Windows Authentication. Now this same kind of setup was working on his old webserver and it was while migrating to a new server that this problem came up. He was getting the error:

Login failed for user 'NT AUTHORITY\ANONYMOUS LOGON'

I was not too sure myself what was going on and a little digging up showed that this was a common problem although most of this was for users in an IIS environment. Eventually we found a solution and I cannot remember which forum the hint came from.

The problem with his setup was that he has set his XAMPP (Apache) to start as a service. However the service was set to log in as a local system account. Changing this to use a particular user in the domain fixed the problem. You do this by going to the Apache service, get to the Log on tab and choose This Account and specify a valid domain account.

Hopefully this will help someone looking for a solution.

Hadoop Interview Questions

noreply@blogger.com (Gideon) — Fri, 30 Apr 2010 20:17:00 +0000

Unlike many other technologies, there is a lack of Hadoop related interview help. Unless I am looking in the wrong places.

Here is a small list of Hadoop Interview Questions.

How much are you worth?

noreply@blogger.com (Gideon) — Mon, 12 Apr 2010 18:36:00 +0000

How many years does it take to build a billion dollar brick and mortar business? In other words, a business that is not classified as a Tech company. Several years? Several decades? On the other hand, it only takes a few years from having an idea on the drawing board to a billion dollar Tech company. To be clear, I am referring to user based companies such as Facebook, MySpace, hi5 (anyone remember that one!). It is no secret that the number of users plays a pivotal role in the valuation of a company.

I am not a business person but I think that users add value to a Tech company such as Facebook because of revenue generating activities such as advertising. In other words, other companies are willing to pay Facebook a lot of money to be able to get their message across to an audience that Facebook owns. However, advertising on Facebook does not always allow you to drill down to your target demographic and often the ads are impersonal. This brings me to the larger point: Some users on Facebook are more valuable than others as viewed by third parties for advertising purposes. Networks tend to be composed of like-minded individuals and finding the "more valuable" users in these networks can help an advertiser or company reach out to a more targeted demographic with a very personal message.

As it turns out, employers are interested in knowing how much influence you have on your personal network. It brings a new perspective to "It's not what you know BUT whom you know." For an employer (with suitable business models), it makes much more sense to employ the person who is mostly likely to be able to tap into their social network for the benefit of the company. So the people (and number of people) in your network are now as much of an asset to you as it is to Facebook.

So if you are worried or guilty about the amount of time you are putting into social networks, viewing it as adding value to your personal worth can help you lessen your worries!

Is the answer to Google Docs here?

noreply@blogger.com (Gideon) — Wed, 31 Mar 2010 14:30:00 +0000

I have one of those friends who always keeps up to speed to what is happening in the technology world. There is nothing wrong with that except that he ends up preaching to anyone who would listen. I have always told him that if I was going to make it big, I would employ him as a technology evangelist in a heartbeat. He is always quick to point out that he will be employing me instead not for what I know but just for pity.

Anyway today was one of those days that I received a phone call from him. Apparently he got wind of some rumors that Microsoft is going to release Office 2010 this summer. He tells me that this new version of Office will have a focus on mobile computing and collaboration. That got my attention. Ever since I moved from academia to the corporate world, I have grown frustrated with the restrictions placed on the use of some freely available technologies. Since the organization already has a relationship with Microsoft, this may be the only chance I have to finally take some of the applications and systems to the next level.

Microsoft Office 2010 will include free web-based versions of office products which might pose a credible challenge to Google Docs, which I adore much. Some new tools such as video-editing in PowerPoint and another feature I might readily find useful for my work is the data snapshots in Excel. An even sweeter feature is the ability to store files with SkyDrive cloud service, something that I am looking forward to. You can also share files to Sharepoint which is also another sweet functionality that will make some of my work responsibilities that much easier. Email and blog distribution capabilities is also one of those features that I know a lot of people are going to find very useful.

Before you get overly excited about this, there is a problem if you are in financial red as I am or if you are one of those people who believe software should be free (I am talking to you!) You will have to part with about $500 for the Professional edition for two computers. I probably wont be buying this software anytime soon and I know my company many not upgrade to Office 2010 until much later.

I don't think Microsoft Office 2010 would put a big dent into Google Docs market share especially at that price tag, at least for now. I am not sure if one can use the online version without buying Office first.

Do you think Office 2010 poses a threat to Google Docs?

Apple iPhone Multitasking Problem Solved

noreply@blogger.com (Gideon) — Tue, 30 Mar 2010 17:08:00 +0000

Did the title to this posting catch your eye? I really got worked up after reading a teaser to an article on the Newsweek website. What made me really want to read the article was the person writing the article and the person who was being interviewed. The author is Daniel Lyons, also known as The Fake Steve Jobs. The person being interviewed was Steve Wozniak, Apple co-founder. Noone can really blame me for falling for this. Anyway here is a snippet of the interview on the Newsweek Website that really got me laughing:

What's your favorite phone?
The iPhone, because of the apps. By the way, I solved the problem of battery life and [the lack of] multitasking on the iPhone.

Really?

Yeah. I just have two iPhones, so if the battery runs down on the first one, I can use the other. And if I'm talking on one, I can use the other one to look something up. You would not believe how much use I get out of that.

The rest of the interview can be found here.

Going home

noreply@blogger.com (Gideon) — Sat, 27 Mar 2010 23:37:00 +0000

Today I start a journey back to my roots. Computing roots that is. I was introduced to computers using some old Macintosh computers a long time ago and Macs were the only thing I knew until my high school acquired/received a batch of new PC running on Windows NT and later some running on Windows ME. We were among the only schools that have computers labs in my native country let alone 2 : a Mac and a PC lab.

Today I realized why I fell in love with Macs (and possibly computers in general) with my delivery of a new Mac. I took delivery of a Mac Mini this weekend which I had hesitated to do because I am still officially in the PC world due to the nature of my work. I decided not to go overboard because, well, it was the financially sound thing to do and also because I was not sure how much time I was going to be using it. For now though, that might be much more time than I had in mind. I just love how easy it is and how things just work. It did not help the situation that my HP laptop received an update which wiped away all of my settings. I am not sure whether this was an HP or Microsoft update.

Someone used to say Mac for Productivity, Linux for Development and Windows for Solitaire. Now I can relate to that.

Snow in Texas

noreply@blogger.com (Gideon) — Mon, 22 Mar 2010 01:03:00 +0000

It's usually warm in Texas and we get a 3-5 inch snow people go crazy (read excited). This is the second time we have had this much snow in about a 30 day period.

Netflix API Revisited

noreply@blogger.com (Gideon) — Mon, 15 Mar 2010 20:34:00 +0000

In my last posting, I mentioned my quest to get some movie information from Netflix through the Netflix API. I was a bit surprised that there are very few tools or tutorials out there to help use the API. In my last post, I mentioned that I was having some problems mainly because of my limited knowledge of JSON. Because I was pressed with time, I just decided I would use REST instead of JSON.

I used SimpleXML to parse the data although I initially had problems getting some data that was enclosed in the CDATA tags. After a few Google searches, I managed to get that fixed by changing using:

$xml = simplexml_load_string($buffer,'SimpleXMLElement', LIBXML_NOCDATA);

So now I have a better working solution without any need for screen scrapping. I forgot to mention in my first post that I am using OAuthSimple (http://github.com/jrconlin/oauthsimple/).

Netflix API

noreply@blogger.com (Gideon) — Wed, 10 Mar 2010 15:31:00 +0000

For sometime now I have been trying to find an easy way to get movie related information especially newly released. Like many people, I have been surprised by the lack of an API on IMDB. If my memory serves me right, it is now illegal to use (read screenscrap) IMDB although I am not too sure the full extend of the restrictions.

That tidbit of information was actually a blessing in disguise for me as I shifted my focus away from IMDB. This is how I encountered the Neflix API. From the looks of things there has not been a lot of fanfare as evidenced by the lack of tools to tap into the API.

I am in the processes of developing a PHP helper class to interface with the API. My problem for now is that I need to get up to speed with JSON. For now I am parsing an atom result which I am too pleased about.

If I manage to develop something that I am proud of, I will post my code on this blog. For now it is working for what I need it for only that it is not stable.

If anyone knows of or comes up with a good class that can accomplish this, please let me know.

Web Personalization and Recommendation

noreply@blogger.com (Gideon) — Thu, 26 Nov 2009 22:37:00 +0000

One of my routines that I have been doing for a long time has been to log on to a few technology related websites and read up a bit so I can always stay on top of what is happening in the field. One of these websites is GIGAOM and I am really happy with the excellent content that they provide. Today, (yes I have been browsing on Thanksgiving day!), I came across an article that really brought back a combination of very good and sad memories. Here is qoute from the article:

I'm not really comfortable making broad technology predictions, but I've run across two things recently that perfectly illustrate for me the next generation of the web. Some of us are already living it — those of us that use offerings like Twitter or Hunch, or any service that instead of forcing us to search for information asks for our interests, preferences or even location, then delivers it. The next-gen version of the web will consist of services that tell us what we want to know at the exact moment we want to know it — possibly even if we don't ask.

The good memories is that I was lucky to join a team whose focus was to solve such a similar problem. That was almost 4 years ago! I just happened to stumble into this research group at the proding of one of my programming professors. It was a privilege to join some very smart people who, thinking back now, were visionaries who were thinking several years ahead into the future.

The sad bit is that due to poor leadership, the team crumbled and fell apart. The lesson I learned is that no-matter how smart the team is, if the leader is a wimp and not bold, very little can be done. The sad bit too is that I and my colleagues let the poor leadership, not only break the team but break our spirits.

Web personalization, in my opinion, in huge and very important. Many people depend on it. With information overload and the multiplicity of sources now available, it is important that technologies tell us what we need to know. I use Google News for that reason. I depend on Google to tell me what I should know that is happening around me.

There are many aspects of our information consumption that can greatly benefit from personalization and recommendations!

Hadoop and Data Mining Status

noreply@blogger.com (Gideon) — Tue, 24 Nov 2009 20:56:00 +0000

A number of people who have followed my work have been asking me whether I no longer had interests in Hadoop and Data Mining. I can understand where this is coming from since I have not posted anything either on this blog or elsewhere. The main thing is that after graduating from college, I mostly lost access to the cluster that I was using for most of my work. I had always thought that as soon as I graduated I could land a good job and possibly acquire a few good desktops and make my own cluster. That did not happen. The economy decided to completely ignore my wishes. So without resources I have not been able to do much.

So santa if you are listening, I do not need more candy this season. Cash is more like it. As soon as I can get myself a reasonable cluster, I am going to dive back into Hadoop and continue my research. Of course I have been reading a lot about Hadoop but I usually prefer writing about my experiences rather than what someone else wrote. So until then there won't be much. But as soon as I am set up, I am going to finalize/extend my findings on Hadoop and Distributed Data Mining and write about it. I have had requests to write about my experiences learning parallel programing and I am also going to do that once I get the chance.

Will Fox delist from Google

noreply@blogger.com (Gideon) — Tue, 24 Nov 2009 01:09:00 +0000

I have been doing a lot of studying up Search Engine Optimization and Web Marketing in general. I just have so much time in my hands and those 2 just happened to be both challenging and not too involving to do. Because of this, a friend of mine forwarded a link to me about Fox delisting from Google and exclusively index with Bing. I then did a little bit of reading trying to find out more about the impact of this and other related stories such as the rumor that Google is going to make pageload speed a factor to ranking.

One of the articles that I found interesting, from a different perspective anyway, is from Marc Cuban on his blog. Here is a quote from the article:

Or they can target to pay sites about mesothelioma and other diseases that ambulance chasers covet and pay huge dollars per click through, or other high paying PPC searches. The advertisers for these categories go where they can get the most clicks. It wont change marketshare, but it could change how the battle between Googe and Bing is fought. If they can win enough categories, all of the sudden they have some bragging rights that set a platform for people to question googles positioning.

It just reminded me of what a friend of mine used to call "guerrilla marketing techniques" and a line from one of my favorite movies, Enemy of the State "In guerrilla warfare you use your weaknesses as your strengths; they are big and you are small; you are fast and they are slow" something to that effect.

Facebook killed me!

noreply@blogger.com (Gideon) — Tue, 10 Nov 2009 23:05:00 +0000

The title to this posting is not mine. That was the first statement of a conversation I had on my way home. You see, way before social networking was very popular (read as before facebook became popular) I created a social-networking-like site for my high school with Joomla and some custom modules. That was way in 2005 before facebook became mainstream and I was still in college. The site took off very well with a huge number of people signing up and people catching up with friends.

Because of the success of the first site, I started getting a lot of requests to create similar websites for school related sites. The very first person to ask me to do this is this guy who called me today and as I college student this became a way for me to make a little bit of money. Now almost 5 years later, most of these sites are barely making it because people moved to facebook.

So this is why this guy was calling me. Facebook had killed him. His investment in the project was not paying off anymore. That's when it dawned on me that yes facebook had also killed me. The requests for similar projects have all but stopped. Yes its always a tough situation to have to compete for people's attention with well-oiled corporate machines where there is constant innovation.

New Facebook Chat -> Good News

noreply@blogger.com (Gideon) — Fri, 06 Nov 2009 01:02:00 +0000

Like many people I know, one of the main problems with a presence of facebook is that as you connect with more people, the upkeep of your profile almost becomes a full time job. At times you end up just deserting facebook because once you log in, you can end up spending a long time tending to the profile. One feature I have always found to be very useful is facebook chat although the fact that it was not a full blown chat interface always bugged me. So it was relief that very soon I can be able to chat to my facebook friends via a XMPP client of my choice.

Here is a brief note from GIGAOM:

The instant messaging world should prepare for a major quake — thanks to Facebook, which seems to be all set to launch a new connection interface that would allow Facebook Chat to work with any kind of XMPP client.
The news of this development was first reported by Mickaël Rémond on the company blog of Process One, a Paris-based messaging startup. “It now seems the launch is close as the XMPP software stack has been deployed on chat.facebook.com,” writes Rémond, who is a leading expert on instant messaging and ejabberd and is an active member of the XMPP Standard Foundation.

About a year-and-a-half ago, Facebook had announced that it would build “a Jabber/XMPP interface for Facebook Chat” and that “users will be able to use Jabber/XMPP-based chat applications to connect to Facebook Chat to” communicate, check their friends’ profiles, and set their statuses.

Extensible Messaging and Presence Protocol, or XMPP, has surely become the de facto standard for messaging and presence. After a big push from Google Talk, XMPP is going to get the next major push from Facebook. The world’s largest social-networking service, with over 350 million subscribers, is about to launch the XMPP connection interface. That will allow users to use Facebook Chat with any XMPP client — whether on the desktop or mobile. A good example of how this works is Adium, a popular open-source IM client that allows you to communicate with disparate IM networks. The latest version of Adium supports Facebook Chat.

You can read the rest of the article here.

Installing Pligg : Lessons learned

noreply@blogger.com (Gideon) — Wed, 14 Oct 2009 02:35:00 +0000

Yes, this lesson was learned the hard way. The main problem I think was that I had installed Pligg several times before and I guess the confidence was proactively preventing me from arriving at a solution. Yes the problem might be very trivial but looks like there were a lot of people having this problem and I did not find any solution online.

The error:

Fatal error: TPL: [in line 0]: syntax error: file 'blank.tpl' does not exist in ...

The problem:

To me at least, the problem was possibly caused by the fact that I was moving to a new host. My old host was giving me all sorts of problems (after almost 4 years! I tried to be patient). For some strange reason blank files that are part of Pligg were not being uploaded to the server. The "blank.tpl" file is blank and so it was always missing. The problem is that this error message is misleading. There is another blank.tpl file in the template folder which I thought was the one it was complaining about. The missing file however was the blank.tpl file in the main templates folder.

There were some other folders that were not being created because they were blank and I was always having to create them manually.

The Solution:

Easy! I uploaded the zip file to my server and unzipped it from there. From there everything went on smoothly, several hours later. Hope my frustration and this posting will help someone (maybe even myself in the future!)

And then there was Twitter

noreply@blogger.com (Gideon) — Fri, 07 Aug 2009 21:42:00 +0000

I don't usually say things such as this openly but I realized I am not the only one: I have recently started using Twitter. I don't really have much of a strong reason for not doing so before now but I suppose its just that I did not see how Twitter could benefit me. As it turns out, Twitter is a very good marketing tool something that I did not appreciate before now. I started using Twitter when someone told me of the real time search. Eventually I created accounts for websites I manage so that I can periodically tweet any updates to the sites. Anyone interested to know whenever an update was made could follow that account plus any other announcements.

I have decided to play with the API a bit and see if there are any interesting things I can do. (I have been playing with the Facebook Stream in the past few weeks and will post my experiences shortly.)

Of course there is a lot of junk and spam on Twitter which can be annoying at times. Here is an excerpt of an article at readWriteWeb about Twitter that I found interesting:

Only 5% of Twitter's users account for 75% of all the activity on the service, and almost one third of all the tweets posted by the most active users come from bots that each generate more than 150 tweets per day. According to a new report from Sysomos, the up-and-coming social media monitoring and analytics service, one quarter of all the messages posted on Twitter are currently generated by bots. Some of these are obviously spambots, though a large number of bots are also run by legitimate organizations, including @diggupdates, @imdb, and @dogbook, which posts updates from pets on Facebook to Twitter.

The rest of the article is here:

15 minutes offline

noreply@blogger.com (Gideon) — Fri, 31 Jul 2009 16:27:00 +0000

I got to realize the extend to which I am dependent on the internet last night when we lost power for a while last night (more than 15 minutes actually). I was in the middle of preparing a document on my laptop. I rarely reference books because with Google you can always find anything on the internet. Now, the document I was preparing was time sensitive and I went into panic when suddenly I could not search for information on the internet.

This led me to imagine how people did without the internet let alone the computer!

Hadoop to the rescue?

noreply@blogger.com (Gideon) — Sun, 07 Jun 2009 21:20:00 +0000

I have mentioned elsewhere on this blog that I am currently doing a thesis in which I am using Hadoop and MapReduce. One thing I have always liked whether in school or in life in general is the applicability of what I am learning in the "real world". Once you get past why I really need to learn and know this stuff, I usually can put my mind into it and get motivated to learn. In high school, I took Advance Math, not out of love for Math but it was the most convinient way to take Computer Science. Looking back into it now, I wish someone had really explained to me why I needed to pay more attention. I really did not foresee how some of the stuff we were doing could possibly come in handy later on in life.

Anyway back to my thesis: This afternoon, I bumped into an article at GigaOm by Gary Orenstein entitled "Digging Deeper Into Data With Hadoop". In the article, the author gave some kind of an overview of Hadoop and how its being used by some of the top companies. To me, this made my work very relevant because coincidentally, I had just finished writting a MapReduce program that processes a server log to find out query terms and their frequency. According to Gary, Yahoo is using Hadoop to build a database for its Search Assist feature. I was smiling when I read that before the use of Hadoop, the task used to take 26 days to complete! Thanks to Hadoop, it now takes only 20 minutes! I guess thats a cluster with thousands of nodes. I am also encourage that even Microsoft is using Hadoop although its not clear if its limited to Powerset.

I am glad I found this article because it gives my work relevancy and applicability to real world situations. Although I am working on a relatively small cluster with only 12 nodes, I know all this is not in vain!

My first Facebook Application

noreply@blogger.com (Gideon) — Thu, 30 Apr 2009 20:52:00 +0000

I had never though of looking into developing an application for Facebook mainly because I had been annoyed by many apps that cluttered and wasted my time on my notifications page. This week, after reading on how Facebook was opening the stream, I realized that getting to know about the Facebook platform may be of some use in my Machine Learning research and so decided to familiarise myself with Facebook's API.

I then threw up a very basic application : the application just consist of pulling the showtimes rss feeds for my local theater - nothing fancy here. Since this application was not going to utilize many of the features of the API, I created another application and just experimented with some of the features. I am not sure yet if this will turn out to be a useful thing.

The app (for the readers who are from around here) can be accessed at http://apps.facebook.com/majestictheater/

Including external jars in a Hadoop job

noreply@blogger.com (Gideon) — Thu, 16 Apr 2009 04:04:00 +0000

One of the disadvantages of setting up a Hadoop development environment in Eclipse is that I have been dependent on Eclipse to take care of job submission for me and so I had never worried about doing it by hand. I have been developing mostly on a single node cluster (i.e my laptop) which meant I never had the need to submit a job to an actual cluster, a remote cluster in this case. Also, the first MapReduce programs I have written and run on the cluster (more to follow) were not dependent on third party jars. However, the program I am working on depends on a third-party xml parser which in turn depends on another jar.

As it turns out, I had to specify 3 external jars everytime I submit a job. I knew there was a -libjars option that you could use as I had seen it somewhere (including the hadoop help when you don't specify all arguments for a command) but I did not pay attention since I did not need it then. Googling around, I found a mention of copying the jars to the lib folder of the Hadoop installation. It seemed a good solution untill you think about a multi-node cluster which means you have to copy the libraries to every node. Also, what if you do not have complete control of the clusters. Will you have write permissions to lib folder.

Luckily, I bumped into a solution suggested Doug Cutting as an answer to someone who had a similar predicament. The solution was to create a "lib" folder in your project and copy all the external jars into this folder. According to Doug, Hadoop will look for third-party jars in this folder. It works great!