We thought it would be fun to give a sense of all the systems that power Instagram, at a high-level; you can look forward to more in-depth descriptions of some of these systems in the future. This is how our system has evolved in the just-over-1-year that we’ve been live, and while there are parts we’re always re-working, this is a glimpse of how a startup with a small engineering team can scale to our 14 million+ users in a little over a year.

via Instagram Engineering • What Powers Instagram: Hundreds of Instances, Dozens of Technologies.

No surprises, but it looks like a solid stack. I’ve been using most of the tools myself in earlier projects. I was a bit surprised about the size of their instances. I would’ve used more but smaller instances, but apparently the IO is not the bottleneck. Food for thought..

“Companies can see pieces [of software] as far more core to their business,” said Recordon. “But our ability to serve PHP faster is not core to our business. But cheaper/faster development tools for other companies is a real competitive advantage.” Hence, logic dictates that because there’s no business loss if Facebook’s infrastructure is open-sourced, then open-sourced it should be.

/via Facebook opens up about open-source software

Very nice quote by David Recordon, creator of OAuth amongst other things and currently Facebook’s open source guru. The rest of the interview is worth reading as well, but this quote is something to think about. Which parts of your infrastructure can you open-source?

There is an old parable about the concept of commitment when it comes to breakfast. The story goes that when looking at a plate of the traditional fare of ham and eggs, it’s obvious that the chicken is an interested party, but the pig is truly committed.

I heard this parable before and it keeps intriguing me. Am I a chicken or a pig?

Jeff continues:

When I tell this story to entrepreneurs, my point is usually to contrast the approach VCs have to start-ups as compared to entrepreneurs. The VC is an interested party, but at the end of the day, if their start-ups live or die, they typically still have their job, their office and their portfolio of other investments. The entrepreneur, on the other hand, is the pig – truly committed to the outcome, with no fallback.

But lately I’ve been thinking about the parable of the pig and the chicken in the context of the characteristics that make a great entrepreneur – and the kind of entrepreneur that we VCs in general, and my firm Flybridge Capital in particular, like to back. In short, we like to back pigs – entrepreneurs who are truly and completely committed to the outcome of their venture, have a lot of stake, and no fallback.

via Why Venture Capitalists Invest In Pigs, Not Chickens.

So the question is: Are you a Pig or a Chicken?

Do you wait to start your venture until you are fully backed or do you start right away and see where you end up? Gary Vaynerchuck said it over and over again, there is plenty of time after office hours to start your company. It’s all about commitment.

Image source: San Diego Shooter

Many businesses realize the potential of social media and are savoring a seemingly compelling way to reach customers. Many are even boasting millions of fans and followers.

But what they too underestimate is the power of people. It’s not so much about technology as much as it is about the democratization of information and the equalization of societal influence. People are now part of the equation and are willingly shedding their “audience” moniker and vacating the branded auditoriums of yore in favor of building their own stages, their own personal theaters.

via Social Media is not Going to Save Your Business Brian Solis.

Social Media is not about a new audience and Social Media cannot be bought like traditional ‘Reach’. It is a new and better way to connect with your customers and therefor priceless.

A couple of weeks ago I wrote a small post on The ROI of Social Media linking to a video by Gary Vaynerchuck.  I wrote that “Having a visible, understandable and mutual relationship with your customer is priceless.” That is what Social Media is providing us. Not a new way to advertise our products, but engagement and a connection with customers we last had when small stores in small towns knew their customers personally and could rapidly adapt to changing needs.

You don’t ask for the ROI of mobile phones, because they are an integral part of peoples way of communicating. They improve efficiency and enable faster and more direct communication. Social Media should be an integral part of a business’ way of communicating, not a separate devision handling the ‘social media strategy’.

As a wise executive of a leading global company once told me, “If you come to me with a request for budget and resources for social media, to make it a priority for our business, you will lose every time. If you tie social media to our business priorities and objectives and demonstrate how engagement will enable progress, you will win every time. Social media must be an enabler to our business, just show me how.”

Social Media is not going to save your business, but it will help you better understand your customer and help you to adapt, to do better.

I had never heard about the position of Entrepreneur in Residence (EIR in short) until it was offered to me last week by Boris from The Next Web.

An Entrepreneur in Residence can have different roles depending on the type of company it is working for, usually Venture Capital firms, Law firms or Universities. The NY Times describes it as:

Entrepreneur-in-residence is one of those only-in-Silicon-Valley jobs. Smart people get paid to sit around and think about new ideas, and investors get the chance to join an entrepreneur early in a new project, betting that lightning will strike twice.

But what does that really mean? There are two clear types of EIR in my opinion:

Entrepreneur is looking to start his next company

The Entrepreneur usually just finished his last venture and is in between startups. He might be looking for the next big thing, a new co-founder, a new investment round or is just shaping his ideas for execution. To keep him off the streets he is offered the position of EIR at an investment firm. Entrepreneurs have a unique skill set only Entrepreneurs possess. They know how to build successful companies. It’s an unusual position with no pre-scripted work days. EIRs are present at the firm and give advise where needed, do due diligence on new startups coming in, mentor new Entrepreneurs and help out where needed until they fall in love with one of the companies coming in (and join them) or they start a company of their own (usually funded by the investment firm they work for).

Entrepreneur is looking to mentor new Entrepreneurs

Usually a seasoned Entrepreneur with years of experience willing to give something back to young Entrepreneurs. This role is usually held at Universities, but Jason Putorti, the lead Designer at Mint.com holds a Designer in Residence position at Bessemer Venture Partners, helping new companies to build  “simple, intuitive and engaging web sites”. The Harvard Business School has a long running program inviting HBS alumni Entrepreneurs to become an Entrepreneur in Residence for a full academic year to mentor and inspire HBS students.

The EIR working at a law firm usually has one of the roles mentioned above to assist clients of the firm.

Pros and Cons

Being an Entrepreneur in Residence gives you a unique insight into the VC or Startup Hub perspective. Normally an Entrepreneur sits on the other side of the table, but now he has full access to all knowledge of the firm he is working for. The EIR also has the possibility to vet its new ideas to the guys who should know if something will be a hit or not and can discuss new business models without fear of not winning a pitch.

Being an EIR at a VC also has some disadvantages. The VC is probably going to invest in your new startup and your EIR could scare other VC’s away, but these cons are minor compared to the advantages.

Being an EIR

The position of Entrepreneur in Residence is no longer an only-in-Silicon-Valley job or Amsterdam really looks more like Silicon Valley every day, because starting today I am an Entrepreneur in Residence at The Next Web.

The Next Web is a unique combination of an Event organizer, Tech blog and startup incubator. Most of the people at the TNW Offices here in Amsterdam are running or working for startups like Twitter Counter, Feest.je and Shoudio. My Entrepreneur in Residency is not meant to be a full-time job, but encourages me to try out new concepts, meet everyone I could possibly want to meet and at the same time share my knowledge and experience with the TNW team. I feel honored that this position is offered to me and am exited to be part of The Next Web for a while. Certainly to be continued.

I love Gary Vaynerchuck. A lot of people hate him and I understand why, but I can’t help to get passionate about Social Media every time I hear him speak. In the video above he answers a question on LeWeb’10 how to measure the ROI of Social Media. Blunt as ever he throws in some jokes and weird connections, but he does make some valid points.

All of us know how much money we’ve spent with people we either know or that we are recommended to. This is how we buy things. Word of Mouth is the currency of how people spent their money. That is where this entire ecosystem is build on.

This is how we have spent our money since the dawn of time. Peer influence and recommendations have always been a major influence on our buying pattern, together with price of course. The difference with Social Media is that is has become more visible and more measurable. The current state of internet technology gives us the tools to bridge the gap between the real world and the virtual world.

The word Media in Social Media is the whole problem. The word Media requires ROI. If we rename Social Media to ‘Relationship with the Consumer‘, the question about ROI fades instantly. Everyone knows how valuable that relationship is.

I truly believe in that. Having a visible, understandable and mutual relationship with your customer is priceless. The difference with the internet is that it brings scale into the equation. The internet brought us a communication cost of nearly zero and a scale we couldn’t apprehend before. When with the introduction of Social Media our customers could not only consume but also interact with us as producers a new type of relationship was born. When you are able to interact on a near personal level with your customers by using Social Media then that is your ROI and Word of Mouth is your currency.

Of course Nielsen or McKinley will come up with a number like Gary says and maybe it will indeed be $3.09. But the real value of this new relationship will be far greater and we all know that.

There is a fairly accepted rule that 1% of a site’s users will create content, 10% will interact with it while 100% consumes the content. That means that 90% of your users will probably be logged out. Consuming users can’t upgrade to Interacting users without having to create a profile. Registering has become far easier with the introduction of OpenID and OAuth connections offered by Twitter and Facebook, but it still is a big step.

At Mobypicture we are experimenting with ways where people leave a comment first and are then guided through the login or registration process, placing the comment afterwards. This way Consuming users can say what they want to say, before getting distracted by login screens.

Fred Wilson, VC and principal of Union Square Ventures, goes a step further and proposes more interaction for logged out users by giving them “phantom profiles“, storing activity against their cookies and building user profiles on logged out users.

This does two things. First, if those logged out users eventually register and become logged in users, this “phantom profile” can help the user get a lot of value from the service right away.

If you visit the profiles of Boris Veldhuijzen van Zanten, Robert Scoble and Lord Voldemort on Twitter a lot and you finally decide to register your account with Twitter, Twitter could show you the profiles of @Boris, @Scobleizer and @Lord_Voldemort7 with a follow button next to it. This might be a far more effective way of turning new users into regular users than showing random profiles based on follower counts and the fact those profiles are currently ‘trending’.

And second, this “phantom registration” might allow the service to permit lightweight engagement without logging in. Lightweight engagement might be favoriting an item on Etsy, hearting something on Tumblr, or starring something on Twitter.

The great thing about Instapaper, a simple tool to save web pages for reading later, is that you can start using it without a password, without real registration, just by claiming a username. You are storing articles you want to read later, not your bank account details, so why should you worry too much about your account? Only if you become a regular user and you would like to make sure no one else can add articles to your reading list you can add a password, an email address and other personal details. If Instapaper was a browser only application mostly used from one computer, claiming a username wouldn’t even be necessary. A “phantom profile” should suffice. Twitter could add favoriting of tweets for logged out users and Facebook could allow you to like content so you can start with a partially filled profile when you register.

There is a lot to gain from the 90% of your users that are logged out, do not forget about them.

Illustration courtesy of Esther Gons.

I’m reading a lot about Continuous Integration and Test-Driven Development lately, to work out the best ways to develop code as agile and with as much flexibility for deployment as possible. I wrote a small post earlier on Continuous Integration in PHP about how to build a CI server with Jenkins. In this post I would like to go deeper into the Why of Continuous Integration and Test-Driven Development.

In the early stage of Sugababes.nl we worked live on our production codebase, changing and testing things while users were visiting the website. That provided a lot of trouble and was soon discarded. Ever since we (my team at Sugababes first and Mobypicture later) work from SVN on development environments, commit code, try to test it in a staging environment, but most of the time we just deploy. That is, export parts of the codebase to our live environment. That is not an ideal situation and proves to be very catchy for bugs that could have been prevented.

The first thing we should need is automated deployment, for example with Ant or Capistrano. Capistrano gives the option to rollback a deploy, which can be very useful. Capistrano copies the code to servers using SCP in parallel and can perform tasks before or after a deploy, like Minifying your Javascript/CSS and upload them to your CDN. This does not help in the prevention of deploying bugs and broken code. When you change a Class in one area of your code it will often break your codebase, how hard you look for all instances of your Class and change the code where necessary. Unit Testing should find most of these breaks.

So the second thing we need is Unit Testing. You write a test for every single function in every single Class. When you change Class A and Class B uses Class A, your Unit Test will tell you when Class B breaks on that change. A good way to make sure you write Unit Tests is by using Test-Driven Development. Test-Driven Development forces you to write your Unit Tests before you write any code. It thereby also forces you to better think about what you are going to code, because you define all income and outcome before writing any line of code.

Before you commit any code you should run your Unit Tests to test and make sure your code is still working. When your project grows your tests will last longer and besides Unit Tests, there could be more tests you would like to run. Like Code Duplication Detection, Javascript and HTML testing and generation of documentation. This can take a lot of time every time you would like to commit. And if you program as iterative as we do, commits happen multiple times a day. It would be nice if this could all be done automatically right?

With a Continuous Integration server you can. After someone commits any code, the CI server exports the codebase and runs all the tests automatically and will notify the person who comitted or even the whole team about errors. Those errors should be fixed as fast as possible because the codebase should be ready to be deployed at any time. And that is probably the most important reason to use Continuous Integration. I ask my team all the time if the codebase is ready for a deploy and that should not be the case. It should always be ready, because your main codebase in the repository it THE place where every developer starts to work from and new versions are build from.

Setting up a Continuous Integration server is no easy task, but Jenkins looks interesting. It supports Ant-like build files, has a nice web-based front end and has tons of plugins to provide further functionality. Because it is the leading CI server there are also a lot of tutorials about implementation for almost all development languages. Eric Hogue wrote a nice tutorial on setting up Jenkins for Continuous Integration on PHP, with PHPUnit and a lot of quality assurance. I will write more about my experiences with Jenkins and Continuous Integration when I learn more about it, because I really believe this will improve my deployment cycles in the future.

Because Backtype is processing all tweets for URLs to calculate your influence, they have to process a massive amount of data from the Twitter Firehose. The Firehose can go as fast as 7000 tweets per minute during New Years Eve in Tokyo. That’s a massive 117 tweets per second!

The big problem with realtime is that you can not or not easily process it in batches, because the data keeps coming. When you batch this amount of data you have to be able to process the data faster than realtime or create an always growing backlog. During the World Cup finals last year when The Netherlands was playing against Spain, we (Mobypicture’s MobyNow) had a small flaw in our code processing the tweets with #ned and #wk2010. During roughly 90 minutes we had built a backlog of 18 hours worth of processing. Because people kept using #ned and #wk2010 it was almost impossible to remove the backlog, we had to go many times faster than realtime to remove it. While displaying realtime tweets you don’t want to be more then a couple of seconds behind. With batched processing this process of removing the backlog is a fight you are fighting every time you process a batch.

Backlog also recognized that batched processing wasn’t the way to go for their analytics. So they recently developed a new system for doing realtime processing called Storm to replace their old system of queues and workers:

Storm is a distributed, reliable, and fault-tolerant stream processing system. Its use cases are so broad that we consider it to be a fundamental new primitive for data processing. That’s why we call it the Hadoop of realtime: it does for realtime processing what Hadoop does for batch processing. We are planning to open-source Storm sometime in the next few months.

The realtime processing system of Mobypicture consists, just as Backlog’s used to be, out of  a series of queues and workers. We kick a piece of content from one queue to another, trying not to slow down the whole process when more data than we can handle comes in. The system isn’t really fault tolerant. Every queue has to stay online and we can’t process anything when one of the queues goes down. It’s a complex system and I understand why Backlog has tried to replace it with a more generic solution, they can deploy anywhere.

[Storm] abstracts the message passing away, automatically parallelizes the stream computation on a cluster of machines, and lets you focus on your realtime processing logic. Even more interesting, Storm enables a whole new range of applications we didn’t anticipate when we initially designed it.

Properties of Storm

I won’t quote them all, you can find the longer descriptions in the original blogpost by Backtype, but these are the main characteristics:

1. Simple programming model: Just like how MapReduce dramatically lowers the complexity for doing parallel batch processing, Storm’s programming model dramatically lowers the complexity for doing realtime processing.
2. Runs any programming language
3. Fault-tolerant: To launch a processing topology on Storm, all you have to do is provide a jar containing all your code. Storm then distributes that jar, assigns workers across the cluster to execute the topology, monitors the topology, and automatically reassigns workers that go down.
4. Horizontally scalable
5. Reliable
   
6. Fast: Storm is built with speed in mind. ZeroMQ is used for the underlying message passing, and care has been taken so that messages are processed extremely quickly.
   

I’m really curious about Storm. The use-cases are numerous, from stream processing to continuous calculations. Love to get my hands on it and do some testing. Maybe in a couple of months.

In the meantime, check out Preview of Storm: The Hadoop of Realtime Processing by BackType Technology. They have more interesting tech articles and also have some very interesting tech presentations.

Yesterday was the last MobileMonday Amsterdam, an inspiring bi-monthly event with great speakers and inspiring visitors. I had a lengthy talk with Martijn Rijntjes about entrepreneurship and he introduced me to something I had never heard of: The Lifestyle Entrepreneur.

Startups are the things all the cool kids do, Martijn told me, but he never felt it was the right fit for him. Martijn likes to travel and see the world, not work his ass off, give up his social life, all in favor of his internet startup. He almost felt guilty he had not the same dreams and goals a lot of his friends have. Until he read an article by Corbett Barr about Startup vs Lifestyle Business.

A startup business has the primary responsibility to grow as big and successful as possible, whatever the impact on the lives of the entrepreneurs. Founders of a startup are competing for success, fame and glory, although some of them, including me, are also in it because they really like the process. After hitting the jackpot, a lot of them jump back into the startup life to repeat their success. Once your lucky twice your good, Sarah Lacy wrote and I believe in this statement. Running a startup is a lifestyle.

Martijn on the other hand is a lifestyle entrepreneur running a lifestyle business. The primary responsibility of a lifestyle business is to allow the entrepreneurs to live their lives the way they want it, not the other way around. They became entrepreneurs because it allows them a way of life, a way of freedom, normal employment can’t give them. They have other definitions for success.

A startup’s job is to grow big enough to provide a return to investors. A lifestyle business’s job is to provide a great quality of life to its owners.

Nobody can tell you what type of entrepreneur you are, you have to see it for yourself. There also is no good or right answer.

Corbett gives 37signals as the example of a lifestyle business. They probably could have been many times bigger, VC-funded, all over the news, but 37signals decided to stay relatively small and are in this for the long run, not for the quick money. They are building a sustainable business, not looking for an exit and are living the lives they want.

David Heinemeier Hansson already told it in the famous interview with Jason Calacanis. He rants about taking someone else’s money and just burn it on too many staff, too many computers and bad choices. “I’m doing 37signals for the next 20 years. I found what I want to do with my life.”. The gap between 2 million and 10 million in the bank, he continues, is so much smaller than the gap between $15,000 and 1 million dollar. The big jump already happened at that point. “How much better will my life be if I owned a private jet, versus having a job I really care about, a company I really care about? I’d much rather have that”, David says.

I do not exactly know what kind of entrepreneur I am. I’m not the guy chasing the quick money, but I am willing to sell for the right amount of money, as I proved in 2007, to start a new adventure. Although I’m in it for the long run, I am thinking what’s best for the company, not what’s best for me. I have had my omens of a beginning burnout and I think I have lost some friends along the way. But this is what makes me happy. And with happy I mean really, really happy. I think it is all worth it.

Martijn doesn’t on the other hand and I envy him for that. I think it’s beautiful how he can live his life with a certain freedom I can’t. After I sold Sugababes I took a sabbatical of a year and what did I do? I didn’t travel the world, emptied my bucket list. No, I started new services and helped people out for I-owe-you’s instead of money. And I loved it!

It is beautiful to see how Martijn and I probably envy each other for the life the other one is living. Both entrepreneurs for life, but with completely different goals.