What Changed Externally

api.gnip.com is alive, well, and fully supported. From a product standpoint we’re now also pursuing a decentralized data access model to broaden our offering. The original centralized product continues to serve its customers well, but it doesn’t fit all the use cases we want to nail. It turns out that while many folks want to be completely hands-off WRT how their data is collected (”just get me the data”), they still want full transparency into, and control of, the process. That transparency and control is on its way via Gnip Data Collectors that customers configure through an easy to use GUI.

To summarize, externally, you will soon see additional product offerings/approaches around data movement.

What Changed Internally

A lot. api.gnip.com is a phenomenal message bus that can reliably filter & move data from A to B at insane volumes. In order to achieve this however, we left a few things by the wayside that we realized we couldn’t leave there any longer. Customer demand, and internal Product direction needs (obviously coupled with customer needs) were such that we needed to approach the product offering from a different technical angle.

GUI & Data

We neglected a non-trivial tier of our customer base by almost exclusively focusing on the REST API to the system. Without the constraint of a GUI, technical/architectural/implementation decisions that come with building software were blinded by “the backend.” As a result, we literally cut our data off from the GUI tier. Getting data into the GUI was like raising the Titanic; doable, but hard and time consuming. Too hard for what we needed to do as a business. We’d bolted the UI framework onto the side, and customized how everything moved in/out of the core platform to the GUI layer. We weren’t able to keep up with Product needs on the GUI side.

Statistics

Similar to GUI, getting statistics out of the system in a consumer friendly manner was too hard. Business has become accustomed to running SQL queries to collect information/statistics. While one can bolt SQL interfaces onto customized systems, you have to ask yourself whether or not you really want to? What if you started with something that natively spoke SQL?

So…

We introduced a stack that supports a decentralized data collection approach, as well as off-the-shelf GUI, statistics collection/display, and SQL interface; “Cloud” instances, running Linux (obviously), MySQL, and Rails. We have prototypes up and running internally, and things are going great.

Product Details

I’ve been vague here on purpose. We’re still honing all the features, capabilities, and market opportunities in front of us, and I don’t want to commit to them right now.

The People

I want to end on a personal note. My mind was blown by the people we decided to “let go” in this process; all of them incredibly high quality.

All I can say here is that it’s all in the people. You build teams that meet the needs of the business. For the sand that shifted, Eric and I are to blame. We undoubtedly burned bridges with amazing people during this process, and that is excruciating. Those no longer with us are great, and all of them have either already jumped into new projects/companies, or are weighing their options. The best of luck to you, and I hope to work with you again someday.

I would like to address a few specific points for our partners, customers and friends:

1. We believe as strongly as ever in providing data aggregation solutions for our customers.  If we didn’t, we would have returned to our investors the year of funding we have in the bank (now two years).
2. We are still delivering the same data as yesterday. The existing platform is highly stable and will continue to churn out data as long as we want it to.
3. The changes in personnel revolve around rebuilding the technology stack to allow for faster, more iterative releases. We’ve been hamstrung by a technology platform that was built under a very different set of assumptions more than a year ago. While exceptionally fast and stable, it is also a beast to extend.  The next rev will be far more flexible and able to accommodate the many smart feature requests we receive.

To Alex, Shane, Ingrid, JL, Jenna, Chris and Jen, it has been a honor working with you and I hope to have the privilege to do so again some day.

To our partners and customers, Gnip’s future is brighter than ever and we look forward to serving your social data needs for many years to come.

Sincerely,

Eric Marcoullier, CEO

As part of the new service Gnip also provides dedicated integration capacity for companies as we now are able to segment individually managed nodes on our platform for specific company accounts.   This means that a company with a developer key on Flickr, a whitelist account on Twitter, an application key on Facebook and a developer key on YouTube receives dedicated capacity on the Gnip platform to support all their data integration requirements.

Gnip will also continue to maintain the existing public data integration services which do not require authentication for access and distribution, and we expect most companies with use a blend of our data integration services.

Using the new support for authenticated data service requires contacting us at sales@gnip.com so we can enable your account. Please contact us today to leverage your existing whitelisting or authenticated account on Flickr, YouTube, Twitter or other APIs and feeds.

Today we rolled out meta-data updates to the <actor> and <to> elements of the Gnip schema.   The updates today are new optional attributes that provide a place to map additional user information that is available on some social media services like Twitter and others.    Initially we will just add the new meta-data to activities where the information is available inline with activities and then in the near future we are adding more platform features to support the scenario where a second API call is required to add this meta-data to the activity.

Starting today the <actor> element has support for the numeric userID, friends, followers, and posts.   In addition, we are now mapping the fullname and username to individual attributes in order to better support services that allow end users to create custom screen names and change those names.   The <to> element was updated to provide a new attribute for numeric userID.

Overview of updates to <actor> and <to> elements of Gnip schema:

• <actor> is the person who performed the activity on the service
  • <posts> is the number of updates made by the user
  • <followers> is the number of people following the user
  • <friends> is the number of people the user has friended
  • <fullname> is the descriptive name or screen name of the user
  • <username> is the username of the user on the service
  • <uid> the unique numeric ID for the user on the service
  • <metaURL> is the user profile link on the service

• <to> is the person who the activity is in response
  • <uid> the unique numeric ID for the user on the service
  • <metaURL> is the user profile link on the servic

These updates will be put in place this Friday, August 28th.    The impact of the new licensing will be the following for our existing users.

1. The Gnip Community Edition license will be disabled as it is no longer being offered.   Accounts that were created before August 1st will be set to inactive and no longer will be able to access the Gnip API or Developer Website.   If your company is in the process of evaluating Gnip for a commercial project and needs a longer amount of time to complete your project please contact us at info@gnip.com and we can extend your account on a longer trial.
2. Gnip Standard Edition user accounts using the Commercial, Non-profit and Startup partner license options will continue to be available as they are not impacted by the change on Friday.   If you are a standard edition user and we accidentally your disable your account on Friday please contact us at info@gnip.com and we will reactivate the account.
3. New users who created an account starting after August 1st will receive an email notification on the day their 30-day trial expires informing them they need to contact Gnip to obtain the appropriate license for their commercial, non-profit or partner use case.

We appreciate all the companies and developers who have built solutions using Gnip and look forward to continuing to deliver real-time data to power these solutions.   By making these adjustments in our licensing we will be able to focus on innovating the Gnip Platform and supporting the many companies and partners we are fortunate to work with every day.

I’ve spent the better part of the past couple of days playing a game. I was chasing down some odd polling behavior observed in one of our internal prototype applications. It ultimately turned out to be some bad assumptions I made around how some code I wrote should behave. The rate limiting policy around one of the open APIs I was using was obfuscated.

The scenario reminded me of a challenge I faced earlier in my career at Netscape. We were trying to figure out how Netscape/Mozilla open source should function (early on in Mozilla’s life… pre-independence from AOL; e.g. 1998). We struggled managing corporate needs, sometimes around confidentiality, in the context of open source. Mozilla wouldn’t work if things that impacted the open source software on the Netscape-side of the engineering house weren’t openly discussed. As predicted, innovation suffered when significant code contributions being made by Netscape weren’t transparent. Netscape was faced with staying quiet about its intentions, or being open with them. Open sourcing the code (e.g. having an “open” API) wasn’t enough. The process by which the code/API was to evolve and function had to be open.

Netscape/AOL weren’t able to let go of key, though seemingly small, aspects of the project and innovation waned. Mozilla/Firefox didn’t explode until there was a formal transition from AOL to Mozilla Foundation many years later. While Firefox has pushed the industry forward in bounds since then, there were years of browser industry confusion and impedance due to a non-committal controlling interest.

The parallel I’m drawing between Netscape/Mozilla’s history and today’s “open” web APIs is that there are key players chokeholding the rest of the industry with inadequately supported, poorly communicated, API access policies. Access policies, while sometimes documented, are highly irregular and poorly communicated. The result is a developing ecosystem around these APIs that has to decide whether or not to play the API access guessing game. When a developer using some of today’s open APIs wakes up and rolls out of bed each morning they wonder “will my application work today?” That’s untenable in the long term.

Just as it was Netscape’s right to control the bits it wanted to, Kings of today’s API hill have a right to do whatever they want. To those who’ve been successful at creating unyielding demand; hats off! Use that power wisely however, and learn from history’s mistakes.

To APIs crying uncle due to the operational overload of their popularity I recommend moving to an event driven API access model (ala Gnip). When that’s simply not possible (though I’d argue it always is) use something like SUP to minimize constipation in the rest of the digestive system.

If you’re throttling access to your API because you don’t know what your business model is, hurry up, get it sorted, and communicate intentions. If you don’t, industry will find a way to pass you by.

First, many thanks to Adjix, awe.sm, betaworks, bit.ly, Cligs,  URLizer, and urlShort who have joined with us to launch this new organization.  And. you can read the actual 301works announcement that is posted on Gnip.com

Why is Gnip involved?  We are part of the Internet software community and most of us are also active social media users.   While there is debate on the pros and cons of short URLs in some parts of the industry it is obvious that over the last few years there has been a huge growth in the adoption of short URL formats across the web and increasingly custom short URLs are being used by businesses and individuals.

People generate short URLs everyday and they need to know that these mappings will continue to function as they were intended when generated, that their mappings will be available for them to use in the future, and that their privacy preferences will be respected. With short URL formats having reached general acceptance by millions of users in their daily activities there was a need for the industry to ensure the connections provided by these mappings exist over time.

In providing the technology to power the 301works solution Gnip is ensuring that the social connections and data represented by the millions of URL mappings done everyday continue to be available across the web.

We are thrilled to be able to participate and to do our part in helping sustain and grow an open web.

Team

We continue to grow, primarily on the engineering side. Checkout our jobs page if you’re interested in working on a hard problem, with smart people, in a beautiful place (Boulder, CO).

Technology

We’ve built a serious chunk of back-end infrastructure that I’d break into two general pieces: “the bus”, and “the pollers.”

“The Bus”

Our back-end moves large volumes of relatively small (usually <~3k bytes) chunks of data from A to B in a hurry. Data is “published” into Gnip, we do some backflips with it, then spit it out the other side to consumers.

“The Pollers”

Our efforts to get Publishers to push directly into Gnip didn’t pan out the way we initially planned. As a result we had to change course and acquire data ourselves. The bummer here was that we set out on an altruistic mission to relieve the polling pain that the industry has been suffering from, but were met with such inertia that we didn’t get the coverage we wanted. The upside is that building polling infrastructure has allowed us to control more of our business destiny. We’ve gone through a few iterations on approach to polling. From complex job scheduling and systems that “learn” & “adapt” to their surroundings, to dirt simple, mindless grinders that ignorantly eat APIs/endpoints all day long. We’re currently slanting heavily toward simplicity in the model. The idea is to take learning’s from the simple model over time, and feed them into abstractions/re-factorings that make the system smarter.

Deployment

We’re still in the cloud. Amazon’s Ec2/S3 products have been a solid (albeit not necessarily the most cost effective when your CPU utilization isn’t in the 90%+ range per box), highly flexible, framework for us; hats off to those guys.

Industry

“The Polling Problem”

It’s been great to see the industry wake up and acknowledge “the polling problem” over the past year. SUP (Simple Update Protocol) popped up to provide more efficient polling for systems that couldn’t, or wouldn’t, move to an event-driven model. Providing a compact change-log for pollers, you can poll the change-log, and then go do heavier polls for only stuff that has changed. PubSubHubbub popped up to provide the framework for a distributed Gnip (though lacking inherent normalization). A combination of polling and events spread across nodes allows for a more decentralized approach.

“Normalization”

The Activity Streams initiative grew legs and is walking. As with any “standards” (or “standards-like”) initiative things are only as good as adoption. Building ideas in a silo without users makes for a fun exercise, but not much else. Uptake matters, and MySpace and Facebook (among many other smaller initiatives) have bitten off chunks of Activity Streams, and that’s a very big, good, sign for the industry. Structural, and semantic, consistency matters for applications digesting a lot of information. Gnip provides highly structured and consistent data to its consumers via gnip.xsd.

In order to meet its business needs, and to adapt to the constantly moving industry around it, Gnip has adjusted it’s approach on several fronts. We moved to incorporate polling. We understand that there is more than one way of doing and will incorporate SUP and PubSubHubbub into our framework. Doing so will make our own polling efforts more effective, and also provide data to our consumers with flexibility. While normalized data is nice for a large category of consumers, there is a large tier of customers that doesn’t need, or want, heavy normalization. Opaque message flow has significant value as well.

We set out to move mind-boggling amounts of information from A to B, and we’re doing that. Some of the nodes in the graph are shifting, but the model is sound. We’ve found there are primarily two types of data consumers: high-coverage of a small number of sources (”I need 100% of Joe, Jane, and Mike’s activity”), and “as high as you can get it”-coverage of a large number of sources (”I don’t need 100%, but I want very broad coverage”). Gnip’s adjusted to accommodate both.

Business

We’ve had to shift our resources to better focus on the paying segments of our audience. We initially thought “life-stream aggregators” would be our biggest paying customer segment, however data/media analytics firms have proven significant. Catering to the customers who tell you “we have budget for that!” makes good business sense, and we’re attacking those opportunities.

Digg-2000 Publisher — What is it?

Lots of people submit stories to Digg and lots of other people Digg the stories which allows more popular information to rise to the top in being discoverable.    Several Gnip customers asked if we could make it possible for them to only receive stories that had a specific number of Diggs.   We asked Digg about the idea and they said it sounded great since they have a Twitter account that provides a similar type of feature, so the Digg-2000 Gnip publisher was born.

Digg-2000 Publisher — How it works?

On the Gnip platform we have set up a publisher that is listening to activities on Digg.

• When new stories are submitted to Digg we pick those new activites up along with the Digg count on the story and they are posted to the standard Gnip Digg publisher.
• With every new Digg on a story we increment our tracking of the story and when we hit 2000 diggs we re-post the original story to the Digg-2000 publisher.
• The configuration of the Digg-2000 publisher allows for us to turn two different dials.
  • The default configuration only will re-post stories that were first posted on Digg in the last two days.   This means we are looking for current active stories and not stories that were posted months ago and through a slow and steady interest finally hit 2000 Diggs.
  • The default configuration re-posts at 2000 Diggs.   This can be set to any number of Diggs — 100, 1000, 2000, etc.