ReadWriteHack

Sign o' the Times: IBM, SAP Now Support Microsoft's Standardization Move

Scott M. Fulton — Thu, 24 May 2012 17:51:00 -0700

It's perhaps the one way that database interaction can work reliably using any format, any server and any client on the Web today. It happens to be a protocol created by Microsoft. But in a symbol of how Microsoft is now perceived today as just another major player instead of a dominant force, the leading platform makers are joining Microsoft in a formal move to standardize OData, the Open Data Protocol. The reason Microsoft and IBM are no longer fighting over this? The real competition is no longer just amongst these old-line technology companies, but largely between them and a new breed of competitors often based around new mobile devices and consumer platforms.

Nearly six years ago one of the most bitter standards battles in all of computing was fought between OASIS, caretaker of the OpenDocument format, and Microsoft. The issue was whether, by making the document formats for Microsoft Office available as open standards, the company was unfairly leveraging Office's broad base of installation to unseat its only real competitor, the storage format behind OpenOffice.

That year, 2006, marked the turning point in Microsoft's public stance towards openness and interoperability. This despite bitter opposition led by IBM, which was working to resurrect the Symphony brand it had acquired through Lotus. IBM argued that Microsoft was undermining the standards process to suit its purposes. Just how much the computing environment has changed since then was made evident today as IBM (whose heart grew three sizes this day) joined with SAP and Citrix in backing a move by Microsoft to declare the OData HTTP-based database interaction prototol - also known as Open Data Protocol - an OASIS standard.

"OData is an application-level protocol for interacting with data via RESTful Web services," reads a white paper on an OData extension for XML published by OASIS (PDF available here). "An OData Service’s contract is defined by simple, well-defined conventions and semantics applied to the data model exposed by the service, providing a high level of semantic interoperability between loosely coupled clients and services."

Phrasing data in XML, ATOM or JSON format does allow it to be visible to many different database engines. They can see data in these formats, so that's not the problem. Essentially, how does a Web client (any browser on any device) request data from any server (any database engine or framework), and then initiate a dialog with that server so it can update or change one or more records? Remember, HTTP is a stateless protocol: Clients make GET requests and the occasional POST statement, and servers acknowledge and, when they can, comply.

So OData enables communication using any of these three formats for data expression, setting up the mechanism for clients to authenticate themselves and enter into a secure dialog. It already works. And as IBM discovered two years ago, OData was pretty much the only way of creating a single Web client for WebSphere data that worked with .NET, AJAX and PHP clients.

While an XML document contains the body of data, expressed explicitly within the markup, OData expresses the schema - the arrangement and relations between data in a database. It also allows for vendor-specific properties that enable features that one vendors database might utilize while another's does not, though these properties are typically flagged within the metadata using obvious vendor-specific tags; for example, sap:filterable.

Writing on behalf of the OData organization today, Microsoft's senior program manager for OData Alex James said today, "With interoperability front and center in OData we saw more and more technology stacks that started to work with it. Now there are a number of companies that use OData in their products to ensure the data they manage is easily accessible beyond the boundaries of their applications. Many of these companies regularly collaborate on the ongoing design effort for OData. While so far we’ve run the OData design process as transparently as we could (sharing designs, taking feedback through the odata.org blog and distribution list, etc.), we are at a point where the level of adoption and the scale at which organizations are betting on OData require a more formal commitment to shared ownership of the design of this protocol."

It’s tempting to draw the conclusion that Microsoft and IBM are “joining forces,” in coalition against a common enemy. The facts are more economic than melodramatic: Waging platform wars, especially over something that should be as ubiquitous as data exchange, costs way too much money. While the Open Office XML/OpenDocument fracas generated plenty of headlines and even sparked a few campaign rallies, it was a drain on both sides’ resources at a time when the platforms supporting both standards – client-side applications – were starting to become obsolete anyway. The fact of the matter is, OData works, and creating a competing format just to rally the troops and produce headlines is a waste of time and money. It’s best that companies pick their battles more wisely. What’s more, IBM, Microsoft, and SAP alike have come to the realization – perhaps a mutual one – that the best platforms are never islands unto themselves.

IBM sign by Simon Greig.

A Utility That Makes You Master of the Twitterverse

Joe Brockmeier — Wed, 23 May 2012 07:32:00 -0700

You can do a lot more with Twitter than the official Web, mobile and desktop clients allow. To do it, though, you'll need to access Twitter's API — or use a tool that digs deeper into the API than the usual clients. If you're comfortable with a command line interface, or willing to learn how to use one, t gives you simple, fine-grained control over your Twitter stream.

The one-letter Ruby gem, t, is written by Erik Michaels-Ober. The source is available on GitHub, or you can just grab it using Ruby's gem install command. Ruby is on most *nix type systems by default (Linux distributions, Mac OS X) but Windows users will need to download the installer to get started.

After you've installed the t gem, you'll need to authorize it. You can use it without getting API access, but then you're limited to 150 requests per hour, and you can't use t to post a Twitter message or make changes to your account.

Using t

To see all of the commands available via t, just run t help. You'll find that t supports more than 40 commands, or tasks as its documentation calls them. The syntax is simple: t followed by the task name, and any options and arguments.

For example, you can use t to get all of the followers for an account by running t followers username. Note that you don't need to include the @ in front of a username.

Many of the commands are mundane operations that you can do with any Twitter client. For example, posting a status update or following another user. Running t update "message goes here" will post an update, running t follow username will follow the user that you specify, and so on.

If you're a command-line diehard, then you might prefer using t for those operations. But its real value is in doing things you can't do with most Twitter clients. Want to see a list of users who follow you but whom you don't follow? Run t groupies. You can also use that command to check on other users, t groupies username.

The most interesting thing about t is that it works well with other *nix commands. For example, you can create a list (t list create listname) and add all users who are followed by another user (t followings user | xargs t list add listname. That's just a small sample. You can use any of the standard *nix utilities with t, like uniq, which might be useful in generating lists of users to follow by examining users that other people follow (and much more).

You may bump up against arbitrary limits when adding people to lists and such. According to the Twitter API docs, you can only add 100 users at a time to a list, and a list can only have 500 users. When I tested the limit, though, I was able to add 195 users to a list with one shot. The operation failed (sort of) when I tried to add another large chunk of users (more than 400), leaving my list with 582 members. To put it another way, trial and error is called for.

Some of the t commands also support a --csv option, which you can use to dump output to a comma-separated value file and then import it into a spreadsheet.

The Bad

While t has a lot to offer, it also has a few problems. First and foremost, a few commands don't quite work. For instance, the t stream timeline command is supposed to provide an on-going stream of your timeline. It provides the first 20, and then errors out. The "matrix" command, likewise, returns an error. The "trends_locations" task doesn't seem to be implemented at all, as it returns a "could not find task 'trends_locations'" error.

The error messages, in general, are pretty minimal. You may be notified "too many terms specified in query" for instance - but no information on what part(s) of a command succeeded or failed. I've seen a few operations that appeared to mostly succeed, like adding users to a list, but received an error notice. When that happens, it's not trivial to figure out what worked.

Sparse documentation is another problem. Many of the commands have options that are not well-explained. You may need to go spelunking through the code to see what options are available.

Finally, it's pretty easy to hit Twitter's rate limit. Twitter has an hourly limit of 350 requests per hour with an authenticated client. You can bump up against the limit if, for example, you try to request a list of all users for an account with tens or hundreds of thousands of followers. Note: If you wish to check on your current rate limit, use t rate_limit. You'll see the current limit (should be 350) and how many remaining hits you have, and the time your limit will be reset.

For the most part, though, t works quite well and provides a great way to do more with Twitter than the usual clients allow. If you're doing more than idle chit-chat on Twitter, take a look at t.

Computer Programming for All: A New Standard of Literacy

Dan Rowinski — Thu, 17 May 2012 17:30:00 -0700

Everyone ought to be able to read and write; few people within the global mainstream would argue with that statement. But should everyone be able to program computers? The question is becoming critically important as digital technology plays an ever more central role in daily life. The movement to make code literacy a basic tenet of education is gaining momentum, and its success or failure will have a huge impact on our society.

The democratization of literacy in the late 19th century created one of the great inflection points in human history. Knowledge was no longer confined to an elite class, and influence began to spread throughout all levels of society. Any educated person could command the power of words.

What if any educated person had equal sway over the power of machines? What if we were to expand our notion of literacy to encompass not only human languages but also machine languages? Could widespread facility in reading and writing code come to be as critical to society as the ability to manipulate spoken and written language?

The usual definition of computer literacy stops at the UI: If a user knows how to make the machine work, he or she is computer-literate. But, of course, the deeper literacy of the programmer is far more powerful. Fortunately, computer languages and human languages are basically very similar. Like human languages, computer languages vary in form and character (Python to Java to Ruby) and can be implemented in infinite ways. My Python may not look like your Python, but it can do the same thing; likewise, a single idea can be expressed using a variety of combinations of English words. And both kinds of language are infinitely flexible. Just as a person literate in English can compose everything from a sonnet to a statute, a person literate in programming languages can automate repetitive tasks, saving time for things only a human can do; distribute access to systems of communication and control to large groups of people; and train machines to do things they've never done before. Computer programming already does marvelous things like deliver this article to your mind, operate life-sustaining medical devices and enable IBM's Watson to win at Jeopardy. The current potential for innovation would be many times greater if every schoolchild had a firm grasp of programming concepts and how to apply them.

Among programmers, a movement is forming around this idea. Shereef Bishay, founder of San Francisco-based Developer Bootcamp, believes that coding is destined to become a new form of widespread literacy within the next 20 to 30 years. Everybody should learn to code, he says, because machine/human and machine/machine interaction is becoming as ubiquitous as human/human interaction. Those who don't know how to code soon will be in the same position as those who couldn't read or write 200 years ago.

300 years ago, Bishay said, "you would have to hire to write a letter for you, and hire them to read the letter for you. It is just insane." Today most people hire a skilled programmer to write computer programs for them.

The code literacy movement began to gather steam in late 2011, when Codecademy started teaching basic programming skills for free. The debate came to a head this week as two blog posts took the top spots on the tech website Hacker News. The first, dubbed “Please Don’t Learn to Code,” came from noted developer and StackOverflow.com creator Jeff Atwood on his blog Coding Horror. The second, a rebuttal entitled "Please Learn to Code," came from Sacha Greif, a Parisian designer whose clients include HipMunk and MileWise.

“I do think (or at least, hope) that computer programming will become the next version of literacy,” Greif wrote in an email to ReadWriteWeb. “When I watch my 4 year old niece interact with an iPhone, I see her intuitively using interaction patterns that older people often have trouble with, even when they're computer-literate. And kids can easily memorize huge quantities of facts about complex abstract systems like Pokemon games. So clearly they have the potential to learn how to code.”

Not everyone in the programming community agrees. Atwood argues that verbal literacy is a different kind of skill, and more fundamental. “Literacy is the new literacy,” he told ReadWriteWeb. “As much as I love code, if my fellow programmers could communicate with other human beings one-tenth as well as they communicate with their interpreters and compilers, they'd have vastly more successful careers.”

Atwood stresses learning, and mastering, the basic skills of communication. Learn to read. Learn to write. Learn to hold a conversation. Learn some basic math. These skills, he says, are more essential than being able to program a computer.

Of course, the path to universal code literacy is not without roadblocks. The skills necessary depend on how computing evolves over the next several decades. How will quantum computing affect our relationship with computers? However, the human capacity to learn is not at issue. If it becomes necessary for me to code to interact with my machine, I will likely learn to code. It is no different than if I was dropped off in Cambodia without a place to stay or food to eat - I'd learn the local language posthaste.

At present, the ability to program computers is vocational, like carpentry or learning to cook. There's little impetus to make it universal. But imagine if it were.

Should computer programming become the new literacy? Or should it remain a vocation? Let us know in the comments.

Images courtesy of Shutterstock

Google Easy Dashboard Library Makes Using Analytics API Easier

Joe Brockmeier — Thu, 10 May 2012 15:30:00 -0700

Google has long provided an API for automating Google Analytics, but it required developers to jump through a few more hoops than many would like. Yesterday, the company announced its Easy Dashboard Library, which should let developers speed up custom-tailored dashboards and reports.

Prior to the library, getting things out of the Google Analytics API wasn't a trivial process. According to the post announcing the feature, developers had to learn the API, then figure out how to handle authorization, and then meld the data with another visualization library. Fun for some developers, but definitely not speedy. And when has management ever said "we'd really like a better dashboard for our Web traffic, but take as long as you like to come up with it"? Right, so something better was obviously needed.

The Easy Dashboard Library

Google worked with students at the University of California, Irvine to come up with something better. The Easy Dashboard Library has three basic steps: Set up API access with an OAuth 2.0 client ID; copy and paste some code; and configure the code to query data and select your chart type.

The post from Google demonstrates how to create a quick-and-dirty chart for pageviews, visits and visitors over the last 30 days. You can test out a demo on Google Code, where the library lives.

Developers have the option of using line, bar, pie, table or column charts. The demo shows a pretty simple query, but developers can set queries using all kinds of dimensions and metrics supported by the Core Reporting API. It looks like developers can also use the data with another chart API if they prefer.

If you still don't think it's quite easy enough to use or would like to see additional features, you may be in luck. Google is planning to work with another group of students at the university for the next three quarters. The main goal is simplifying the library, but Google is also encouraging feedback via the Google Group for the Easy Dashboard Library.

Given the prevalence of Google Analytics, this should be good news for a lot of developers. It should also make custom dashboards more accessible even to more casual users, who might not have been eager to spend the time needed to get up to speed with the Analytics API but can ramp up pretty quickly with the Easy Dashboard Library. If you've taken a stab at using it, or have any recommendations for working with Analytics data, let us know in the comments.

Why Is Microsoft Trying to Hobble Firefox on Windows 8 Tablets - and Why Does It Matter?

Joe Brockmeier — Thu, 10 May 2012 13:00:00 -0700

As Windows 8 approaches, Mozilla developers have been working hard on a Metro version. If you're using Windows 8 on the desktop, no problem. Tablet users, however, are going to be denied a fully functional Firefox - and will face restrictions on many other third-party applications. In the name of security, Microsoft is forcing them into a "sandbox" on ARM devices. The lockdown renegs on the company's prior promises, and it's going to have some far-reaching effects on many applications.

Mozilla's Asa Dotzler touched on this issue yesterday, saying that Microsoft "is trying to lock out competing browsers when it comes to Windows running on ARM chips." But it actually goes farther than that.

Microsoft is restricting access to some APIs on ARM-architecture devices that are, as Dotzler says, "absolutely necessary for building a modern browser that it won't give to other browsers so there's no way another browser can possibly compete with IE in terms of features or performance."

Dotzler is focused on the implications of Microsoft's win32 API restrictions on ARM because they affect Firefox. This makes sense because Dotzler works for Mozilla and focuses on Firefox in general, not to mention Microsoft's long history of anticompetitive behavior towards third-party browsers. Make no mistake, though: Limiting access to the win32 APIs is likely to impact many other applications as well. How can LibreOffice or Apache OpenOffice compete with Microsoft Office if they're shut out of the win32 APIs?

In the Name of Malware

Microsoft is getting cut a lot of slack for its anticompetitive stance, because it is casting the anti-features for developers in the name of "protecting users from malware." It's OK if Microsoft cuts off competing applications at the knees, because it's trying to prevent malware.

Leaving aside Microsoft's intentions - perhaps it truly is motivated only by the best interests of users - this argument fails on a number of levels. First, it assumes that Microsoft's own applications won't be exploitable. Given Microsoft's history with security, this isn't likely. Why does Microsoft get the assumption of secure applications, while third parties do not?

And let's not forget who got us to this juncture in the first place. Microsoft users have been worn down by more than a decade of security issues that trace back to Microsoft itself. Microsoft is essentially using its own failings to excuse its blocking of third-party apps that may well have better security than its own applications.

Sandboxing third-party apps into limited parts of the machine does nothing to ensure that Microsoft's own browser won't be ownable by malware. Since Internet Explorer code isn't open source, security researchers can't audit the code directly. Firefox, which can be independently audited, won't be available on the new ARM tablets.

Why Not Complain About Apple?

Some folks have tried to dismiss complaints about Microsoft's ARM policies by pointing at Apple. Since Apple also discriminates against developers on iOS, why shouldn't Microsoft?

Yes, Apple's iOS developer policies suck, but they've sucked since the operating system's inception. What's more, there's little chance that Apple is going to change its policies unless users start abandoning iOS, or there's some sort of legal interference. Given that it'd be hard to make a case that Apple has a monopoly, legal interference seems unlikely.

That doesn't mean that third parties should just shrug their shoulders and accept the same treatment from Microsoft. If Microsoft is successful in the tablet market, ceding the Windows 8 ARM tablets is going to be a big loss for third parties. Loss of one platform is difficult, but being shut out of two tablet platforms in a three-horse race is going to spell major problems for Mozilla.

Dotzler distinquishes between a tablet OS and a general-purpose OS, though. Right now, at least, iOS is just for phones and tablets. Firefox can still compete with Safari on Mac OS X. Whether the distinction really makes sense, I'm not sure, given iOS' dominance on tablets so far.

But Windows 8 is not tablet-only. As Dotzler points out, tablets may be a "tiny sliver" of the PC universe now, but if you're looking ahead, that's not going to be the case in a few years. "ARM will be migrating to laptop PCs and all-in-one PCs very quickly," he says. "If you read Microsoft's blog posts about Windows on ARM, you'll see that they expect ARM PCs to cover the whole spectrum. ARM chips are already being used in servers. This is not a tablet-only concern."

Tablets and Tightening

If trends continue, users will do much more computing on their tablets and phones in the future. Even if this was a tablet-only issue, it'd be worth standing against. The amount of lockdown being exhibited on tablets is troubling, to say the least.

Giving Microsoft (or Apple) so much control over what applications run on their platforms is not good for developers or users. It should be assumed that users have control over their computing devices, and that means having the option to choose their own applications for Web browsing and everything else.

It's not at all puzzling that Mozilla is complaining about being shut out of Windows 8 tablets. What's puzzling is how many developers and industry pundits are willing to give Microsoft a pass.

US Government Has More "Big Data" Than It Knows What to Do With

Scott M. Fulton — Thu, 10 May 2012 09:29:00 -0700

It's a valid question: "Why has all the data the government has been collecting turned out to be too big to handle?" The results of a U.S. and state government IT survey released this week by the public sector IT community MeriTalk sheds a bright, halogen spotlight on the answer: It's because it's being collected in an unfiltered format and is waiting for someone - anyone - to claim it and write viable applications for it.

If you've followed along with RWW's expanding coverage of technology trends in the public sector, you'll recall we take an occasional look at the MeriTalk survey, which is an ongoing dialogue with IT professionals in the U.S. federal government. In a report released Monday summarizing the views of 151 respondents, only 60% say they capture "big data" to any degree. All respondents in that segment said they then analyze the data, but when asked to qualify what that meant, fewer than half (49%, or about 44 respondents) said they extrapolate any meaningful trends from it. Some 28% of respondents collecting big data say they don't do any collaboration with other agencies whatsoever to discover what it all means.

But that may not be the whole picture, as 31% of the IT professionals polled whose agencies are affiliated with the Department of Defense or with intelligence (about 23 in all) say they cannot even discuss the subject of big data. Among all agencies polled, approximately 52% say they are in the learning stage about what they can actually do with all this data. But when asked how long this phase should be expected to last, the average response was three years!

What will happen to government big data during that time? MeriTalk's calculations conclude that the agencies it polled currently store some 1.61 petabytes (quadrillion bytes) of total data, both structured and unstructured. (MeriTalk estimates that 31% of the data collected by agencies polled is actually unstructured.) In an astounding conclusion based on its respondents' estimates, MeriTalk predicts that the rate of expansion of public-sector data stores is about 0.51 petabytes per year. So by the time these agencies have completed their assessment of what to do with all the data they have collected throughout their history, the size of those data stores will have nearly doubled.

[Source: MeriTalk Government IT Network]

Based on the rough conversion table provided by WhatsAByte.com, we calculated that if each individual agency polled was to store all the data it's currently collecting on paper, the (very) old-fashioned way, it would need to requisition at least 20 four-drawer filing cabinets every minute of every day simply to contain it all. When asked to select the top three challenges to being able to manage all this data from a long list, 40% of respondents checked storage capacity, 36% being able to distribute and share it, 35% being able to query the data in any form and 34% said having enough time to actually process the results.

Of those who could accurately estimate their current big data storage capacities, some 57% said it was already too late: The infrastructure is not in place for them to be able to work with what they have, and that includes cloud capacity.

What appears to have happened is this: During the past few years, once it became feasible for government agencies to begin amassing data from Internet-based sources rather than through direct collection or communication, they deployed the first generation of big-data tools immediately, with the idea that they'd plan for how to use it later. The result was like the plan for how to pay for the first wave of tax credits in 2001: We'll do it now, and figure out how to pay for it when the time comes. The trick, of course, is to know when that time comes, but now more than half of federal IT workers speaking with MeriTalk say that time has already passed.

Lead image courtesy of Shutterstock.

Bye, Bye Waterfall: 5 Steps to Implement Responsive Web Design

Travis Sheppard — Thu, 10 May 2012 06:05:00 -0700

Congratulations, you’ve finally convinced the powers that be that your next Web-design project needs to be responsive. It was tough work convincing them, but you can’t rest on your laurels now. The most critical decision of the project remains: how is your team going to build it?

Guest author Travis Sheppard is VP of Technology at BGT Partners, a 2010, 2011 and 2012 Ad Age Top 15 Best Place to Work in the U.S. BGT creates interactive marketing and technology solutions for global corporations that strengthen brands, develop more engaging relationships and transform businesses. For more information on BGT, please visit www.bgtpartners.com, www.facebook.com/bgtpartners and www.twitter.com/bgtpartners.

Responsive Roundup

You're probably familiar with the typical "waterfall" process: start with strategy, then onto design, followed by front- and back-end development and finally analytics implementation. The main attribute of the waterfall process is its linear nature: once a stage ends, the next one begins with a near-perfect dovetail. The progression through the stages only goes in a single direction, and unfortunately as problems accumulate, they inevitably flow downhill and changes are not handled very gracefully.

Waterfall process:

If you’re unfamiliar with the term "responsive Web design", it describes a website-building process that utilizes the same set of code to display properly on desktop, tablet and smartphone browsers. Gone are the days of creating entirely separate websites in parallel desktop and mobile versions. Now you can construct an extremely flexible website to handle all environments.

Responsive approach:

So why implement responsive design over the waterfall process? One of the biggest disadvantages of the waterfall is that little thought is given to design and development for environments beyond the standard desktop browser. An agile approach with responsive design considers these issues from the start, but will require more upfront wireframing, design and testing that are omitted in the waterfall process. Once implemented, your site will render properly on desktops, mobile devices and tablets.

Responsive approach:

So how can you implement responsive Web design in your organization? Below we’ll review the steps of the typical waterfall process and explain how they can become responsive.

How to Implement Responsive Web Design

Step 1: The Plan

Waterfall Process

In the waterfall process, wireframes are built and consist primarily of layouts and widgets. They’re set to a specific size (usually pixel-based), and have little room for flexibility. These wires promote very specific grid/layout sizes, but when the layout changes due to different screen resolutions, things will shift in transit. The results are that navigation menus become unusable, forms become inaccessible and your interface is rendered inadequate.

Responsive Web Design

The fix for this problem isn't so difficult. You’ll need to design widgets for different views, and not think of a page as a "page." Pages are not atomic units - instead, the sliders, content, forms and other components are atomic pieces making up the whole. Wireframes must represent different screen sizes, and therefore layouts must be fluid. They can go from three columns to two, and perhaps scale down into a single column for the smallest displays (mobile smartphones).

You’ll need to change the user experience as well - a slider cannot only be controlled by a mouse, but might require a user’s finger for interaction on smaller screen sizes. Wireframes need to become prototyping tools rather than blueprints, and some development and testing is needed to ensure they’re fully functional across the display spectrum. If design commences prior to this initial testing, then unknown development issues may arise at a later point. Regardless, the ultimate vision for the project must be sustained, so keeping lines of communication open between departments is essential.

Step 2: Design Time

Waterfall Process

In the waterfall process, the next step would be handing off the wireframes to design, and breathing life into them via fonts, colors, spacing and other tools of the craft. Oftentimes, there is some back and forth on design direction, with updates to the design comps made as more knowledge about the brand and its design guidelines surface.

Responsive Web Design

To make better use of allotted project time and resources, design should include a few layouts and widgets at different sizes. Responsive Web design means letting go of pixel-perfect designs. Making those designs work on desktop browsers is challenging enough, but when we think in terms of flexible widgets on a flowing grid, the number of designs needed becomes manageable.

Let the medium of HTML enhance the qualities of the design using a fluid layout in all environments. Creating the states for each browser width is a huge waste of time - instead focus on the totality of the user experience. For example, ensure that the atomic components of a rotating banner are touch-sensitive on smaller mobile devices, and use an industry-recommended minimum size of 44 pixels to allot for the typical human fingertip size. The design of the experience is just as important as the look of the site on all screen sizes.

Step 3: Build It

Waterfall Process

With the waterfall approach, once the designs are approved by the client, front-end development ensues where issues regarding rendering on smaller screens could arise. Unfortunately, due to the linear nature of the waterfall, these unforeseen problems appear well into the project progression.

Responsive Web Design

In our agile and responsive process, the design must live on a flexible grid. The widgets need to be planned out and prototyped by the developers, and they need to be tested along the way. The code also needs to be optimized to ensure that the widgets are the smallest possible unit. The widgets can be easily inserted and removed from layouts that were not originally planned for, and testing these options grants peace of mind. Constant collaboration between the developer, designer and strategist circumvents issues with the inevitable changes. With the different members of the team on the same page, problems are identified and resolved earlier in the process.

Step 4: Testing Your Patience

Waterfall Process

The final stage in a standard waterfall process is to evaluate the site via unit and functional testing methods. Issues discovered at this stage may require the original vision to be tempered, and sometimes a new device that’s just hit the market can throw a wrench in the works. The strategy and design team members, who may have begun work on new projects, must be brought back in the loop to accommodate these changes, and more time must be spent on updates.

Responsive Web Design

In the responsive process, you need to test along the way on multiple browsers and screen sizes so any problems are revealed early on. Issues with the mobile environment, which don’t coincide with the wireframes, can be recognized, as well as the capabilities of the design on a number of different platforms. A working project prototype will be ready early on allowing clients to review sooner, making it a win-win for all parties.

Step 5: Rinse, Lather, Repeat

Waterfall Process

The traditional waterfall process doesn’t have a step where you iterate through the designs and the interface. Passing over minor details at certain stages of the project construction permits issues to arise and client-expectation conflicts. Despite consistent and timely communication with the client, until a presented working model is shown, the full gravity of these accumulated poor decisions is not known.

Responsive Web Design

With the responsive approach, the same amount of progress is attained and, as a bonus, there is live code for client presentation every step of the way. The discoveries made at these earlier stages will help drive subsequent stages and anticipate critical changes ahead of any deadlines.

In Summary

Adopting an agile responsive approach will free you from the constraints of the waterfall process. Your design and development will be streamlined, you’ll be more productive and efficient and your online brand presence will be maximized on all possible platforms and screens. The real challenge is breaking out of the waterfall mold and becoming a responsive organization. Follow these five steps and you might just say "bye, bye" to the waterfall and "hello" to responsive Web design.

Waterfall photo courtesy of Shutterstock.

Improvements in New York Times' Fech Makes It Easier to Follow the Money

Joe Brockmeier — Mon, 07 May 2012 17:15:47 -0700

Having data available electronically is not the same thing as the data being useful. Campaign finance disclosures provided electronically by the Federal Elections Commission (FEC), are a good example of that. The New York Times's Fech (not "fetch") is a RubyGem - a packaged application - designed to help journalists and public interest organizations access and make sense of FEC filings.

Here's the NY Times' description of Fech from its first release last year:

Journalists who work with these filings need to extract their data from complex text files that can reach hundreds of megabytes. Turning a new set into usable data involves using the F.E.C.'s data dictionaries to match all the fields to their positions in the data. But the available fields have changed over time, and subsequent versions don't always match up. For example, finding a committee's total operating expenses in version 7 means knowing to look in column 52 of the “F3P” line. It used to be found at column 50 in version 6, and at column 44 in version 5. To make this process faster, my co-intern Evan Carmi and I created a library to do that matching automatically.

Fech (think “F.E.C.h,” say “fetch”), is a Ruby gem that abstracts away any need to map data points to their meanings by hand. When you give Fech a filing, it checks to see which version of the F.E.C.'s software generated it. Then, when you ask for a field like “total operating expenses,” Fech knows how to retrieve the proper value, no matter where in the filing that particular software version stores it.

Derek Willis of the NY Times announced the 1.0 release of Fech last month. This release covers "all of the current form types that candidates and committees submit." Perhaps most importantly, this release allows comparing two filings against one another.

Why Fech Matters

Fech is already being used by the NYT for its reporting and interactive visualizations of campaign spending. But that's just one editorial team. Putting this tool in the hands of any developer or reporter that wants to work with the data opens a lot more possibilities.

For example, there's ProPublica, which is using Fech and the NY Times' APIs for its reporting and interactive graphics. ProPublica is able to show not just what campaigns are spending, but how much and with whom. (So far the biggest winner is Mentzer Media Services, an ad agency that specializes in GOP campaigns - including the Swift Boaters. Fech doesn't automatically point that out, of course, but it helps journalists uncover it.

Data without context is useless. By helping developers and journalists work with the filings in a more structured way, Fech helps newsrooms (or any other group) put the data in context to find the story behind the data. It's a long way from being simple to use, but it represents a significant improvement over the raw data. It's Apache-licensed, so it might find its way into all kinds of data analysis tools over time.

With Fech maturing well before the elections this fall, it could help all kinds of organizations follow the money trails much more efficiently. Here's hoping that happens.

A Gaming Replacement for Those Annoying CAPTCHAs

David Strom — Thu, 03 May 2012 15:00:00 -0700

We all know about those authentication blocks of text called CAPTCHAs, perhaps too well. (Today's fun trivia: The acronym stands for Completely Automated Public Turing Test to Tell Computers and Humans Apart.). A new idea from PlayThru is to embed a small Flash or HTML5-based game that a human plays with a mouse to prove he or she really is a carbon-based life form. It is intriguing, potentially less annoying, and has captured (if you will excuse the pun) a few supporters already. The service is just getting started, and it is free to try out.

The CAPTCHA process was developed in 2000 by several computer scientists at Carnegie Mellon University. It has since spread like kudzu to various websites, where site owners try to prevent automated bots from bombarding their pages.

The original thought of Alan Turing with his namesake test was to develop some way for a human to tell when it was talking to a computer. The CAPTCHA is actually this process in reverse: It is administered by a machine, but tries to distinguish humans. It hasn't been working all that well, though. Many spammers employ a variety of techniques to defeat them, such as by paying low wages to actual humans or running optical character recognition software to ferret out the CAPTCHA codes.

In the process, as so often happens with the Internets these days, the bad guys are making for a miserable user experience for the rest of us. The codes have gotten harder to read by ordinary humans, and many users will abandon a Web page rather than try to enter the right code. Updates such as user refreshes to get a new code or audio translators haven't helped much.

Enter PlayThru's attempt. Their service, perhaps the first game-based CAPTCHA, invites users to solve a game by identifying and interacting with dynamic objects, such as dragging and dropping toppings onto a pizza or food items into a refrigerator. You can see an example here.

Beta deployments of PlayThru are seeing submission rates increase by up to 40% over text-based CAPTCHAs, and in a company-sponsored survey of 100 people, 98% of the users preferred PlayThru over traditional text-based CAPTCHAs. Granted, that isn't a scientific sample, but it's still an indication of how much we all hate the regular CAPTCHAs, and of how much opportunity there is for their replacement.

PlayThru isn't the first company to invent a better CAPTCHA. Oregon-based Vidoop came out with their own innovation a few years ago, but it hasn't caught on.

But this just illustrates the problems in fighting spammers and still making our computer systems usable for the rest of us who just want to go about our business and get work done. The spammers always seem to have ways to defeat the latest technology, no matter how sophisticated. The PlayThru game-based CAPTCHAs could turn into a miniature Space Invaders or World of Warcraft as the automated tools used by the bad guys get better, which would make the simple originals useless. In the meantime, though, try out the demo on PlayThru's site, and let us know what you think of the idea.

BUSTED: Security Exploit Exposed by Skype “Tool” Not a Hole

Scott M. Fulton — Tue, 01 May 2012 12:02:27 -0700

News of a potential security leak in Skype’s network protocols may be overblown, an investigation by ReadWriteWeb reveals today. Though it is possible for a program to expose IP addresses that have, at some point in history, been utilized by Skype users, this particular program is not Skype itself or anything that exploits a flaw in Skype.

Rather, it’s a separate, nonendorsed, reverse-engineered form of Skype 5.5. Though the reverse engineering project responsible for this program calls itself “open source,” in actuality, its proprietors appear to have merely de-obfuscated Skype’s proprietary code, made adjustments to it, and recompiled it. Those adjustments can, we discovered, produce human-readable IP addresses.

Underexposure

The commercial Windows-based editions of Skype are capable of logging details of complete user sessions (minus the dialog). Third-party developers who partner with Skype can make two adjustments to their System Registries, as explained on this Skype page, to produce log files that can be shared with Skype’s own developers. However, those files are encrypted so as not to be legible even by the developers. RWW confirmed this fact in its own tests this morning.

The project calling itself “skype-open-source” produces an executable file that demonstrates the use of reverse-engineered code to send a test message through the Skype network. (So as not to further infringe upon intellectual property, ReadWriteWeb will not post links.) This particular file is not the tool in question, and in our tests, could not be used to leak IP addresses. (One form of the tool, however, does appear to be advertising itself as a manipulator of Skype credits.) However, one of the contributors to the project, whose handle is Vilko, did post a link to his own privately hosted package, which contains both the binaries and de-obfuscated source code for a recent version of Skype.

Once the Registry changes suggested by Skype to its developers have been entered, Vilko’s program produces a nonencrypted version of the log file that is completely human-readable. A small portion of one of these session logs is shown above, with potentially identifying data blurred. Note the PresenceManager keyword, which Skype uses to log the use of its own tool for querying the availability and status of a contact.

An examination of these session logs does reveal - as any engineer might expect - that IP addresses are associated with Skype users. As the program retrieves entries for individuals in the user’s contacts list, the log file shows, it retrieves an IP address that is apparently the beginning of a known route to an address the user has employed at some point in history. Our research indicates that this history may go back several years, meaning that a user may have moved to a different city any number of years since an address was first catalogued.

What’s more, the address that leads to a known Skype user, after a number of hops, may or may not be an IP address of the user’s direct Internet host. Several of the IP addresses we found in our research are traceable to Microsoft, though the users we traced are not Microsoft employees, nor do they use Microsoft’s servers. (Microsoft is presently the owner and operator of Skype.) My own Skype data, for example, which I located in the file, traces to a Microsoft server and not to my own company’s network.

So it may be inaccurate to say that even the reverse-engineered program exposes users’ IP addresses, but rather that it exposes addresses that enable Skype servers to make contact with them. In and of themselves, IP addresses are not personally identifiable data, although researchers have demonstrated that, with access to certain databases, it may be easy to deduce users from addresses they have used.

Damage Control

In a recent message, the “skype-open-source” project’s owners suggest that their users try looking up any Skype username. The act of doing so triggers a search for the user’s active status, which is then recorded in the log. That log entry will contain two IP addresses: often one associated with the ISP, and another used for internal routing to the loop associated with the user. Though legal experts have said this is not necessarily personally identifiable data, law enforcement officers have sought this level of information in obtaining evidence on suspects’ Internet activities.

The fact that the “skype-open-source” project first bore fruit in June 2011, and the first revelation by the project’s owners of its unencrypted logging was made just last week, suggests that even the project’s self-described developers may not know exactly what Vilko is up to. On their blog, they congratulate Vilko for his contributions and quote a message from Vilko describing how the de-obfuscated source code may be compiled using Microsoft Visual C++.

As long as code for encrypting one’s log files exists within a program that can be de-obfuscated and reverse-engineered (and many can be), it may be trivially easy for someone like Vilko to simply omit the encryption routines and recompile. In a statement released to the press, a Skype spokesperson contended that problems such as this are “industry wide.” One short-term solution for this problem may be for Skype to change its encryption scheme for its log files, although conceivably such a solution may be very short-lived indeed.

A longer-term solution may be for Skype and/or Microsoft to develop an entirely new logging system, whose files are never human-readable even in their unencrypted form. It may take more skill to decode such a system - skill which the ordinary de-obfuscator of other people’s code may lack.

Confirmed: Too Many Open Source Projects Remain Arbitrary Mashups

Scott M. Fulton — Fri, 27 Apr 2012 11:02:00 -0700

One benefit many developers perceive from working with a proprietary platform is that its components are generally updated and deployed according to a single agenda. (Sometimes that agenda is so slow that this fact becomes undeniable.) By contrast, the tremendous pace of open source development can be overwhelming for some. Back in 2007 the Open Source Census noted one big problem, especially among enterprises: Development teams have continually demonstrated an inability to manage the multitude of dependencies that OSS components have upon one another, and to use the proper or most secure components for their projects. A new Sonatype survey confirms that things haven't really changed that much since then.

Sonatype, producers of an open source repository manager called Nexus, surveyed some 2,550 information technology workers. Among those surveyed (52% software developers or engineers, 22% software architects, and 13% team leads or project managers), 52% say their organizations are standardizing on some form of open source development infrastructure stack - a 3% gain over last year's survey (which had about half as many participants).

Drilling down, however, only 20% of respondents say their organizations’ standards have evolved to the point where developers use only approved OSS components in their projects. The remainder either work with standards that aren’t actually enforced, or work in environments where developers choose whatever components they feel are best at the time.

The split last year was 13:87, indicating an improvement among at least 7% of respondents. But that uptick pales in comparison to the embrace of Twitter or of RESTful APIs.

So how are developers finding the components they use? Among the developers surveyed, some 74% pretty much type something into Google and see what pops up. Approximately 42% ask their colleagues for advice, knowing that they probably do a Google search, as well. About two-thirds read the websites for prospective components to help them make their decision, although - as Sonatype points out - such sites often don’t help developers iron out dependency issues.

“Open source has become the backbone of custom application development. Yet it brings with it a complex environment ecosystem with no notification infrastructure in place,” reads a statement from Sonatype CMO Charles Gold. “The compounding reality is that when issues do arise, the effects are viral when the fixes are not.”

Five years ago, OpenLogic released a free tool called Discovery, hoping that developers integrating OSS components would use it to inventory what’s installed in their systems. That tool helped compile the first comprehensive data about open source components in use and how they’re being managed. Since that time, with the almost total shift to cloud-based development, OpenLogic moved the tool into the cloud, deploying it as OpenLogic Exchange (OLEX).

Now Sonatype has picked up the mantle of surveying enterprises, offering a bit more insight into the structure of organizations that are responding. The larger the organization, the more likely it is to use a repository manager, with 75% of respondents from organizations with more than 500 employees saying they have installed such a device (whether or not they make good use of it).

But who makes sure these repositories are being used properly? According to Sonatype’s survey, perhaps it’s the wrong people: Only 28% of respondents say that application development managers are responsible for application development - just greater than one-fourth managing governance of OSS component deployment by the people who actually do the job. The remainder say governance may be handled by a vast array of other departments, as depicted in the chart above.

While Sonatype (as might be expected) states the survey reflects the desire for a tighter notification infrastructure, that’s the same message that the Open Census sent back in 2007. If it’s a desire, it’s not a strong enough one to produce the needed changes in behavior.

Charts credit: Sonatype 2012 Open Source Software Development Survey

Inside IBM’s Huge “Big Data” Buying Spree

Scott M. Fulton — Thu, 26 Apr 2012 16:30:00 -0700

Haven't we seen this before: IBM making a big acquisition of a “V” company in the big data space? Indeed, last April 13, IBM purchased analytics software maker Varicent for an undisclosed amount. And then this morning, IBM announced its acquisition of enterprise search facilitator Vivisimo.

IBM usually doesn’t acquire applications software makers unless it has something very specific in mind for them. IBM is clearly piecing together a “big data” platform - a comprehensive package for storing, accessing and analyzing unstructured data.

That IBM wants to be perceived as the master of data stores is no surprise. What’s interesting here is that the company is taking an almost Oracle-like approach to the subject, buying up as many as 30 separate big data-related entities in recent months (Vivisimo being #30) and wedging them together in hopes that they form a picture that looks something like a platform.

The man in charge of stitching together the collected “big data” shards into a cohesive whole is Arvind Krishna, the general manager for IBM’s information management division. Big data, Krishna informs us, is big. But that’s really just characteristic number one of three. The second most important characteristic is its variety of formats, and the third is its velocity. (Remember that word; it’ll come up again in just a bit.) These nebulous qualities along with the quickly mutating nature of big data are keys to what Krishna perceives as its most important phenomenon: what he calls its perishability. Understanding Krishna’s attitude toward this phenomenon provides the best insight into IBM’s platform-by-acquisition strategy.

“If you don’t react to it in a few minutes, it might not be worth it to react at all,” remarked Krishna in an interview earlier today with ReadWriteWeb (using words that tugged at my very heart strings). “The way we look at this, every single company does a great job of dealing with structured information. Examples of this are like payroll or sales data - what product did I sell at which store at which geography at what time? I can slice it and dice it and see areas of weakness and strength. But if I look at my interactions with my clients, not just my transactions, what they’re saying about me could be in social media, what they’re writing to me could be in email, what they’re saying to my call center reps... Do I really have a full view across all of that? And the answer today would be, not really, right?”

Vivisimo began in the late 1990s as a very public effort at creating a next-generation Web portal, applying sophisticated semantics to refine the power of queries - just before Google’s rise to prominence. As it became more difficult for Vivisimo or any company to succeed on a broad technological scale by simply making better technology (and its query system, which I helped test, was indeed that), it found itself scaling back its customer base to more reasonable, conquerable verticals, such as the customer service field.

This played directly into some of today’s headlines, which speculated that IBM was purchasing a customer service software provider. That’s not how Arvind Krishna described it. Indeed, there’s a piece of Vivisimo’s original, broader plan that plays into IBM’s bigger, broader plan today, having to do with the creation of ontologies - structures of subjects that are related to a broader topic.

“When I want to index a document, simple indexing would count all the words in the document and say, ‘These words appear so many times,’ and then relevance becomes a matter of whether words appeared more than once. Next, you could go to grammar structure and semantics, so you can do ontologies. Suddenly, the way that Vivisimo does document capture and classification allows them to leverage these ontologies and develop their semantics. That is what is really interesting about their indexing technology.”

Simple indexing becomes inefficient “at scale” - meaning that as document sizes scale up linearly, time to index and time to read the index scale up exponentially. That fact alone renders traditional database methodologies pointless for the waves of incoming text and media that broadband Internet has enabled. Despite having been marketed largely to the customer service sector, Vivisimo has in its arsenal a potentially very potent weapon: a kind of ontology development tool whose name just happens to be (I told you this word would come in handy) Velocity.

“Vivisimo Velocity can find, extract and deliver content regardless of format or where it resides,” reads a recent Vivisimo white paper (PDF available here). “Administrators have access to a library of pre-built Information Optimization connectors and a universal connect framework to enable indexing from common data repositories (e.g., file shares, databases, e-mail systems, content management and collaboration systems, customer relationship management applications, e-mail archives and other archival solutions). In addition, content from third-party feeds, subscriptions and other search engines can be added via federation.”

That’s another very important word in IBM’s emerging big data strategy, which Krishna characterized as being more flexible like a platform, as opposed to HP's Autonomy, which he portrays as more specific like an application. “[Calling it] indexing is almost a disservice to what this really does,” said Krishna. “It’s the ability to do rich indexing across a set of federated data sources at a scale that has not been done before and, in addition, preserving the security semantics of the underlying data sources.”

What does Krishna mean by this last part? How Velocity responds to its user depends on its understanding of who that user is - more specifically, whether she’s entitled to the information she’s searching for. Vivisimo has a rule definition system that tailors responses to known restrictions about the user - a later generation of the system originally intended for the Web portal, which could apply responses to known characteristics of the user.

“Federation implies that I don’t need to actually copy the data; it can stay where it is,” added Krishna. “Security semantics means that, when someone tries to do a query, I can preserve the semantics so only those who are authorized to look at [the response] in the first place can get an answer that contains that information. And scale means, I’m not confined to a single machine or a single cluster.

“We are clearly acquiring technology which goes into a general platform that people can use to build custom applications and create an ecosystem of ISVs in addition to direct clients,” the general manager said. “We are also interested in bringing analytics to the big data space, in the form of a complete solution as opposed to having to write a custom answer for a client.”

Why Apple's WWDC 2012 Conference Sold Out Again - in Just 2 Hours

Alicia Eler — Wed, 25 Apr 2012 13:45:00 -0700

Tickets to Apple's Worldwide Developers Conference (WWDC) sold out in a record two hours this morning. The conference will be held in San Francisco from June 11-15.

"Sorry, tickets are sold out," the site says. It explains that Apple will post videos of the sessions shortly after the conference, making it available to the Internet public at large. While the event always sells out, this is the fastest conference sellout in Apple's history.

Last year, the WWDC sold out in 10 hours, which was a record at the time. In 2010, the event sold out in eight days. Last year's WWDC unveiled iOS 5 and highlighted features from OS X 10.7 Lion. This year, insiders speculate that Apple will focus on the new iOS 6 and details of OS X 10.8 Mountain Lion.

Even though the conference is technically sold out, there are still scholarships available to 150 student developers. The program is open to full- and part-time students who are 13 years of age and older. Kids must provide proof of school enrollment and be paid members of the iOS Developer Program, iOS Developer Enterprise Program or Mac Developer Program, or an iOS Developer University Program Student team member.

Apple's Q1 Earnings Reflect a Huge Surge in Popularity

The jump to a fast two hours is indicative of Apple's soaring popularity, and it comes on the heels of yesterday's blowout Q1 earnings. For comparison, Google sold out its Google I/O developers conference in just 20 minutes, though Google's event often includes thousands of dollars worth of free hardware for attendees.

ReadWriteWeb's Dan Frommer illustrated Apple's huge quarter in charts. Its sales have grown 59% year-over-year, making sales its biggest growth statistic. iPhones make up Apple's biggest business, with 35 million shipped during the quarter. Meanwhile, iPads account for 17% of products sold, trailing the iPhone's impressive 58% of sales. With its Q1 earnings alone, Apple could actually buy T-Mobile.

4 Cool Things You Can Do With Wappwolf and ifttt

Joe Brockmeier — Tue, 24 Apr 2012 05:20:30 -0700

Wappwolf and ifttt are a bit like the chocolate and peanut butter in Reese's Peanut Butter Cups. Separate, they're pretty good. Together, though, is when the magic happens.

For folks new to the services, Wappwolf is an application that performs actions when you drop a file into your Dropbox. ifttt, on the other hand, can interact with websites and services and then do things like send an email or save a file in Dropbox.

Let's take a look at a several ways to combine the services for maximum effect:

For example, Wappwolf can watch Dropbox and convert files to Kindle format and then send them to your Kindle. You could use this to save company documents for later reading on your ebook.

ifttt might watch Instagram or Flickr for photos, and save them to your Dropbox when they appear. You could use that to make sure you never miss a polar bear picture again.

Separately, they're pretty handy. Together, you can do some really interesting things.

(If you haven't tried Wappwolf or ifttt before, don't worry. They're dead easy to use. See the How to use Wappwolf Automator page, and/or the About ifttt page.)

1. Send a Text Message When Files Are Saved to Dropbox

Say you share a Dropbox folder with co-workers, and you want to be notified when there's a new file in the shared folder. ifttt only has triggers for new files in the public folder for Dropbox, while Wappwolf can take action based on pretty much any folder you give it access to.

Start with Wappwolf and create an action for the appropriate folder. Choose the "email it" action and set up an email to go to trigger@ifttt.com (be sure to set up your email address with ifttt). Set the Subject to #Dropbox Notify, and the body to the filename (or whatever you like). You can add whatever you like to the body, but you probably want to uncheck the "Add file as attachment" box.

Next, go to ifttt and choose the Email channel. Choose "Send ifttt an email tagged and use #Dropbox as the tag. Create the trigger and then go choose your action channel, in this case SMS. Choose "Send me a text message," and complete the task. Now when a file is saved in the Dropbox folder, you should get a text message.

Don't care for text messages? You could also use ifttt to send Boxcar notifications if you prefer that app to texts.

2. From Google Reader to Kindle

iPads are great for reading, especially with the Reeder app. But for reading longer text, nothing beats the Kindle. What we'll do here is save an article from Google Reader to a PDF, then convert that and send it to your Kindle. This one should be very easy, because ifttt already has a recipe for it, so you're halfway there. Just visit the recipe and create the task.

Next, head over to Wappwolf and pick the folder you specified, then tell Wappwolf to "Send it to your Kindle." Add the action, save, and you're ready to go.

3. Copy Files from ifttt to FTP Servers, Box or SkyDrive

ifttt supports saving files to Dropbox, but it doesn't support the breadth of services that Wappwolf does. If you want to copy files over to Microsoft SkyDrive, Box, or upload them to an FTP server, you'll need to combine ifttt and Wappwolf.

Create any action in ifttt that saves a file to Dropbox in ifttt, then select the folder you're saving to in Dropbox to create the Wappwolf action for saving to the appropriate service.

Of course, you're not limited merely to uploading files from one service to the other. Wappwolf lets you combine actions, so you can save a file to Dropbox, use Wappwolf actions on the file, and then upload it to its destination.

4. Process Files via Email

Wappwolf works on files only when they're saved to Dropbox. Usually that's not a problem, since Dropbox clients are available on just about every popular platform. But what if you want to use Wappwolf with files coming in via email? No problem. You can use ifttt to save to a Dropbox folder, then create the appropriate Wappwolf action.

This might be useful if you want, for instance, to email yourself a file using a chron job on a server. Just attach the file to the email, mail it to ifttt and have it saved to Dropbox, then use Wappwolf to perform whatever actions you like.

Patience Is a Virtue

Note that some patience is required in setting up tasks with Wappwolf and ifttt. It can take a few minutes for each service to perform its tasks. For example, the chain required to save a file to PDF via Google Reader and ifttt, then convert and send to Kindle, then for Amazon to ready it for the Kindle took about five to seven minutes. That's not a long time, but it's slower than some folks might be used to with most modern Web services.

Have any cool tips that you want to share with the rest of the ReadWriteWeb audience? We'd love to hear about any cool hacks that use ifttt and/or Wappwolf. Share and enjoy!

Using SPDY on Your Web Server

Joe Brockmeier — Thu, 19 Apr 2012 23:00:00 -0700

Google's SPDY protocol offers several advantages over serving traffic via HTTP/HTTPS. But, if you want to use SPDY, you're going to have to take a few additional steps to set it up. The good thing is that if you happen to be using Apache on recent Debian or RPM-based systems, installing and using SPDY is a simple matter.

Installing SPDY Packages

Right now, Google is providing binary packages for Debian or RPM-based systems with Apache 2.2.4 or greater. If your system is Debian or Fedora/Red Hat-based, you should have little trouble getting SPDY running. If not, Google also provides the source code for the SPDY module, and you can compile the module if necessary.

Before installing SPDY, you will need to have an SSL certificate to use SPDY since it uses SSL. This can be a self-signed certificate, though Chrome will give some pretty evil-looking errors when accessing a site with a self-signed certificate. For testing SPDY, this is merely a small annoyance. If you're actually looking to deploy SPDY to a production site, though, it's probably much better to go ahead and buy a certificate signed by a recognized authority. (There is a way to turn off SSL, but it's not recommended for anything but debugging.)

If this is the first time you've set up SSL on Debian-based systems, see the Debian Administration tutorial on configuring SSL. The Linode library has a Fedora 14 tutorial that should work with later versions of Fedora. See the CentOS guide if you're using RHEL, CentOS or another RHEL-based distribution.

Next step is to get the right package for your system. My server is a Linode VPS and runs Ubuntu 10.04 LTS, and it's a 32-bit system. The 32-bit Debian package worked just fine for that, but Google also provides a 64-bit Debian package and 32-bit and 64-bit RPMs as well. All of the packages are on the mod_spdy page under the Google Developers site.

You might run into a few snags if you're using an RPM-based system, like openSUSE, that isn't quite what the Google packagers were expecting. Nick Robinson has provided some instruction for getting the binaries installed on openSUSE 12.1 that may prove useful if the RPM doesn't work for you.

For Debian systems, you should use dpkg to install the module, like so:

dpkg -i mod-spdy-beta_current_i386.deb

Note that that's the current package name; it will likely change soon when Google decides to move the module from beta to stable. (One hopes this will happen faster than Gmail moving out of beta status...)

Google also says that you need to run apt-get -f install, but this had no effect on my system. Presumably, some systems require installing additional dependencies.

Next, you'll need to restart Apache for the new module to be recognized. The packages automatically add the spdy.conf and spdy.load files for Apache, so that SPDY will start automatically when Apache starts.

Testing

Once you have the SPDY module installed and enabled, use Chrome to visit a page on your site with HTTPS. You can go to chrome://net-internals/#spdy in Chrome to see whether SPDY is enabled, and chrome://net-internals/#events&q=type:SPDY_SESSION%20is:active will show live and active sessions.

I do recommend doing extensive testing before pushing a SPDY deployment live. So far, I've been having no problem with my WordPress blog, but I have found a few glitches with my Piwik install. Apparently, other users have encountered this as well. I suggest watching the mod-spdy-discuss list if you're working with SPDY.

If you decide you don't want to use SPDY until it moves out of beta, you can turn it off without needing to remove the packages. Open the spdy.conf file (under /etc/apache2/mods-available on Debian systems) and turn SpdyEnabled to off. You'll need to restart Apache for that to take effect.

There's talk of SPDY becoming an official Apache module (though it's written in C++ and official modules need to be in C), so it may be even easier to get SPDY in the not-too-distant future. Until then, it's still pretty trivial to install SPDY, and you should be able to have it up and running in less than 30 minutes.

What Web Users Need to Know About SPDY

Joe Brockmeier — Thu, 19 Apr 2012 04:07:00 -0700

Slowly but surely, SPDY ("speedy") is becoming more widely used. The Google-backed protocol, a modification to HTTP, is designed to help reduce latency and bolster security. Even if you don't manage a Web server yourself, you should know about SPDY and what it offers to you - and the Web at large.

SPDY has been in development for a couple of years, but a few things will start to accelerate its deployment.

First, Google has put out a SPDY module for Apache, which will make it much easier for organizations to deploy SPDY. Nginx is expected to have an implementation by end of May. That covers a huge chunk of the server market already.

Second, SPDY should be on by default in Firefox 13, and Chrome (and Chromium) already supports SPDY. Which means that organizations have more incentive than ever to start turning on SPDY.

What SPDY Is, and What It Offers

SPDY is a two-layer HTTP-compatible protocol. To break that down into more manageable terms, SPDY is like HTTP, but with additional features designed for today's Web. The "upper" layer provides HTTP's request and response semantics, while the "lower" layer manages encoding and sending the data.

The lower layer of SPDY provides a number of benefits over standard HTTP. Namely, it sends fewer packets, uses fewer TCP connections and uses the TCP connections it makes more effectively.

A single SPDY session allows concurrent HTTP requests to run over a single TCP/IP session. As Patrick McManus writes in SPDY: What I Like About You, it's great for high-latency environments "because a resource never needs to be queued on either the client or the server for any reason other than network congestion limits."

SPDY cuts down on the number of TCP handshakes required, and it cuts down on packet loss and bufferbloat. Says McManus, "SPDY's parallelism, by virtue of being on a single TCP stream, leverages one busy shared congestion control block instead of dealing with 36 independent tiny ones. Because the stream is much busier it rarely has to guess at how much to send (you only need to guess when you're idle, SPDY is more likely to be getting active feedback), if it should drop a packet it reacts to that loss much better via the various fast recovery mechanisms of TCP, and when it is competing for bandwidth at a choke point it is much more responsive to the signals of other streams - reducing the over buffering problem."

There's currently a lot of redundancy and bandwidth wasted in HTTP headers. SPDY compresses HTTP headers, which means that fewer bytes have to be transmitted between client and server.

All that adds up to serious performance improvements. According to Google's initial whitepaper on SPDY, you could see "a speedup over HTTP of 27% - 60% in page load time over plain TCP (without SSL), and 39% - 55% over SSL."

Security

In addition to better performance, SPDY is also more secure. Despite the best efforts of the Electronic Frontier Foundation (EFF) and others, we're a long ways away from HTTPS Everywhere - which means that most Web traffic is still sent unencrypted.

That will be a thing of the past with SPDY. Current implementations of SPDY mandate SSL, which isn't universally liked but seems the best way to nudge the Web forward to encrypting traffic most of the time.

Push and Hint

Finally, SPDY adds two new mechanisms that will also help speed up the Web. Server Push and Server Hints.

Just what it sounds like, Server Push will send resources to clients without being asked. If you request a Web page URL, for example, SPDY might also decide to send down images associated with the page even if they've not been requested yet. Note that there's a potential downside here, since servers could wind up sending redundant or unneeded content.

Server Hints doesn't send the content, but it does send the URL so that the client can decide if it needs it. If the content isn't cached, a browser or other SPDY client can then make the request a bit faster than it might have otherwise.

Getting SPDY

The only, or at least the major, problem with SPDY? You need SPDY support on two ends to make it work. You need a browser that supports SPDY, and Web servers that are delivering content using SPDY. If you're one of millions using Chrome, you already have SPDY support. If you're using Firefox, SPDY support will be the default with Firefox 13. Note that Firefox 11 already supports SPDY; you just have to turn it on manually. It's unclear when other browsers will support SPDY, but it may be awhile before you see SPDY in Internet Explorer or Safari.

Very few websites support SPDY at the moment. Google, of course, has been rolling out SPDY. Twitter is also offering SPDY. But it's going to be some time before most users see the effects of SPDY across all or even most of the sites they visit. But the odds are good that you'll start seeing a significant benefit from SPDY before you're using IPv6 at home.

3 Reasons Why Everyone Needs to Learn Markdown

Jon Mitchell — Tue, 17 Apr 2012 01:00:22 -0700

You've probably heard of Markdown. Maybe you've heard the name for years. Perhaps you just encountered it, since it's enjoyed a renaissance lately.

But do you know what it is? Are you using it? You should be. Here are three good reasons to use Markdown. There are no good reasons not to.

Wait... What Is Markdown Again?

If you don't know what Markdown is, here's the introduction from the Markdown project page:

"Markdown is a text-to-HTML conversion tool for web writers. Markdown allows you to write using an easy-to-read, easy-to-write plain text format, then convert it to structurally valid XHTML (or HTML).

Thus, 'Markdown' is two things: (1) a plain-text formatting syntax; and (2) a software tool, written in Perl, that converts the plain text formatting to HTML."

We're here to discuss No. 1, using Markdown whenever you're typing to format your text - whether or not you plan to post that text on the Web. Installing it allows you to directly post Markdown documents as blog posts or Web pages, but that's up to you.

One of its key strengths is that you can use HTML in Markdown. If there's something you can't do in Markdown, or if you can't remember the Markdown syntax, you can switch back and forth freely between HTML and Markdown within one document. It understands both.

You don't have to have Markdown installed on your site in order to use it. It's amazingly useful just as a writing language. Even if you don't have to convert to HTML at all, it's still an appealing way to format plain text without having to deal with Microsoft Word or another goofy rich-text editor.

But if you write for the Web, or you work with people who do, you just have to try it. Here's why.

Easy On The Eyes

"The overriding design goal for Markdown's formatting syntax is to make it as readable as possible," writes John Gruber of Daring Fireball, creator of Markdown. "The idea is that a Markdown-formatted document should be publishable as-is, as plain text, without looking like it's been marked up with tags or formatting instructions."

HTML tags add lots of in-line noise. They make a document hard to read. For people unfamiliar with HTML, it could be impossible. Reading a Markdown document should make plain sense to anyone.

Here's an example of the difference:

HTML

<h1>Why <em>you</em> should use Markdown to write your next blog post</h1>

<p><a href="http://daringfireball.net/projects/markdown/">Markdown</a> is just so dang legible, it will make your <em>whole life</em> easier. <strong>I promise.</strong></p>

Markdown

# Why *you* should use Markdown to write your next blog post

[Markdown][1] is just so dang legible, it will make your *whole life* easier. **I promise.**

[1]: http://daringfireball.net/projects/markdown/basics

Not only is Markdown easier to type and read, it's accessible to someone who doesn't know the first thing about HTML. There are no extra words or letters, the link looks like a footnote readers are used to seeing, and asterisks around a word convey emphasis even if you don't know which is italics and which is bold.

For Web developers and designers, this clarity will make life easier for nontechnical members of your team. For people not used to writing in hypertext, the Markdown characters are far more intuitive and easier to remember.

Fewer Errors

HTML just begs for typos. Even the smallest, one-letter tags require three characters to open them and four characters to close. If you forget a slash or accidentally type an apostrophe instead of a quotation mark, your whole page could be screwed up. And especially if you're in a hurry, the errors can be hard to spot.

Plenty of tools assist with HTML writing by highlighting errors on the fly, but why rely on those? Markdown's simplicity and flexibility helps you make fewer mistakes, and errors are much easier to find.

Kill Your CMS

Writing straight into a browser window is a dangerous game. It's so easy to accidentally lose or save over data. Furthermore, most content management systems offer the temptation to use WYSIWYG tools, and those tend to create awful HTML code that could make your page display in funky ways.

Markdown can be written anywhere there's a blinking cursor and shared in any format. It's just plain text. You don't need any WYSIWYG controls, because the Markdown characters actually look like the formatted results you'll get.

Markdown syntax is intuitive and, in many cases, it allows for multiple options, so writers can choose the formatting characters that make the most sense for them.

Plus, thanks to the Markdown renaissance, there are tons of new text apps that help Markdown writers on Mac, Windows, iOS and the Web. The good ones can even preview and export your Markdown writing as HTML, which you can then paste or upload into your CMS once you're done.

Markdown is so easy to learn. Don't let inertia stop you. Just download a Markdown text editor or use Gruber's browser-based dingus and start writing. You'll get it under your fingers in no time.

Lead image courtesy of Shutterstock.

Realizing the Promise of a Real-Time Web

Greg Slovacek_3 — Mon, 16 Apr 2012 01:00:00 -0700

It's a fantastic time to be a web developer. Society and the software economy that serves it are experiencing a dramatic change in the way successful Web applications are built. Previous shifts in software development have had massive, far-reaching consequences: the toppling of old technology giants, crowning of new industry kings, and the very transformation of how people communicate and how work gets done.

At the same time, these shifts also ripple through the software development ecosystem, changing everything from the processes and tools developers use down to the programming languages that are taught in schools. This time, the change is the real-time Web.

Guest author Greg Slovacek is an engineer at Asana, a real-time collaborative Web application that's reimagining the way teams work together.

Understanding Real Time

The advances in computing leading up to now have had a major impact on the software experience: richer and more powerful clients, access to more information and greater connectivity to other users. We now have the opportunity to get a desktop software-quality interaction experience combined with a Web-quality network experience, empowering us to build apps that are far better than the best of either of the two experiences alone. The first wave of these applications - highlighted by Google Docs - brought us the power and speed of desktop word-processing applications with the ability to jointly edit in near-real time from any browser.

The real-time Web takes this initial step and extends it with two essential attributes: responsiveness and freshness.

Responsiveness

Anyone who has used Apple's FaceTime on a bad connection understands how difficult - and frustrating - it is to have a conversation when the other party isn't responsive. If you attempt your natural conversational style in such a situation, the delays throw off your rhythm and cause cross-talk and miscues. You must become much more conscious of what you're saying and how you say it, stripping your dialogue of nuance and quality, just to ensure that your basic message is getting across. The medium gets in the way.

But when a software is at peak responsiveness, it can have an equally dramatic (yet overwhelmingly positive) effect: The software itself disappears. Truly responsive software should feel as if it doesn't even exist - it's natural, fluid and nonintrusive - becoming an extension of the user and creating a sense of flow when in use.

A real-time experience isn't just faster - it's way, way better.

Freshness

When you track a package, it's frustrating if the only available information is that it "left the origin." Stale information seems almost worse than no information at all because it creates anxiety: Has my package really not moved in three days, or am I just not getting the whole story? You suspect that someone must know exactly which truck it's on and exactly where that truck is. A better system would propagate that information to you, giving you confidence in its authenticity.

Without freshness, information is simply not trustworthy or accurate. This is increasingly unacceptable in collaborative Web applications. The benefits of fresh information go far beyond organizational agility, which requires users to respond quickly to new information. When you can trust that information is truly authentic, you feel more in control; you're observing the information directly rather than via some proxy, which might not be filtering it in your best interest. Timely information lets users collaborate with each other more intimately.

Fresh information isn't just more valuable - it's indispensable.

Architecture of a Real-Time Web Application

If responsiveness and freshness are so important, what does that mean for software developers? Maximizing these qualities in a Web application requires architectural changes in fundamental ways.

While the advancements and benefits of software in the Web 2.0 era are indisputable, outdated client-server models still persist, forcing developers to heavily invest time and engineering resources to achieve the semblance of real-time. The current software development model is not sustainable. Indeed, to build a truly functional real-time Web application, software must be built from the ground up.

If developers are going to start rethinking the way software is built, there are obviously a number of things they'll need to consider. But there are three architectural components that are especially important: the application's data model, sync protocol and the publication-subscription system. Ultimately, developers will need to optimize the way data is stored and accessed, make sure the server is publishing information as it's received, and make use of a balanced pub-sub system.

Let's take a deeper dive into each of these components.

Data Model

Your application's data model itself is primarily affected by two factors: the number of users in the system who may be viewing or updating a document/file/page, and the number of objects a user is viewing. Rich, real-time Web applications crank up both of these numbers.

Many systems treat an object as a monolithic blob, transmitted and stored wholesale in at least some layers each time it's changed (with no way to change its aspects independently). This is a suboptimal strategy for a real-time Web app, for two reasons. First, the volume of changes means transmitting more data than necessary would bog down the system. Second, unnecessary data collisions would be more frequent, complicating application logic or impacting the user experience.

Instead, each object needs to be divided into a collection of aspects - information with individual and identifiable properties, each of which can be stored, modified and transmitted independently. Clients will want access to many objects, but not all of their data; they are automatically subscribed to only the aspects they need to display or operate on, given the view they are presenting at that moment.

Sync Protocol

The means of synchronizing data between client and server - and from server to server - is of paramount importance. In a Web 2.0 application, data is rarely pushed to a user. More modern applications may poll for new data in the background and show some indicator that users should update their displays (for example, Twitter's "You have new tweets" message). This behavior is often limited to one aspect of the application, such as a feed; however, it is not achieved or delivered across the entire application.

A real-time application receives information relevant to any part of the user's view at any time, and minimizes the delay between when it is available on the server and when the user can actually see it in the client view. In this situation, the comet model shines brighter than Ajax - the server must be effectively pushing information as it receives it.

As the client view may change with each keystroke, a proper sync protocol must be able to handle a lot of data churning in and out of view. The client needs to cache some data, but manage that cache carefully. In some contexts, it may be unacceptable to show cached, stale data, and the app should be able to choose where to take advantage of a cache and where to require fresh data from the server.

Publication-Subscription System

Any system that provides notifications of changes must employ some kind of publication and subscription (pub-sub) model. In a real-time application, the units of subscription must be balanced carefully. If they are too granular (e.g., mirroring every single data dependency in the system), then the pub-sub system can become overwhelmed with bookkeeping. If they are too coarse (e.g., subscribing to any change of a particular property on any object), then some operations can cause explosions of notifications.

Developers must make use of pub-sub both on object properties and query results; they must find the right balance of granularity in a hybrid approach. This allows for object-level subscriptions, as well as scoping queries to at least one collection or index.

Development Implications

It comes as no surprise that to keep up with the complexity of the real-time Web, software engineers will have to write more sophisticated applications. But it's not going to be easy to achieve that sophistication, especially since the way developers build and maintain apps is evolving rapidly. To keep up with the pace of change, it's time to rethink the way apps are built. The effort will pay off, and the results will be incredible.

Let's Start Building

We have massive computing power on our desktops and in our pockets. We have a mind-boggling amount of data in the cloud representing the contributions of millions of individuals to collective knowledge. And we have fast conduits to move necessary information between any user or device that needs it. This bountiful technology beckons us to make the next big leap in software. It's time to realize the promise of the real-time Web.

- Greg Slovacek

Microsoft Spins Off Open Source, Hopes to "Build Bridges"

Scott M. Fulton — Fri, 13 Apr 2012 10:00:49 -0700

Microsoft is spinning off its open source unit into Microsoft Open Technologies, Inc., and promoting open source veteran Jean Paoli to the role of president of the wholly owned subsidiary. The move gives the new company a measure of open source credibility and is likely to give Paoli more latitude in determining Microsoft's open source policy.

Paoli helped create XML, cementing his reputation among developers, and in 2007, he pledged his company's support for multiple document formats so long as the other format did not restrict customers' free choice. The masterstroke of diplomacy calmed a brewing rebellion among champions of the existing ISO standards format, OpenDocument, who claimed competing standards (specifically, Microsoft's Open Office XML) would sow confusion.

Building a Bridge

In an interview with ReadWriteWeb this afternoon, Paoli characterized the creation of the new Microsoft subsidiary not so much as a reorganization but a confirmation of the evolving role of open source development at Microsoft. Some will continue to regard "Microsoft" and "open source" as opposite sides of a coin. But Paoli described his new subsidiary as a liaison between Microsoft's corporate entities and the open source community, and more than the masked voice of a commercial developer.

"The business case is literally around enabling scenarios for our customers that bridge Microsoft and non-Microsoft technologies," the new president told RWW. "That's basically the business goal."

One example Paoli cited was MongoDB, the open source NoSQL database system. While MongoDB is already effectively promoting the database, Microsoft Open Technologies is positioned to promote the use of MongoDB in conjunction with Microsoft technologies, especially the Windows Azure cloud service.

"That's an example of a non-Microsoft technology that originated with a brilliant open source community who understands a lot about big data, and we're very interested to start working with them to see how MongoDB can work on Windows Azure," Paoli said. "It's always going to be a bridge between a Microsoft technology and a non-Microsoft technology. Those are the use cases that we will really be working on."

Microsoft Can Now Speak With Two Voices

In many cases, Microsoft will have interests as both a commercial developer and an open source participant. The creation of the spinoff gives it a little wiggle room with respect to its open source positions. Microsoft Open Technologies can effectively stand for public and open standards if it so wishes, while Microsoft corporate continues to be the actual liaison for standards.

"Let me be frank: There are a number of differences between the process of developing proprietary software and the processes of the open source community," explained Paoli. "So in some cases, it is important to keep those processes separate. And in other cases, there is really room for great collaboration, interaction. Sometimes we want to keep things separate, and sometimes we need to have greater collaboration. This needs to be properly managed."

He cited a circumstance where a new community is developing around an open source product or technology - one with a deep technical brain trust and mixture of talents. Microsoft needs to be able to volunteer its participation, including helping to fix bugs and add features. "We need to be able to have it turn over very quickly, in the next 24 hours."

But as Paoli's response implies, the liaison with the community must not appear bound to corporate interests. That, too, will help expedite simple fixes and feature additions, and help Microsoft become perceived as an equal player in the open source community.

"All our customers expect the software they use will work in a heterogeneous environment. Customers expect their phone to connect to any cloud device; if you cannot receive your email on your phone, well, that's not good, irrespective of which phone or which cloud or which operating system they are connected to," Paoli said. "We live in a mixed IT world, and the goal is to provide customers with even more choice to bridge Microsoft and non-Microsoft technologies, because that's what customers expect."

SAP's HANA: Accelerating Your Apps by 6 Orders of Magnitude

Scott M. Fulton — Fri, 13 Apr 2012 02:00:28 -0700

Oracle CEO Larry Ellison likes to walk into a presentation with raw numbers in hand. When he touts Oracle technology as faster, he prefers to say how much... sometimes inflating the number as he goes along. "A factor of five. A factor of 10! A factor of 15!"

What would Ellison have given for the opportunity to tout a factor of 400,000? Evidently not enough. The notion that a database manager would run six orders of magnitude faster if it resided in DRAM rather than on a hard drive, is decades old, just waiting for an Oracle or a Microsoft or a Dell to make it hapen. Instead, SAP is the company delivering the dramatic database performance boost.

Most database technology today is an upgraded form of the same database technology from yesterday, and it's usually in the upgrader's best interest to support its native platform. So it only makes sense that perhaps the most revolutionary database technology of the last quarter-century should come from a company that only recently acquired Sybase and entered the database industry: SAP, known as a leader in enterprise resource planning.

Quiet Riot

Quietly, SAP has been building momentum behind a technology called HANA. It's not a particularly sophisticated concept, which is partly why SAP engineers don't embellish its descriptions with charts. Essentially, it's a data-organization system based around memory rather than hard-drive storage arrays. HANA may use storage arrays for backup and redundancy and traditional database engines - like SAP's own Sybase - for managing data prior to its delivery to memory.

"In our customers' minds, there is now absolutely no doubt that HANA now represents the database architecture for the future," said Vishal Sikka, SAP's CTO, in a press conference Tuesday afternoon. "For OLTP and OLAP [transactional and analytical processes], for structured and unstructured data, for legacy applications and new applications, a single database with, as you can see... pretty much unlimited scale." From there, Sikka proceeded to tell stories of HANA deployments since it became generally available last June. One customer story independently cited the surreal-sounding 400,000x speed improvement, others cited less but were still awe-inspiring.

"When we built our first HANA box internally with commodity hardware, it cost us something like $700,000 to build it," related SAP Global Solutions President Sanjay Poonen (right), in an exclusive interview with ReadWriteWeb. "Within six months, the costs of these [same] parts and commodity hardware to assemble and test it, decreased by 30%. The price of hardware is falling so rapidly."

The trick for SAP is infusing its in-memory platform into the enterprise with a minimum of upheaval. It sees a place for itself in a new space that could reside alongside business' existing database investments, and even alongside their new investments in Hadoop and "big data" stores. Think of a turbocharger that improves whatever car it's in, but works so well that customers end up replacing most or all of the car over time anyway.

In fact - and this is perhaps the most impressive aspect of the story - SAP has established a pair of giant investment funds to support the effort. It will literally invest its own money in companies and customers who can develop new means for integrating HANA more seamlessly in existing environments.

"The Acceleration Fund will kick in to help customers fund some of that migration, so we can help ease them off that [older platform]... The database vendors, you'd think, are all set in concrete; they never change. That's not always the case. Part of the trigger factor is, if you look at the amount of spending that companies have on databases - you add the license cost, the maintenance cost and, the biggest part, the DBA service cost - it's too big. If you're spending $100 million on database costs, and by consolidating a lot of this onto a platform that includes HANA, we can take that cost down from $100 million down to $70 million, or by 30%, would you spend an extra $5 million or $10 million with us, or what have you, to ensure you get the best, next-generation data platform? That's the proposition... You can keep adding more bells and whistles, bells and whistles, to a Model T, but at some point in time, a Model T is a 1920s car. It's not going to scale for what your [travel requirements] need to be."

Forward Motion

Not since the 1980s has a database manager been a single application. Today's database platforms are comprised of multiple components, many of which come in shiny, metal boxes. Oracle likes to demonstrate that a platform migration can take as little as five hours, but in practice, the actual time is often measured in years. One factor is the lack of one-to-one correspondence between various manufacturer's components.

If you consider HANA a different platform rather than an extension platform, this issue is magnified. SAP intentionally designed HANA to have fewer components. On Tuesday, the company's executive vice president, Steve Lucas, tried to spin this into an advantage.

"Our customers asked us for help," Lucas told the press. "They said, 'The existing database players are not innovating quickly enough. We're not getting from them what we need. Please help us!' We said, 'Okay.' Vendors are pushing more, not less.

"Think about that for a minute: We've gone from transactional databases... to needing data warehouses, and on top of those data warehouses we need multidimensional stores and OLAP cubes, and we need data marts, and the list goes on and on, because it makes people more money. SAP is focused on helping customers consolidate those systems, solve many things with one solution - one - and save them money at the same time."

SAP's Poonen told RWW: "If you want to build an analytical infrastructure today, check out all that a CIO has to buy. If you wanted to buy an enterprise data warehouse, pay $5 [million] or $10 million to Teradata." For data reporting, he added, you may go to SAP; for planning and budgeting, you might go to Oracle's Hyperion. The cost for SAP's BusinessObjects business intelligence (BI) platform may be another million. For predictive analytics, you might go to SAS and pay another $2 million.

With the exception of BusinessObjects, each of these extra components has a database engine, noted Poonen, that would be redundant to the underlying Teradata layer. "If you use Hyperion Planning on top of Teradata, you take the data out of Teradata, you flatten it and put it in a new structure called Essbase - a multidimensional cube structure. That's completely redundant, it has all kinds of overlap, and you could ask the question, 'Why isn't that Essbase database that serves OLAP, multidimensional processing sitting inside Teradata? Why is it different?' 'Because they're separate companies.' That's it. There's no other reason. So when we looked at this architecture, we said, 'Collapse all those layers into one in-memory database, because now we know what the destination state of that architecture needs to be.' "

So HANA includes capabilities for reporting, planning and predictive analytics. "It really is kind of a better architecture than Teradata, Hyperion and SAS put together. That's what we mean by collapsing those layers, and by doing that, you save an enormous amount of costs for the average CIO."

Throwing the Switch

Because these multicomponent architectures and SAP's HANA architecture share fewer "seams" with one another, it would appear on the surface that a migration plan would be rocket science. Poonen told RWW that SAP got a leg up on such planning early on by mastering the fine art of migrating data between other companies' platforms, as part of SAP services.

For installations of Oracle databases atop SAP's NetWeaver Business Warehouse (BW), the SAP president says his company has already constructed procedures that enable the move from Oracle to HANA, which one customer reports was completed in its entirety in two weeks. "You're talking in weeks, maybe months, but not years. That's because data structure is something we know about. And there's 15,000 of them, not just five."

For the last quarter-century, software architects have asked themselves how much ammunition and firepower would be required to achieve an upheaval in system architecture - not just storage and capacity, as is addressed by Hadoop, but in the way data is processed inside the hardware. Now we're seeing SAP amassing numbers in the six-digit range. If SAP manages to attach this firepower to the cloud any time soon, Oracle, HP NonStop, IBM and the Dell/Microsoft combo may need to recompute and reload to compete.

Stock photo by Shutterstock.com