Google Code: News

Use compression to make the web faster

2009-11-09T12:00:00.000-08:00

Every day, more than 99 human years are wasted because of uncompressed content. Although support for compression is a standard feature of all modern browsers, there are still many cases in which users of these browsers do not receive compressed content. This wastes bandwidth and slows down users' interactions with web pages.

Uncompressed content hurts all users. For bandwidth-constrained users, it takes longer just to transfer the additional bits. For broadband connections, even though the bits are transferred quickly, it takes several round trips between client and server before the two can communicate at the highest possible speed. For these users the number of round trips is the larger factor in determining the time required to load a web page. Even for well-connected users these round trips often take tens of milliseconds and sometimes well over one hundred milliseconds.

In Steve Souder's book Even Faster Web Sites, Tony Gentilcore presents data showing the page load time increase with compression disabled. We've reproduced the results for three highest ranked sites from the Alexa top 100 with permission here:

Data, with permission, from Steve Souders, "Chapter 9: Going Beyond Gzipping," in Even Faster Web Sites (Sebastapol, CA: O'Reilly, 2009), 122.

The data from Google's web search logs show that the average page load time for users getting uncompressed content is 2.0 seconds, whereas the time for users getting compressed content is 1.6 seconds. In a randomized experiment where we forced compression for some users who would otherwise not get compressed content, we measured a latency improvement of 300ms. While this experiment did not capture the full 0.4 second difference, that is probably because users getting forced compression have older computers and older software.

We have found that there are 4 major reasons why users do not get compressed content: anti-virus software, browser bugs, web proxies, and misconfigured web servers. The first three modify the web request so that the web server does not know that the browser can uncompress content. Specifically, they remove or mangle the Accept-Encoding header that is normally sent with every request.

Anti-virus software may try to minimize CPU operations by intercepting and altering requests so that web servers send back uncompressed content. But if the CPU is not the bottleneck, the software is not doing users any favors. Some popular antivirus programs interfere with compression. Users can check if their anti-virus software is interfering with compression by visiting the browser compression test page at Browerscope.org.

Internet Explorer 6 removes the Accept-Encoding request header when it sends requests via a proxy that only supports HTTP 1.0 requests. The table below, generated from Google's web search logs, shows that IE 6 represents 36% of all search results that are sent without compression. This number is far higher than the percentage of people using IE 6.

Data from Google Web Search Logs

There are a handful of ISPs, where the percentage of uncompressed content is over 95%. One likely hypothesis is that either an ISP or a corporate proxy removes or mangles the Accept-Encoding header. As with anti-virus software, a user who suspects an ISP is interfering with compression should visit the brower compression test page at Browerscope.org.

Finally, in many cases, users are not getting compressed content because the websites they visit are not compressing their content. The following table shows a few popular websites that do not compress all of their content. If these websites were to compress their content, they could decrease the page load times by hundreds of milliseconds for the average user, and even more for users on modem connections.

Data generated using Page Speed

To reduce uncompressed content, we all need to work together.

Corporate IT departments and individual users can upgrade their browsers, especially if they are using IE 6 with a proxy. Using the latest version of Firefox, Internet Explorer, Opera, Safari, or Google Chrome will increase the chances of getting compressed content. A recent editorial in IEEE Spectrum lists additional reasons - besides compression - for upgrading from IE6.
Anti-virus software vendors can start handling compression properly and would need to stop removing or mangling the Accept-Encoding header in upcoming releases of their software.
ISPs that use an HTTP proxy which strips or mangles the Accept-Encoding header can upgrade, reconfigure or install a better proxy which doesn't prevent their users from getting compressed content.
Webmasters can use Page Speed (or other similar tools) to check that the content of their pages is compressed.

For more articles on speeding up the web, check out http://code.google.com/speed/articles/.

By Arvind Jain, Engineering Director and Jason Glasgow, Staff Software Engineer

Announcing the Worldwide Ning Appathon Competition

2009-11-06T08:28:00.000-08:00

Let the games begin! At last night's Ning Appathon kickoff event at the Ning offices in Palo Alto, Ning started a week-long worldwide app development competition for the recently launched Ning Apps platform.

WHAT: The Ning Appathon is a week-long OpenSocial development competition with prizes for both original and ported applications. Judges include Ning Chairman and Co-founder Marc Andreessen, Wired magazine Editor-in-Chief Chris Anderson and Managing Director of building43 Robert Scoble.

WHERE: The competition is open to participants worldwide. Visit the Ning Developer Network for details and Ning Apps documentation.

WHEN: The competition began at our kickoff event on November 5th. All entries must be submitted via http://developer.ning.com/appathon by 10 p.m. Pacific on Thursday, November 12th.

PRIZES: The Grand Prize for Best Original App is $5,000 (US). The Grand Prize for Best Ported App is $4,000 (US). Two Original App and Ported App finalists (four total) will each receive $500 (US). All six winners will also receive prominent editorial placement in the Ning Apps directory.

Please visit the Ning Developer Network for more information. Good luck!

Posted by Kyle Ford, Director of Partner Design & Development, Ning

ADC 2 Round 2 Voting Open

2009-11-05T21:15:00.000-08:00

The results from ADC 2 Round 1 are now tabulated and verified. With the top 200 applications identified, it's time to begin the final round judging. Be sure to download the ADC 2 judging application, or update your existing application, and help us select the final winners!

For the final round, both users and a Google-selected panel of industry judges will provide votes to determine the final winners. Prizes will be distributed to the top 3 entrants in each of the 10 categories, and the top 3 overall entrants will receive additional prizes. Please see our reference page for full challenge information.

Your vote is critical! We will keep voting open until we have received sufficient votes for all of the applications. We encourage you to download the ADC 2 judging application and evaluate entrants for yourself.

Download Android Developer Challenge 2:

Audrey Roy Wins the App Engine + Twilio Developer Contest

2009-11-05T18:03:00.000-08:00

Congratulations to Audrey Roy -- her "Price It By Phone" application won the Google App Engine + Twilio developer contest that ran from September 21st to October 4th. Audrey will receive $1000 in Google App Engine credit, and a Dell Netbook from Twilio.

Audrey hacked together her contest submission in under 48 hours after going to an art museum and discovering that the $60 box she wanted was only $37 on Amazon.com. She used the Amazon Product Advertising API to look up the best used and new prices for books by ISBN number, using any touch tone phone.

Look up Amazon.com book prices by calling (877) 265-8137. Enter the book's 10-digit ISBN number after the prompt. The voice will tell you Amazon.com's lowest new and used prices for the book. Then, to see the list of books that you've already price-checked, with links to their Amazon detail pages, enter your phone number at http://price-it.appspot.com.

You can find an interview of Audrey on Twilio's blog.

Thank you to everyone who participated in the contest.

Twilio runs weekly developer contests to encourage developers to explore the many use cases, technologies, and industries where voice applications can provide useful solutions. They announce a new category each Monday along with the previous week's winner. This week's category is "Twilio for Salesforce, with Appirio" and the deadline is midnight on November 19th.

Questions? Drop us a line at help@twilio.com

Guest Post by Danielle Morrill, Director of Marketing, Twilio

Introducing Closure Tools

2009-11-05T10:30:00.000-08:00

Millions of Google users worldwide use JavaScript-intensive applications such as Gmail, Google Docs, and Google Maps. Like developers everywhere, Googlers want great web apps to be easier to create, so we've built many tools to help us develop these (and many other) apps. We're happy to announce the open sourcing of these tools, and proud to make them available to the web development community.

Closure Compiler
Closure Compiler is a JavaScript optimizer that compiles web apps down into compact, high-performance JavaScript code. The compiler removes dead code, then rewrites and minimizes what's left so that it will run fast on browsers' JavaScript engines. The compiler also checks syntax, variable references, and types, and warns about other common JavaScript pitfalls. These checks and optimizations help you write apps that are less buggy and easier to maintain. You can use the compiler with Closure Inspector, a Firebug extension that makes debugging the obfuscated code almost as easy as debugging the human-readable source.

Because JavaScript developers are a diverse bunch, we've set up a number of ways to run the Closure Compiler. We've open-sourced a command-line tool. We've created a web application that accepts your code for compilation through a text box or a RESTful API. We are also offering a Firefox extension that you can use with Page Speed to conveniently see the performance benefits for your web pages.

Closure Library
Closure Library is a broad, well-tested, modular, and cross-browser JavaScript library. Web developers can pull just what they need from a wide set of reusable UI widgets and controls, as well as lower-level utilities for the DOM, server communication, animation, data structures, unit testing, rich-text editing, and much, much more. (Seriously. Check the docs.)

JavaScript lacks a standard class library like the STL or JDK. At Google, Closure Library serves as our "standard JavaScript library" for creating large, complex web applications. It's purposely server-agnostic and intended for use with the Closure Compiler. You can make your project big and complex (with namespacing and type checking), yet small and fast over the wire (with compilation). The Closure Library provides clean utilities for common tasks so that you spend your time writing your app rather than writing utilities and browser abstractions.

Closure Templates
Closure Templates grew out of a desire for web templates that are precompiled to efficient JavaScript. Closure Templates have a simple syntax that is natural for programmers. Unlike traditional templating systems, you can think of Closure Templates as small components that you compose to form your user interface, instead of having to create one big template per page.

Closure Templates are implemented for both JavaScript and Java, so you can use the same templates both on the server and client side.

Closure Compiler, Closure Library, Closure Templates, and Closure Inspector all started as 20% projects and hundreds of Googlers have contributed thousands of patches. Today, each Closure Tool has grown to be a key part of the JavaScript infrastructure behind web apps at Google. That's why we're particularly excited (and humbled) to open source them to encourage and support web development outside Google. We want to hear what you think, but more importantly, we want to see what you make. So have at it and have fun!

By the Closure Tools team

London Open Source Jam 14

2009-11-04T16:31:00.000-08:00

We held the 14th Google London Open Source Jam at our Victoria HQ on September 24th. The topic this time was "Video and Sound", and our Jammers had some real treats to share.

Steven Goodwin told us how his open source SGX 3D graphics engine deals with three key problems of other computer game engines. On a similar theme, Themis Bourdenas discussed the vine engine, a modular game engine for 2d and 3d games.

Borys Musielak presented Filmaster, an open source film recommendation engine. Neil Harris told us about an attempt by the Kendra Initiative to foster a common meta data format for content discovery on the semantic web.

In an Open Source Jam first, Jagannathan gave a performance of his Din software musical instrument. Din is designed for playing live Indian music, is based on Bezier curves and really has to be heard to be fully appreciated.

Sam Mbale gave us an update on his projects to help Africans build online communities using open source. Mike Mahemoff discussed some web tools frameworks for intranets, bookmarklets and trails in Scrumptious.

The UK government has plans to introduce a law to allow content-owners to force ISPs to disconnect the internet connection of users suspected of file sharing, without any proof. Glyn Wintel gave us an overview of how the proposed law will affect us, how the Open Rights Group is campaigning against it, and how we can help.

Douglas Squirrel talked about the difficulty blind people have in finding information on websites, and presented BlindPages.com - a new project to reformat the web in a screen-reader friendly way. He also demoed a prototype telephone interface to the service.

Much pizza was eaten and free beer drunk, and we all ended up in the pub next door to continue our discussions. A big thank you to all our speakers and attendees, and we hope to see you at the next Jam!

By Matt Godbolt, Mobile Engineering

Upcoming US Events with Google Wave Presentations

2009-11-04T15:26:00.000-08:00

Similiar to the Google Wave European tour, we're stopping by several developer events in the states:

LISA 2009

Nov. 1-6, 2009, in Baltimore MD
Nov. 4 (today!) - "Google Wave Federation Protocol" with Joe Gregorio

SF Java User Group

Nov. 10, 2009, in San Francisco, CA
"Google Wave APIs: Now & Beyond" with Marcel Prasetya

Dreamforce

Nov. 17-20, 2009, in San Francisco, CA
Nov. 19 - "Ready to Ride the Google Wave?" with Dan Peterson

Massachusetts GTUG Wave Hackathon

Nov. 21, 2009, in Boston, MA

All of these events are still open for registration - so grab a seat while you still can. See you there!

Posted by Pamela Fox, Developer Relations

Final Decommission Notice for the Legacy YouTube API

2009-11-04T13:03:00.000-08:00

It's been several years since we've released the Google Data-based YouTube API, and in that time we've been encouraging developers who used the legacy YouTube API to upgrade before we pull the metaphoric plug on that older version. At this point, all but a handful of holdouts have upgraded, and as of November 11, 2009, the legacy YouTube API will cease operation.

If you're using one of our YouTube API client libraries, then you're definitely making use of the modern Google Data YouTube API. If you're manually making HTTP requests to a URL whose hostname contains gdata.youtube.com, then you're also good to go. If you think you might still be using the legacy YouTube API but aren't sure, take a look at some of the example legacy API calls in this migration guide. If it turns out that you are still using the older API, then the migration guide will give you the information you need to upgrade – and be sure to do so before November 11!

New insights into web application performance

2009-11-04T10:23:00.000-08:00

I've sometimes thought that optimizing web applications is as much a science as dowsing. (No offense intended, dowsers of the world — but you have to admit it's a hard thing to explain even when it does work out.) Even when you are completely willing to invest time and energy into optimizing an application, how do you actually go about it?

Our team, along with everyone else in the world who cares about web application performance, has had to essentially guess at where time goes inside the browser. We've spent countless hours debating each others' wild-eyed speculations as to the true sources of latency afflicting a wide variety of applications. Indeed, web apps can be slow for all sorts of opaque and unintuitive reasons. Don't be fooled into thinking that bloated, slow JavaScript is the only culprit. We've seen pathological situations in which a few seemingly insignificant CSS tweaks can improve performance by a factor of 5 or more. Another dark gem: rearranging a mere few lines of JavaScript that were inadvertently calling DOM-related methods in an unfortunate sequence (which caused multiple redundant layouts) turned a life-negating 5 second operation into a sprightly 50 millisecond blink of an eye. That one took 5 days to find the offending 4 lines of JavaScript and then about 3 seconds to actually make the code change. #nowiamwaybehindingooglereader

We couldn't take it anymore. We decided we had to find a way to transform the witchcraft of optimizing web apps into a legitimate engineering task, once and for all.

My team began working on a series of changes to WebKit and Chrome to collect precise metrics on where time is, in fact, going within the browser. When, exactly, does layout occur? How long does each layout take? Can layout happen synchronously while my JavaScript is executing, or is it deferred? How much time is spent on CSS selector matching? How long does parsing (versus executing) JavaScript take? Does the process of actually painting pixels on the screen take much time? We instrumented the browser way deep down inside to produce a stream of such metrics, being very careful to keep observer effects to a minimum.

Happily, we've managed to land these changes into both WebKit and Chrome over the last several weeks. Soon you'll see the first examples of tools that make these metrics available to web developers using WebKit-based browsers. Of course, we have a lot more instrumentation planned, but the ball is really rolling now thanks to lots of help from the friendly folks on the WebKit and Chrome teams (especially Pavel Feldman and Timothy Hatcher). We've gained many new insights, some of which I mentioned in my Measure in Milliseconds talk at Google I/O earlier this year. When you see these metrics yourself for your own web apps, you'll likely be surprised — and you'll almost certainly wonder how anyone tried to write high-performance code without this sort of insight. Be sure to keep an eye on the Chrome Dev Channel and the WebKit nightly builds for new Inspector features based on our work.

New personalization features in Google Friend Connect

2009-11-04T08:50:00.000-08:00

Today, we're excited to announce several new features for Google Friend Connect that make it possible for website owners to get to know their users, encourage users to get to know each other, and match their site content (including Google ads) to visitors' interests. Check out the Google Social Web Blog for an overview of these new features.

We also want to point out that there are APIs for developers who want to play with the interests data programmatically. With the new interest data described on the Social Web Blog, developers can write custom polls and access the interests data directly. Friend Connect provides API level access to both individual interests information as well as aggregate information for all users of a site. Interests information can be added programmatically for the signed-in user or via the poll gadget, and it can be accessed via both the JavaScript API and the OpenSocial REST API. The Guitar Universe example site should give you an idea of some of the things that are possible with this new launch.

As always, feel free to ask technical questions related to the Friend Connect APIs in the developer forum.

By Mussie Shore, Product Manager, Google Friend Connect