Heroku

Introducing the Europe Region, Now Available in Public Beta

Zeke — Wed, 24 Apr 2013 17:35:38 GMT

Today we’re happy to announce Heroku’s Europe region, available in public beta. With more than 3 million apps running on our platform from developers all over the globe, it's not surprising that we've had high demand for Heroku in more regions of the world. After collaborating closely with customers during private beta, we're now ready to offer Heroku services in Europe to all customers as part of a public beta. The Europe region runs Heroku applications from datacenters located in Europe, offering improved performance for users in that region.

One Heroku, Two Continents

The Europe region offers all the features of the existing US region. Both regions run apps on infrastructure dedicated to each region, but are managed by the same unified Heroku interface you already know. This provides powerful global app management with isolated, geolocated infrastructure:

Faster Apps

The physical proximity of the Europe region to European end-users means reduced latency, often resulting in a dramatic improvement in app responsiveness to those users. We’ve observed performance improvements of 100ms per request or more for European end-users:

A Familiar Workflow

All Heroku users can now create and deploy apps to the Europe region:

$ heroku create --region eu
$ git push heroku master

Once the app is created, you can interact with it just like any other Heroku app.

Easy App Migration with Heroku Fork

To ease the process of migrating existing applications to the Europe region, we created heroku fork, a new addition to the Heroku CLI. Heroku fork copies an app's Heroku Postgres data and config vars, and re-provisions all its add-ons. If you have an existing app you’d like to run in the Europe region, check that you’ve got the most recent Heroku toolbelt installed and use Heroku fork to create a running copy of your app in the new region:

$ heroku fork --region eu
Creating fork myapp-332... done
Copying slug... done
Adding newrelic:professional... done
Copying config vars... done
Fork complete, view it at http://myapp-332.herokuapp.com/

Note: heroku fork will not move any domains or scale your app past a single dyno, so you're free to decide when your app will be made available to your customers. For more information about this powerful new feature, see the Dev Center article on heroku fork.

Add-ons

More than 60 add-ons are currently available in the Europe region, with more on the way. To ensure that your app has fast access to its data wherever it’s deployed, Heroku automatically provisions latency-sensitive add-ons in your app’s region. Many add-ons are already available to apps in the Europe region, including:

To discover which add-ons are available in the Europe region, search for 'europe' on the addons homepage or use the new --region flag in the heroku CLI:

$ heroku update
$ heroku addons:list --region eu

Customer Success

Several Heroku customers have already deployed their production Heroku apps to the Europe region for improved performance. For example, top Swedish television network TV4 is currently running its video on demand service TV4 Play out of Heroku’s Europe region. TV4 CTO Per Åström says:

Deploying our app closer to our users in Heroku's Europe region gave us a 150ms improvement in web performance. Based on this win for our users, we’re moving all of our apps to the Europe region.

Digital agencies and other customers delivering user-facing mobile and social apps have also benefited from improved performance in the Europe region. For example, app development company Betapond COO Conor Ryan says:

Deploying our Facebook apps live to the Heroku cloud in the EU was pain-free and as easy as staging. We're delighted to see an average of 11% improvement in page load times since making the switch. Consumers don't tolerate waiting, particularly on mobile devices, so Heroku's EU contribution to a snappy user experience is a big boost for Betapond and our customers.

Safe Harbor Compliance is Coming Soon

Heroku is not yet a registered participant in the Safe Harbor program. We’ve laid the groundwork for becoming Safe Harbor certified and expect to have it soon. The Europe region public beta is designed to let you build high-performance apps for European users. It does not currently address data residency or jurisdiction concerns. You should assume that some portions of your app and its data will be in, or pass through, datacenters located in the US.

We Want Your Feedback

If you have comments or questions about the Heroku Europe region, email us at eu-beta@heroku.com. To receive email updates about the Heroku Europe region as it progresses to general availability, subscribe to the beta mailing list below.

Additional Resources

Expanded HTTP Method Support

Blake — Mon, 22 Apr 2013 18:26:48 GMT

HTTP and its secure variant, HTTPS, are the protocols used by every web client to talk to web applications. A key part of the protocol is the HTTP method. Traditionally, web applications used a very limited set of HTTP methods. It is common for application servers, proxies and routers (such as the Heroku HTTP router) to prevent unknown methods from being used. This unnecessary constraint of the Heroku HTTP router has increasingly become a limitation to developers building modern web applications.

In the past, if you tried to use an unsupported HTTP method in your Heroku application, clients would receive a 405 METHOD_NOT_ALLOWED error.

As of today, that's no longer the case. The Heroku routers now accept any HTTP method, allowing you to use newer methods that have recently gained adoption, such as PATCH.

Why is this important?

Many new web frameworks use recently introduced HTTP methods such as PATCH or even custom methods that are not yet standardized. Ruby on Rails is one such framework. Rails has constantly evolved since its birth to always incorporate the latest development techniques. One such evolution is happening in Rails 4 where the framework uses the PATCH HTTP method to perform partial updates of resources.

Before Heroku removed restrictions on HTTP method, it was not possible to leverage this new functionality in Rails 4. Now that any method is supported, you can take full advantage of PATCH support in Rails 4.

Background

For the curious, a little more background is in order.

HTTP is extensible

A huge factor in HTTP's popularity has been its extensibility. HTTP is a simple protocol that can be used for many different purposes. For instance, the HTTP/1.1 Spec defines the following set of methods: OPTIONS, GET, HEAD, POST, PUT, DELETE, TRACE, CONNECT.

But the spec also states that new methods can be added. In fact, a number of other RFCs have standardized additional HTTP methods, often for specific applications such as version control or calendaring. Yet in most cases, regardless of the underlying application, developers can leverage the same supporting infrastructure: load balancers, proxies, caching proxies; all designed for HTTP.

The HTTP `PATCH` Method

A recent extension to HTTP is RFC5789, which introduced the PATCH method. This method is particularly useful for web application developers, as it standardizes a common way to make partial updates to resources. Rails 4 and many other libraries now make use of it.

We can quickly demonstrate how it is used by deploying a basic Sinatra app to Heroku using the quick start guide. Replace the code in app.rb with:

require 'sinatra'
require 'json'

class App < Sinatra::Base

  # fake persistent data.
  DATA = { "a" => 1, "b" => 2, "c" => 3 }

  put '/data' do
    # put replaces the existing resource so we simply return the new resource
    new_data = JSON.parse(request.body.read)
    new_data.to_json
  end

  patch '/data' do
    # patch 'patches' the existing resource
    new_data = DATA.merge(JSON.parse(request.body.read))
    new_data.to_json
  end

end

Once your app is deployed, perform a PATCH and PUT request with curl:

$ curl -X PATCH http://shiny-object-1234.herokuapp.com/data -d '{"a": 2, "b": 3}'
{"a":2,"b":3,"c":3}
$ curl -X PUT http://shiny-object-1234.herokuapp.com/data -d '{"a": 2, "b": 3}'
{"a":2,"b":3}

The PATCH logic merges the new values into the existing data structure and returns the whole data structure, whereas PUT simply replaces the old data structure with a new one. While this example doesn't persist the changes, it demonstrates the semantic difference between these two HTTP methods.

Other HTTP methods

PATCH is just one example of how a new HTTP method was defined, standardized and successfully achieved widespread use. Because Heroku no longer imposes any restrictions on HTTP methods, future extensions will work seamlessly without requiring explicit support. This also means that the Heroku platform can be used as a proving ground for new, non-standardized HTTP methods.

Why restrict HTTP methods in the first place?

The original motivation for application servers, proxies and HTTP routers to restrict HTTP methods was that these intermediaries often performed more functions than just passing through requests. For example, if an intermediary wants to retry a failed request, it needs to understand whether a retry is safe. POST and PATCH requests are generally not safe to retry while GET, PUT, and DELETE are.

Heroku's routers pass requests straight through to the application, without special handling based on the particular HTTP method; the routers therefore handle all requests the same, regardless of which method was used. This results in maximum flexibility for the application developer to use any HTTP method and fully control the semantics of each request.

For more on HTTP routing visit our Dev Center article.

Heroku Postgres – Version 9.2 now Default

Craig — Thu, 18 Apr 2013 18:43:46 GMT

Over a year ago we began working with the community as to how we could help to make Postgres better. Much of this came to fruition with PostgreSQL version 9.2 and three months ago we released support of Postgres 9.2 into GA. PostgreSQL version 9.2 is now the new default when provisioning a Heroku Postgres database.

You can read more about the powerful features in this version over on the Heroku Postgres blog.

Empowering Change: Programming Literacy for All

Richard — Fri, 12 Apr 2013 16:00:45 GMT

There has never been a better time to be a programmer. Every day more and more gadgets get connected or over-clocked. Programming is so prevalent that it often goes unnoticed in our daily lives. Whether we're scripting out social presence with IFTTT, or doing taxes with Excel, automation and programming has become an inescapable part of the modern world.

Heroku believes that to invest in our future, we must invest in programming literacy. While we're waiting for recursion to be a staple in our children’s classrooms, we can work on continuing and higher education today.

Heroku engineers are given opportunities and encouragement to be part of this movement. They’ve done so through supporting and participating in a number of groups including Hungry academy, Rails Girls, PyLadies, and more.

As a Heroku engineer I had a recent opportunity to teach a class in Ruby on Rails at the University of Texas in Austin. While nothing beats an in-classroom experience, it's not modular or scalable. In an effort to further scale programming literacy, we’ve been working to make this content available for everyone. After many re-takes, re-writes and hours of editing, we are happy to provide you with over 40 hours of video, lectures, exercises and quizes for free: Heroku Presents: UT on Rails.

The course will take a brand new developer up through the ranks, until they can build and deploy a fully functional website. If you or someone you know is interested in learning web programming, it's a great opportunity.

Staying Connected

At Heroku, we are encouraged to get involved with our respective communities. While we dogfood our product on a daily basis, there is no substitute for seeing how other developers actually work with it.

Developers don't have to teach a full length course to get involved with their communities. We've had developers help out at a number of community sponsored events that focus on getting more people engaged with technology. One of them, Rails Girls, has been so successful that we've had engineers participate at events in over 6 different countries.

Terence Lee at Rails Girls Amsterdam, photos by: Konstantin Haase

By staying connected, we help a new generation of programmers while making our own products better. It's a win-win situation.

Feedback Cycles

Learning is a feedback loop. You take an action, see the result, learn a lesson. The smaller the loop, the less time from action to result: the quicker you learn, the faster you advance. For students, using Heroku means they can spend less time worrying about their production environments and more focusing on their application logic.

Millions of application developers choose Heroku to get their features to market as quickly as possible. It's that same speed of delivery and deployment that also makes Heroku a natural choice for beginners.

The Next Generation

As we move further into an ever-connected society, developers and companies must do their part to promote programming and technology in the classroom. While the demand for tech jobs seems to be constantly increasing, where will tomorrow's senior developers come from? Perhaps they will come from programs like Let’s Learn Python at this years PyCon or through intensive focus courses like those offered by gSchool, or perhaps from general education courses. Wherever the next generation of programmers comes from, we want to set them up to succeed.

Heroku has always had a free tier which is perfect for prototyping and learning. Putting the right tools in a student’s hands can empower them to succeed. If you are picking up web development or know someone who is, consider introducing them to this Rails course and volunteering to help mentor them. The reward is far greater than the effort.

If you're teaching technology or programming and are using Heroku, we want to get to know you better. Please reach out and contact us, we're interested in your story. Help us invest in programming literacy, and make the future a better place.

2X Dynos in Public Beta

Nicolas — Fri, 05 Apr 2013 17:33:20 GMT

A dyno, the unit of computing power on Heroku, is a lightweight container running a single user-specified command. Today we’re announcing a dyno with twice the capacity: 2X dynos.

Existing dynos are now called 1X dynos. They come with 512MB of memory and 1x CPU share. They cost $0.05/hr. 2X dynos are exactly what their name implies: 1GB of memory and twice the CPU share for $0.10/hr. To support the growth of current and future apps on the platform, you can now control your dyno resources on two axes: size and quantity.

Let’s try them out.

Getting started with 2X dynos

Using the Heroku Toolbelt, resize your dynos with the resize command:

$ heroku ps:resize web=2X worker=1X
Resizing dynos and restarting specified processes... done
web dynos now 2X ($0.10/dyno-hour)
worker dynos now 1X ($0.05/dyno-hour)

To view the dyno size of a process type, use the ps command:

$ heroku ps
=== web (2X): `bundle exec unicorn -p $PORT -c ./config/unicorn.rb`
web.1: up 2013/03/27 14:27:58 (~ 6h ago)
web.2: up 2013/03/27 14:47:04 (~ 6h ago)
web.3: up 2013/03/27 15:08:23 (~ 5h ago)

=== worker (1X): `bundle exec rake worker:job`
worker.1: up 2013/03/27 14:39:04 (~ 6h ago)
worker.2: up 2013/03/27 15:08:24 (~ 5h ago)
worker.3: up 2013/03/27 14:30:55 (~ 6h ago)

Using the Dashboard use the app’s resources page:

For full instructions, see the Dev Center article.

Use Cases

You can already scale horizontally, adding more dynos with heroku ps:scale. When you want to scale vertically instead, use larger dynos with heroku ps:resize.

1. Concurrency for Rails with Unicorn - The first use case is increasing concurrency on single-threaded Rails apps using Unicorn. Due to the improved in-dyno queuing efficiency that results from increasing the number of Unicorn workers, doubling the workers in 2X dynos while halving the dyno count often results in better performance.

2. JVM Languages - The modern JVM has an explicit memory model designed for multi-threaded concurrency, and has many frameworks explicitly designed to take advantage of this property. Utilizing more threads requires more memory for both the thread stacks and for objects created by these threads. The JVM is fully capable of taking advantage of the vertical scale in a 2X dyno.

3. Memory-intensive background jobs - image processing and geospatial processing often need larger dynos. If you’ve gotten an R14 out-of-memory error and this is not a leak, 2X dynos might be for you.

Horizontal vs. Vertical Scale

2X dynos give you a new knob to turn: vertical scale. Vertical scaling is appealing when the consolidation of compute power and memory increases overall performance. This can translate into better app performance at the same cost, and possibly lower costs if the set of dynos can be reduced by more than half.

It's impossible to make blanket statements about app performance due to each app's unique performance characteristics. I/O-bound apps may not benefit from an increase in memory and compute, while process-heavy apps would. The best way to determine in what configuration your app runs best is to measure it. Heroku provides several tools to help you understand your application's behavior.

Use log2viz to understand how much memory and CPU is being consumed by your current app/dyno configuration. If the memory usage or web dyno activity indicators are close to their limits, 2X dynos may provide much-needed increase in resources. The latest New Relic add-on accurately displays request queue times and can be used to visualize the impact a higher-concurrency configuration has on your app. Use 2X dynos to double your in-dyno Unicorn workers and mitigate inconsistent and variable queue times.

Summary

Through the duration of the beta period, 2X dynos will cost the same as 1X dynos, that is $0.05 per hour. Once generally available they will cost $0.10 per hour.

If you’re looking to improve concurrency on Rails apps, making use of JVMs or other memory-hungry tasks, 2X dynos have a lot of potential to make apps faster. Give them a try.

FAQ

How can I measure how much memory my dynos are using?

The log-runtime-metrics Labs feature provides logged metrics that can be consumed and inspected with an add-on that consumes logs such as Papertrail, or displayed by a visualization tool like log2viz.

How do 2X dynos affect the 750 free dyno hours?

2X dynos consume twice as many free dyno-hours per hour as 1X dynos. Example: A 2X one dyno app will run for free for 375 hours compared to 750 hours for a 1X one dyno app.

Do single 2X dyno apps idle?

Yes. Idling affects single 1X and 2X dyno apps alike.

Can dyno size be configured on a per process-type basis?

Yes. Dyno size can be applied on a per process-type basis. For example: heroku ps:resize web=2X worker=1X worker2=2X

Can I get 4X (or larger) dynos?

We are exploring the option. If you have an app that may need these, send us a note.

Heroku Postgres Databases Patched

Craig — Thu, 04 Apr 2013 14:43:21 GMT

This post originally appeared on the postgres blog. We are also posting it in full here because we believe the content is so important.

Data is one of the most valuable assets of any company. As a database-as-a-service provider, one of our biggest responsibilities is ensuring your data is kept safe. A few weeks ago, one of the worst security vulnerabilities to date in PostgreSQL was discovered. To address this issue, Heroku deployed a point release upgrade across the entire Heroku Postgres service earlier this week. This resulted in a period of database unavailability, typically with a duration of less than one minute. Every database running on Heroku Postgres is now appropriately patched and is unaffected by the vulnerability.

PostgreSQL Vulnerability Details

The PostgreSQL project has provided official detail on CVE-2013-1899.

Several weeks ago there was a responsible disclosure of a serious security vulnerability within PostgreSQL by Mitsumasa Kondo and Kyotaro Horiguchi. The vulnerability allows unauthenticated remote users to use the ‘postmaster‘ process to write data to any accessible file, including critical internal database files.

The vulnerability was fixed and then committed to the PostgreSQL’s private git repository, but only after updates to anonymously accessible copies were disabled. Updated versions of PostgreSQL were released today to most large packaging repositories, as well as source code and installers.

Heroku Postgres Patching

The Heroku Postgres team worked with the PostgreSQL community to ensure we would be able to rapidly apply this patch. However, due to the nature of the issue, and aiming to mitigate risk for others, we were not able to discuss specifics until now. Our goal — in addition to ensuring your data was safe — was to continue monitoring this upgrade as it was deployed, providing early feedback to the community should bugs be found, and not jeopardizing in any way the coordinated public disclosure process stewarded by the PostgreSQL community. Most importantly, the PostgreSQL source code that included the patch was held in the utmost secrecy. In addition, the deployment plan was reviewed by PostgreSQL community members in advance.

Once the source code was released to the PostgreSQL packagers—of which a member of the Heroku Postgres staff is a part of—we began applying this patch to all Heroku Postgres databases, with the first updates starting on Monday. As of Wednesday at 6:30 PM PDT, all Heroku Postgres databases had been upgraded to their appropriate point release and were no longer vulnerable to CVE-2013-1899.

Conclusion

We realize that having no control over a maintenance window, however brief, is among the worst possible experiences. We are very sorry. Two reasons prevented us from working with you to schedule the security update. First, we prioritize ensuring your data is safe above all else, as a result making sure that every database was patched before this exploit was weaponized was paramount. Secondly, this was the first time we've had to deal with a security update of this scale, and have no machinery in place to schedule upgrades of this sort. Spending time to build such machinery would have prevented us from having every database patched in time. We will continue to work on improving our process around such maintenance to provide a better experience in the future.

As of late Wednesday all Heroku Postgres databases were upgraded and no longer at risk of CVE-2013-1899. No further action is required on your part to ensure your data remains safe.

Routing and Web Performance on Heroku: a FAQ

Adam — Wed, 03 Apr 2013 17:02:27 GMT

Hi. I'm Adam Wiggins, cofounder and CTO of Heroku.

Heroku has been my life’s work. Millions of apps depend on us, and I take that responsibility very personally.

Recently, Heroku has faced criticism from the hacker community about how our HTTP router works, and about web performance on the platform in general. I’ve read all the public discussions, and have spent a lot of time over the past month talking with our customers about this subject.

The concerns I've heard from you span past, present, and future.

The past: some customers have hit serious problems with poor web performance and insufficient visibility on their apps, and have been left very frustrated as a result. What happened here? The present: how do you know if your app is affected, and if so what should you do? And the future: what is Heroku doing about this? Is Heroku a good place to run and scale an app over the long term?

To answer these questions, we’ve written a FAQ, found below. It covers what happened, why the router works the way that it does, whether your app is affected by excessive queue time, and what the solution is.

As to the future, here’s what we’re doing. We’re ramping up hands-on migration assistance for all users running on our older stack, Bamboo, or running a non-concurrent backend on our new stack, Cedar. (See the FAQ for why this is the fix.) We’re adding new features such as 2X dynos to make it easier to run concurrent backends for large Rails apps. And we're making performance and visibility a bigger area of product attention, starting with some tools we've already released in the last month.

If you have a question not answered by this FAQ, post it as a comment here, on Hacker News, or on Twitter. I’ll attempt to answer all such questions posted in the next 24 hours.

To all our customers who experienced real pain from this: we're truly sorry. After reading this FAQ, I hope you feel we're taking every reasonable step to set things right, but if not, please let us know.

Adam

Overview

Q. Is Heroku’s router broken?

A. No. While hundreds of pages could be written on this topic, we’ll address some of this in Routing technology. Summary: the current version of the router was designed to provide the optimum combination of uptime, throughput, and support for modern concurrent backends. It works as designed.

Q. So what’s this whole thing about then?

A. Since early 2011, high-volume Rails apps that run on Heroku and use single-threaded web servers sometimes experienced severe tail latencies and poor utilization of web backends (dynos). Lack of visibility into app performance, including incorrect queue time reporting prior to the New Relic update in February 2013, made diagnosing these latencies (by customers, and even by Heroku’s own support team) very difficult.

Q. What types of apps are affected?

A. Rails apps running on Thin, with six or more dynos, and serving 1k reqs/min or more are the most likely to be affected. The impact becomes more pronounced as such apps use more dynos, serve more traffic, or have large request time variances.

Q. How can I tell if my app is affected?

A. Add the free version of New Relic (heroku addons:add newrelic) and install the latest version of the newrelic_rpm gem, then watch your queue time. Average queue times above 40ms are usually indicative of a problem.

Some apps with lower request volume may be affected if they have extremely high request time variances (e.g., HTTP requests lasting 10+ seconds) or make callbacks like this OAuth example.

Q. What’s the fix?

A. Switch to a concurrent web backend like Unicorn or Puma on JRuby, which allows the dyno to manage its own request queue and avoid blocking on long requests.

This requires that your app be on our most current stack, Cedar.

Q. Can you give me some help with this?

A. Certainly. We’ve already emailed all customers with apps running on Thin with more than six dynos with self-migration instructions, and a way to reach us for direct assistance.

If you haven’t received the email and want help making the switch, contact us for migrating to Cedar or migrating to Unicorn.

Routing technology

Q. Why does the router work the way that it does?

A. The Cedar router was built with two goals in mind: (1) to support the new world of concurrent web backends which have become the standard in Ruby and all other language communities; and (2) to handle the throughput and availability needs of high-traffic apps.

Read detailed documentation of Heroku’s HTTP routing.

Q. Even with concurrent web backends, wouldn’t a single global request queue still use web dynos more efficiently?

A. Probably, but it comes with trade-offs for availability and performance. The Heroku router favors availability, stateless horizontal scaling, and low latency through individual routing nodes. Per-app global request queues require a sacrifice on one or more of these fronts. See Kyle Kingsbury’s post on the CAP theorem implications for global request queueing.

After extensive research and experimentation, we have yet to find either a theoretical model or a practical implementation that beats the simplicity and robustness of random routing to web backends that can support multiple concurrent connections.

Q. So does that mean you aren’t working on improving HTTP performance?

A. Not at all. We're always looking for new ways to make HTTP requests on Heroku faster, more reliable, and more efficient. For example, we’ve been experimenting with backpressure routing for web dynos to signal to the router that they are overloaded.

You, our customers, have told us that it’s not routing algorithms you ultimately care about, but rather overall web performance. You want to serve HTTP requests as quickly as possible, for fast page loads or API calls for your users. And you want to be able to quickly and easily diagnose performance problems.

Performance and visibility are what matters, and that’s what we’ll work on. This will include ongoing improvements to dynos, the router, visibility tools, and our docs.

Retrospective

Q. Did the Bamboo router degrade?

A. Yes. Our older router was built and designed during the early years of Heroku to support the Aspen and later the Bamboo stack. These stacks did not support concurrent backends, and thus the router was designed with a per-app global request queue. This worked as designed originally, but then degraded slowly over the course of the next two years.

Q. Were the docs wrong?

A. Yes, for Bamboo. They were correct when written, but fell out of date starting in early 2011. Until February 2013, the documentation described the Bamboo router only sending one connection at a time to any given web dyno.

Q. Why didn’t you update Bamboo docs in 2011?

A. At the time, our entire product and engineering team was focused on our new product, Cedar. Being so focused on the future meant that we slipped on stewardship of our existing product.

Q. Was the "How It Works" section of the Heroku website wrong?

A. Yes. Similar to the docs, How It Works section of our website described the router as tracking which dynos were tied up by long HTTP requests. This was accurate when written, but gradually fell out of date in early 2011. Unlike the docs, we completely rewrote the homepage in June of 2011 and it no longer referenced tracking of long requests.

Q. Was the queue time metric in New Relic wrong?

A. Yes, for the same 2011—2013 period from previous questions. The metric was transmitted to the New Relic instrumentation in the app via a set of HTTP headers set by the Heroku router. The root cause was the same as the Bamboo router degradation: the code didn't change, but scaling out the router nodes caused the data to become increasingly inaccurate and eventually useless. With New Relic's help, we fixed this in February 2013 by calculating queue time using a different method.

Q. Why didn’t Heroku take action on this until Rap Genius went public?

A. We’re sorry that we didn’t take action on this based on the customer complaints via support tickets and other channels sooner. We didn’t understand the magnitude of the confusion and frustration caused by the out-of-date Bamboo docs, incorrect queue time information in New Relic, and the general lack of visibility into web performance on the platform. The huge response to the Rap Genius post showed us that this touched a nerve in our community.

The Future

Q. What are we doing to make things right from here forward?

A. We’ve been working with many of our customers to get their queue times down, get them accurate visibility into their app’s performance, and make sure their app is fast and running on the right number of dynos. So far, the results are good.

Q. What about everyone else?

A. If we haven’t been in touch yet, here’s what we’re doing for you:

Migration assistance: We’ll give you hands-on help migrating to a concurrent backend, either individually or in online workshops. This includes the move to Cedar if you’re still on Bamboo. If you’re running a multi-dyno app on a non-concurrent backend and haven’t received an email, drop us a line about Thin to Unicorn or Bamboo to Cedar.
2X dynos: We’re fast-tracking the launch of 2X dynos, to provide double the memory and allow for double (or more) Unicorn concurrency for large Rails apps. This is already available in private beta in use by several hundred customers, and will be available in public beta shortly.
New visibility tools: We’re putting more focus on bringing you new performance visibility features, such as the log2viz dashboard, CPU and memory use logging, and HTTP request IDs. We’ll be working to do much more on this front to make sure that you can diagnose performance problems when they happen and know what to do about it.

Want something else not mentioned here? Let us know.

Helios - open source framework for mobile

Mattt — Tue, 02 Apr 2013 19:11:59 GMT

Heroku has a strong tradition with open source projects. Engineers have dedicated countless hours to the projects that developers count on every day. Open Source Software is in our DNA.

Speaking personally, I’m passionate about building tools like AFNetworking and cupertino, in order to help developers build insanely great experiences for mobile devices. It’s with great pleasure that I introduce something new I’ve been working on:

Helios is an open-source framework that provides essential backend services for iOS apps. This includes data synchronization, push notifications, in-app purchases, and passbook integration. It allows developers to get a client-server app up-and-running while seamlessly incorporating functionality as necessary.

Helios is designed for "mobile first" development. Build out great features on the device, and implement the server-side components as necessary. Pour all of your energy into crafting a great user experience, rather than getting mired down with the backend.

Built on the Rack webserver interface, Helios can be easily added into any existing Rails or Sinatra application. Or, if you're starting with a Helios application, you can build a new Rails or Sinatra application on top of it. This means that you can develop your application using the tools and frameworks you love, and maintain flexibility with your architecture as your needs evolve.

Give it a try and let me know what you think!

Waza 2013: How Ecosystems Build Mastery

Dana — Wed, 20 Mar 2013 19:57:35 GMT

When we think of the concept of Waza (技) or "art and technique," it's easy to get caught up in the idea of individual mastery. It's true that works of art are often created by those with great skill, but acquiring that skill is neither solitary nor static. Generations of masters contribute to a canon and it is in that spirit that we built the Heroku platform and the Waza event. This year's Waza was no exception.

On February 28th, more than 900 attendees participated in Waza including Ruby founder Yukihiro "Matz" Matsumoto, Django co-creator Jacob Kaplan-Moss and Codeacademy’s Linda Liukas. True to form, we offered you a platform for experimentation and you surprised us with your contributions.

From your origami creations, to your Arduino hacks, to the technical conversations over craft beer -- you taught us that the definition of software development is ever-evolving. Thank you for allowing us to help you change lives and push boundaries. We will continue our commitment to growing the platform for you and look forward to collaborating with you in the future.

For more event highlights visit the Waza videos and photos. To learn more about Heroku, add yourself to our mailing list.

log2viz: Logs as Data for Performance Visibility

Adam — Tue, 19 Mar 2013 18:36:53 GMT

If you’re building a customer-facing web app or mobile backend, performance is a critical part of user experience. Fast is a feature, and affects everything from conversion rates to your site’s search ranking.

The first step in performance tuning is getting visibility into the app’s web performance in production. For this, we turn to the app’s logs.

Logs as data

There are many ways to collect metrics, the most common being direct instrumentation into the app. New Relic, Librato, and Hosted Graphite are cloud services that use this approach, and there are numerous roll-your-own options like StatsD and Metrics.

Another approach is to send metrics to the logs. Beginning with the idea that logs are event streams, we can use logs for a holistic view of the app: your code, and the infrastructure that surrounds it (such as the Heroku router). Mark McGranaghan’s Logs as Data and Ryan Daigle’s 5 Steps to Better Application Logging offer an overview of the logs-as-data approach.

Put simply, logs as data means writing semi-structured data to your app's logs via STDOUT. Then the logs can be consumed by one or more services to do dashboards, long-term trending, and threshold alerting.

The benefits of logs-as-data over direct instrumentation include:

No additional library dependencies for your app
No CPU cost to your dyno by in-app instrumentation
Introspection capability by reading the logs directly
Metrics backends can be swapped out without changes to app code
Possible to split the log stream and send it to multiple backends, for different views and alerting on the same data

Introducing log2viz, a public experiment

log2viz is an open-source demonstration of the logs-as-data concept for Heroku apps. Log in and select one of your apps to see a live-updating dashboard of its web activity.

For example, here’s a screenshot of log2viz running against the Rubygems Bundler API (written and maintained by Terence Lee, André Arko, and Larry Marburger, and running on Heroku):

log2viz gets all of its data from the Heroku log stream — the same data you see when running heroku logs --tail at the command line. It requires no changes to your app code and works for apps written in any language and web framework, demonstrating some of the benefits of logs as data.

Also introducing: log-runtime-metrics

In order to get memory use stats for your dynos, we’ve added a new experimental feature to Heroku Labs to log CPU and memory use by the dyno: log-runtime-metrics.

To enable this for your app (and see memory stats in log2viz), type the following:

$ heroku labs:enable log-runtime-metrics -a myapp
$ heroku restart

This inserts data into your logs like this:

heroku[web.1]: measure=load_avg_5m val=0.0
heroku[web.1]: measure=memory_total val=209.64 units=MB

log2viz reads these stats and displays average and max memory use across your dynos. (Like all Labs features, this is experimental and the format may change in the future.)

Looking under the hood

log2viz is open source. Let’s look at the code:

It uses OAuth via heroku-bouncer to connect to your Heroku account and make API calls to access information about your apps.
It reads from your app's log stream. This is the same API endpoint that the Heroku CLI connects to when you run heroku logs --tail.
It's written in Ruby and uses the Sinatra streaming API (an EventMachine implementation of Server-sent Events) to handle both the incoming log data and outgoing live updates to the user's browser.
It uses the Heroku API when you connect to the page to fetch the app's name and number of web dynos. It takes a hint for concurrency per dyno based on the WEB_CONCURRENCY config var.
All of the metrics are derived from 60 seconds of data, stored in-memory in the user’s browser. (Storing historical data would enable trend analysis, but we chose to leave that out of scope for this experiment.)
log2viz only subscribes to the app’s logs while the page is open in your browser. Closing the window also closes the log connection.

You can deploy your own copy of log2viz on Heroku, so fork away! For example, Heroku customer Timehop has experimented with trending graphs via Rickshaw.

Logs-as-data add-ons

log2viz isn't the only way to take advantage of your log stream for visibility on Heroku today. Here are a few add-ons which consume your app's logs.

Loggly offers a web console that lets you search your log history, and graph event types over time. For example, let’s search for status=404, logged by the Heroku router whenever your app serves a page not found:

Papertrail offers search archival and history, and can also alert on events when they pass a certain threshold. Here’s how you can set up an email alert every time your app experiences more than 10 H12 errors in a 60 second period. Search for the router log line:

Click “Save Search,” then:

Other add-ons that consume logs include Treasure Data and Logentries.

You can also use non-add-on cloud services, as shown in thoughtbot's writeup on using Splunk Storm with Heroku.

Conclusion

Visibility is a vast and challenging problem space. The logs-as-data approach is still young, and log2viz is just an experiment to get us started. We look forward to your feedback on log2viz, log visibility via add-ons, and your own experiments on performance visibility.

Running Rails on Heroku Update

Oren — Wed, 13 Mar 2013 21:11:12 GMT

On February 16th, we published a blog post outlining five specific and immediate actions we would take to improve our Rails customers' experience with Heroku. We want to provide you with an update on where these things stand. As a reminder, here’s what we committed to do:

Improve our documentation so that it accurately reflects how our service works across both Bamboo and Cedar stacks
Remove incorrect and confusing metrics reported by Heroku or partner services like New Relic
Add metrics that let customers determine queuing impact on application response times
Provide additional tools that developers can use to augment our latency and queuing metrics
Work to better support concurrent-request Rails apps on Cedar

We have resolved the first two items:

Improving Documentation: We’ve updated our Dev Center docs and website to more accurately describe how routing occurs on Heroku (e.g., HTTP Routing on Bamboo and How Routing Works on the Bamboo Stack).
Removing Incorrect Metrics: We’ve worked with New Relic to release an updated version of their monitoring tools, which we documented in Better Queuing Metrics With Updated New Relic Add-on.

We are working on the remaining three items in partnership with our early beta users:

Adding New Metrics: We are improving the data available through our logs in two ways. First, we’re adding a transaction ID to every request log. Second, we are working on application middleware to provide additional metrics in customer logs.
Providing Additional Measurement Tools: We have created a data visualization tool that analyzes log data in real-time and helps observe an app’s performance across a few key Heroku-specific metrics. This tool is currently being beta tested.
Improving Cedar Support: Unicorn is now the recommended Rails app server. We are currently beta testing larger dynos to support additional unicorn processes.

Fulfilling these commitments is Heroku’s number one priority. We have dedicated engineering and product teams focused on improving the performance of Rails apps on Heroku.

We’ll detail the availability of our improvements in future blog posts. If you would like to beta test any of these new features, please drop us a note and let us know which features you are interested in testing.

Jacob Kaplan-Moss, Django Co-Creator, Talks Ecosystems at Heroku's Waza

Craig — Sat, 09 Mar 2013 19:36:15 GMT

The value of a network is proportional to the square of the number of users connected to the system, according to Metcalfe’s law. Jacob Kaplan-Moss, co-creator of Django, highlights this as a value in creating communities or as he puts it, “ecosystems”. In his talk at Waza last week on building ecosystems, he went on to highlight three key principles of creating ecosystems:

APIs to support extensibility
Conservatism as a value
Empowering the community

Whether building a platform or a framework, these key principles ring true. Check out his talk or read more on creating ecosystems below:

APIs to support extensibility

You don’t choose to become an ecosystem, but you can choose to do the groundwork. By defining APIs you enable others to build on the foundation you lay. Django core member Russ Keith-McGee summed this up quite clearly on the Django mailing list – "I’m not sure if this is appropriate, but it should be possible. If its not possible we should build the APIs to support it." This sentiment is at the core of building ecosystems.

Django succeeded at this with its apps API, and has a full directory in Django Packages to help others discover what has already been built. At Heroku we’re seeing the same phenomon with our buildpacks API resulting in numerous third-party buildpacks. By building APIs and empowering users, you solve more problems.

Conservatism as a value

There’s great value in the “move fast and break stuff” concept. However, creating explicit APIs and contracts that are stable is critical for an ecosystem to thrive longer term. For Django as a framework this means being slow moving and reliable when it comes to modifying any APIs in the core framework. For Heroku as a platform it means a dedication to erosion resistance. Jacob cleanly highlights this by pointing out that “Interoperabilty only works when the common parts don’t change.”

Empowering the community

Ecosystems create ways for the community to engage and contribute. With an ecosystem there’s even additional ways in which users can contribute and those contributions don’t have to be code. When a project first starts, it’s creator tests it and consumes it acting as developer and QA. As it gains popularity a project may attract new developers, then more users, who submit their own bug reports and issues. These newcomers contribute documentation patches, triage tickets, and recruit others to help with the project. Where once a single developer committed quietly, now a thriving ecosystem of contributors and consumers exists.

This change from product to ecosystem doesn’t happen overnight, it must be nurtured and nudged. Kaplan-Moss, a self-proclaimed “documentation nut” has done just that by cultivating a better understanding of Django through documentation and carefully working with patch and issue submitters. Jacob explains that if you document your process, you can increase transparency and understanding of the way things work. Django does this by clearly documenting how users can contribute back to Django; at Heroku we’ve done this by documenting our methodology on how apps should be built.

Conclusion

Building a great product isn’t enough, as Jacob points out: you’ve got to create a vibrant community and build an ecosystem. If it wasn’t for Django’s community there wouldn’t be over 3500 Python packages for Django. If it wasn’t for our community, you wouldn’t be able to deploy Go, common lisp, or GeoDjango to Heroku with just one command.

Learn more about ecosystems by listening to Jacob’s talk, check out other talks from Waza available now, or engage with the ecosystem of your choice today.

Matz on Ruby 2.0 at Heroku's Waza

Craig — Wed, 06 Mar 2013 22:53:28 GMT

Matz, the creator of Ruby, spoke at Waza for the 20th anniversary of the language and the release of Ruby 2.0. If you weren't in the sold out crowd, not to worry. Information should flow free and experiences should be shared; in line with those concepts you can watch Matz's talk right here, then read about what's new in this version of Ruby and how to run it on Heroku.

With slides available on speakerdeck

Keep reading for more information on Ruby 2.0 or check out our first batch of videos from Waza 2013. To stay up to date as we post new videos, follow us @heroku.

Compatibility

Iterating quickly means happier developers and happier customers. We optimize our platform to allow you to develop, stage, and deploy faster. Not only does this make running software easier, but it makes trying new technologies like Ruby 2.0 as simple as spinning up a new app. During Matz's talk at Waza, he mentioned that, while 1.9.3 is popular now, it took years after 1.8.7 was released to gain traction. With the release of Ruby 2.0 Matz hopes to reduce upgrading barriers and allow developers to iterate quicker using newer, faster and better tools.

Ruby 2.0 was written to be backwards compatible and it works with Rails 3.2.13 out of the box. If your Ruby apps are running using 1.8.7, you should upgrade. Ruby 1.8.7 is approaching End of Life (EOL) in three months on June 2013. EOL for Ruby 1.8.7 means no security or bug patches will be provided by the maintainers. Not upgrading means you're potentially opening up your application and your users to vulnerabilities. Don't wait till the final hour, upgrade now to be confident and secure.

Speed

Ruby 2.0 has a faster garbage collector and is Copy on Write friendly. Copy on Write or COW is an optimization that can reduce the memory footprint of a Ruby process when it is copied. Instead of allocating duplicate memory when a process is forked, COW allows multiple processes to share the same memory until one of the processes needs to modify a piece of information. Depending on the program, this optimization can dramatically reduce the amount of memory used to run multiple processes. Most Ruby programs are memory bound, so reducing your memory footprint with Ruby 2.0 may allow you to run more processes in fewer dynos.

If you’re not already running a concurrent backend consider trying the Unicorn web server.

Features

In addition to running faster than 1.9.3, and having a smaller footprint, Ruby 2.0 has a number of new features added to the language including:

Keyword arguments
Kernel#require optimization which makes Rails startup very fast
Copy on write
Dtrace support
Module#prepend
Enumerable#lazy
Use __dir__ to get the current directory of __FILE__
UTF-8 is now the default encoding
Much, much more

The list of new features is more than we can cover here. If you really wanted to dig in you can check the Ruby changelog

Running 2.0 on Heroku

If you’re interested in taking advantage of these new features give it a try on Heroku today. To run Ruby 2.0 on Heroku you'll need this line in your Gemfile:

ruby "2.0.0"

Then commit to git:

$ git add .
$ git commit -m "Using Ruby 2.0 in production"

We recommend that you test your app using 2.0 locally and deploy to a staging app before pushing to production. Now when you $ git push heroku master our Ruby buildpack will see that you've declared your Ruby version and make sure you get the right one.

20 years of simplicity, elegance, and programmer happiness

Heroku, since its founding, has been aligned with the key values of Ruby – simplicity, elegance, and programmer happiness. Heroku still believes in the power and flexibility of Ruby, and we've invested in the language by hiring Yukihiro "Matz" Matsumoto, Koichi Sasada and Nobuyoshi Nakada. We would like to thank them and the whole Ruby core team for making this release happen. Join us in celebrating Ruby's successes and in looking forward to the next twenty years by trying Ruby 2.0 on Heroku today.

Idea to Delivery: Application Development in the Modern Age. Adam Wiggins at Waza 2012 [video]

Craig — Wed, 27 Feb 2013 23:05:55 GMT

Great coders know their technology intimately. And they know how to choose it. Truly awesome application developers know more. They know the human side of technology. They know technique. They focus on their method—their practice.

In 2000 Heroku co-founder and CTO Adam Wiggins saw this more clearly than ever before. He read The Pragmatic Programmer by Andy Hunt and Dave Thomas. The book, as Adam explains in this thought-provoking (and method-shifting) Waza talk, showed him that his work could not only be about technology. It HAD to be about technique.

Heroku Co-founder, CTO and general badass, Adam Wiggins

Adam discusses techniques—historical ones such as agile development and the power of rapid and flexible response to change; frameworks which helped drive speed by allowing developers to focus on the specifics of their application without having to reinvent basic wheels of, say, state management and request handling; the cloud which removed a huge burden of selecting, purchasing and maintaining hardware; and Software as a Service (SaaS), which, in addition to providing incredible benefit to end-users has created a development culture of continual, rapid improvement.

Technique means a lot. How we think about and describe what we do means a great deal too. Adam makes a strong point when he compares thinking and describing ourselves as programmers vs. application developers. Application developers, Adam says, think more broadly. They think about the end-to-end process of developing and deploying an application. vs.

vs.

By taking ownership of thinking across the full spectrum from idea to delivery, application developers play a far more strategic role in their app—and their company’s—growth and success. Developers who think and work like this are truly “the new kingmakers” and a “powerful force to be reckoned with.”

Adam shares newer, even more powerful techniques that will help a developer who wants to think more broadly, act more strategically, and increase efficiency in her or his organization. Adam also discusses a number of these in The Twelve-Factor App:

Deploying from Day One and Continuous Development/Deployment

Having one (version controlled) codebase that is deployed in various states of completion across several instances from a live production app/site to development environments of different employees’ machines means we can all move faster and we can all stay in tune with one another. It means we can be deploying from day one and we can rapidly improve our products via continuous development and deployment.

Development and Production Parity: Keeping development, staging, and production as similar as possible

Historically, there have been substantial gaps between development (a developer making live edits to a local deploy of the app) and production (a running deploy of the app accessed by end users). These gaps, as Adam discussed at Waza and at The Twelve-Factor App manifest in three areas:

The time gap: A developer may work on code that takes days, weeks, or even months to go into production.
The personnel gap: Developers write code, ops engineers deploy it.
The tools gap: Developers may be using a stack like Nginx, SQLite, and OS X, while the production deploy uses Apache, MySQL, and Linux.

Staying Close to Production

As we've come to appreciate agile development techniques and put such a sharp focus on shipping features, minimizing each of these gaps allows developers to:

Make the time gap small: a developer may write code and have it deployed hours or even just minutes later.
Make the personnel gap small: developers who wrote code are closely involved in deploying it and watching its behavior in production.
Make the tools gap small: keep development and production as similar as possible.

Conclusion

We have come along way since The Pragmatic Programer. But it remains highly influential and set much of the tone for the many pragmatic developments in technique and practice that have come to the fore in recent years. You could say that Adam reading The Pragmatic Programmer back in 2000 is one of the reasons we invite developers to come together for Waza, which happens tomorrow. And one of the reasons we share the talks freely online for those who cannot make it. Waza is all about technique, about personal improvement for developers, about, as the subtitle of The Pragmatic Programmer says, the journey from journeyman to master.

Adding Concurrency to Rails Apps with Unicorn

Richard — Wed, 27 Feb 2013 03:35:10 GMT

With support for Node.js, Java, Scala and other multi-threaded languages, Heroku allows you to take full advantage of concurrent request processing and get more performance out of each dyno. Ruby should be no exception.

If you are running Ruby on Rails with Thin, or another single-threaded server, you may be seeing bottlenecks in your application. These servers only process one request at a time and can cause unnecessary queuing. Instead, you can improve performance by choosing a concurrent server such as Unicorn which will make your app faster and make better use of your system resources. In this article we will explore how Unicorn works, how it gives you more processing power, and how to run it on Heroku.

Concurrency and Forking

At the core of Heroku is the Unix Philosophy, and we see this philosphy at work in Unicorn. Unicorn uses the Unix concept of forking to give you more concurrency.

Process forking is a critical component of Unix's design. When a process forks it creates a copy of itself. Unicorn forks multiple OS processes within each dyno to allow a Rails app to support multiple concurrent requests without requiring them to be thread-safe. This means that even if your app is only designed to handle one request at a time, with Unicorn you can handle concurrent connections.

Unicorn leverages the operating system to do most of the heavy lifting when creating and maintaining these forks. Unix-based systems are extremely efficient at forking, and even take advantage of Copy on Write optimizations that are similar to those in the recently released Ruby 2.0.

Unicorn on Rails

By running Unicorn in production, you can significantly increase throughput per dyno and avoid or reduce queuing when your app is under load. Unicorn can be difficult to setup and configure, so we’ve provided configuration documentation to make it easier to get started.

Let's set up a Rails app to use Unicorn.

Setting up Unicorn

First, add Unicorn to your application Gemfile:

gem 'unicorn'

Run $ bundle install, now you are ready to configure your app to use Unicorn.

Create a configuration file for Unicorn at config/unicorn.rb:

$ touch config/unicorn.rb

Now we're going to add Unicorn-specific configuration options, that we explain in detail in Heroku's Unicorn documentation:

# config/unicorn.rb
worker_processes 3
timeout 30
preload_app true

before_fork do |server, worker|

  Signal.trap 'TERM' do
    puts 'Unicorn master intercepting TERM and sending myself QUIT instead'
    Process.kill 'QUIT', Process.pid
  end

  defined?(ActiveRecord::Base) and
    ActiveRecord::Base.connection.disconnect!
end

after_fork do |server, worker|

  Signal.trap 'TERM' do
    puts 'Unicorn worker intercepting TERM and doing nothing. Wait for master to sent QUIT'
  end

  defined?(ActiveRecord::Base) and
    ActiveRecord::Base.establish_connection
end

This default configuration assumes a standard Rails app with Active Record, see Heroku's Unicorn documentation for more information. You should also get acquainted with the different options in the official Unicorn documentation.

Now that we've got your app setup to use Unicorn, you’ll need to tell Heroku how to run it in production.

Unicorn in your Procfile

Change the web command in your Procfile to:

web: bundle exec unicorn -p $PORT -c ./config/unicorn.rb

Now try running your server locally with $ foreman start. Once you're happy with your changes, commit to git, deploy to staging, and when you're ready deploy to production.

A World of Concurrency

With the recent release of the Rails 4 beta, which is threadsafe by default, it's becoming increasingly clear that Rubyists care about concurrency.

Unicorn gives us the ability to take multiple requests at a time, but it is by no means the only option when it comes to concurrent Rack servers. Another popular alternative is Puma which uses threads instead of forking processes. Puma does however require that your code is threadsafe.

If you've never run a concurrent server in production, we encourage you to spend some time exploring the ecosystem. After all no one knows your app's requirements better than you.

Whatever you do don't settle for one request at a time. Demand performance, demand concurrency, and try Unicorn today.

Concurrency is not Parallelism. Rob Pike at Waza 2012 [video]

Craig — Sun, 24 Feb 2013 20:11:50 GMT

In planning Waza 2013 we went back to reflect on last year’s speakers. And we want to make the talks readily availble to anybody who could not make it last year—or who wants a refresher. Check back soon for more talks from Waza 2012. And we hope to see you in person at Waza 2013 coming up FAST on Feb. 28 in San Francisco.

In a world of evolving languages, frameworks and development patterns, we developers must continually improve our craft. Innovative developers already have jumped on board many of these shifts. We’ve seen this with the adoption of more agile frameworks (such as Rails, Django, and Play). We’ve seen it too with a shift towards asynchronous programming patterns such as in Node.js and with evented programming in Rails.

One clear example of this evolution is the re-emergence of a focus on concurrency.

Rob Pike—with the help of a few gophers—gave this fantastic educational talk on concurrency at last year’s Heroku waza conference. Rob covered big themes that are important to developers—speed, efficiency and productivity. And he covered parallelism and concurrency in programming processes—making it very clear that they are not the same thing. If you want to click through Rob’s slides while watching, they are hosted at GoogleCode.

Rob (@rob_pike) is a software pioneer. His influence is everywhere: Unix, Plan 9 OS, The Unix Programming Environment book, UTF-8, and most recently the Go programming language.

Waza is the Japanese word for art and technique and it's where we celebrate craft and the creative process of software development with technical sessions and interactive artistic happenings.

Better Queuing Metrics With Updated New Relic Add-On

Jesper — Thu, 21 Feb 2013 14:00:28 GMT

Today our partner, New Relic, released an update to the Ruby New Relic agent that addresses issues brought up by our customers. The new version corrects how New Relic reports performance metrics for applications running on Heroku. Queueing time is now reported as the total time from when a request enters the router until the application starts processing it. Previous versions of New Relic only reported queueing time in the router. The new approach will result in more accurate queueing metrics that allow you to better understand and tune the performance of your application.

Update, Feb 22:

New Relic has released a similar update for Python. Python developers should update to this latest version to benefit from the improved metrics. JVM language developers do not need to take any action. The current New Relic Java agent already includes the improved queue time metrics.

Install or update the New Relic Ruby Add-on

If you are already using New Relic with your Ruby apps, then simply update your Gemfile to reference the new agent version:

gem "newrelic_rpm", "~> 3.5.7.59"

then run

$ bundle update newrelic_rpm
$ git add Gemfile Gemfile.lock
$ git commit -m 'update new relic agent'
$ git push heroku master

If you are not yet using New Relic, you can learn how to install and configure the add-on on Dev Center.

How It Works

The updated New Relic agent uses an improved strategy for reporting request queue times on Heroku. Prior to this update, New Relic reported request queue time using a value set by the Heroku routing mesh. This only reflected the time a request spent in the router queue and did not properly include time spent waiting in the dyno’s request queue.

Our routing performance update documents our finding that some applications have requests that may spend significant time queued on dynos. To help our customers understand precisely where their applications are being delayed, the updated New Relic agent includes dyno wait time in the total queue time metric. The new queue time is calculated as the difference between the time the Heroku router first gets a request and the time the request begins processing on the dyno. The result is a more accurate picture of how long requests wait in queues.

Clock Skew

The new version of New Relic calculates queue times using two different clocks — the dyno and router clocks. While Heroku servers regularly sync their clocks, it’s common for clocks to drift apart between syncs. This phenomenon is known as clock skew and it can affect the queue time metric collected by New Relic. In our experience, even though clock skew can cause small inaccuracies, the overall trend data displayed by New Relic will still accurately reflect your application’s queue times.

How to Learn More

If you’d like more information on how to install and configure the New Relic add-on, please see the New Relic Dev Center article and the Unicorn specific instructions. For general suggestions on how to improve the performance of your app, check out our performance overview page.

What’s Happening at Waza

Sara — Wed, 20 Feb 2013 16:04:55 GMT

Waza (技) 2013 is only a week away and the schedule is packed with amazing speakers and hands-on craft experiences. We can’t wait to share this day with all of you. If you haven’t yet, register now before it’s too late!

This year, Waza will have three stages with a total of 20 talks. The rest of the venue is packed with lounges, co-working spaces, snack and beverage stations, and, thanks to our sponsors, all kinds of interactive, craft-based activities to fuel your creative mind.

Hands-on Crafts

In addition to our great sponsored happenings, we have quilting, dye-making and printmaking artists on hand. Come experience their unique crafts, hands-on and up-close.

Quilting and Dye-making: Maura Grace Ambrose is bringing her Folk Fibers all the way from Austin, TX. Maura collects natural materials to dye fabrics then uses them to stitch together special quilts. Join Maura in the hands-on creation of a custom Waza quilt.

Printmaking: Marissa Marquez joins us at Waza for the second time. Marissa uses woodworking tools to hand-carve original designs into blocks and stamp them onto paper. She has created some beautiful prints for Waza which you can use to print your own postcard, or she can teach you how to make your own.

All Things Delicious

Blue Bottle Coffee: We’re a bit obsessed with coffee at Heroku. And it’s an obsession we like to share. Doors open at 10am for badge pickup, show up early and enjoy a cup of pour-over coffee while you get to know some Herokai. But don’t worry, this cup-at-a-time coffee service will be available all day.

Tea Lounge: We know some people prefer tea, including many of our own staff, so we’ve set aside space for the Waza tea lounge where you’ll find a variety of loose-leaf teas.

Food Trucks: We will have an assortment of local food trucks offering a selection of lunch specials. Use the ticket on your badge and select your favorite.

Don’t Want to Deal with Parking?

Secure bike parking will be available thanks to the fine folks at the San Francisco Bicycle Coalition.

Meet our Sponsors

Atlassian

Well-known for their collaboration tools that help teams build better products, Atlassian is providing Waza with co-working spaces to get the job done.

DODOcase

Creating artisan products for technology is the core of DODOcase’s business. Stop by and create your own Waza-branded, hand-bound notepad to take home .

Github

We all know that Github knows how to throw a great party, so we’re pumped that they are sponsoring the Waza afterparty. The fun starts at 9 p.m., and everyone with a Waza badge is invited.

MongoHQ

We are pleased to have an origami artist, Linda Mihara as part of the Waza experience. And thanks to MongoHQ for adding rockets to her repertoire. Learn the ancient art of paper folding and make your own shiny silver rocket to take home.

Neo4j

Adding an art experience to one of the lounges, Neo4j is bringing an amaizng Zen Table to Waza. They’ll also use their superstar graphing skills to monitor and display the event’s Twitter activity.

Neon Roots

A full service interactive agency specializing in custom web & mobile development that contributed to our web site.

New Relic

Cheers! Thanks to New Relic, we’ll be serving craft beers at Happy Hour. To kick things off at 5 p.m., an expert from 21st Amendment Brewery will share how those delicious flavors you’re enjoying came to be.

SendGrid

SendGrid is bringing us a Waza first: Arduino hacking in the Garden! If you are new to Arduino, SendGrid will be leading two intro talks to get you started. If you’re already an Arduino, just sit down and hack! You might even win an Arduino kit to take home.

Treasure Data

Saving us all from huddling in a corner near a power source, or missing anything to charge our laptops, Treasure Data is providing Waza attendees with two power valet stations. Drop off your electronics to be safely stored and charged while you enjoy the talks and activities.

Register Now

February 28th is less than two short weeks away. If you haven’t registered yet, now is the time. Looking forward to seeing you all at the Concourse for a sure-to-be-epic Waza!

Routing Performance Update

Jesper — Sat, 16 Feb 2013 06:10:38 GMT

Over the past couple of years Heroku customers have occasionally reported unexplained latency on Heroku. There are many causes of latency—some of them have nothing to do with Heroku—but until this week, we failed to see a common thread among these reports. We now know that our routing and load balancing mechanism on the Bamboo and Cedar stacks created latency issues for our Rails customers, which manifested themselves in several ways, including:

Unexplainable, high latencies for some requests
Mismatch between reported queuing and service time metrics and the observed reality
Discrepancies between documented and observed behaviors

For applications running on the Bamboo stack, the root cause of these issues is the nature of routing on the Bamboo stack coupled with gradual, horizontal expansion of the routing cluster. On the Cedar stack, the root cause is the fact that Cedar is optimized for concurrent request routing, while some frameworks, like Rails, are not concurrent in their default configurations.

We want Heroku to be the best place to build, deploy and scale web and mobile applications. In this case, we’ve fallen short of that promise. We failed to:

Properly document how routing works on the Bamboo stack
Understand the service degradation being experienced by our customers and take corrective action
Identify and correct confusing metrics reported from the routing layer and displayed by third party tools
Clearly communicate the product strategy for our routing service
Provide customers with an upgrade path from non-concurrent apps on Bamboo to concurrent Rails apps on Cedar
Deliver on the Heroku promise of letting you focus on developing apps while we worry about the infrastructure

We are immediately taking the following actions:

Improving our documentation so that it accurately reflects how our service works across both Bamboo and Cedar stacks
Removing incorrect and confusing metrics reported by Heroku or partner services like New Relic
Adding metrics that let customers determine queuing impact on application response times
Providing additional tools that developers can use to augment our latency and queuing metrics
Working to better support concurrent-request Rails apps on Cedar

The remainder of this blog post explains the technical details and history of our routing infrastructure, the intent behind the decisions we made along the way, the mistakes we made and what we think is the path forward.

How routing works on the Bamboo stack

In 2009, Heroku introduced the Bamboo stack. It supported only one language, one web framework and one embedded webserver. These were: Ruby (MRI 1.8), Rails (2.x) and Thin, respectively.

The Bamboo stack does not support concurrency. On Bamboo, a single process can serve only one request at a time. To support this architecture, Heroku’s HTTP router was designed to queue requests at the router level. This enabled it to efficiently distribute requests to all available dynos.

The Bamboo router never used a global per-application request queue. The router is a clustered service where each node in the cluster maintains its own per-application request queue. This is less efficient than routing with a global request queue, but it is a reasonable compromise as long as the cluster is small.

To see why, let’s look at a simplistic example. In the two diagrams below, requests are coming in through three router nodes and being passed to two dynos. The majority of requests take 50ms, while a rare slow request takes 5000ms. In the first diagram, you can see how a slow request, coming in to Router 1, is passed to Dyno 1. Until Dyno 1 is finished with that request, Router 1 will not send any more requests to that dyno. However, Routers 2 and 3 may still send requests to that dyno.

Meanwhile, as illustrated in the next diagram, because Routers 2 and 3 are not aware that Dyno 1 is busy, they may still queue up one request each for Dyno 1. These requests are delayed until Dyno 1 finishes processing the slow request.

The inefficiency in request routing gets worse as the number of routers increases. This is essentially what’s been happening with Rails apps running on the Bamboo stack. Our routing cluster remained small for most of Bamboo’s history, which masked this inefficiency. However, as the platform grew, it was only a matter of time before we had to scale out and address the associated challenges.

Routing on Cedar

As part of the new Cedar stack, we chose to evolve our router design to achieve the following:

Support additional HTTP features like long polling and chunked responses
Support multi-threaded and multi-process runtimes like JVM, Node.js, Unicorn and Puma
Stateless architecture to optimize for reliability and scalability

Additionally, to meet the scalability requirements of Cedar we chose to remove the queuing logic and switch to random assignment. This new routing design was released exclusively on Cedar and was significantly different from the old design. What’s important to note is we intended customers to get the new routing behavior only when they deployed applications to Cedar.

Degradation of Bamboo routing

In theory, customers who had relied on the behavior of Bamboo routing could continue to use the Bamboo stack until they were ready to migrate to Cedar. Unfortunately that is not what happened. As traffic on Heroku grew, we added new nodes to the routing cluster rendering the per-node request queues less and less efficient, until Bamboo was effectively performing random load balancing.

We did not document this evolution for our customers nor update our reporting to match the changing behavior. As a result, customers were presented with confusing metrics. Specifically, our router logs captured the service time and the depth of the per app request queue and present that to customers, who in turn were relying on these metrics to determine scaling needs. However, as the cluster grew, the time-and-depth metric for an individual router was no longer a relevant way to determine latency in your app.

As a result, customers experienced what was effectively random load balancing applied to their Bamboo applications. This was not caused by an explicit change to the Bamboo routing code. Nor was it related to the new routing logic on Cedar. It was a pure side-effect of the expansion of the routing cluster.

No path for concurrent Rails apps on Cedar

We launched Cedar in beta in May 2011 with support for Node.js and Ruby on Rails. Our documentation recommends the use of Thin, which is a single-threaded, evented web server. In theory, an evented server like Thin can process multiple concurrent requests, but doing this successfully depends on the code you write and the libraries you use. Rails, in fact, does not yet reliably support concurrent request handling. This leaves Rails developers unable to leverage the additional concurrency capabilities offered by the Cedar stack, unless they move to a concurrent web server like Puma or Unicorn.

Rails apps deployed to Cedar with Thin can rather quickly end up with request queuing problems. Because the Cedar router no longer does any queuing on behalf of the app, requests queued at the dyno must wait until the single Rails process works its way through the queue. Many customers have run into this issue and we failed to take action and provide them with a better approach to deploying Rails apps on Cedar.

Next Steps

To reiterate, here is what we are doing now:

Improving our documentation so that it accurately reflects how our service works across both Bamboo and Cedar stacks
Removing incorrect and confusing metrics reported by Heroku or partner services like New Relic
Adding metrics that let customers determine queuing impact on application response times
Providing additional tools that developers can use to augment our latency and queuing metrics
Working to better support concurrent-request Rails apps on Cedar

If you have thoughts or questions, please comment below or reach out to me directly at jesperj@heroku.com.

Bamboo Routing Performance

Oren — Fri, 15 Feb 2013 03:55:28 GMT

Yesterday, one of our customers let us know about significant performance issues they have experienced on Heroku. They raised an important issue and I want to let our community know about it. In short, Ruby on Rails apps running on Bamboo have experienced a degradation in performance over the past 3 years as we have scaled.

We failed to explain how our product works. We failed to help our customers scale. We failed our community at large. I want to personally apologize, and commit to resolving this issue.

Our goal is to make Heroku the best platform for all developers. In this case, we did not succeed. But we will make it right. Here’s what we are working on now:

Posting an in-depth technical review tomorrow
Quickly providing more visibility into your app’s queue of web requests
Improving our documentation and website to accurately reflect our product
Giving you tools to understand and improve the performance of your apps
Working closely with our customers to develop long-term solutions

I am committing to listening to you, acting quickly to meet your needs and making sure Heroku is a platform that you trust for all of your applications. If you have additional concerns, please let me know. My email address is oren.teich@heroku.com.

Oren Teich GM, Heroku

Heroku

Introducing the Europe Region, Now Available in Public Beta

One Heroku, Two Continents

Faster Apps

A Familiar Workflow

Easy App Migration with Heroku Fork

Add-ons

Customer Success

Safe Harbor Compliance is Coming Soon

We Want Your Feedback

Additional Resources

Expanded HTTP Method Support

Why is this important?

Background

HTTP is extensible

The HTTP PATCH Method

Other HTTP methods

Why restrict HTTP methods in the first place?

Heroku Postgres – Version 9.2 now Default

Empowering Change: Programming Literacy for All

Staying Connected

Feedback Cycles

The Next Generation

2X Dynos in Public Beta

Getting started with 2X dynos

Use Cases

Horizontal vs. Vertical Scale

Summary

FAQ

Heroku Postgres Databases Patched

PostgreSQL Vulnerability Details

Heroku Postgres Patching

Conclusion

Routing and Web Performance on Heroku: a FAQ

Overview

Q. Is Heroku’s router broken?

Q. So what’s this whole thing about then?

Q. What types of apps are affected?

Q. How can I tell if my app is affected?

Q. What’s the fix?

Q. Can you give me some help with this?

Routing technology

Q. Why does the router work the way that it does?

Q. Even with concurrent web backends, wouldn’t a single global request queue still use web dynos more efficiently?

Q. So does that mean you aren’t working on improving HTTP performance?

Retrospective

Q. Did the Bamboo router degrade?

Q. Were the docs wrong?

Q. Why didn’t you update Bamboo docs in 2011?

Q. Was the "How It Works" section of the Heroku website wrong?

Q. Was the queue time metric in New Relic wrong?

Q. Why didn’t Heroku take action on this until Rap Genius went public?

The Future

Q. What are we doing to make things right from here forward?

Q. What about everyone else?

Helios - open source framework for mobile

Waza 2013: How Ecosystems Build Mastery

log2viz: Logs as Data for Performance Visibility

Logs as data

Introducing log2viz, a public experiment

Also introducing: log-runtime-metrics

Looking under the hood

Logs-as-data add-ons

Conclusion

Running Rails on Heroku Update

Jacob Kaplan-Moss, Django Co-Creator, Talks Ecosystems at Heroku's Waza

APIs to support extensibility

Conservatism as a value

Empowering the community

Conclusion

Matz on Ruby 2.0 at Heroku's Waza

Compatibility

Speed

Features

Running 2.0 on Heroku

20 years of simplicity, elegance, and programmer happiness

Idea to Delivery: Application Development in the Modern Age. Adam Wiggins at Waza 2012 [video]

Deploying from Day One and Continuous Development/Deployment

Development and Production Parity: Keeping development, staging, and production as similar as possible

Staying Close to Production

The HTTP `PATCH` Method