Unfortunately, many companies attempting to move towards agile methodologies struggle to align with the agile manifesto because they are conditioned in older techniques like waterfall. The following list is an attempt at humor (it’s really therapy) for the Top 10 reasons why agile fails. Enjoy!

10. The team thinks that 8 week sprints are really sprints

9. All scrum masters recently got certified after 10+ years of being PMP certified waterfall PMs

8. Management thinks that implementing an open workspace environment equals agile

7. After implementing the open space environment, the white noise is turned up so loud it throws people’s equilibrium out of whack.

6. The new open space environment becomes a place to hold 5 concurrent meetings, many on speaker phones

5. The new open space environment is always empty because everyone is still in meeting rooms all day.

4. The “Scrum Master” announces that they need a QA sprint (Say what?!)

3. There are no user stories for architecture and design because there is “no time for those in 2 week sprints”

2. The user stories are so long they have to staple multiple index cards together.

1. The team actually implements Wagile – Waterfall with 15 minute daily standups.

Let me hear what your top 10 are.

Public clouds are not safe for storing sensitive data due to the Patriot Act which gives the government the power to seize the data in an attempt to mine data in the name of preventing terrorist attacks.  Instead, companies can use a private cloud or forgo the cloud altogether.

The first sentence is true.  The part that is untrue is that leveraging private clouds or on-premises data centers are safe from the Patriot Act.  The Patriot Act allows the government to access the data no matter where it is stored;  public cloud, private cloud, or on-premises.

Building private clouds in the name of protecting data from the Patriot Act is misguided.  Private clouds are no safer from the government’s prying eyes than public clouds.  In fact, by choosing private clouds, the onus to protect data falls squarely on the enterprise’s shoulders and studies have shown that private cloud solutions typically fair no better than public cloud solutions when dealing with security.  Here is an excerpt from the Alert Logic report.

The Spring 2013 Cloud Security Report analyzes six months of new threat data from over 1,800 Alert Logic customer environments, finding that cloud and hosted environments are not inherently less secure than enterprise data centers. With over two years of data analyzed in three reports, the conclusion is consistent: fears about cloud security are not supported by the data.

Summary

In summary, most excuses for leveraging private clouds over public clouds are based on myths. The main reason for using private clouds is the desire to have more control.  Sometimes control is a valid reason but often it is IT just not letting go. Many of my clients want to use public clouds and get out of the infrastructure business but their clients buy into all of the myths about the public cloud so they are forced to go to a hybrid solution to satisfy those customers.

Here are some Key Take Aways

• The further down the cloud stack the more responsibility you own and must build
• Three aspects of securing systems are apply, detect, prevent
• Three key strategies: Centralize, Standardize, Automate
• Along with the control that comes with private clouds comes all of the responsibility of security (Have fun with that!)
• Leverage web frameworks to assist in protecting against Top 10 web threats
• Don’t roll your own if it is not your core competency
• Don’t be proprietary.  Make it easy and secure to integrate
• Separate policies from APIs.  Makes it easier to change, manage.

If there are security related topics that you wish I included let me know. Enjoy!

Redis as a Service

Redis is an open source database with tons of valid uses cases that are implemented in the real world today. Caching is one common use case. The challenge with Redis is that it has scaling limitations when datasets grow beyond the capacity of the master node. Maintaining a highly available Redis database is a complex undertaking and the performance may degrade over time. In addition, open source distributions of Redis lack various features making many Redis implementations not enterprise worthy. This is the exact problem Garantia set out to solve. Garantia adds sharding, clustering, automation, and numerous other enhancements to the core Redis distribution thus making their Redis as a Service offering enterprise class.

Garantia Data delivers Redis as a service so their customers can focus on building their applications while the database simply manages itself. With Garantia’s Redis as a Service solution, customers no longer need to deal with clustering, software upgrades, patching, server management and much more. One simply pays for what they use and they get the robustness and autoscaling of Redis without the headaches.

Partner integrations

Garantia also provides reliability and zero management for memcacheDB (see video below).

With these database as a service offerings, Garantia makes Redis and MemcacheDB enterprise class databases without the limitations that exist with unmanaged open source versions. Currently, Garantia is available on AWS (US East and EU West), Microsoft Azure, Heroku, AppFog, and AppHarbor. On the road map is a do-it-yourself private cloud version.

Moving forward

Garantia Data was founded in March of 2011 and raised $3.8M from angel investors. They are currently raising a series A round of $5M. They were recently named “Cool Vendor” by Gartner in early May and named to the Inaugural CRN Big Data 100 back in April.  Keep an eye on these guys and look for their services to become integrated in more IaaS and PaaS solutions in the future. You can check them out at www.garantiadata.com.

Fast forward to the current decade and ASPs have been replaced with SaaS, IaaS and other types of cloud services. Jamcracker saw this shift to the cloud coming and built a cloud platform to broker various cloud services to their customers. Jamcracker provides a single point of management for managing and controlling billing, auditing, support, and security.  Jamcracker provides a catalog of cloud services for their customers to choose from and provides single sign-on (SSO) capabilities so the customers can seamlessly work across multiple services with one set of credentials.

How it works

As you can see from the image below, the Jamcracker broker platform sits in the middle of the cloud ecosystem connecting the cloud vendors, both public and private, and cloud solution providers (distributors, service providers, enterprise IT organizations, etc.) to end users whether they are from large enterprises, SMBs, or industry channels (for example Telecom providers). Jamcracker adds automation, workflow, integration, and single sign-on services so that the end users can simply pick what services they want from the catalog to use without having to install, manage, and configure those services.

Addressing Shadow IT challenges

The Jamcracker solution provides a way for IT to proactively address the challenges of Shadow IT. Instead of fighting the battle of employees bringing in their own unmanaged cloud based solutions, Jamcracker provides a platform for enabling people to leverage cloud services on demand. IT gets the benefit of controlling the enterprise environment thus reducing risks created by unsupported one off solutions and reducing sprawl.

Summary

Jamcracker supports over 100 pre integrated services and that list continues to grow each day. Whether the need is to service internal IT customers or to provide services to enterprises as an external third party, Jamcracker is there to make this process seamless, simple, on-demand, and easy.  Check them out at http://www.jamcracker.com/

Why REST?

There are many reasons why RESTful services are a critical component of any cloud service.  First of all, when building services in the cloud one typically builds on top of an Infrastructure as a Service (IaaS) or Platform as a Service (PaaS) provider and integrates with one to many Software as a Service (SaaS) offerings. All of these vendor or open source solutions have exposed their APIs using RESTful services.

In addition, clouds are heterogeneous ecosystems that connect many different services from many different companies written in many different technology stacks. The complexities of the underlying stacks and protocols should be abstracted away from the business logic so this ecosystem of services can easily connect and work in harmony. A great example of this concept in practice is how simply we can “plugin” social media functionality from Facebook, Twitter, Pinterest, and other social media touchpoints. Underneath those widely used APIs are some very diverse and complex systems. All of that complexity is hidden from us as developers and literally in a few minutes we can connect our applications and leverage all of that wonderful social functionality. That is agility at its finest.

Second, gone are the days of building separate systems for individual touchpoints. Today the preferred method is to build multiple user interfaces (web, mobile, tablet, etc.) that leverage the same services and are always in sync. We have to build things this way because our users are bouncing around between devices and browsers and will leave in droves if each touchpoint behaves differently. To make things even easier, there are a few companies that are delivering mobile platforms so that developers can maintain a single codebase while the platforms transform that code into the various different mobile and tablet user interfaces. Did I mention agility?

Third, and most important, cloud infrastructure is virtual and dynamic, meaning resources come and go in an elastic matter and every piece of virtual infrastructure is expected to fail. The cloud is built so a failure in one part of the infrastructure does not impact the entire system as additional infrastructure can be dynamically created to take over. In order to design for failure, one must refrain from relying on application state because the virtual cloud infrastructure maintaining the state can fail. How does RESTful services get by without maintaining state? By leveraging hypermedia as the engine of application state (HATEOAS). HATEOAS means that state is represented by a series of links (URIs) much like following the site map of a website by following the URLs. When cloud infrastructure fails or is unavailable, other available cloud resources can take over. These cloud resources get their instructions from the URI and process the request.

This first example shows a successful REST call. No application state is stored on the client. The client uses hypermedia links to instruct the server what to execute. The server performs the get customer call and returns XML as instructed by the Accept command.

In the second example, the REST call returns a 504 Timeout error because something happened to the server during processing. The call is retried and redirected to the next available server. No application state is required because once again the server simply follows the instructions of the hypermedia links (URI). Another important fact is that the client leverages DNS (mydomain.com) and does not have to know anything about the cloud infrastructure on the right hand side. The client leverages an elastic IP address which never changes (static) while the internal IP addresses of the various cloud infrastructure changes as servers scale up and down (dynamic).

A good analogy of HATEOAS is the way a GPS works in a car.  Punch in a final destination on the GPS and the application returns a list of directions. You start driving by following these directions. The voice on the application tells you to turn when the next instruction is due. Let’s say you pull over for lunch and shut the off the car. When you resume driving, the remaining directions in the trip list pick right up where they left off. This is exactly how REST works via hypermedia. A node failing is similar to shutting your car off for lunch and another node picking up where the failed node left off is similar to starting back up the car and the GPS. Make sense?

ACID vs. BASE

Building stateless services also requires thinking differently about data consistency. Here are two really good articles that dive deeper into this topic (from ZapThink  and HighScalability.com). ACID (Atomicity, Consistency, Isolation, Durability) transactions are focused on ensuring that a transaction is complete and consistent. ACID transactions will lock part of a database to ensure that the transaction is completed and committed to the database. With ACID, a transaction cannot leave the database in an inconsistent state. This creates issues in a high volume distributed system when many services may be competing for the same resources. For example, let’s say that Walmart.com promotes a “Deal of the Day” that is available at all of their stores. If they design for ACID transactions, each time a web user clicked on the deal certain parts of the database table would be locked momentarily while the database ensured the consistency of the transaction. Sure it may only take a fraction of a second but if thousands of consumers are trying to click on the deal concurrently the website performance would be horrendous due to locking.

A better way to solve that problem is with BASE (Basically Available, Soft State, Eventually Consistent) transactions. BASE is focused on ensuring that resources are always available and the data will eventually become consistent. With BASE, the database does not lock thus allowing many consumers to get their deals with no waiting. The downside to this approach is that a small number of transactions may fail and those transactions will need to be reconciled. If there was a counter showing total deals issued, the number would be mostly accurate but it might be a slightly off due to timing or to a failed transaction. The developers have to account for these minor issues through retries, audits, reconciliation processes, etc. BASE transactions are harder to develop for but they allow systems to achieve high scalability and better performance than ACID based systems.

One of the challenges companies have when they decide to port applications from on-premises to the cloud is that some of their legacy systems are reliant on ACID transactions. To port an ACID based application without making drastic design changes, they would have to deploy the application in a single partition in order to get the same results as expected from the on-premises solution. By doing this they lose much of the value of the cloud because the ACID based application cannot be elastic. BASE transactions are the key to elasticity in the cloud.

Summary

Architecting solutions for cloud computing requires a solid understanding of how the cloud works. To build resilient solutions that scale, one must design their solution with the expectation that everything can and will fail. Architects should become familiar with how RESTful services work. It takes a shift in thinking to build things properly in the cloud including moving away from ACID transactions to BASE. To truly take advantage of the benefits of cloud computing, architects must build RESTful services the right way by leveraging hypermedia as the application state instead of storing state within the services themselves.

A great book on this topic is Jason Bloomberg’s latest called The Agile Architecture Revolution: How Cloud Computing, REST-Based SOA, and Mobile Computing are Changing Enterprise IT.

SaaS Strategies

There are many strategies for avoiding downtime in the cloud for all three cloud service models. Many people don’t consider any strategies for SaaS solutions and this could lead to serious business impacts if a disaster was to occur. What would a company do if its SaaS based financial system went offline for a week? It would not be pretty. For SaaS solutions, if the data or business processes are mission critical, there better be a plan for operating if the service is not available.  Minimally, the SaaS contract from the vendor should have a software escrow.  A SaaS software escrow protects the buyer if the SaaS vendor goes out of business or is purchased by another company and severs the existing contract.  The escrow holds the vendor’s IP in an independent third party’s holding area where it may be released to the buyer if the vendor goes out of business or kills the product. This essentially gives the buyer the ownership of the data.

Escrows are great for protecting your rights and ownership, but they don’t do much for getting your business back up and running. Businesses should document and practice manual processes as a plan to combat a major outage for any mission critical SaaS functionality. In some cases, it may even be feasible to use two different SaaS vendors to protect against outages. For example, let’s say that a business is running an eCommerce site that generates $1M a day in online sales. Instead of reinventing the wheel they decide to leverage a best in breed shopping cart and credit card processing SaaS solution. If this SaaS solution fails, they risk losing $700 a minute. It would be wise for them to use a second SaaS solution either in a round robin fashion or as a hot backup. Most of these solutions charge by transaction so a hot backup method would not be an expensive undertaking.

Some SaaS providers have an export feature where the buyers can download their data on demand. A best practice is to write automation scripts that run on a schedule (daily, weekly. or monthly) that will extract this data and store it securely in case of a disaster.

IaaS Strategies

IaaS disaster recovery strategies are much more involved than PaaS and SaaS, hence one of the advantages of the higher up service models. With IaaS, the buyer does not have control over the uptime of the data centers unless they are building their own private clouds. For this blog post, I will not discuss disaster recovery at the data center level, but will focus on the application stack and higher layers of the architecture.

Avoiding lock-in

Public cloud providers like Amazon, Google, Rackspace, and Microsoft have had their share of outages over the years. It would be foolish to expect otherwise. There are many ways to design around these failures. Building redundant solutions across datacenters or zones is a best practice to avoid outages.  But some companies simply refuse to relinquish control and build their own private clouds. This strategy has huge tradeoffs. Private clouds sacrifice many business benefits such as rapid elasticity, cost efficiencies, and others. For some companies, control is the highest priority so building private clouds are a viable option. Standard datacenter disaster recovery strategies apply.

Another approach is to build code that does not lock you into the IaaS vendor. This is easier said than done. For example, one of the huge benefits of AWS and other IaaS providers is the large selection of APIs that can be leveraged to quickly build applications and focus more on solving business problems. To be cloud agnostic one must refrain from using these proprietary APIs thus devaluing the vendor offering. An alternative is to use open source cloud software like OpenStack.

A third option is to leverage multiple public cloud vendors. This is a very unattractive option because the cloud vendors APIs are very different and require developers to build way more code than they would had they leveraged a single vendor. One exception is leveraging Eucalyptus with AWS.  Eucalyptus supports many of the AWS APIs which allow the buyer to build hybrid cloud solutions and move workloads between the public and private cloud when outages occur.  Apache Cloudstack also has support for AWS APIs. OpenStack supporters will also be able to move workloads across public and private clouds if both datacenters are using OpenStack.

All in with public clouds

For those who believe in and thrive in public clouds and are far past the concerns of lock-in and control, here are four common strategies for dealing with outages and disasters.

1. Classic backup and restore method
2. Redundant datacenters – Active/Passive “Cold”
3. Redundant datacenters – Active/Passive “Warm”
4. Redundant datacenters – Active/Active “Hot”

The term datacenter here represents virtual datacenters provided by the IaaS vendor. In the world of AWS, these are known as availability zones (AZ) but regions can be used as well.

Backup and recovery

In this model, daily full backups and incremental backups are created during the day and stored to a disk service provided by the cloud vendor. The backups are also copied to the secondary datacenter and possibly to some other third party vendor just to be extra safe.

If the database goes offline, gets corrupted, or encounters some other issue, we can restore the last good backup and apply the latest incremental backups on top of that.  If those are unavailable we can go to the secondary site and pull those backups.

This method has been used for years in our brick and mortar datacenters.

Active/Passive Cold

In this model, all of the backup virtual machines are scripted and ready to launch in case of an emergency. When a disaster occurs, the team runs automated scripts that fire up the database server and restores the latest backups. It also fires up the other servers and essentially creates a duplicated environment in a few minutes hence the term “cold”. This method is a cost effective way to deal with outages but if the toleration for the length of time for an outage is less than a few minutes it will not be an acceptable plan.

Active/Passive Warm

The warm method runs the database server hot, meaning that it is always on and always in sync with the master data center. The other servers are cold or off and are only fired up when needed. This method costs a little more than the cold method due to the database server costs but greatly reduces the amount of downtime if an outage occurred because no database restore would be required. In addition, adhoc and business intelligence workloads could be pointed to this database instance so that it is not running idle during normal times thus improving the overall efficiency of the master database.

Active/Active Hot

The most expensive but most resilient method is to run fully redundant datacenters at all times. The beauty of this model is that all of the compute resources are being used at all times and in many cases a complete failure of one datacenter may not cause any downtime at all. This is the model that I have used with my first startup that has never missed a transaction due to any AWS outage.

PaaS Strategies

With PaaS, the entire platform is the responsibility of the vendor and the buyer is responsible for the applications built on top of the platform. The disaster recovery strategies for PaaS are similar to those of SaaS and IaaS. Your PaaS contracts should also have a provision in it for software escrow. Like IaaS, the same thoughts about control and lock-in apply. If those are a concern there are numerous open source PaaS solutions like Open Shift or Cloud Foundry that can be run in private or public clouds and run on top of open source solutions like OpenStack.

PaaS platforms handle scaling and failover for you but it is still up to the buyer to configure the platform to deploy enough resources to handle redundancy. For example, PaaS provider Heroku deploys dynos that can provide redundancy across multiple AWS zones. However, the buyer must configure the platform to have at least 2 dynos for each layer of the architecture in order for each layer to fail over successfully.

Summary

Cloud computing is still relatively new and immature. We should expect to see occasional outages, vendors closing their doors, and natural disasters like hurricanes, earth quakes, and floods impacting our ability to keep our systems up all of the time. Planning for disasters is a critical function regardless of the cloud service model. I have highlighted a few options but there are certainly more. Apply the option that makes sense for your business and make sure your hands are not tied if and when your cloud service fails.

* NOTE:  Special thanks to Brian Gracely for proof reading this for me!

Cloud computing has come a long way since I first swiped a credit card on AWS in 2008 and started building a platform from scratch in the cloud. Back then cloud services were not mature and many of today’s SaaS and PaaS solutions that assist in building cloud based solutions did not exist. Back in 2008 we built everything from scratch and there were few success stories to learn from. Long story short, we made a ton of mistakes and learned from them. To save you all from making those same mistakes, here are some hard lessons learned with a twist of humor.

1. Spending money like a drunk college kid with a credit card

Cloud computing is supposed to be a money saver. It can be but not without a view into how many servers are running and how well they are being utilized  When a team consists of only 3 people it is easy to manage the cloud. When the team ramps up to 30+ people you better have a good process in place for managing and monitoring cloud resources. In a blink of an eye the monthly cloud bill will hockey stick the way you hoped your revenue would and the developers will create dependencies on these assets like a host parasite making it hard to shut down the excess resources.

Recommendation:

Put processes and controls in place to manage cloud resources and invest in cost optimization tools. These tools pay for themselves every month.

2. Walking to school and back uphill both ways

Remember the joke where the old timer says “When I was your age I walked to school uphill, both ways”? Well the cloud equivalent is “When I started in the cloud, I built everything from scratch, all the time”. I was involved in a project where the first several (and I mean several) sprints were all focused on IT plumbing features like setting up message queues, event processing, caching, email engines, notification systems, and on and on. To make matters worse, we had very high SLAs so each piece of plumbing had to be redundant across zones and withstand an entire AWS zone failure. After a few months of these plumbing sprints we did not have a lot to show to our customers. This is why I am such a fan of PaaS. Today’s PaaS solutions provide all of these IT plumbing features as simple APIs.

Recommendation:

If it not core to your business, don’t build it! Leverage PaaS and SaaS solutions where it makes sense and don’t do all of the heavy lifting. The goal should be to get to market fast. Leave the plumbing to the plumbers.

3. Needle in the haystack

Trying to debug a system distributed across hundreds or thousands of servers is like trying to find a needle in a haystack. Imagine having the following server farms that autoscale: web servers, API servers, cache servers, database servers. Now imagine a customer reports an error with one of the values of one of the fields on the user interface. What server and which server farm would you search? Which datacenter (zone or region in AWS terms) would you search first? There needs to be a centralized logging process in place that is searchable and can be segregated by customer.

Recommendation:

Put a logging strategy in place and consolidate all logs to a central location while locking down all of the production servers to prevent accidental issues from developers poking around while troubleshooting. This is another area where time and money is better spent leveraging logging solutions in the marketplace rather than rolling your own.

4. Don’t go out without making reservations

AWS offers reserved instance which allows a company to prepay for servers upfront thus giving deep hourly discounts that cut the costs of a server by about 50%. Studies show that a majority of companies do a poor job of optimizing server costs especially around leveraging reserved instances.

Only 33% of instances are reserved instances today, while some 94% of instances, if converted to reserved instances, would result in expected cost savings.

Some of this is due to a lack of understanding on how reserved instances work. When you buy a reserved instance, it is not assigned to an actual virtual server. Instead, at the end of the month AWS applies the number of reservations per server size to the usage of servers for that size.

Recommendation

Have a strategy around reserved instances. If you reserve 15 XLs and use only 10, you over paid. But if you know that your system fluctuates between 10 and 15 servers, but always has at least 10, buy 10 reserved. Just because the count fluctuates does not mean you can’t take advantage of reserved instances. This is just another reason to use one of the many cost optimization tools in the market place. These tools will figure out the optimal number of reserved instances for you.

5. Giving developers an all you can eat buffet for free

Imagine telling a developer that they can have any size virtual machine they want and as many as they want. Nobody would do that…on purpose. But giving developers self service provisioning rights without any process or controls is the equivalent of sending a professional football team’s offensive line to an all you can eat Chinese buffet. Nobody is making any money in that scenario.

Recommendation

In order to do self provisioning right, a good deal of planning and governance must be put in place. The DevOps team should only allow developers to self provision from a certified image that can pass all of the security and regulatory audits. In addition the process for self provisioning should have features that track costs and chargebacks, has hours of operations which can automatically shut of these resources in off hours, and have the proper permissions in place so only authorized people can self provision. I have seen a number of companies build very nice self service portals to do this but I would look to the market place first and see if any tools can meet the needs before building.

6. Governance is not a dirty word. Your CIO’s reaction to another outage is.

People just hate the word governance. It reminds them too much of government. The cloud environment is all about agility so why do we keep demanding governance? Well, a fool with a tool is still a fool. Just because the cloud offers us ways to be more agile does not mean we should not apply a set of processes and controls around building software and services. We still need to control costs, pass audits, keep the bad guys out, protect customer data, meet high SLAs, and much more. Don’t let the agility of the cloud turn your operation into the wild west.

Recommendation:

At the end of the day it is all about architecture. Building software in the cloud requires best practices in design and an in-depth understanding of the strengths and weaknesses of cloud computing. DevOps plays a key role in governance as well. Centralizing, abstracting, and automating the processes in operations, builds, and deployments allows developers to focus on development and protects the overall platform from chaos.

7. Pay no attention to the man behind the curtain.

Remember the scene in the Wizard of Oz where Toto pulls back the curtain and the true identity of the Wizard of Oz was revealed? The perception of the wizard sure went down in flames when he was exposed. Make sure the same does not happen to your cloud services.

Recommendation:

Live by the mantra “Automate Everything”.  Invest in continuous integration and continuous deployments. Monitor everything. Know when your system is showing signs of stress and act before things get catastrophic. Make everything as “push button” as possible and stay away from the man behind the curtain syndrome.