Necessity is the Mother of the Invention of Amazon’s EC2

October 26, 2010

Since the early days of the computer era there has been a desire to create a ubiquitous computer utility which can be accessed by anyone from anywhere.  With the advent of Amazon’s EC2 in 2006 the dream of a computer utility is fast becoming a reality.

How and why did Amazon, an on-line retailer, “invent” Cloud Computing? 

Urban legend has it that Amazon invented Cloud Computing as a means of selling the excess capacity of its production serves.  As the legend goes, Amazon had to have enough spare server capacity to handle its peak order period from Thanksgiving to Christmas, but these servers were idle most of the rest of the year.  So, why not sell that excess capacity.

The real story is much different.  As you will see in the following link, Jesse Robbins, Amazon’s IT Operations Manager at that time, states that like any good IT manager, he would never let non-production applications on his infrastructure:  “NOBODY gets any of it if you can humanly prevent it. It’s ‘mine, mine, mine’ as the duck said.

From its initial founding, Amazon’s online business grew rapidly.  By 2003, Amazon was having difficulty maintaining its large number of servers and vast network and it was still adding servers and network capacity at a frantic pace.  As the number of servers and network grew more complex, they became more difficult to manually manage and a more automated mechanism was needed. 

In 2003 / 2004, Amazon’s Chris Pinkham, VP of Global IT Systems and Benjamin Black, a Network Operations Manager wrote up a paper describing how to go about automating the provisioning and management of Amazon’s servers and network.  This paper became the foundation for Amazon’s Elastic Compute Cloud.  Shortly thereafter, Jeff Bezo gave Chris Pinkham, along with Chris Brown and Willem Van Biljon the go ahead to develop an automated provisioning and management system working out Amazon’s Cape Town Development Centre. 

In one meeting with Bezo, the idea was discussed that if Amazon itself wanted automation management capabilities other computer users might want similar capabilities and the idea of selling these services was born.  So was the dream of Amazon’s Elastic Compute Cloud (EC2).  See Benjamin Black Blog and a video interview with Amazon CEO, Jeff Bezo talking about creating Amazon EC2:

So now you know that “Necessity” not excess capacity was the mother of the invention of Cloud Computing at Amazon!

Computing in the Cloud versus Cloud Computing

October 12, 2010

Lately, we are seeing the term Cloud Computing everywhere. What is Cloud Computing and how does it differ from “Computing in the Cloud”?

Most of us have been “Computing in the Cloud” for years. We use a wide variety of on-line services such as search, email, on-line shopping, on-line banking, social networking, etc. These services are all examples of computing services that are delivered to users via the internet, WWW or the “Cloud”.

Cloud Computing refers to something different and can be defined as: Ultra-available, virtually limitless compute capability that can easily grow or shrink (elastic), is easy to set-up and is paid for with a pay as you go model.

In the early days of Computing in the Cloud, the main service offered was browser based search. As the demand for search grew, more and more people used Google and Yahoo and they quickly became household names. After people found the information they were searching for, they wanted to share that information and Email services became the next big growth spurt. At the same time, Amazon and eBay began to offer the ability to purchase goods, banks started to provide on-line banking services and Social Networking services like Facebook, Twitter, LinkedIn and were offered.

The demand for these services grew at an unbelievable rate. Today, Google is said to process billions of transactions a day and Face-book has over 500 million users.

In order to meet demand, Amazon, Google, Yahoo, and the other major service providers developed highly available computer infrastructures with enormous capacities to handle demand. To make their own lives easier, they developed automated software tools to manage, update and deploy their vast numbers of servers. Google is said to have over 1 million computers to support its growing list of Cloud services.

As these automated server management services matured, several service providers thought that end users might like similar capabilities and Cloud Computing was born. Amazon was the first service provider to offer a Cloud Computing service, now called Amazon Elastic Compute Cloud (EC2).

Although Computing in the Cloud and Cloud Computing are different, they have much in common with each other. Most Computing in the Clouds services are provided to end users using Cloud Computing infrastructure capabilities to handle the enormous demand.

So, from here on in, I will refer to either of these services as “Cloud” services.