Cloud Computing

I almost missed a really important anniversary! Yesterday marked Amazon EC2's fourth birthday. Here are some of the ways that EC2 has grown and changed in the last four years:

Category 2006 2010 Regions One Four Availability Zones One Ten Availability Zones Instance Types One Nine Pricing Models One Three Storage Ephemeral Storage
Ephemeral Storage
Elastic Block Store
Operating Systems Linux Linux, Windows, OpenSolaris
Management Tools Command-Line Tools Command-Line Tools
AWS Management Console
Third-Party Tools Ancillary Services - Elastic Load Balancing, Auto Scaling, CloudWatch
High Performance Computing - Elastic Map Reduce, Cluster Compute Instances

We've done quite a bit, but we're not resting, not for a minute. We have a lot of open positions on the AWS team, including a really interesting developer position within the EC2 team. This developer will focus on EC2's dynamic market pricing features. In addition to experience with Ruby, Perl, Java, C, or C++, candidates should have some experience building large-scale distributed systems and an interest in operational scheduling, optimization, and constraint satisfaction. You can read more here and you can send your resume directly to amazon-ec2-spot-jobs@amazon.com.

While I am on the subject of anniversaries, eight years ago this month I abandoned my full-time consulting practice to take a development position with the Amazon Associates Team, with the agreement that I could spend some of my time helping out with the effort to create and market the E-Commerce Service (which has since become the Product Advertising API). A few months in, I was asked if I would mind speaking at a conference. I guess I did ok, because they asked me to do another one, and before too long they invited me to apply for the position of Web Services Evangelist. I took on that title in the spring of 2003 and have been spreading the word about our web service efforts ever since. All things considered, this is a really awesome place to work. Day after day, week after week, things get more and more exciting around here. The pace is quick and I do my best to keep up. We do our best to understand and to meet the needs of our customers with regard to features, reliability, scale, business models, and price. I get to work with and to learn from a huge number of world-class intellects. If this sounds like the kind of place for you, check out our list of open jobs and apply today!

-- Jeff;

Good news for all those interested in security...we've released the fourth version of our Overview of Security Processes whitepaper. It contains ten pages of new and additional detailed information. Highlights of new content include:

• A description of the AWS control environment
• A list of our SAS-70 Type II Control Objectives
• Some discussion of risk management and shared responsibility principles
• Greater visibility into our monitoring and communication processes and our employee lifecycle
• Descriptions of our physical security, environmental safeguards, configuration management, and business continuity management processes and plans
• Updated summaries of new AWS security features
• Additional detail about the security attributes of various AWS components

The additional information and greater level of detail should help to answer many common questions. As always, feel free to reach out to us if you're still needing more information.

> Steve <

We just released version 1.0.8 of theAWS SDK for Java. In addition to some bug fixes, the SDK includes the following new features:

• Support for the new Reserved DB Instances.
• Improved constructors for the Amazon RDS model classes.
• A new StepFactory class to simplify the process of creating Elastic MapReduce job flows.
• Improved support for EC2 security groups.
• Improved constructors for the EC2 model classes.
• Additional diagnostic information in AWS responses including request IDs, S3 host IDs, and SimpleDB box usage.

The SDK includes the AWS Java library and some helpful code samples. You may also want to check out the AWS Toolkit for Eclipse.

-- Jeff;

We're ready to start accepting entries for the AWS Start-Up Challenge 2010!

Each year we run the challenge to help young, promising start-ups get noticed and to compete for $100,000 (USD) in cash and credits (full prize list).

We've made some important changes this year:

• We can now accept applications from 22 countries across the Americas, Europe, and Asia.
• We'll now recognize five regional semi-finalists from each of the three regions, at least six finalists, and one global grand prize winner.
• Start-ups must be currently using or in the process of using one or more of the Amazon Web Services. We will now accept entries from start-ups that have yet to launch.

We will be looking for applications which can grow into significant, meaningful businesses over time. We will look for implementation and integration of AWS services, originality and creativity, likelihood of long-term success and scalability, and overall effectiveness in addressing a need in the marketplace. Full information on the contest can be found here.

Previous winners include Good Data (2009), Yieldex (2008), and Ooyala (2007).

Submit your entry today!

-- Jeff;

PS: Several of you have asked why the Start-Up Challenge isn’t available world-wide. It turns out that the rules and regulations governing competitions vary widely from country to country so this isn’t as simple as it sounds. The list of eligible countries has grown from one (2007) to four (2009) and now to 22. We’ll do our best to expand the list in the future.

The AWS Start-up Challenge is back with an expanded world footprint – start-ups in 22 countries across the Americas, Asia, and Europe can compete for a chance to win $100,000 in cash and credits. This year, we will recognize 15 regional semi-finalists, 6 global finalists, and select one global grand prize winner. We’re looking for the most promising start-ups that can grow into significant, meaningful businesses that leverage AWS for their infrastructure. The contest submission closes October 31. To learn more and enter the competition, visit the AWS Start-Up Challenge home page.

I can't even leave my neighborhood without getting feature requests! My neighbor Rick was walking his dog past my house and stopped to ask me when we were going to support Reserved Instances for Amazon RDS. Such is life in the Pacific Northwest part of the US.

Hey Rick, we've got your RDS Reserved Instances ready now. You can get a lower price and know that an instance has been reserved for your use.

After you make a low, one-time payment, you can use a DB Instance at an hourly rate that is significantly lower than the On-Demand pricing for the same instance class. This will result in a decrease in AWS charges for any application that requires full-time access to a relational database.

When you purchase an RDS Reserved DB Instance, you choose a specific AWS Region and instance class. You do not need to choose an Availability Zone. You can purchase up to 20 Reserved DB instances. If you need more DB Instances, you need to complete the Amazon RDS DB Instance request form. You cannot move a reservation from one Region to another and they always apply to a particular DB instance class, so make your purchase with care.

You can make your purchase using the RDS APIs, or the RDS command-line tools. Here are the commands:

• rds-describe-reserved-db-instances - Returns the list of DB Instance reservations for your account or details for one of your reserved database instances.
• rds-describe-reserved-db-instances-offerings - Returns the list of DB Instance offerings that are available for purchase.
• rds-purchase-reserved-db-instances-offering - Purchases one or more reserved DB Instances.

The one-time fee starts at $227.50 for a Small DB Instance with a one-year term.

-- Jeff;

PS - Clean up after your dog, Rick.

I've got a pair of related news items for users of the Amazon Relational Database System (RDS):

1. We now support version 5.1.49 of MySQL, with the InnoDB Plugin as the InnoDB storage engine.
2. We now provide a new, optional level of control over when and if your DB Instance(s) are upgraded to new MySQL versions supported by Amazon RDS. This new functionality is called DB Engine Version Management, and this post will help you understand what it can do and how to use it.

MySQL Version 5.1.49 with InnoDB Plugin

Version 5.1.49 of MySQL includes a number of bug fixes and enhancements. For DB Instances running MySQL 5.1.49, the most significant change is the inclusion of InnoDB Plugin as the InnoDB storage engine, replacing InnoDB Builtin. InnoDB Plugin offers (according to the feature page) performance and scalability enhancements (especially on multi-core platforms), fast index creation, data compression, and more. The InnoDB Builtin (InnoBase) is not available in 5.1.49 and will not be available in future Amazon RDS Supported versions.

New DB Instances that you create will now run MySQL 5.1.49 with InnoDB Plugin, unless otherwise specified using the new DB Engine Version Management feature described in the next section.

DB Engine Version Management

With DB Engine Version Management, Amazon RDS gives you additional (yet optional) control over the version of relational database software (i.e. MySQL) powering your DB Instance. The goal of this functionality is to provide you the flexibility to maintain compatibility with specific MySQL versions, test new versions with your application before deploying in production, and perform version upgrades on your own terms and timelines. Let’s take a look:

DB Instance Creation

Amazon RDS now allows you to create new DB Instances using any supported version of MySQL. Right now this means version 5.1.45 and 5.1.49, but we plan to support additional minor and major versions (e.g. MySQL 5.5) in the future. There's a new EngineVersion attribute associated with each DB Instance, and you can specify your desired Version when issuing a CreateDBInstance API call (or the associated command).  New DB Instances will now use MySQL 5.1.49 with InnoDB Plugin by default.  If you still wish to use MySQL 5.1.45 (which uses InnoDB Builtin) for a new DB Instance, you can do so using this new functionality.

Automatic Upgrade Schedule

Your existing DB Instances will be upgraded to the new version of MySQL with InnoDB Plugin during your scheduled maintenance window on the week of October 4, 2010. If you don't want to upgrade, you need to use ModifyDBInstance to set the AutoMinorVersionUpgrade attribute of your DB instances to "false."

Find Your Engine

The new DescribeDBEngineVersions API call returns a list of all supported versions of MySQL. You can use it (or the rds-describe-db-engine-versions command) to drive your configuration process. You could also use it to set up a really cool automated testing framework. You could call the function, iterate through the resulting list of versions, and create a new DB Instance for each one. Then you could test your application to make sure that it works with each version of MySQL. Or you could create several DB Instances (one per version) simultaneously and test the same queries against each instance.

Choose to Upgrade Today

With DB Engine Version Management, you can also upgrade any of your existing DB Instances to InnoDB Plugin and the new version of MySQL today by setting the EngineVersion attribute to "5.1.49" using the ModifyDBInstance API call (or the associated command).  Amazon RDS will take care of the rest. This will happen immediately if you set the ApplyImmediately flag when you modify the EngineVersion attribute, or at the next maintenance window for your DB Instance if you don't.

Learning More

To learn more about DB Engine Version Management and associated API changes, please visit our Forum Post on the subject. You can also read more about the new functionality via our DB Engine Version Management FAQs, which include information about our guidelines for supporting new versions and deprecating old versions.

With these changes, Amazon RDS becomes even more powerful. If you want to verify that your application works with the new version of MySQL, simply create a snapshot of your existing DB Instance, create a new one from the snapshot, and test away.

-- Jeff;

We’re excited to announce today that we’ve added Reserved Database Instances (Reserved DB Instances) for Amazon RDS. With Reserved DB Instances, you can now make a one-time, up-front payment to create a one or three year reservation to run your DB Instance in a specific Region and receive a significant discount off of the ongoing hourly usage charge. Your DB Instances will work just as they do today, but can lower your costs over the life of your database. For more information on Reserved DB Instances, visit the Amazon RDS detail page.

Since the announcement of Recovery.gov last March, Amazon has seen an accelerating adoption of the cloud by our Federal customers.  These include Treasury.gov, the Federal Register 2.0 at the National Archives, the Supplemental Nutrition Assistance Program at USDA, the openEI.org project at DoE's National Renewable Energy Lab, and the Jet Propulsion Laboratory at NASA. 

On September 23, 2010 we'll be conducting a half-day event in Crystal City, Virginia to discuss the use of the AWS Cloud by the Federal Government.

Speakers will include Amazon CTO Werner Vogels, AWS CISO Steve Schmidt, and a number of AWS customers including representatives of the agencies and organizations mentioned above. There will also be time for Q&A and a cocktail reception afterward for networking.

The event is free but you need to sign up now in order to reserve your spot.

-- Jeff;

PS - You may also enjoy the story behind the Federal Register 2.0 makeover.

If you have ever set up a web site from scratch, you know that you have to handle the root of the web site in a special way so that requests for the site's root URL (e.g. http://aws.amazon.com) are handled properly. You generally map the root URL to an HTML document such as index.html using an entry in the web server's configuration file.

You can now set a default root object for any of your Amazon CloudFront distributions to duplicate this behavior for your own content. This object must be stored within the Amazon S3 bucket associated with the distribution. Once you have set the default root object, a request for the root URL of the distribution will return the contents of the default root object. 

With this change, you can now create a distribution that acts just like a static web site.

If you don't set a default root object for a distribution, the response to a request for its root URL has not changed. Depending on the ACL on the distribution's bucket and on the objects inside, the request could return a list of the contents of the bucket or a 403 error.

You can learn more about this new feature by reading the CloudFront documentation.

These partners support this new feature:

• CloudBuddy Personal
• Bucket Explorer
• CloudBerry Explorer
• boto

-- Jeff;

Amazon CloudFront, the easy to use content delivery network, now supports the ability to assign a default root object to your HTTP or HTTPS distribution. This default object will be served when Amazon CloudFront receives a request for the root of your distribution – i.e., your distribution’s domain name by itself. This feature should make it easier for you to serve all of your static content from edge locations close to your end users and provide them with a better experience. You can read more about the default root object feature in the Amazon CloudFront Developer Guide. For more information on Amazon CloudFront, visit the Amazon CloudFront page.

My previous trips to China, India, Japan, London have been super productive. I get a chance to meet tons of new people, make a lot of friends and talk about something that I am truly passionate about: Cloud Architectures and Amazon Web Services Cloud.

Next month, I will be in Brazil and traveling to 3 main cities to keynote and present at different conferences and user groups. My complete plan is as follows:

Aug 6 - Aug 11, 2010 in Sao Paulo:

• Keynote at iMasters Pro's Ecommerce Forum Brazil 2010
• Presentation at the Cloud Computing Summit 2010 - Organized by Tecla

Aug 12 - Aug 16, 2010 in Rio De Janero:

• Meeting Customers and Open for meetings

Aug 17 - Aug 21, 2010 in Brasilia:

• Keynote at CONSEGI 2010 - Third International Congress of Free and Electronic Government
• General session on Architecting for the cloud : Best Practices and Design Patterns
• Getting Started with Amazon Web Services - Hands-on Workshop

If you are in Brazil and passionate about cloud computing, I would like to meet you. If you are an aspiring cloud developer or architect, system integrator trying to win a local SaaS contract or an ISV trying to build a cloud strategy around your product, send me an email at evangelists [[at]] amazon [[dot]] com to schedule a meeting. I would love to exchange ideas, learn more about the local market and discuss the future. If you are a leader of a local user group and would like us to present to your group, please contact me in advance.

-- Jinesh

Security is a top priority for Amazon Web Services. Providing a trustworthy infrastructure for you to develop and deploy applications is a responsibility we take very seriously. One important aspect of gaining your trust is being open and transparent about our security processes and continually working toward achieving industry-recognized certifications. Other important aspects include providing you with mechanisms for contacting us about potential security issues and enabling you to conduct security tests of the applications you deploy on AWS. I'm pleased to announce today two new policies: one that outlines our vulnerability reporting process and one that describes how to receive permission to conduct penetration tests of the applications running on your EC2 instances.

A new page in the AWS Security Center describes our vulnerability reporting process. The process is high-priority for us, it's human-driven, and is governed by a service level commitment. Like other technology providers, we believe in the concept of responsible disclosure: let's work together to protect everyone.

Another page in the Security Center describes our penetration testing procedure. Normally, conducting such tests violates our Acceptable Use Policy because these tests are often indistinguishable from real attacks. However, to ensure higher degrees of application security, external testing is an important phase of development and deployment. We put the procedure in place so that we won't respond to your testing as if your instances were under attack.

The e-mail address aws-security@amazon.com is your single point of contact for all things security-related. If you need to contact us about a particularly sensitive issue, you can encrypt your message with our PGP public key. And, of course, if you suspect abuse of EC2 or other AWS services, our abuse reporting process remains in place.

Finally, a small navigational change. We've moved the bulletins off the main page and onto a separate security bulletin list and changed the format so that all bulletins are displayed rather than just the most recent five.

As always, we welcome your comments and feedback. We're here to help you succeed!

> Steve <

You can now use the Linux kernel of your choice when you boot up an Amazon EC2 instance. 

We have created a set of AKIs (Amazon Kernel Images) which contain the PV-Grub loader. This loader simply chain-boots the kernel provided in the associated AMI (Amazon Machine Image). Net-net, your instance ends up running the kernel in the AMI instead of the kernel specified in the boot process.

You need to install an "EC2 compatible" kernel and create an initrd (initial RAM disk) as part of your AMI. You also need to create a menu (/boot/grub/menu.lst) for the Grub boot loader. Once you've done this you can create the AMI and then launch instances by using one of the PV-Grub "kernels" as described above. You may find this document to be helpful if you want to learn more about the Linux boot process.

To be compatible with EC2, a Linux kernel must support Xen's pv_ops (paravirtual ops) infrastructure with XSAVE disabled or the Xen 3.0.2 interface. The following kernels have been tested and/or have vendor support:

• Fedora 8-12 Xen kernels
• SLES/openSUSE 10x, 11.0, and 11.1 Xen kernels
• SLES/openSUSE 11.x EC2 Variant
• Ubuntu EC2 Variant
• RHEL 5.x
• CentOS 5.x

Other kernels may not start reliably within EC2. We're working with the providers of popular AMIs to make sure that they will start to use PV-Grub in the near future.

You can read more about this in our "Enabling User Provided Kernels in Amazon EC2" document.

-- Jeff;

PS - You could (if you are sufficiently adept) use this facility to launch an operating system that we don't support directly (e.g. FreeBSD). If you manage to do this, please feel free to let me know.

One thing that I love (among many) about working at Amazon.com is the customer-driven innovation cycle. We introduce a new product or service with a useful yet somewhat minimal feature set. We do this to get it out into the real world as soon as possible so that our customers can start to use it and to provide us with feedback on it. Then we put an ear to the ground and do our best to listen and to learn. The information that we gather in this way feeds directly in to the product planning process. I hear the phrase "voice of the customer" several times per week as I wander the halls.

The Amazon CloudFront team has been improving their product in this way since they launched it at the end of 2008. In response to requests from customers they have added a number of great features including more edge locations, private content, streaming media content, HTTP request logging, a reduced TTL (Time To Live), private streamed content, streaming access logs, console support, additional pricing tiers, support for HTTPS, and out-and-out price reductions.

Our customers have been asking for additional information in the CloudFront access logs. Specifically, they have asked us to include the URL's query string (the part after the "?") in each log entry so that they can implement better and more detailed tracking of the source of each request.

We have implemented this feature and it is available now.

Here's how it works. The basic URL to the image above is:

http://d1nqddva888cns.cloudfront.net/amazon_product_cycle.png

Let's say that I want to use the same image in this blog post and in a white paper about corporate innovation. I could simply append two distinct query strings to the URL, like this:

http://d1nqddva888cns.cloudfront.net/amazon_product_cycle.png?bloghttp://d1nqddva888cns.cloudfront.net/amazon_product_cycle.png?white_paper

My log analysis software can use the "?blog" and "?white_paper" strings to figure out which source is more popular.

Many customers have told us that they use (or plan to use) this technique to track marketing campaigns and microsites, as well as targeted use of their content. People used to say that "content is king." These days, based on what I am seeing and hearing, numbers and analytics are about to depose the king. The ability to track, analyze, and understand the behavior of site visitors (perhaps using some A/B testing and a healthy dose of Elastic MapReduce) has become a critical success factor.

You can generate these query strings yourself, but I'd assume that sophisticated blogging and content management tools will start to do so over time. CloudFront logs and then ignores the query string. It is not passed along to Amazon S3.

As I said earlier, this new feature is available now and I look forward to hearing how you put it to use. If you develop content management or analytic tools and add support for it, let me know by posting a comment or by sending me some email.

A great way for you to influence our future investments is by sharing your use case with us by means of our CloudFront survey. We always appreciate it when our customers suggest ways to make CloudFront even better.

-- Jeff;

We're starting to wire various AWS services to each other, with interesting and powerful results. Today I'd like to talk to you about a brand new connection between Amazon S3 and the Amazon Simple Notification Service.

When I introduced you to SNS earlier this year I noted that "SNS is also integrated with other AWS services" and said that you could arrange to deliver notifications to an SQS message queue.

We're now ready to take that integration to a new level. Various parts of AWS will now start to publish messages to an SNS topic to let your application know that a certain type of event has occurred. The first such integration is with Amazon S3, and more specifically, with S3's new Reduced Redundancy Storage option.

You can now configure any of your S3 buckets to publish a message to an SNS topic of your creation (permissions permitting) when S3 detects that it has lost an object that was stored in the bucket using the RRS option.Your application can subscribe to the topic and (when the event is triggered) respond by regenerating the object and storing it back in S3. The message will include the event, a timestamp, the name of the bucket, the object's key and version id, and some internal identifiers.

Let's say that you are using S3 to store an original image and some derived images. You would use the STANDARD storage class for the original image and the REDUCED_REDUNDANCY storage class for the derived images. You would also need to store the information needed to regenerate a derived image from the original image. You could store this in SimpleDB or you could create a naming convention for your S3 object keys and then extract the needed information from the URL.

Consider this image:

http://faces.s3.amazonaws.com/jbarr_2007_web.jpg

It is the original image and would be stored with the STANDARD storage class. Derived images (scaled to a new size in this case) would use a suffix containing the needed information, and would be stored with REDUCED_REDUNDANCY:

http://faces.s3.amazonaws.com/jbarr_2007_web_120x168.jpg

A notification would be stored on the faces bucket and routed to a topic such as faces_web_app_errors. Your application need only await events on the topic and respond as follows:

• Confirm the event is of the expected type (s3:ReducedRedundancyLostObject)
• Extract the bucket and key name from the event
• Parse the key name to identify the key of the original object and the transform to be applied
• Fetch the original object
• Apply the transform (image scaling in this case)
• Store the derived object in S3 using the REDUCED_REDUNDACY storage class

Over time, we'll wire up additional events (for S3 and for other services) to SNS. You can prepare for this now by creating general purpose event handlers in your application, and by keeping your code properly factored so that it is easy to create an object when needed. For the case listed above, I would think about structuring my application so that the only way to create a derived object is in response to an event. I would then generate synthetic "lost" events and use them to materialized the derived objects for the first time.

-- Jeff;

The AWS Management Console now supports Amazon S3's Reduced Redundancy Storage. You can view and change the storage class of an S3 object in the object's Properties pane:

You can also select multiple objects and change the storage class for all of them at the same time.

Finally, you can set the option when you upload one or more objects:

Are you putting RRS to use in your application? I'd like to learn more. Send me an email or leave me a comment.

-- Jeff;

We are excited to announce enhancements to Amazon S3’s Reduced Redundancy Storage (RRS) option: Notifications for RRS object loss and RRS support in the AWS Management Console. Customers using the console can now select the RRS storage option when uploading new files and can easily move a set of objects or even an entire folder from Standard storage to RRS. Additionally, Amazon S3 customers can now configure their bucket so that when Amazon S3 detects the loss of an RRS object, a notification will be sent through Amazon Simple Notification Service. For more information on these new features, attend the Introducing Amazon S3 Enhanced Support for Reduced Redundancy Storage webinar on July 20. For more information on RRS, visit the Amazon S3 detail page.

We've added two new features to the Amazon Virtual Private Cloud (VPC) to make it more powerful and easier to use. Here's the scoop:

• IP Address Control - You can now assign the IP address of your choice to each of the EC2 instances that you launch in your Virtual Private Cloud. The address must be within the range of addresses that you designated for the VPC, it must be available for use within the instance's network subnet, and it must not conflict with any of the addresses that are reserved for internal use by AWS. You can specify the desired address as an optional parameter to the RunInstances function. This will allow you to have additional control of your network configuration, and has been eagerly anticipated by many of our customers. Two use cases that we've heard about already are running DNS servers and Active Directory® Domain Controllers.
• Config File Generation - VPC can now generate configuration files (example at right) for several different types of devices including the Cisco ISR and a number of Juniper products including the J-Series Service Router, the SSG (Secure Services Gateway), and the ISG (Integrated Security Gateway). The files can be generated from the command line or from within ElasticFox. Generating the config files in this way lets you avoid common configuration issues and allows you to be up and running in minutes.

 

If you want to connect a Linux-based VPN gateway to your Virtual Private Cloud, take a look at Amazon VPC With Linux. This article will show you how to set up IPSec and BGP routing and includes detailed configuration information.

If you are running OpenSolaris, take a look at the OpenSolaris VPC Gateway Tool.

-- Jeff;

A number of AWS users have been using Amazon EC2 to solve a variety of computationally intensive problems. Here's a sampling:

• Atbrox and Lingit use Elastic MapReduce to build data sets that help individuals with dyslexia to improve their reading and writing skills.
• Systems integrator Cycle Computing helps Varian to run compute-intensive Monte Carlo simulations.
• Harvard Medical School's Laboratory for Personalized Medicine creates innovative genetic testing models.
• Pathwork Diagnostics runs tens of thousands of models to help oncologists to diagnose hard-to-identify cancer tumors.
• Razorfish processes huge datasets on a very compressed timescale.
• The Server Labs helps the European Space Agency to build the operations infrastructure for the Gaia project.

Some of these problems are examples of what are called “embarrassingly parallel” computing.  Others leverage the Hadoop framework for data-intensive computing, spreading the workload across a large number of EC2 instances.

Our customers have also asked us about the ability to run even larger and more computationally complex workloads in the cloud.

It is clear that people are now figuring out that they can do HPC (High-Performance Computing) in the cloud. We want to make it even easier and more efficient for them to do so!

Our new Cluster Compute Instances will fit the bill. With Cluster Compute Instances, you can now run many types of large-scale network-intensive jobs without losing the core advantages of EC2: a pay-as-you-go pricing model and the ability to scale up and down to meet your needs.

Each Cluster Compute Instance consists of a pair of quad-core Intel "Nehalem" X5570 processors with a total of 33.5 ECU (EC2 Compute Units), 23 GB of RAM, and 1690 GB of local instance storage, all for $1.60 per hour.

Because many HPC applications and other network-bound applications make heavy use of network communication, Cluster Compute Instances are connected using a 10 Gbps network. Within this network you can create one or more placement groups of type "cluster" and then launch Cluster Compute Instances within each group. Instances within each placement group of this type benefit from non-blocking bandwidth and low latency node to node communication.

The EC2 API's, the command-line tools, and the AWS Management Console have all been updated to support the creation and use of placement groups. For example, the following pair of commands creates a placement group called biocluster and then launches 8 Cluster Compute Instances inside of the group:

$ ec2-create-placement-group biocluster -s cluster

$ ec2-run-instances ami-2de43f55 --type cc1.4xlarge --placement-group biocluster -n 8

The new instance type is now available for Linux/UNIX use in a single Availability Zone in the US East (Northern Virginia) region. We'll support it in additional zones and regions in the future. You can purchase individual Reserved Instances for a one or a three year term, but you can't buy them within specific cluster placement groups just yet. There is a default usage limit for this instance type of 8 instances (providing 64 cores). If you wish to run more than 8 instances, you can request a higher limit using the Amazon EC2 instance request form.

The Cluster Compute Instances use hardware-assisted (HVM) virtualization instead of the paravirtualization used by the other instance types and requires booting from EBS, so you will need to create a new AMI in order to use them. We suggest that you use our Centos-based AMI as a base for your own AMIs for optimal performance. See the EC2 User Guide or the EC2 Developer Guide for more information.

The only way to know if this is a genuine HPC setup is to benchmark it, and we've just finished doing so. We ran the gold-standard High Performance Linpack benchmark on 880 Cluster Compute instances (7040 cores) and measured the overall performance at 41.82 TeraFLOPS using Intel's MPI (Message Passing Interface) and MKL (Math Kernel Library) libraries, along with their compiler suite. This result places us at position 146 on the Top500 list of supercomputers. The input file for the benchmark is here and the output file is here.

Putting this all together, I think that we have put together a true fire-breathing dragon of an offering. You can now get world-class compute and network performance on an economical, pay-as-you-go basis.  The individual instances perform really well, and you can tie a bunch of them together using a fast network to attack large-scale problems. I'm fairly certain that you can't get this much compute power so fast or so economically anywhere else.

I'm looking forward to writing up and sharing some of the success stories from the customers who've been helping us to test the Cluster Compute instances during our private beta test. Feel free to share your own success stories with me once you've had a chance to give them a try.

Update - Here's some additional info:

• Werner Vogels: Expanding the Cloud - Cluster Compute Instances for Amazon EC2.
• TechCunchIT: AWS Launches Cluster Compute EC2 Instances For High Performance Applications.
• ZDNet: Amazon Web Services tackles high performance computing instances.
• And our own HPC Applications page.
  

-- Jeff;