Feed aggregator

I almost missed a really important anniversary! Yesterday marked Amazon EC2's fourth birthday. Here are some of the ways that EC2 has grown and changed in the last four years:

Category 2006 2010 Regions One Four Availability Zones One Ten Availability Zones Instance Types One Nine Pricing Models One Three Storage Ephemeral Storage
Ephemeral Storage
Elastic Block Store
Operating Systems Linux Linux, Windows, OpenSolaris
Management Tools Command-Line Tools Command-Line Tools
AWS Management Console
Third-Party Tools Ancillary Services - Elastic Load Balancing, Auto Scaling, CloudWatch
High Performance Computing - Elastic Map Reduce, Cluster Compute Instances

We've done quite a bit, but we're not resting, not for a minute. We have a lot of open positions on the AWS team, including a really interesting developer position within the EC2 team. This developer will focus on EC2's dynamic market pricing features. In addition to experience with Ruby, Perl, Java, C, or C++, candidates should have some experience building large-scale distributed systems and an interest in operational scheduling, optimization, and constraint satisfaction. You can read more here and you can send your resume directly to amazon-ec2-spot-jobs@amazon.com.

While I am on the subject of anniversaries, eight years ago this month I abandoned my full-time consulting practice to take a development position with the Amazon Associates Team, with the agreement that I could spend some of my time helping out with the effort to create and market the E-Commerce Service (which has since become the Product Advertising API). A few months in, I was asked if I would mind speaking at a conference. I guess I did ok, because they asked me to do another one, and before too long they invited me to apply for the position of Web Services Evangelist. I took on that title in the spring of 2003 and have been spreading the word about our web service efforts ever since. All things considered, this is a really awesome place to work. Day after day, week after week, things get more and more exciting around here. The pace is quick and I do my best to keep up. We do our best to understand and to meet the needs of our customers with regard to features, reliability, scale, business models, and price. I get to work with and to learn from a huge number of world-class intellects. If this sounds like the kind of place for you, check out our list of open jobs and apply today!

-- Jeff;

Good news for all those interested in security...we've released the fourth version of our Overview of Security Processes whitepaper. It contains ten pages of new and additional detailed information. Highlights of new content include:

• A description of the AWS control environment
• A list of our SAS-70 Type II Control Objectives
• Some discussion of risk management and shared responsibility principles
• Greater visibility into our monitoring and communication processes and our employee lifecycle
• Descriptions of our physical security, environmental safeguards, configuration management, and business continuity management processes and plans
• Updated summaries of new AWS security features
• Additional detail about the security attributes of various AWS components

The additional information and greater level of detail should help to answer many common questions. As always, feel free to reach out to us if you're still needing more information.

> Steve <

We just released version 1.0.8 of theAWS SDK for Java. In addition to some bug fixes, the SDK includes the following new features:

• Support for the new Reserved DB Instances.
• Improved constructors for the Amazon RDS model classes.
• A new StepFactory class to simplify the process of creating Elastic MapReduce job flows.
• Improved support for EC2 security groups.
• Improved constructors for the EC2 model classes.
• Additional diagnostic information in AWS responses including request IDs, S3 host IDs, and SimpleDB box usage.

The SDK includes the AWS Java library and some helpful code samples. You may also want to check out the AWS Toolkit for Eclipse.

-- Jeff;

We're ready to start accepting entries for the AWS Start-Up Challenge 2010!

Each year we run the challenge to help young, promising start-ups get noticed and to compete for $100,000 (USD) in cash and credits (full prize list).

We've made some important changes this year:

• We can now accept applications from 22 countries across the Americas, Europe, and Asia.
• We'll now recognize five regional semi-finalists from each of the three regions, at least six finalists, and one global grand prize winner.
• Start-ups must be currently using or in the process of using one or more of the Amazon Web Services. We will now accept entries from start-ups that have yet to launch.

We will be looking for applications which can grow into significant, meaningful businesses over time. We will look for implementation and integration of AWS services, originality and creativity, likelihood of long-term success and scalability, and overall effectiveness in addressing a need in the marketplace. Full information on the contest can be found here.

Previous winners include Good Data (2009), Yieldex (2008), and Ooyala (2007).

Submit your entry today!

-- Jeff;

PS: Several of you have asked why the Start-Up Challenge isn’t available world-wide. It turns out that the rules and regulations governing competitions vary widely from country to country so this isn’t as simple as it sounds. The list of eligible countries has grown from one (2007) to four (2009) and now to 22. We’ll do our best to expand the list in the future.

The AWS Start-up Challenge is back with an expanded world footprint – start-ups in 22 countries across the Americas, Asia, and Europe can compete for a chance to win $100,000 in cash and credits. This year, we will recognize 15 regional semi-finalists, 6 global finalists, and select one global grand prize winner. We’re looking for the most promising start-ups that can grow into significant, meaningful businesses that leverage AWS for their infrastructure. The contest submission closes October 31. To learn more and enter the competition, visit the AWS Start-Up Challenge home page.

I can't even leave my neighborhood without getting feature requests! My neighbor Rick was walking his dog past my house and stopped to ask me when we were going to support Reserved Instances for Amazon RDS. Such is life in the Pacific Northwest part of the US.

Hey Rick, we've got your RDS Reserved Instances ready now. You can get a lower price and know that an instance has been reserved for your use.

After you make a low, one-time payment, you can use a DB Instance at an hourly rate that is significantly lower than the On-Demand pricing for the same instance class. This will result in a decrease in AWS charges for any application that requires full-time access to a relational database.

When you purchase an RDS Reserved DB Instance, you choose a specific AWS Region and instance class. You do not need to choose an Availability Zone. You can purchase up to 20 Reserved DB instances. If you need more DB Instances, you need to complete the Amazon RDS DB Instance request form. You cannot move a reservation from one Region to another and they always apply to a particular DB instance class, so make your purchase with care.

You can make your purchase using the RDS APIs, or the RDS command-line tools. Here are the commands:

• rds-describe-reserved-db-instances - Returns the list of DB Instance reservations for your account or details for one of your reserved database instances.
• rds-describe-reserved-db-instances-offerings - Returns the list of DB Instance offerings that are available for purchase.
• rds-purchase-reserved-db-instances-offering - Purchases one or more reserved DB Instances.

The one-time fee starts at $227.50 for a Small DB Instance with a one-year term.

-- Jeff;

PS - Clean up after your dog, Rick.

I've got a pair of related news items for users of the Amazon Relational Database System (RDS):

1. We now support version 5.1.49 of MySQL, with the InnoDB Plugin as the InnoDB storage engine.
2. We now provide a new, optional level of control over when and if your DB Instance(s) are upgraded to new MySQL versions supported by Amazon RDS. This new functionality is called DB Engine Version Management, and this post will help you understand what it can do and how to use it.

MySQL Version 5.1.49 with InnoDB Plugin

Version 5.1.49 of MySQL includes a number of bug fixes and enhancements. For DB Instances running MySQL 5.1.49, the most significant change is the inclusion of InnoDB Plugin as the InnoDB storage engine, replacing InnoDB Builtin. InnoDB Plugin offers (according to the feature page) performance and scalability enhancements (especially on multi-core platforms), fast index creation, data compression, and more. The InnoDB Builtin (InnoBase) is not available in 5.1.49 and will not be available in future Amazon RDS Supported versions.

New DB Instances that you create will now run MySQL 5.1.49 with InnoDB Plugin, unless otherwise specified using the new DB Engine Version Management feature described in the next section.

DB Engine Version Management

With DB Engine Version Management, Amazon RDS gives you additional (yet optional) control over the version of relational database software (i.e. MySQL) powering your DB Instance. The goal of this functionality is to provide you the flexibility to maintain compatibility with specific MySQL versions, test new versions with your application before deploying in production, and perform version upgrades on your own terms and timelines. Let’s take a look:

DB Instance Creation

Amazon RDS now allows you to create new DB Instances using any supported version of MySQL. Right now this means version 5.1.45 and 5.1.49, but we plan to support additional minor and major versions (e.g. MySQL 5.5) in the future. There's a new EngineVersion attribute associated with each DB Instance, and you can specify your desired Version when issuing a CreateDBInstance API call (or the associated command).  New DB Instances will now use MySQL 5.1.49 with InnoDB Plugin by default.  If you still wish to use MySQL 5.1.45 (which uses InnoDB Builtin) for a new DB Instance, you can do so using this new functionality.

Automatic Upgrade Schedule

Your existing DB Instances will be upgraded to the new version of MySQL with InnoDB Plugin during your scheduled maintenance window on the week of October 4, 2010. If you don't want to upgrade, you need to use ModifyDBInstance to set the AutoMinorVersionUpgrade attribute of your DB instances to "false."

Find Your Engine

The new DescribeDBEngineVersions API call returns a list of all supported versions of MySQL. You can use it (or the rds-describe-db-engine-versions command) to drive your configuration process. You could also use it to set up a really cool automated testing framework. You could call the function, iterate through the resulting list of versions, and create a new DB Instance for each one. Then you could test your application to make sure that it works with each version of MySQL. Or you could create several DB Instances (one per version) simultaneously and test the same queries against each instance.

Choose to Upgrade Today

With DB Engine Version Management, you can also upgrade any of your existing DB Instances to InnoDB Plugin and the new version of MySQL today by setting the EngineVersion attribute to "5.1.49" using the ModifyDBInstance API call (or the associated command).  Amazon RDS will take care of the rest. This will happen immediately if you set the ApplyImmediately flag when you modify the EngineVersion attribute, or at the next maintenance window for your DB Instance if you don't.

Learning More

To learn more about DB Engine Version Management and associated API changes, please visit our Forum Post on the subject. You can also read more about the new functionality via our DB Engine Version Management FAQs, which include information about our guidelines for supporting new versions and deprecating old versions.

With these changes, Amazon RDS becomes even more powerful. If you want to verify that your application works with the new version of MySQL, simply create a snapshot of your existing DB Instance, create a new one from the snapshot, and test away.

-- Jeff;

We’re excited to announce today that we’ve added Reserved Database Instances (Reserved DB Instances) for Amazon RDS. With Reserved DB Instances, you can now make a one-time, up-front payment to create a one or three year reservation to run your DB Instance in a specific Region and receive a significant discount off of the ongoing hourly usage charge. Your DB Instances will work just as they do today, but can lower your costs over the life of your database. For more information on Reserved DB Instances, visit the Amazon RDS detail page.

Since the announcement of Recovery.gov last March, Amazon has seen an accelerating adoption of the cloud by our Federal customers.  These include Treasury.gov, the Federal Register 2.0 at the National Archives, the Supplemental Nutrition Assistance Program at USDA, the openEI.org project at DoE's National Renewable Energy Lab, and the Jet Propulsion Laboratory at NASA. 

On September 23, 2010 we'll be conducting a half-day event in Crystal City, Virginia to discuss the use of the AWS Cloud by the Federal Government.

Speakers will include Amazon CTO Werner Vogels, AWS CISO Steve Schmidt, and a number of AWS customers including representatives of the agencies and organizations mentioned above. There will also be time for Q&A and a cocktail reception afterward for networking.

The event is free but you need to sign up now in order to reserve your spot.

-- Jeff;

PS - You may also enjoy the story behind the Federal Register 2.0 makeover.

If you have ever set up a web site from scratch, you know that you have to handle the root of the web site in a special way so that requests for the site's root URL (e.g. http://aws.amazon.com) are handled properly. You generally map the root URL to an HTML document such as index.html using an entry in the web server's configuration file.

You can now set a default root object for any of your Amazon CloudFront distributions to duplicate this behavior for your own content. This object must be stored within the Amazon S3 bucket associated with the distribution. Once you have set the default root object, a request for the root URL of the distribution will return the contents of the default root object. 

With this change, you can now create a distribution that acts just like a static web site.

If you don't set a default root object for a distribution, the response to a request for its root URL has not changed. Depending on the ACL on the distribution's bucket and on the objects inside, the request could return a list of the contents of the bucket or a 403 error.

You can learn more about this new feature by reading the CloudFront documentation.

These partners support this new feature:

• CloudBuddy Personal
• Bucket Explorer
• CloudBerry Explorer
• boto

-- Jeff;

Amazon CloudFront, the easy to use content delivery network, now supports the ability to assign a default root object to your HTTP or HTTPS distribution. This default object will be served when Amazon CloudFront receives a request for the root of your distribution – i.e., your distribution’s domain name by itself. This feature should make it easier for you to serve all of your static content from edge locations close to your end users and provide them with a better experience. You can read more about the default root object feature in the Amazon CloudFront Developer Guide. For more information on Amazon CloudFront, visit the Amazon CloudFront page.

Motivation
The application uses URL containing dynamic parameters and so a record/replay fails
http://www.mail-archive.com/jmeter-user@jakarta.apache.org/msg30317.html

Solution
This short tutorial deals with test scripts that interact with dynamic data, and we will choose a Spring Webflow example to write a test script against.
So we look around on Google for a sample Spring Webflow application which leads us to
Spring By Example Sample
I'm not going to use the JMeter Proxy Recorder because it seems to harm beginners more than it helps. Testers seem to get the impression that all they need to do is record and replay the script and they are done. So I shall use Firefox and LiveHttpHeaders. We are going to Login to the site and create a User

Step 1: Request the Login Page

http://www.springbyexample.org/simple-webflow/login.jsp
GET /simple-webflow/login.jsp HTTP/1.1

Response
HTTP/1.1 200 OK
Set-Cookie: JSESSIONID=[uniquevalue1]; Path=/simple-webflow

So the first request response shows that the system sets a cookie and the name should tell us that this is a session id. [TODO]. This should also indicate to us that we need to add a cookie manager. Note the status is 200 in the Response. This is the server telling us all is well.

Step 2: Submit Login

http://www.springbyexample.org/simple-webflow/loginProcess;jsessionid=[uniquevalue1]

POST /simple-webflow/loginProcess;jsessionid=[uniquevalue1]
HTTP/1.1
Cookie: JSESSIONID=[uniquevalue1]
Content-Type: application/x-www-form-urlencoded
Content-Length: 48
j_username=david&j_password=newyork&submit=Login

Response
HTTP/1.1 302 Moved Temporarily
Date: Tue, 17 Aug 2010 05:03:48 GMT
Set-Cookie: JSESSIONID=[uniquevalue2]; Path=/simple-webflow
Location: http://www.springbyexample.org/simple-webflow/index.jsp
----------------------------------------------------------
http://www.springbyexample.org/simple-webflow/index.jsp
GET /simple-webflow/index.jsp HTTP/1.1
Cookie: JSESSIONID=[uniquevalue2]

HTTP/1.1 302 Moved Temporarily
Location: http://www.springbyexample.org/simple-webflow/index.html
----------------------------------------------------------
http://www.springbyexample.org/simple-webflow/index.html
GET /simple-webflow/index.html HTTP/1.1
Cookie: JSESSIONID=[uniquevalue2]
HTTP/1.1 200 OK

a. The browser sends back the cookie that was previously set. However note that the URL also has a jsessionid which has the same value as the cookie that was set. This is known as URL rewriting. Our test will ignore this value because we will be using a Cookie manager
b. The server sends a 302 response to the request. This basically asks the browser to request a new URL (the one that the Location Response Header has). Note this can happen more than once as the next request too results in a 302 response, which finally results in a 200 status code.
HTTP Status Codes are discussed here
c. The server also sends a Set-Cookie for JSessionID again with a new value. This is usually done for security reasons. We can ignore this for JMeter because the Cookie manager will handle this for us. However the sampler that we use needs to have follow redirects set. If redirect automatically is set , JMeter would not get to see this Set-Cookie header and since the application has switched the session id , if you sent it the old session id (which was associated to the before login session)

Click the Create Link

http://www.springbyexample.org/simple-webflow/person.html
GET /simple-webflow/person.html HTTP/1.1
Cookie: JSESSIONID=[uniquevalue2]

Response
HTTP/1.1 302 Moved Temporarily
Location: http://www.springbyexample.org/simple-webflow/person.html?execution=e1s1

----------------------------------------------------------
http://www.springbyexample.org/simple-webflow/person.html?execution=e1s1
GET /simple-webflow/person.html?execution=e2s1 HTTP/1.1
Cookie: JSESSIONID=[uniquevalue2]

HTTP/1.1 200 OK
Date: Tue, 17 Aug 2010 05:06:04 GMT

Clicking the Create link also resulted in a 302 code so the browser issued a new request. But take a look at the URL. It has a parameter named execution with a value e1s1. Now click the link again.

Click Create link again

http://www.springbyexample.org/simple-webflow/person.html
GET /simple-webflow/person.html HTTP/1.1
Cookie: JSESSIONID=[uniquevalue2]

Response
HTTP/1.1 302 Moved Temporarily
Location: http://www.springbyexample.org/simple-webflow/person.html?execution=e2s1
----------------------------------------------------------
http://www.springbyexample.org/simple-webflow/person.html?execution=e2s1

GET /simple-webflow/person.html?execution=e2s1 HTTP/1.1
Cookie: JSESSIONID=[uniquevalue2]

HTTP/1.1 200 OK

The "execution" parameter value changes! So if you run the same test multiple times , the value has to be different. This is a common cause of failure of Record / Replay tests. Some of the data is dynamic. This data can be of various forms
a. Something the application framework needs
b. Something where the data varies (E.g. the UserId, the Product ID). Typically a user friendly string is shown on the screen but when the user performs an action , an id (usually a key in the database) is what is used by the application
c. Dynamic fields (E.g. DHTML based applications) which get added on the fly.
To deal with this parameter/URL we need to be able to extract the value from previous requests in our JMeter scripts.
If you haven't developed the application it may be difficult to identify whats dynamic and whats not. The easy way is to record the script twice(With different data) and observe what changes. e.g. if you browse a product catalog , change the product.

Submit the form

http://www.springbyexample.org/simple-webflow/person.html?execution=e2s1

POST /simple-webflow/person.html?execution=e2s1
HTTP/1.1
Cookie: JSESSIONID=[uniquevalue2]
Content-Length: 56
id=&firstName=deepak&lastName=shetty&_eventId_save=Save

Response
HTTP/1.1 302 Moved Temporarily
Location: http://www.springbyexample.org/simple-webflow/person/search.html
----------------------------------------------------------
http://www.springbyexample.org/simple-webflow/person/search.html
GET /simple-webflow/person/search.html HTTP/1.1
Cookie: JSESSIONID=[uniquevalue2]

HTTP/1.1 200 OK

Unfortunately the application has a visual bug, the added user does not show. However if you do a view source on the page and search for the username you created you can find it.

Click Logout

http://www.springbyexample.org/simple-webflow/logout
GET /simple-webflow/logout HTTP/1.1
Cookie: JSESSIONID=[uniquevalue2]

Response
HTTP/1.1 302 Moved Temporarily
Set-Cookie: SPRING_SECURITY_REMEMBER_ME_COOKIE=""; Expires=Thu, 01-Jan-1970 00:00:10 GMT; Path=/simple-webflow
Location: http://www.springbyexample.org/simple-webflow/logoutSuccess.jsp
----------------------------------------------------------
http://www.springbyexample.org/simple-webflow/logoutSuccess.jsp
GET /simple-webflow/logoutSuccess.jsp HTTP/1.1
Cookie: JSESSIONID=[uniquevalue2]

HTTP/1.1 200 OK
Set-Cookie: JSESSIONID=[uniquevalue3]; Path=/simple-webflow


The JMeter Script
Now that we know the above we can get to writing the script

Most of the script is pretty much standard(All Samplers use follow redirects and we have added a Cookie Manager), till we get to the point where we need to extract out the URL.
The Request Create Sampler extracts out the URL. Since we know that the application is going to redirect us , the extracter will extract out the last URL we will be on



The Post Create Data Sampler simply uses this extracted URL



Note that all samplers have assertions that verify the behavior, so if something goes wrong the Assertion should fail. For e.g. after creating the user we verify that the user we created exists on the page. When requesting a page the assertions verify some text on the page that is unique to the page.
Run the test, we get no errors! Modify the ThreadGroup to have two threads and two iterations. Run the test , everything should pass. Now look at the View Results Tree listener. You should see the redirects being followed (multiple children results under a parent). But do you see different values for the execution parameter? No - This is because each thread is its own session and each user will start with the same parameter value. Even having multiple iterations on the thread group doesnt give you different values because we are logging out at the end of the script so the next iteration is a new session.

Try adding a Loop controller to create 10 users. Note that the loop controller must enclose the Request Create and Post Create samplers (because we need to extract a new value for the execution parameter every time!). Using View results tree you can now observe the execution parameter changing.



Sample Script
Spring Webflow.jmx

Spring Webflow with loop.jmx

My previous trips to China, India, Japan, London have been super productive. I get a chance to meet tons of new people, make a lot of friends and talk about something that I am truly passionate about: Cloud Architectures and Amazon Web Services Cloud.

Next month, I will be in Brazil and traveling to 3 main cities to keynote and present at different conferences and user groups. My complete plan is as follows:

Aug 6 - Aug 11, 2010 in Sao Paulo:

• Keynote at iMasters Pro's Ecommerce Forum Brazil 2010
• Presentation at the Cloud Computing Summit 2010 - Organized by Tecla

Aug 12 - Aug 16, 2010 in Rio De Janero:

• Meeting Customers and Open for meetings

Aug 17 - Aug 21, 2010 in Brasilia:

• Keynote at CONSEGI 2010 - Third International Congress of Free and Electronic Government
• General session on Architecting for the cloud : Best Practices and Design Patterns
• Getting Started with Amazon Web Services - Hands-on Workshop

If you are in Brazil and passionate about cloud computing, I would like to meet you. If you are an aspiring cloud developer or architect, system integrator trying to win a local SaaS contract or an ISV trying to build a cloud strategy around your product, send me an email at evangelists [[at]] amazon [[dot]] com to schedule a meeting. I would love to exchange ideas, learn more about the local market and discuss the future. If you are a leader of a local user group and would like us to present to your group, please contact me in advance.

-- Jinesh

Security is a top priority for Amazon Web Services. Providing a trustworthy infrastructure for you to develop and deploy applications is a responsibility we take very seriously. One important aspect of gaining your trust is being open and transparent about our security processes and continually working toward achieving industry-recognized certifications. Other important aspects include providing you with mechanisms for contacting us about potential security issues and enabling you to conduct security tests of the applications you deploy on AWS. I'm pleased to announce today two new policies: one that outlines our vulnerability reporting process and one that describes how to receive permission to conduct penetration tests of the applications running on your EC2 instances.

A new page in the AWS Security Center describes our vulnerability reporting process. The process is high-priority for us, it's human-driven, and is governed by a service level commitment. Like other technology providers, we believe in the concept of responsible disclosure: let's work together to protect everyone.

Another page in the Security Center describes our penetration testing procedure. Normally, conducting such tests violates our Acceptable Use Policy because these tests are often indistinguishable from real attacks. However, to ensure higher degrees of application security, external testing is an important phase of development and deployment. We put the procedure in place so that we won't respond to your testing as if your instances were under attack.

The e-mail address aws-security@amazon.com is your single point of contact for all things security-related. If you need to contact us about a particularly sensitive issue, you can encrypt your message with our PGP public key. And, of course, if you suspect abuse of EC2 or other AWS services, our abuse reporting process remains in place.

Finally, a small navigational change. We've moved the bulletins off the main page and onto a separate security bulletin list and changed the format so that all bulletins are displayed rather than just the most recent five.

As always, we welcome your comments and feedback. We're here to help you succeed!

> Steve <

You can now use the Linux kernel of your choice when you boot up an Amazon EC2 instance. 

We have created a set of AKIs (Amazon Kernel Images) which contain the PV-Grub loader. This loader simply chain-boots the kernel provided in the associated AMI (Amazon Machine Image). Net-net, your instance ends up running the kernel in the AMI instead of the kernel specified in the boot process.

You need to install an "EC2 compatible" kernel and create an initrd (initial RAM disk) as part of your AMI. You also need to create a menu (/boot/grub/menu.lst) for the Grub boot loader. Once you've done this you can create the AMI and then launch instances by using one of the PV-Grub "kernels" as described above. You may find this document to be helpful if you want to learn more about the Linux boot process.

To be compatible with EC2, a Linux kernel must support Xen's pv_ops (paravirtual ops) infrastructure with XSAVE disabled or the Xen 3.0.2 interface. The following kernels have been tested and/or have vendor support:

• Fedora 8-12 Xen kernels
• SLES/openSUSE 10x, 11.0, and 11.1 Xen kernels
• SLES/openSUSE 11.x EC2 Variant
• Ubuntu EC2 Variant
• RHEL 5.x
• CentOS 5.x

Other kernels may not start reliably within EC2. We're working with the providers of popular AMIs to make sure that they will start to use PV-Grub in the near future.

You can read more about this in our "Enabling User Provided Kernels in Amazon EC2" document.

-- Jeff;

PS - You could (if you are sufficiently adept) use this facility to launch an operating system that we don't support directly (e.g. FreeBSD). If you manage to do this, please feel free to let me know.

One thing that I love (among many) about working at Amazon.com is the customer-driven innovation cycle. We introduce a new product or service with a useful yet somewhat minimal feature set. We do this to get it out into the real world as soon as possible so that our customers can start to use it and to provide us with feedback on it. Then we put an ear to the ground and do our best to listen and to learn. The information that we gather in this way feeds directly in to the product planning process. I hear the phrase "voice of the customer" several times per week as I wander the halls.

The Amazon CloudFront team has been improving their product in this way since they launched it at the end of 2008. In response to requests from customers they have added a number of great features including more edge locations, private content, streaming media content, HTTP request logging, a reduced TTL (Time To Live), private streamed content, streaming access logs, console support, additional pricing tiers, support for HTTPS, and out-and-out price reductions.

Our customers have been asking for additional information in the CloudFront access logs. Specifically, they have asked us to include the URL's query string (the part after the "?") in each log entry so that they can implement better and more detailed tracking of the source of each request.

We have implemented this feature and it is available now.

Here's how it works. The basic URL to the image above is:

http://d1nqddva888cns.cloudfront.net/amazon_product_cycle.png

Let's say that I want to use the same image in this blog post and in a white paper about corporate innovation. I could simply append two distinct query strings to the URL, like this:

http://d1nqddva888cns.cloudfront.net/amazon_product_cycle.png?bloghttp://d1nqddva888cns.cloudfront.net/amazon_product_cycle.png?white_paper

My log analysis software can use the "?blog" and "?white_paper" strings to figure out which source is more popular.

Many customers have told us that they use (or plan to use) this technique to track marketing campaigns and microsites, as well as targeted use of their content. People used to say that "content is king." These days, based on what I am seeing and hearing, numbers and analytics are about to depose the king. The ability to track, analyze, and understand the behavior of site visitors (perhaps using some A/B testing and a healthy dose of Elastic MapReduce) has become a critical success factor.

You can generate these query strings yourself, but I'd assume that sophisticated blogging and content management tools will start to do so over time. CloudFront logs and then ignores the query string. It is not passed along to Amazon S3.

As I said earlier, this new feature is available now and I look forward to hearing how you put it to use. If you develop content management or analytic tools and add support for it, let me know by posting a comment or by sending me some email.

A great way for you to influence our future investments is by sharing your use case with us by means of our CloudFront survey. We always appreciate it when our customers suggest ways to make CloudFront even better.

-- Jeff;

We're starting to wire various AWS services to each other, with interesting and powerful results. Today I'd like to talk to you about a brand new connection between Amazon S3 and the Amazon Simple Notification Service.

When I introduced you to SNS earlier this year I noted that "SNS is also integrated with other AWS services" and said that you could arrange to deliver notifications to an SQS message queue.

We're now ready to take that integration to a new level. Various parts of AWS will now start to publish messages to an SNS topic to let your application know that a certain type of event has occurred. The first such integration is with Amazon S3, and more specifically, with S3's new Reduced Redundancy Storage option.

You can now configure any of your S3 buckets to publish a message to an SNS topic of your creation (permissions permitting) when S3 detects that it has lost an object that was stored in the bucket using the RRS option.Your application can subscribe to the topic and (when the event is triggered) respond by regenerating the object and storing it back in S3. The message will include the event, a timestamp, the name of the bucket, the object's key and version id, and some internal identifiers.

Let's say that you are using S3 to store an original image and some derived images. You would use the STANDARD storage class for the original image and the REDUCED_REDUNDANCY storage class for the derived images. You would also need to store the information needed to regenerate a derived image from the original image. You could store this in SimpleDB or you could create a naming convention for your S3 object keys and then extract the needed information from the URL.

Consider this image:

http://faces.s3.amazonaws.com/jbarr_2007_web.jpg

It is the original image and would be stored with the STANDARD storage class. Derived images (scaled to a new size in this case) would use a suffix containing the needed information, and would be stored with REDUCED_REDUNDANCY:

http://faces.s3.amazonaws.com/jbarr_2007_web_120x168.jpg

A notification would be stored on the faces bucket and routed to a topic such as faces_web_app_errors. Your application need only await events on the topic and respond as follows:

• Confirm the event is of the expected type (s3:ReducedRedundancyLostObject)
• Extract the bucket and key name from the event
• Parse the key name to identify the key of the original object and the transform to be applied
• Fetch the original object
• Apply the transform (image scaling in this case)
• Store the derived object in S3 using the REDUCED_REDUNDACY storage class

Over time, we'll wire up additional events (for S3 and for other services) to SNS. You can prepare for this now by creating general purpose event handlers in your application, and by keeping your code properly factored so that it is easy to create an object when needed. For the case listed above, I would think about structuring my application so that the only way to create a derived object is in response to an event. I would then generate synthetic "lost" events and use them to materialized the derived objects for the first time.

-- Jeff;

The AWS Management Console now supports Amazon S3's Reduced Redundancy Storage. You can view and change the storage class of an S3 object in the object's Properties pane:

You can also select multiple objects and change the storage class for all of them at the same time.

Finally, you can set the option when you upload one or more objects:

Are you putting RRS to use in your application? I'd like to learn more. Send me an email or leave me a comment.

-- Jeff;

We are excited to announce enhancements to Amazon S3’s Reduced Redundancy Storage (RRS) option: Notifications for RRS object loss and RRS support in the AWS Management Console. Customers using the console can now select the RRS storage option when uploading new files and can easily move a set of objects or even an entire folder from Standard storage to RRS. Additionally, Amazon S3 customers can now configure their bucket so that when Amazon S3 detects the loss of an RRS object, a notification will be sent through Amazon Simple Notification Service. For more information on these new features, attend the Introducing Amazon S3 Enhanced Support for Reduced Redundancy Storage webinar on July 20. For more information on RRS, visit the Amazon S3 detail page.

We've added two new features to the Amazon Virtual Private Cloud (VPC) to make it more powerful and easier to use. Here's the scoop:

• IP Address Control - You can now assign the IP address of your choice to each of the EC2 instances that you launch in your Virtual Private Cloud. The address must be within the range of addresses that you designated for the VPC, it must be available for use within the instance's network subnet, and it must not conflict with any of the addresses that are reserved for internal use by AWS. You can specify the desired address as an optional parameter to the RunInstances function. This will allow you to have additional control of your network configuration, and has been eagerly anticipated by many of our customers. Two use cases that we've heard about already are running DNS servers and Active Directory® Domain Controllers.
• Config File Generation - VPC can now generate configuration files (example at right) for several different types of devices including the Cisco ISR and a number of Juniper products including the J-Series Service Router, the SSG (Secure Services Gateway), and the ISG (Integrated Security Gateway). The files can be generated from the command line or from within ElasticFox. Generating the config files in this way lets you avoid common configuration issues and allows you to be up and running in minutes.

 

If you want to connect a Linux-based VPN gateway to your Virtual Private Cloud, take a look at Amazon VPC With Linux. This article will show you how to set up IPSec and BGP routing and includes detailed configuration information.

If you are running OpenSolaris, take a look at the OpenSolaris VPC Gateway Tool.

-- Jeff;