Feed aggregator

We've made two improvements to AWS Import/Export.

You can now send us a "raw" or internal SATA drive all by itself, with no need for an enclosure. You don't have to send connectors, cables, or power cords. Raw SATA drives appear to be the most cost-effective way to send large amounts of data from place to place.

If you have a SATA cradle (I use this one at home; others have told me that they like this one), you can connect the drive to your desktop machine without having to open up the enclosure.

Also, you can now send us drives with capacities up to 4 TB. Customers with the need to import or export large amounts of data will reduce the number of devices needed.

Don't forget that tools like Bucket Explorer, the CloudBerry S3 Explorer, and the S3Fox Explorer make it easy to create your Import and Export jobs.

-- Jeff;

You cannot improve that you cannot measure

-anonymous

Isn’t it so true. What’s the point of investing in scaling up, scaling out, adding caching, accelerators, etc, when you don’t know what you are getting in return. Before doing any performance tuning, it is important to measure the current performance and benchmark the improvements. It will also boost your ego

Jmeter from Apache is probably one of the best open source tools to load the servers and measure response times. There are other tools for performance testing like YSlow which I will not cover here since it cannot do load testing. It will be a separate post. But do yourself a favor and at least install YSlow in Firefox. It’s a drop dead simple tool which gives you the most crucial information. Of course you could also pay me top $$ for the same information.

1. JMeter: If you are anything like most geeks, you have already clicked on the link above and are well on your way to discover the tool yourself. If you are still with me, which I hope, the video below will show you how to set up a simple test. Here is the written summary though:

1. First make sure you have java installed. (not covered in this blog)

2. Download Jmeter zip and unzip it in a directory you would like to install it in - download JMeter.

3. Run jmeter.bat (windows) or jmeter in (X)nix install.

4. Once you see the Jmeter user interface, set up a proxy on your local machine, to route all internet traffic through it (figure 1)

Figure 1 - Proxy setup

5. Create a new thread group, which will have your tests in it (figure 2)

Figure 2 - Threadgroup

5.1 You can define the number of users(threads), iterations(loops), and time between each new thread. This will represent the load.

5.2 Add the HTTP Cookie manager and HTTP cache manager (figure 3). Without Cookie manager JMeter cannot handle the cookies from the server and the tests will not run.

Figure 3 - Add managers

6. Add a new HTTP Request from the sampler (figure 4). Add name of the server in Server Name or IP. This is the web server again which the test will run.

Figure 4 - Add http request

7. Also add listeners which will record the results of the test (figure 5)

Figure 5 - dd listeners

8. Run your test from Run->Start

8. Enjoy the results.

Late last week I met Jim Kaskade of SIOS at a Seattle-area Starbucks for a meeting and a product demo. With the very cool (and appropriate) title "Chief of Cloud", Jim was the right person to demonstrate his company's new cloud-powered high availability and disaster recovery solution.

Jim's Mac laptop was running Centos. He used Xen and Red Hat's Virtual Machine Manager to host a couple of virtual machines representing the web, application, and database tiers of a SugarCRM installation. Each of the guest operating systems was running a copy of the new SIOS CloudStation product. Each copy of CloudStation was configured (using a web-based GUI) to replicate the state of the virtual machine to an Amazon EC2 instance running in a user-selected Region.

Once everything was up and running, Jim showed me how he could selectively kill the local virtual machines while keeping the application running. The demo was designed to feature a very short RPO (Recovery Point Objective) so that changes made locally just seconds before the database was killed were available from the cloud-based virtual mirror. Jim walked me through a number of different failure and recovery scenarios.

It was quite impressive and makes a great demo of the cloud-based DR (Disaster Recovery) and HA (High Availability) that I've been telling my audiences about for the last couple of years. Once configured, CloudStation can fail over from local processing to the cloud, from one cloud region to another, or even from one cloud provider to another. It can also be used as a migration tool, or what is sometimes calls P2V (Physical to Virtual) or P2C (Physical to Cloud).

Read more in the Solution Brief (PDF) or sign up for the March 24th webinar.

-- Jeff;

There is a possibility to save JMeter test results to xml or csv file for further work (e.g. generating reports, importing to a database, etc.). It can be easy done by adding Simple Data Writer listener to a Thread Group and specifiing the filename. Of course you can specify the filename as JMeter variable or property.


You can choose the format of results file (xml or csv) and the fields to be saved clicking on Configure button and checking necessary options. As for me I prefer xml format and default options except Save URL which is useful when I need to know exact URLs of HTTP requests.

We've added two new features to Amazon SimpleDB to make it even easier for you to implement several different data storage and retrieval scenarios.

The first new feature allows you to do a consistent read. Up until now, SimpleDB implemented eventually consistent reads. You now have the option to choose the type of read which best meets the needs of each part of your application. Before I dive into the specifics, here's a quick guide to the two types of reads:

• Eventually consistent reads offer the lowest read latency and the highest read throughput. However, the reads can return data that has been made obsolete or overwritten by a very recent write. In general, the time window where this can occur is smaller than one second.
• Consistent reads always return the most recent data, with the potential for slightly higher read latency and a small reduction in read throughput.

SimpleDB's Select and GetAttributes functions now accept an optional ConsistentRead flag. This flag has a default value of false, so existing applications will continue to use eventually consistent reads. If the flag is set to true, SimpleDB will return a consistent read.

The second new feature allows you to issue SimpleDB PutAttributes and DeleteAttributes operations on a conditional basis. In other words, you can tell SimpleDB to perform the indicated operation if and only if a given single-valued attribute has the value specified in the PutAttributes or Delete call. You can easily implement counters (the value itself is effectively the version number), delete accounts only if the current balance is zero, and insert an item only if it does not exist.

You can combine consistent reads and conditional operations to implement a form of optimistic concurrency control, or OCC. Let's say that your form-based web application allows users to update their accounts and that you built it with SimpleDB. If you store a version number with each SimpleDB item, you can keep the data consistent even if several users or applications attempt to edit the same record at the same time. You would retrieve the data for display using an eventually consistent read, and then display the form so that the user can edit it. You would also read the associated version number from SimpleDB and associate it with the form or the editing session. When the user modifies and then attempts to save the data, you would use a conditional PutAttributes call to make sure that the data hadn't been changed. If the update fails, you'd need to invoke some application-specific action to resolve the conflict before proceeding. OCC often obviates the need for long-term availability-impacting locks, transactions, timeouts, and other complex and sometimes messy programming constructs.

You can read more about consistency models in Werner's newest blog post (he may be in mid-flight as I write this, but I am confident that his post will show up soon).

-- Jeff;

Amazon SimpleDB – the highly available, scalable, and flexible non-relational data store – now enables Consistent Reads and Conditional Puts & Deletes. Consistent Reads provide the ability to specify the consistency characteristic you require for each read call within your application, with an eventually consistent read optimized for lowest latency and highest throughput and a consistent read that provides “read my last write” capability. Conditional Puts & Deletes are mechanisms for inserting, updating, or deleting one or more attributes of an item with full transactional semantics using a single PutAttributes or Delete Attributes API call.

Previously, Amazon SimpleDB provided only eventually consistent reads. By offering a new Consistent Reads option and transactional semantics, AWS aims to facilitate development of additional classes of applications with Amazon SimpleDB. Visit the Amazon SimpleDB page to learn more.

It seems to me that every time we release a new service or feature, our customers come up with ideas and requests for at least two more! As they begin to "think cloud" and to get their minds around all that they can do with AWS, their imaginations start to run wild and they aren't shy about sharing their requests with us. We do our best to listen and to adjust our plans accordingly.

This has definitely been the case with EC2's Reserved Instances. The Reserved Instances allow you to make a one-time payment to reserve an instance of a particular type for a period of one or three years. Once reserved, hourly usage for that instance is billed at a price that is significantly reduced from the On-Demand price for the same instance type. As soon as we released Reserved Instances with Linux and OpenSolaris, our users started asking for us to provide Reserved Instances with Microsoft Windows. Later, when we released Amazon RDS, they asked for RDS Reserved Instances (we've already committed to providing this feature)!

I'm happy to inform you that we are now supporting Reserved Instances with Windows. Purchases can be made using the AWS Management Console, ElasticFox, the EC2 Command Line (API) tools, or the EC2 APIs.

As always, we will automatically optimize your pricing when we compute your AWS bill. We'll charge you the lower Reserved Instance rate where applicable, to make sure that you always pay the lowest amount.

You can also estimate your monthly and one-time costs using the AWS Simple Monthly Calculator.

-- Jeff;

PS - If you have any wild and crazy AWS ideas of your own, feel free to post them in the appropriate AWS forum.

We've added a new EC2 instance type to our repertoire. It is called the High Memory Extra Large (m2.xlarge) and has the following specs:

• 17.1 GB of RAM.
• 420 GB of local storage.
• 64-bit platform.
• 6.5 ECU (EC2 Compute Units), 2 virtual cores each with 3.25 ECU.

You can leverage this new instance type as a lower cost option if you are already using Standard Extra Large instances. The new instance type is available now in all of the EC2 Regions (US-East, US-West, and EU).

-- Jeff;

We are excited to announce the immediate availability of Reserved Instances with Windows, which work like Reserved Instances with Linux/UNIX. You simply pay a low, upfront fee for each Reserved Instances with Windows you want to reserve and in turn receive a significant discount on the hourly price during a 1 or 3 year period. After the one-time payment, that Windows-based instance is reserved for you, and you have no further obligation; you may choose to run that instance for the discounted usage rate for the duration of your term, or when you do not use the instance, you will not pay usage charges on it. You can find more detailed information on Reserved Instances at http://aws.amazon.com/ec2/reserved-instances/.

We are pleased to announce today the availability of Extra Large High Memory instances, for use with all operating systems. The new instance type is specifically designed for memory-intensive workloads such as databases, memory caching, and rendering. You can also leverage this new instance type as a lower cost option if you are already using Standard Extra Large instances. The High Memory Extra Large instance type is defined as:

• Extra Large: 17.1 GB memory, 6.5 ECU (2 virtual cores with 3.25 ECU each), 420 GB storage, 64-bit platform

Please visit the Amazon EC2 detail page for more information on the Extra Large High Memory instance type.

I am very excited about DrupalCon 2010 at SFO. We have proposed the following sessions:

Spreadsheet integration with Drupal – In depth

Business Essentials: Print, Excel and Calendar integration with Drupal

Hands on session: DDBlock, Carousel, Popups

Please vote for the sessions so we can present at DrupalCon. You have to first create a login at drupal.org to be able to vote.

Hi there, this is Simone Brunozzi, AWS Technology Evangelist for Europe and APAC. While Jeff Barr is in Japan, I'll steal his keyboard to tell you a couple of nice Amazon Web Services success stories.

The first one is from our friends at ZapLive.tv, a German company that allows you to launch your own web TV.

On Friday, Jan 21st, 2010, something unprecedented happened: at 11:38 AM CST, Lily the Black Bear, in Minnesota, gave birth LIVE on the internet!  Thousands of people rushed to WildEarth.TV to watch this wonderful event.

The broadcast on WildEarth.TV was produced by Doug Hajicek of Whitewolf Entertainment in association with Dr Lynn Rogers of the North American Bear Center. Peaking at a maximum of about 27,000 concurrent viewers, it was streamed across the zaplive.tv dynamic system.

Luckily for WildEarth, they were using Zaplive's highly scalable infrastructure for load balancing live streams from different locations. The infrastructure is based on Wowza Media Server Pro and Amazon EC2. An Origin Server measures the repeaters and distribute the viewers to different EC2 repeaters, which deliver the stream around the world. Dependent on the load, additional EC2 instances are launched. This combination allows a dynamically auto scaled system to handle hundreds of thousands of unique visitors in a few hours. At their peak, they had about 35 Large EC2 instances running.

As you can see, thanks to the flexibility of Amazon EC2, they were able to quickly launch additional servers when needed, and run them as long as needed, being able to scale down when some of these servers were no longer necessary.

For the second success story we move from Minnesota to green Switzerland.

The Swiss Geoportal, www.geo.admin.ch, is now online and includes a great map viewer, with 35 layers of various Swiss administrations such as Dufour Map, Transport Network, Ground statistic, with more to come in 2010 including English support and up to 50 more datasets. 

Of course, Amazon Web Services (EC2 and S3) were used to implement this web 2.0 mapping application. It allows fast performance and very high load.

Dr. David Oesch, the project coordinator, says:

"Thanks again for providing such a great service...without AWS we would not have been able to achieve such a performance at such costs!"

Hope you liked these two stories. Goodbye from warm Mumbai!
Simone Brunozzi (@simon on Twitter)
Technology Evangelist for AWS, Europe and APAC.

I am pleased to welcome Hitachi Systems to the AWS Solution Providers Program!

They have been providing a broad range of IT consulting, system integration, and professional services to Japanese enterprises in the financial, manufacturing, retail, and telecommunications space for over 30 years. They also provide these services to the government and public sectors. With AWS, they will offer a "Hybrid SI" model that is comprehensive and borderless, spanning any form of IT including on-premise and cloud.

Read more in their Solution Provider page.

As you can read in the new Hitachi Systems case study, they are already creating powerful AWS-based solutions. They have created a unique Mobile Broadcast Solution to deliver text and video service to cell phones capable of handling 1seg, a terrestrial broadcasting system that's popular in Japan. Using 1seg, Hitachi is able to broadcast localized and highly specialized content to individual districts of Tokyo. By its nature, the service is data intensive and generates 8 to 10 TB of data per broadcast area. Instead of purchasing an expensive storage array, Hitachi Systems used Amazon S3. The first installation of this system broadcasts to Tokyo's Shibuya district (pictured at right in a photo that I took last night).

Hitachi estimates that their decision to use S3 instead of a storage array resulted in a cost saviing of $70,000. More information about this solution (in Japanese) can be found here.

-- Jeff;

If you want JMeter to handle cookies automatically you need to add HTTP Cookie Manager to Thread Group.

Be sure that Cookie Policy option is set to "compatibility" value, it will work in most cases. As for "Clear cookies each iteration?" checkbox I always check it and never had the situation when I was need it to be unchecked.

On February 23, 2010, AWS, RightScale, and LTech will present a new webinar!

Titled Leverage the Cloud for High-Traffic, High-Profile Web Marketing Events, the webinar will show you how to protect your brand and your marketing spend by ensuring that your website or promotional site is performing optimally.

You'll see how a cloud-based deployment will give you the infrastructure needed to handle the massive and often unpredictable traffic brought on by a marketing campaign. You'll learn about AWS, RightScale's management offering, and LTech's new Brandscale service.

Registration is free and you can sign up now.

We are looking for some additional Developer Support Engineers to join our Seattle-based team.We also have similar positions in our Dublin, Ireland facility.

I sat down with a couple of the team members to learn more about what they doing during a typical day. Here's a rough time line that we put together for someone working the day shift:

Get to the office and grab a cup of coffee (and maybe a donut too), check out the view of Mount Rainier from my window, and say hello to Mocha (a co-worker’s dog).

Sync up with the overnight crew to take over a high-priority support case: a customer is getting big (fast) and we’re keeping an eye on things to make sure they have everything that they need.

A new customer has questions about getting started with Amazon EC2, so I walk him through some of the basics, highlight a number of best practices, and then let him loose to launch his first instance.

I’ve got a few minutes so I log into one my test instances and update a script that posts some diagnostic data to an SQS queue.

There is an update to a support case that includes a tcpdump of a failed request to one of our APIs. Turns out the customer’s firewall was stripping some headers that we were expecting.

An email just arrived from a customer that we talked with last month. They have a few more questions, this time related to scaling their site. Taking into account what we already know about their setup, we provide them with guidance on how they can use some of the features of Amazon EC2 (Elastic Load Balancing, Auto Scaling, and Monitoring).

The sandwich I brought for lunch isn’t cutting it so a trip to the cafeteria is due. I’m planning to sit in on a talk at lunch by a principal engineer for one of our services. He is covering some of the implementation details about the Amazon Virtual Private Cloud service.

Back at my desk to help a customer migrate his Amazon EC2 instances and EBS volumes from the Northern Virginia region to the new Northern California region.

Another new customer just opened a support case and wants to know how to use the new CloudFront streaming feature, so I walk her through setting up a distribution and streaming content from her site.

The next shift is starting to show up now and there is a training session for a new service that is launching next month. These are great opportunities to pick the brains of the guys who build these services.

If you can handle a day (or a night) like this, check out the job description and apply today!

-- Jeff;

Caching is one of the common ways of improving the performance of a website. Caching aims to reduce the number of trips made to the database by storing the snapshot of the results in a location (like database or file structure or memory) from where it can be retrieved faster the next time. Caching works best for information that do not change often and/or frequently consumed and/or expensive to process. Periodic maintenance need be done on the cached information so that the website users only get the latest information and not ’stale’ information. During development one of the most common frustrations is not seeing the latest changes that have been made, because the webpage information is retrieved from the cache that has old information.

Drupal’s File-based Cache:

Drupal provides a way to consolidate all the css and javascript files into fewer files. This is very useful for pages where many javascript and css files are used to render. Instead of having many roundtrips downloading each of those files, a consolidated file would reduce the roundtrips and can decrease the page loading time significantly.

The setting for optimizing the css and javascript files can be found at Administer->Site Configuration->Performance. If “Optimize CSS files” is enabled the css files are compressed by removing the whitespace and line breaks and stored in the “css” directory within the “files” directory as set on the file system settings page. If “Optimize JavaScript files” is enabled the javascript files (without compression) are stored in the “js” directory within the “files” directory. The “Download method” in the file system settings page has to be set as “Public” for these options to be available.

It is better to turn on these options only in the production environment as it can interfere with the theme and module development. Also if you are running a load-balancer along with two or more servers please make sure that the cached javascript and css files are available and identical on all the servers.

Drupal’s Database-based Cache:

Drupal comes with a nice cache mechanism that is used by the Drupal core. Drupal exposes this as an api called Cache api that can be used by developers to add caching to their own modules.

The Cache api by default stores the information to be cached in a table called ‘cache’. If every module uses the same table for caching, the table can grow exponentially leading to increase in overhead rather than reducing it. Hence it is better to have several cache tables to store your information. If needed module developers can add their own cache table, although it should be identical in structure to the default cache table. It is also a good practice to prepend cache_ to your own cache table.

Drupal core comes with seven cache tables.

1. cache: An “all purpose” table that can be used to store a few rows of cached data. Drupal uses this table to store the following:

Variable data: The table “variable” stores most of the administrative settings. These settings are added to a PHP array(), serialized and stored as a single row in the cache table. This way the settings can be retrieved quickly and avoids making multiple database queries. All variables that uses variable_get() and variable_set() are cached in the cache table.

Theme registry data: Each of the themes registries are cached in the cache table.

Schema data: Information about the structure of all the tables in the database is cached in the cache table.

2. cache_block:

Content generated by the blocks are cached in this table. This reduces Drupal from querying the database repeatedly to get the block contents. The block developer can choose based on the content displayed in the block, whether the block can be cached or not. If the developer decides to cache the block content, he/she has four ways of caching the block content.

• • 1. Cache block content separately for each available role combination
    2. Cache block content separately for each user
    3. Cache block content separately for each page
    4. Cache block content once for all the users

Drupal, in addition to above caches block content separately for each theme and for each language supported.

Block caching can be changed at Administer->Site Configuration->Performance. Also if any of the content access modules that restrict access to different kinds of content are enabled, then block caching is automatically disabled.

3. cache_filter:

Filters are used to filter node content to make it more secure and consumable. This is a very expensive operation to perform every time a node is viewed. Filters are applied to the node content when they are saved or created, and the results are cached in the cache_filter table.

4. cache_form:

The form API uses this table to cache the generated form. This saves Drupal from regenerating the unchanged forms.

5. cache_menu:

The menu system caches all created menus along with its hierarchies and path information in this table. This saves Drupal from having to regenerate the menu information for each page load. Menus are cached separately for each user and for each language.

6. cache_update:

Information about all the installed modules and themes are cached in this table.

7. cache_page:

The page caching is one of the important optimization that can truly improve the performance of a heavily used website. Entire page views are cached in the cache_page table. Drupal’s full page caching only caches anonymous user pages. It does not cache authenticated user pages because they are often customized for the user and so they are less effective. This saves Drupal from making expensive calls to generate the page repeatedly, instead the cached page content can be retrieved in a single query. Administer->Site Configuration->Performance has several settings that affect the page caching.

Caching mode:

Disabled: This disables the page caching although other types of caching would still continue.

Normal: This mode offers a substantial improvement over the “Disabled” mode. Drupal’s bootstrapping is designed such a way that, only the minimum required amount of code and queries are executed to render a page from the cache. Even with this minimum code, the database system is initialized, access and sessions are initialized, module system is initialized and hooks – hook_boot and hook_exit are called on the modules that implemented them. Only then the cached page is rendered.

Aggressive: This mode skips the initialization of the module system and so hook functions are never called. This shaves valuable time each time an anonymous user requests a page. But this means that the modules that have implemented the hooks may not work properly. Drupal warns (see image) by listing all the modules that may not function properly if the Aggressive mode is enabled.

Minimum cache lifetime:

This sets the minimum lifetime of a cached content. If a user changes content, other users would have wait until this expires, to see the changed content. If it is set to “none”, then there is no wait, and all users can see the latest content immediately.

Page compression:

If enabled the page contents would be compressed before caching it. This saves bandwidth and improves the download times. However if the webserver already performs the compression, then this should be disabled. In Apache server you can use module mod_deflate to turn on the compression. IMHO it is better to use this functionality (may be enhance it with modules like css_gzip or javascript_aggregator) rather than mod_deflate because the latter do not have any caching.

Note: Even if the Page cache and/or Block cache are disabled, other types of caching like menus, filters, etc. would still continue to happen and they cannot be disabled.

Clear cached data:

In the performance page at Administer->Site Configuration->Performance, you can use the “Clear cached data” to clear ALL of the caches in the system including the css and javascript optimizations. If you have your own cache table, you can use the hook_flush_caches() to clear the cache when the “Clear cached data” is executed.

Pluggable Cache:

Drupal provides a way to plug in a customized caching solution such as memory or file-based or hybrid (memory and database for fail-safe) caching.

There are 2 levels of plugging:

1. Solutions that uses the Drupal cache API, but stores information in a customized manner (like memory) instead of the Drupal ‘cache*’ tables.

Example: memcache module

2. Solutions that provide their own cache API implementation, together with customized storage of information. Essentially complete Drupal cache system is bypassed. This is called the ‘fastpath’ mode (or is it page_fast_cache?). When a cached page is rendered using this technique most of the Drupal bootstrap technique are skipped, and hence the Drupal statistics would not be updated. So for statistics like page views, Drupal statistics may not be accurate.

Example: cacherouter module

Drupal’s modules:

There are many contributed drupal modules that extend the Drupal cache, or provides integration with an external caching solution. Some of them even work for some parts of the authenticated user pages. It is certainly possible to combine many contributed modules to get better performance.

Here are some of them with a very brief summary (in no particular order):

Memcache: This module lets you use a memcache server to do the caching. The nice thing is it provides 2 types of caching, one memory only, and another memory or database. The latter approach certainly has lesser performance improvement than the first, but it gives a failsafe mechanism where if the memcache is not available, it would use the database for caching.

Authcache: This module can cache authenticated user pages if the pages are same for a particular user role. This module can be combined with Memcache or Cacherouter to have a customized caching.

Advcache: This module extends the caching to areas that the Drupal core does not cover. The main advantage is for authenticated non admin users that have a single role. This module can be combined with Memcache or Cacherouter to have a customized caching.

Cacheexclude: You can use this to exclude certain anonymous user pages from being cached.

Cacherouter: This module allows users to have different caching technology for different cache tables. It has native support to many caching technologies like APC, Memcache, XCache, even database and file system.

Boost: Another external caching mechanism for mostly anonymous users.

Cache: Similar to Cacherouter, provide mechanism to use different caching technologies.
References:

Pro Drupal Development, Second Edition by John K. VanDyk

Caching Modules comparison: http://groups.drupal.org/node/21897

Drupal caching, speed and performance: http://drupal.org/node/326504

Drupal guide to caching: http://lists.drupal.org/pipermail/documentation/2008-March/005949.html

http://httpd.apache.org/docs/2.0/mod/mod_deflate.html

JMeter's variables and properties can do your Test Plan very flexible and prevent you from frequent modifications of the Test Plan. Just add the most frequently used options as variables on the Test Plan page and specify their values using "__P" function as on the screenshot below.

As you can see Test Plan can be executed with default values and at the same time if you need to modify some options you can pass them to Test Plan through the command line without the modification of the Test Plan.

As more and more large organizations make use of AWS in important and mission-critical ways, they have started to ask us for more flexible ways to manage their account. Three of the top requests have been for consolidated billing across multiple AWS accounts, volume pricing across more than one AWS account, and a way to track the cost of a project across an organization. Since we love to listen (and to respond) to our customers, I'm happy to announce that all of these features are available now!

Our new Consolidated Billing feature lets you designate one AWS account as a paying account and a set of other accounts as linked accounts to form a simple one-level hierarchy. The AWS usage within the linked accounts is rolled up into the paying account for volume pricing and billing purposes, so there's just one AWS bill per month.

The account relationship is solely for accounting purposes and normal AWS permissions still apply. The paying account does not have special access to the resources (Amazon EC2 instances, Amazon S3 buckets, and so forth) owned by the linked accounts. Each linked account has its own set of AWS credentials.

You can use this new feature in a number of ways. You can set up AWS accounts for projects, departments, dev/test/staging/production, or even by employee. You can add up to twenty linked accounts to a paying account. Here's the process:

1. Set up a Consolidated Billing account.
2. Use the Consolidated Billing page to send a request to the owner of each of the accounts to be linked.
3. AWS sends an email to the owner of each account.
4. The owner clicks a link in the email, logs in to the AWS portal, and accepts or denies the request.
5. The paying account becomes responsible for the charges incurred by the linked account as soon as the owner of the linked account has accepted the request.
6. The paying account can view an AWS cost report for each linked account.
7. At a later time the owner of the linked account or of the paying account can remove the linked account from the consolidated bill.

Here's a visual representation:

Once two or more accounts have been consolidated into a single paying account, the AWS usage across all of the accounts will be used to determine the appropriate pricing tiers. Let's say that two accounts owned by Stephen and Andy transfer 8 TB and 4 TB of data out of S3 in a given month. The resulting 12 TB of data transfer will be billed at $0.15 per GB for the first 10 TB and $0.11 per GB for the remaining 2 TB.

This should result in a cost savings for customers who previously divided up their AWS usage across a number of accounts. The cost benefits derived from the use of EC2 Reserved Instances also apply across all of the linked accounts. However, the availability of instances favors the  account that purchased the Reserved Instance. Support fees for AWS Premium Support are based on the individual account's portion of the consolidated bill. Taxes are computed separately for each linked account based on the address of the account, but charged to the paying account.

These features will do a lot to address the needs of our larger customers, but we know that we can do more to enhance the account management features of AWS. Later this year we plan to provide additional functionality to allow linking of accounts and user permissions.

-- Jeff;

Scalability can be a confusing topic, because it is usually not defined in easy terms. If I were to characterize scalable system,

1. The system should be able to accommodate increase in data
2. The system should be able to accommodate increase in usage
3. As the load increases on the system, the system still remains relatively accessible and maintainable.

It can be easy to confuse scalability with performance and these are two separate characteristics. A high performing system can quickly become non performing, if it cannot scale (the reverse is usually not true though).

As the load increases on a system, we still want it to keep responding with a good (low) response time. This usually means that the hardware is using more resources to serve the request. How the hardware is provisioned depends on the landscape architecture. We can choose to either scale the hardware vertically (scale up) or horizontally (scale out). These are very different approaches but in a nutshell:

Vertical Scaling (scale up): Scaling is achieved by adding more hardware resources to an existing physical machine. Example, would be allocating more memory, adding more memory, more hard disk, additional CPUs, etc. When a hardware resource starts to run at capacity a bigger box is added to the mix. This hardware upgrade can continue to happen till a limit is reached. Therefore there is a physical limit to vertical scaling.

Horizontal Scaling (Scale out): We also add hardware to scale horizontally, except, horizontal scaling is achieved by adding machines in parallel to an existing machine. We can buy mid range machines and keep adding a new one as each runs out of resource. But of course the scalability will rarely improve proportionally and the TCO will also increase. There is networking supplies and setups required for each new machine, in addition to rack space, etc.

Moreover, some of the resources might be very underutilized in horizontal scaling. For example, in a typical web application, the network I/O and memory might be bottlenecks. As we add more machines, we are also adding CPUs and hard disk, which are now underutilized.

As you can see, capacity planning for a server setup can become a thorny problem very soon and requires a systematic approach to design and scale the landscape, depending on the application.

Another way to understand the issue can be a look at Amdhal’s law. This model explains what should be performance improvement we can expect by adding resources in parallel.

P: proportion that is affected by the computation.

S: Speedup

A simple example: if an improvement can speed up 30% of the computation, P will be 0.3; if the improvement makes the portion affected twice as fast, S will be 2. The overall speedup will be:  1/(0.7 + 0.35) = 1/1.05 = 0.95