I was trying to make the same point throughout the post. EBS is just another tool helping to achieve the purpose but by no means be limited to it. However, rather than just having an instance store without any EBS coupled, having EBS with snapshots could give you more options in a failure.

In the two years I’ve been using AWS in large production environments, random instance termination was the least of our problems. Weird EBS issues can cause horrific outages. So solve both the redundancy and durability problem at the same time: replicate your data among hosts in different Availability Zones and Regions from the get-go.

The application I was speaking of isn’t scaled to the magnitude of your case since it’s still not yet open to the public but all your suggestions are sound and useful, so updated my post mentioning your comment.

Loggly got hit far worse than you did. We’ve rebooted servers before and they usually come back up. This time over 95% didn’t, and it took us all the way down and out for the count. We were down for over 24 hours trying to get our search cluster back online and taking data again. Our system is neither simple nor easy to bootstrap, and even though we have scripts that start and stop instances of our stack at will, for development or testing, bringing a large, live production cluster back up from zero took us WAY longer than we expected. We should have planned for it. We didn’t. We didn’t expect all our machines to go away. What we expected was SOME of them to go away, or SOME interruption of service (because it’s the cloud!), but we never expected all of the boxes to be kicked by humans. All at once.

We should have planned better.

Backing up our database and all customer’s logs (up until we went down) ‘saved’ us, but we still suffered data loss during the outage, dropping data that customers were sending in, and we were DOWN and unusable, which is the greatest sin of all.

Given our experiences, I would add a few more points to your list:

6. Test a full deployment of your current architecture, including size/scale, while still running another instance of it – this ensures you have the resources to start it if you need another one (we did not, and found out post-disaster we could only launch 50 total instance). If required and/or possible, test taking data from the production system and teeing it into the new deployment to see if it works properly.
7. Make sure your deployment management scripts (we use Puppet) work anywhere. We can launch instances of Loggly on VMs, bare metal, Rackspace, etc. Test alternate deployments on other AWS regions AND other providers. You don’t know all the bad things that could be. AWS could go completely tits up, and you’d need to fail to … somewhere.
8. Regarding point 7, make sure you aren’t depended architecturally on AWS services. Where possible, adopt alternate technologies that work across multiple infrastructures. For example, OpenStack supports a S3 like storage system. Make sure your stack works with it.
9. Don’t create technologies in your stack that are hard to scale and/or replicate. We’ve done that at Loggly because we thought we needed a single search cluster. We should have sharded customers/inputs/whatever across zones and regions. That way if part of it goes down, only a few customers are affected.
10. If you run in the ‘cloud’ realize you are offloading the responsibility for running infrastructure to someone else. We expect AWS to be reliable, but yet are limited in our expectations because of limits in the technology and costs that they must manage. Running your own boxes may place more responsibility on you for managing them, but will also allow you to better manage the expectations of what can go wrong.

But the server still failed sometimes. One computer can *never* be 100% reliable. The UPS isn’t reliable. The datacenter isn’t reliable. The network isn’t reliable. People aren’t reliable.

Focus on “MTTR” (Mean Time To Recovery) instead. What are you going to do _when_ your server fails? I’ve seen a doctor’s office down for 3 days because it took 8 hours to restore a backup (and they restored the wrong backup twice.)

Here’s a better plan: Buy several $2K servers, and use software to “RAID” them together. When failure happens, recovery should be seamless and automatic. There’s no reason you should be paged in the middle of the night just because some hardware died. Advanced users should be prepared for the whole datacenter/region going down.

Instead of avoiding failure, embrace failure. That’s the cloud way.