Resources for Amazon Web Services (AWS) migrations from EC2-Classic to EC2-VPC

AWS EC2 (Elastic Compute Cloud) is one of the best known cloud services delivered by AWS. It is a service that allows customers to purchase resizable cloud hosting resources. It is – in my opinion – the best current implementation of IaaS (Infrastructure as a Service) from all cloud service providers. It truly delivers on the promise that within minutes you could spin up a new server instance and proceed with meaningful work without having to wait for days/weeks for a vendor (or IT department) to purchase and deliver properly configured hardware systems.

The original implementation of EC2 (which over time became known as EC2-Classic) had one interesting shortcoming that only became more and more apparent as customers continued to build solutions increasing in complexity: EC2 instances for ALL customers in a given AWS geographic region share private IP addresses in the 10.x.x.x space (technically there are some ranges from that Class A network that are not being used but that’s not relevant for this discussion). For example you could have in your account a web server with the IP address of 10.150.22.220 and another web server with the IP address of 10.20.100.70 (and you basically had no idea or control over who would be using the private IP address of 10.20.100.71).

AWS did provide the technology of security groups (basically software firewalls that wrap around instances) to allow customers to group together EC2 instances of similar functions (and not allow intruders access) but as customers were building more and more complex solutions it was getting harder and harder to manage the instances that one owned in AWS.

This model where one’s servers were spread all over the 10.x.x.x range was not the way that networking professionals were used to run networks in their own data centers.

In 2009 AWS introduced an improvement to the original EC2 approach – VPC (Virtual Private Cloud). In a VPC customers now had the ability to use their own private IP range, divide the network as they saw fit and pretty much come back to networking models that they were used to. There are many, many more features to AWS VPC that make it a clear winner over EC2-Classic but those are not the main focus of this article.

For a while then we had the two technologies side by side – EC2-Classic and EC2-VPC. Customers were able to create EC2 instances using either model. It was becoming clear though that EC2-VPC was the superior technology and AWS proved that on 2013-12-04 because after that date all new AWS accounts only supported EC2-VPC. In new accounts created after that date AWS automatically creates a default VPC and places all EC2 instances in that context.

Many of the older AWS customers were slowly faced with a dilemma: what were they supposed to do with their aging EC2-Classic instances? AWS is constantly innovating, adding new instances types and new features but many of those are now only available on the EC2-VPC side. If customers want to take advantage of the latest AWS features then they need to consider migration paths from EC2-Classic instances to EC2-VPC.

So what are the migration options available?

We need to understand from the beginning that there is no ‘easy’ button we can push that will just magically migrate a resource from EC2-Classic to EC2-VPC. The process that one has to follow is this: shutdown the EC2-Classic resource (if needed), back it up (either via an AMI for EC2 or a RDS snapshot for example), and then restore it to a new instance in EC2-VPC. If we think a little about the process it makes total sense – given the differences between EC2-Classic and EC2-VPC (in networking, instance types and other features) it’s most likely that EC2-VPC resources in a data center representing a particular availability zone would exist separately from hardware resources running EC2-Classic. The backup/restore approach is the only mechanism that allows AWS to move something from the EC2-Classic area to EC2-VPC.

With this in mind we basically have two approaches to perform this migration:

Option 1: Do it all at once – shutdown everything in EC2-Classic, create backups and restore the entire environment to EC2-VPC. This is the simplest and most direct approach. It’s great if your application is not that complex and most importantly if you can afford the downtime that this migration requires.

Option 2: Perform a partial migration – the only reasonable path when downtime is not really acceptable. In this approach we basically have to be strategic about the order in which we move resources so that we keep the application up and running as much as possible. It is the most complex approach but probably the one that most AWS EC2-Classic customers would need to take so we’ll focus on this for the remainder of the article.

When pursuing option 2 we need to be aware of one more piece of bad news that can complicate the process: security groups can’t be shared or visible between EC2-Classic and EC2-VPC. Why is that important? In a well architected solution security groups are critical for establishing allowed contact between the different layers of the application. Web servers for example would be started in a security group called webservers and the database security group allows inbound access on the correct port from any resource found in the webservers group – it does not care how many web servers exist there or what their individual IP addresses are. If security groups can’t be used to establish access rules between EC2-Classic and EC2-VPC then the only alternative is to create rules directly using individual public IP addresses for resources and that can get very, very messy. I won’t even go over that path because thankfully AWS recently gave us something to help manage all this.

In January 2015 AWS introduced a technology called ClassicLink – http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/vpc-classiclink.html. ClassicLink, once enabled on a particular VPC, allows us to take individual EC2-Classic resources and make them appear as members of a security group in the VPC. From that point on we can create access rules between that security group and any other security groups in the VPC. EC2-Classic instances linked this way via ClassicLink can now directly access EC2-VPC resources via their private IP address – a much better way to manage security.

I’d like to mention here something that can easily be missed in the ClassicLink documentation: much of what’s described there talks about how to enable a particular EC2-Classic resource to use ClassicLink. What about instances that are controlled automatically by auto scaling groups in EC2-Classic? What if we need them to automatically use ClassicLink so that they would belong in a particular security group in the VPC? Fortunately there is an answer for that as well as AWS allows us to modify the launch configuration in an auto scaling group such that each new instance started that way can be linked via ClassicLink according to the settings we need:
http://docs.aws.amazon.com/AutoScaling/latest/DeveloperGuide/autoscalingsubnets.html#as-ClassicLink.

ClassicLink is indeed the key to the partial migration approach from option 2. Here are a few more resources about it that were recently made available by AWS:

AWS video from May 2015 called “Deep Dive: Amazon Virtual Private Cloud” (the link goes directly to the point where migrations with ClassicLink are described):
https://www.youtube.com/watch?v=B8vnhRJDujw&feature=youtu.be&t=2553

The set of slides that goes with the video presentation above:
http://www.slideshare.net/AmazonWebServices/deep-dive-amazon-virtual-private-cloud-50093547

If you look at the video & slides you’ll see that AWS suggests the following approach for a partial migration:

  • first migrate any resources that are managed by AWS (such as RDS)
  • next migrate your ELBs
  • finally migrate individual instances

Why does this make sense? Using ClassicLink individual instances from EC2-Classic can connect to an AWS managed resource that now resides in the VPC (such as RDS) but if you first migrate the instances themselves then ClassicLink won’t help those VPC resources connect back to RDS in EC2-Classic – ClassicLink connections really only go one-way from EC2-Classic to EC2-VPC. You’d encounter the same problems that customers were faced with before ClassicLink was introduced – the need to work with public IP addresses without the benefit of security groups.

Here are some more AWS links that are relevant:

Migrating from a Windows Instance in EC2-Classic to a Windows Instance in a VPC
http://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/vpc-migrate.html

Migrating from a Linux Instance in EC2-Classic to a Linux Instance in a VPC
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/vpc-migrate.html

Migrating an Elastic IP Address from EC2-Classic to EC2-VPC (only possible in Amazon EC2 Query API, an AWS SDK, or the AWS CLI – not in the AWS console)
http://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/elastic-ip-addresses-eip.html#migrating-eip

There you have it. I’ve mentioned here a variety of resources to help you with the migration from EC2-Classic to EC2-VPC. It’s certainly not a simple operation but if you’ll continue to be an AWS customer then EC2-VPC is the platform where you’ll want to continue building your applications.