With the endless flood of new products, features and changes from AWS and its surrounding ecosystem, it can be easy to miss an update. Our monthly round up highlights major AWS news, announcements, product updates and behind the scenes changes we think are most relevant.

May welcomed a number of features announced last year out of preview. There were also important announcements for EC2, S3 and EKS.

Amazon Managed Kafka (MSK) Goes GA

Amazon has been working on a preview of their managed Kafka service for the last 6 months, which finally went GA in May. This is pretty exciting, as it takes a lot of the pain out of managing a Kafka cluster. In particular, you don’t need to manage Zookeeper or the relationship between Kafka and Zookeeper. This doesn’t mean you won’t have to deeply understand how Kafka works in order to run it at scale, but it does reduce the toil quite a bit.

The pricing is per hour, and data transfer costs are minimal–you pay for what goes in and out of Kafka but not for any of the traffic that is generated within the cluster, including between Kafka and Zookeeper. Further, your Zookeeper nodes are free. MSK runs inside an Amazon-controlled VPC and attaches to your VPC through ENIs, which are free. The hourly cost is a bit steep, though–a little more than double the comparable EC2 instance price. For example, an m5.large node will set you back $0.21/hr compared to $0.096/hour for an EC2 instance. Reservations are also not possible (yet).

In addition to fully supporting Kafka 1.1.0 and 2.1.0, AWS adds the following useful features:

• Full TLS for all node-to-node (including consumer/producer) communication.
• Mutual TLS (must use AWS ACM Private CA)
• Storage migrations/scale-up
• Multi-AZ redundancy
• CloudTrail integration
• Cluster-wide default topic/broker settings

Unfortunately, you’re going to need to wait for CloudWatch Logs.

App Mesh now GA

Another service long in preview, App Mesh is a managed (and forked) implementation of Envoy, an extremely powerful and popular service mesh. Service meshes externalize discovery, observability and routing from your micro-services applications, freeing the services themselves from having to solve such concerns within code. This allows common capabilities to be rolled out efficiently across a wide range of services.

App Mesh takes away much of the pain of getting a service mesh up and running, mostly by making basic decisions for you. Key features:

• Seamless support for AWS Fargate, Amazon ECS, Amazon EKS, Amazon EC2, meaning all four can be used together interchangeably.
• CloudWatch and X-Ray integration for visibility into intra-service communication, errors, etc.
• It’s free. Obviously, AWS wants you using more and more containerized services, so there’s no additional charge for any of the resources that App Mesh will consume beyond the additional containers that run inside your nodes.

AWS Clarifies S3 Path vs Virtual Host Changes

AWS recently announced they were phasing out path-based access to S3 bucket objects. This created much consternation on the Internet. It was a big deal announced in a small way, and it also has implications for human rights groups throughout the world who rely on the fact that path-based access is virtually impossible to block and monitor. But, it also makes a lot of sense. There are several trillion objects in S3 today. Having all requests funnel through a single URL creates all sorts of complexity and puts the entire system under undue stress.

Despite the valid reasons for this change, it also puts undue pressure on developers to change code, some of which might be buried and forgotten about. Though it is possible to use bucket access logs to determine how objects are being retrieved, few people take the time to set that up for their buckets. That just adds more work just to figure out how much more work you have to do.

Fortunately, AWS has listened to the feedback and struck a perfect balance between the need to move forward and the need to protect their customers. This blog post from Jeff Barr announces that change while also acknowledging that the communications could have been a little better in this area. It’s always nice when a company can acknowledge and move on. The new plan is for path-based access to remain functional indefinitely for any buckets created before September 30, 2020.

Here’s what you need to know to prepare for this change:

• Global and regional endpoints for buckets should be phased out. That means you should be using this:

  https://yourbucket.s3.amazonaws.com/README.md

  and not this:

  https://s3.amazonaws.com/yourbucket/README.md
  https://s3-us-east-1.amazonaws.com/yourbucket/README.md

• You should be mindful of certain character names in your bucket name that may be difficult to serve this way, such as periods.
• This change is coupled to an overall phase out of the V1 and V2 S3 APIs. If you’re using old client access libraries, update. In most cases, modern libraries will use the V3 API automatically.
• Even though this will work indefinitely on buckets created prior to September 30, 2020, you should stop relying on path-based access immediately. If for some reason you need to recreate a bucket (it happens), you’ll be forced into the new V3 API and virtual host based URLs.

Miscellaneous News

• • AWS adds default encryption to EBS volumes. You can now ensure all EBS volumes are encrypted either with an AWS managed key or a customer managed key. See our blog post for more details on this.
  • The new i3en instance class is kind of a big deal. They’re super dense, with super fast networking and NVMe IO. While they’re a little more expensive per class than the i3 type, they’re actually a much better value. If you’re doing a large Kafka or ElasticSearch cluster in AWS, these really need to be a strong consideration.
  • AWS eliminates need for aws-iam-authenticator for EKS. Previously to use EKS, you needed a drop-in binary for Kubernetes to be able to trigger IAM authentication. Now, you can wire the aws eks CLI to do it for you inline, saving that step. Both methods will continue to work, so no need to change. If you want to switch over to the new method, keep in mind that you’ll need the latest AWS CLI.
  • EKS supports public IPs now. But pretend it doesn’t. That’s all.
  • EKS has a new deprecation policy, which you need to read. The long and short of it is that AWS will support EKS versions for a reasonable period of time, but then they’ll force upgrade you if you don’t upgrade yourself. Keep in mind that they’re upgrading your control plane, not your worker nodes. And things can happen if you allow your worker nodes to drift out of date. The long and short of it is if you’re using EKS, you must have regular upgrades as part of your operations plan.
  • CloudFormation finally supports AWS Backup. But be careful. You can’t delete a non-empty backup vault, so this can block deletion of your stacks in some cases. Also, be mindful of your RetentionPolicy when stacks come and go but backups should remain.
  • AWS Config makes pricing much friendlier. If you’re running CIS-compliant on your AWS accounts, you have AWS Config Rules enabled for regions you never use. The prior pricing model charged you for this, which added a meaningful amount to your bottom line. You now solely pay for what the rules match on, not just for having them on.
  • Encrypted AMIs can now be shared across accounts. Previously, you had to copy the underlying EBS snapshot to the account and create a new AMI, which was tedious at best. Now, they can be shared directly. Keep in mind that you have to share the KMS CMK via IAM policy in order to use this effectively, and you’ll want to do so in a way that does not over-permission the key. We’ll post a follow up on the correct policies for that later this month.
  • AWS Cost Explorer supports reservation expiration alerts. You could always do this via CloudWatch Event alarms, but most people don’t take the time to set those up.
  • CloudFormation supports On-Demand Capacity Reservations. Note that capacity reservations are for putting a hold (and paying for the privilege) on resources you might use. This is a common strategy for Disaster Recovery, though there are other use cases. CloudFormation may not be the best place to be making these decisions.