Skip to main content

The Rhythmic Blog

Amazon Blockchain Goes GA, S3 Batch Operations, and More AWS Updates

May 1, 2019       Steven Black       ,         Comments  0

With the endless flood of new products, features and changes from AWS and its surrounding ecosystem, it can be easy to miss an update. Our monthly round up highlights major AWS news, announcements, product updates and behind the scenes changes we think are most relevant.

April brings some features that are actually useful for those who spend their daily lives building in AWS. There’s a few new product launches too, but mostly this month focused on features to make life easier. For which we are grateful to the AWS gods.

Amazon Blockchain goes GA

Amazon Managed Blockchain went GA at the end of the month. That means you, too, can have your own blockchain without all the hard work and toil that comes with blockchain territory. The AWS implementation essentially is just a management layer on top of Hyperledger Fabric, one of the more mature open source blockchains. As is the case with this type of AWS managed service, you’ll be spared from the setup process but still working at a fairly low level with Hyperledger itself. And that’s okay, because if you can’t take the time to learn Hyperledger, you shouldn’t be doing blockchain anyway.

This all begs the question–when should you be using a blockchain? If you have to ask, the answer is you shouldn’t be. And even if you don’t have to ask, the answer is that you still probably shouldn’t be. The premise of a blockchain is to create a trust overlay between untrusted parties. Traditionally, accountability in such situations is enforced with control monitoring–Trust But Verify. Blockchain lets you go further by putting the evidence of each and every transaction into an immutable ledger that is distributed across 2 or more parties. It’s great in the supply chain, for enforcing complex agreements, and even has use cases in security and payments. But if you don’t have untrusted parties and a really hard problem, Blockchain isn’t for you. Even if you could use it. So, don’t.

There are a lot of use cases where you do need a ledger that is provably immutable–where sysadmins and DBAs cannot modify the integrity of a transaction and cover their tracks using their access. This problem has historically been almost impossible to solve, and the best approach is through separation of duties. But, it typically takes as few as 2 people colluding in any organization to bypass that. Amazon QLDB, which is still in preview, is the better answer for that. QLDB allows you to create a ledger that uses chained cryptographic signing, giving Perfect Forward Secrecy and non-repudiation to your transactions. This is similar to a blockchain but with considerably reduced complexity. If you are simply seeking to provide an auditable, immutable transaction log in house, QLDB is what you want. Not blockchain. We promise.

S3 Batch Operations also go GA

S3 Batch Operations went GA after an extended preview. If you work with huge buckets, you care about this. The single most important feature is the ability to get an inventory report, something tedious to do with buckets that scale into millions of objects. The inventory report can be the basis of future bulk operations and can guide decisions on bucket cleanup, governance and permissions processes. Most importantly, it spares you the need to go through a heavily rate limited API to get what you need.

The following operations are currently supported:

  • PUT copy – bulk transfer content to another bucket. This is useful for setting up replication as well as for changing storage classes or encryption keys.
  • Invoke a Lambda – careful! Lambda is dangerously cheap, but nothing is cheap when you invoke it a billion or more times.
  • Replace Tags – supremely useful, developers are notorious for failing to get object tagging right up front and then saying, “well it’s too late to fix it now”. That’s no longer true.
  • Replace ACLs – slightly less useful now that we have Public Access Controls but still handy.
  • Restore – initiate targeted restores from Glacier and Glacier Deep Archive. This is particularly useful for teams that would be gun shy about pushing small objects into Glacier because they might be difficult to get back later.

Keep in mind that the key to S3 Batch Operations is to enable inventory reports on your bucket, either daily, weekly or monthly–and you’ll pay to run those reports, which can generate billions of API calls on your behalf. The reports don’t generate quickly, so if you anticipate needing this functionality, you need to enable that ahead of time. Your batch operations will also run based on whatever inventory report you choose, which is not necessarily what’s in the bucket when you kick your job off.

DirectConnect now supports Transit Gateway

And we are so thankful for it. This is one of those features that is easy to beat AWS up on. Transit Gateway has been out for 2 years now and massively simplifies life in a multi-account, multi-VPC world. Those are exactly the sort of users who would heavily rely on Direct Connect. But, this was also firmly in “hard as hell to implement” territory, so AWS gets a pass for it taking this long.

If you are currently using multiple Direct Connects or VPN for your non-production account, you should start planning your migration to Transit Gateway now. As a refresher on Transit Gateway, it allows you to attach thousands (or dozens if you’re like the rest of us) of VPCs to a single routing policy, enabling you to connect your on-premise infrastructure to your AWS workloads once. When properly configured against your DR strategy, you can achieve concurrently maintainable, fault tolerant global connectivity.

Miscellaneous News

    • AWS rolls out per-second billing for RDS. Something for you to contemplate when you spend 20 minutes waiting for your RDS instance to terminate.
    • EKS now supports A1 ARM instances in preview. If you’re running true microservice containers inside EKS, odds are it can run on an ARM processor with minimal reconfiguration and at considerable cost savings. There’s a common presumption that going cross-platform is difficult, but it often is just a matter of configuration.
    • EC2 T3a AMD instances are available. These instances are just a bit slower than their Intel cousins and about 10% cheaper. If you aren’t depleting your CPU credits on your T3 instances, consider switching over.
    • Amazon ElasticSearch Service finally supports C5/M5/R5 instances. If you’re doing anything small to moderate in ElasticSearch, these are the instance types you want to be running with. If you’re doing anything intense, you want to be in i3’s.
    • Amazon ElasticSearch Service supports native alerts. In conjunction with the gradual rollout of their Open ElasticSearch Distribution, AWS has rolled out built-in ElastAlert support. It’s neat, but you can also use whatever alerting framework you want. If ElastAlert is not your speed, don’t think you are stuck with it when using managed ElasticSearch.
    • Parameter Store supports higher call rates if you ask. But don’t ask. It will make Parameter Store go from “practically free” to “uggh”. The higher call rates allow you to call in excess of 1000 requests per second. You should be caching if you’re calling that much. Since that’s a global limit, there are some narrow use cases for this. But don’t let yourself be one of them. Cache your calls to Parameter Store, even if only for a few seconds.
    • STS tokens have new global endpoint. For most of us, this is just an interesting factoid. But if you’re working with a lot of regions, this is worth checking out. There are implications for when you use regional endpoints vs. the new global endpoint, and many libraries and dependencies are likely to switch over to the global endpoint out of convenience. This can lead to difficult to troubleshoot errors.
    • Beanstalk supports tag-based permissions for everything. Almost everything, anyway. But, this is super useful if you use Beanstalk and want to give fine-grained control to development teams. There are other use cases for this as well.
    • EKS finally delivers logs to CloudWatch Logs. Yes, please.
    • CloudWatch Alarms support tag-based permissioning. This seems trivial but has neat implications. For example, you could prevent your ops team from clearing alarms via the console and instead force them to do so through your ticketing system.
    • CloudFormation coverage for EC2, ECS, ELB. If you’re one of the people affected by these missing resources and properties, you’ll be happy to be able to retire some ugly workarounds.

Leave a Reply