AWS is Shifting Costs Left With Athena
With the endless flood of new products, features and changes from AWS and its surrounding ecosystem, it can be easy to miss an update. Our monthly round-up highlights major AWS news, announcements, product updates, and behind-the-scenes changes we think are most relevant.
AWS Secrets Manager Database
Everyone knows Corey Quinn’s affinity for the cloud-native database, Amazon Route 53. I’ve developed an unhealthy relationship with another one of AWS’s lesser-known databases that, like Route 53, is marketed to solve a problem slightly more complicated than simple storage. In the case of Route53, it allows users to work with the Domain Name System (DNS) easily. Secrets Manager is a service that stores and manages secrets. Technically these are both data stores made for the cloud, “cloud-native,” if you will. I’m not going to tell you how to design your app, but I will let you know that using AWS Secrets Manager as a database is now a more practical option than it was last month. The limit for number-of-secrets-per-account went from 40,000 to 500,000. I can’t imagine what account could reasonably use 40,000 secrets, but AWS isn’t typically inclined to put a limit on the number of reasons they can bill you. I don’t know why the limit is half a million, but you should contact them if you need more than 100k. I’d like to think they want to make sure you don’t dig yourself into any holes that you can’t dig yourself out. That’s the least they can do if you’re planning to spend $200k on secret storage and at least $2.50 on API calls (500,000 secrets per month x 1-month x 0.40 USD per secret per month, 500,000 API calls x 0.000005 USD per API call). If you’ve read this far, I’ll let you know a secret about AWS Secrets Manager. If you’re not going to take advantage of the rotation or replication features of AWS Secrets Manager, then you’re better off using the SSM Parameter store, which can also store secrets with comparable security at one-eighth the cost. Resource link here.
AWS is Shifting Costs Left With Athena
Including cost calculations in the query plan for Athena is one slight shift left for cost optimization. Now engineers will know exactly how much money they’re burning while making Athena queries. What AWS should display is the amount of blood, sweat, and tears Athena saves DevOps personnel from supporting an in-house presto cluster.
On a more serious note, this is really good. You can rack up some serious charges on AWS in as little as a single API call, and I can attest to the fact that developers make lots of API calls 
. This has attracted the attention of the Linux Foundation, which adopted the FinOps foundation recently, a foundation dedicated entirely to controlling costs in the cloud. They’ve certainly embraced informing users about costs, inform is one of their three FinOps “phases”, but haven’t committed themselves to the “shift-left” philosophy. The shift-left for security has worked pretty well; scanning our applications for security vulnerabilities before the attackers is probably a good thing. The shift-left for cost optimization has the same potential. Having the AWS API returning a billing receipt for every call is my favorite idea, but there are a lot of existing tools to which this functionality could be added. A lot of our cloud infrastructure is written in Terraform, and there are already several projects trying to solve this problem specifically for Terraform. Terraform Enterprise has a feature that will show the cost in plans, and a startup, Infracost, offers to bring that to GitHub.
Unrelated to that rambling, I’m curious how this will affect the IAM permissions for Athena queries. I have a hard time imagining how this will be possible without hitting some billing APIs to which a developer would not usually have access. Resource Link Here.
Pulling Lambda Containers From Other Accounts
I’ve seen many AWS organizations where one account, usually dubbed the operation account, contains things shared between application teams like base container images. AWS Lambda has a super-cool feature where you can use a custom container image as the runtime for your function. This announcement made me double-take because I was surprised it wasn’t already a feature. Before this, an ECR repository had to be created in each account to hold a lambda function. This means you have to push your images to that repository and set up IAM permissions, logging, and never-ending fun in each account. Now you can do all the container image work in one account, delegate access to the child accounts, and then create the function with a user who has the appropriate permissions. This is much simpler if you’re familiar with cross-account IAM access and eliminates the possibility of container repository images becoming out-of-sync. Resource link here.
AWS Releases More APIs!
AWS is really good at APIs. I’ve been waiting for them to release the Human 2.0 API so I can program my carbon-based mainframe to read good and learn to do other stuff good too. Sadly I don’t see this on their roadmap, but Elon Musk is on the case.
Back to the AWS APIs. The transit gateway one is great, and you want to verify your transit gateway will work as expected (Companies like Facebook and Slack might want to pay attention). The CloudControl API is cool in that it gives you a CRUD API to work with any resource you can also manage with CloudFormation. This sounds a lot like they’re exposing the APIs cloud formation uses, perhaps after being generalized so that plebeians like me can use them.
Atlassian Embraces AWS
Atlassian, the Australian company, redefining how distributed teams collaborate, has moved Bitbucket cloud to AWS. Many of Atlassian’s other services are already running in AWS, but Bitbucket was in the data center until this last year. Atlassian says they had to cloud-ify the Bitbucket architecture but does not say if they just wrapped AWS CodeCommit in a branded UI like I’d be tempted to do. There was one line in this post that I have to quote here because I think it will resonate with our readership. The holidays bring a lot of parties, but no one is as exciting as a decommissioning party. Resource Link Here.
Because, no matter how carefully you plan your migration and what precautions you have to take, let’s face it: there ain’t no party like a DC-decommissioning party. ®
Here is an image of history’s most iconic decommissioning party:
Welcome AWS Resilience Hub to the AWS Hub Club
The latest nebulous term to get a hub in AWS is “resilience” with the GA release of AWS’ Resilience Hub. This is great news if you want AWS to scan your application configurations so AWS can tell you how AWS can better host your application in AWS. Notice I mentioned their name a few times there. That’s because as much as I love AWS, you need to remember caveat emptor. Don’t let them sell you a complicated web-scale solution when a simple EC2 instance would drive better business value. That being said, the integrations with AWS’ Fault Injection Simulator look like they could be cool, and it can turn recommendations into CloudFormation templates that promise to automatically set up CloudWatch alarms and remediate problems with SSM run commands. Resource link here.
Miscellaneous News
- AWS Fault Injection Simulator now supports Amazon CloudWatch Alarms and AWS Systems Manager Automation Runbooks. AWS now lets you cripple your ability to monitor and respond to incidents so you can see what happens when plans B and C fail! Usually, crippling critical infrastructure would be bad, but if you pay extra for it, it’s chaos engineering. Read More.
- Are lambdas overhyped? Could we be utilizing them better? Why do I have to learn a new technology every year? This, and more, will be answered: The Infinite Loop of Lost Potential
- AWS Backup Picks up more Services. AWS Backup adds support for Amazon Neptune and Amazon DocumentDB (with MongoDB compatibility). AWS Backup’s ability to manage many services by applying tags has saved us an enormous amount of time that would be spent automating snapshots. They’ve been adding features and services at a breakneck pace. I can get excited about anything that will alleviate the operational headache that is disaster recovery with MongoDB.
- CloudFront makes the web a little safer. If you’ve ever used a tool like Mozilla Observatory to lock down the security of your site, you will know how important these headers are to getting A+ security. These headers help protect clients from scary internet ghosties like cross-site scripting and unencrypted connections. AWS is also providing some default policies so you don’t have to learn about the intricacies of HTTP headers to benefit from them! Read More.
- AWS packages CI/CD templates in the SAM CLI. The announcement about simplifying serverless CI/CD piqued my curiosity, so I delved into the documentation for SAM and visited serverlessland for the first time. This announcement amounts to the GA release of some templated files, which isn’t noteworthy, but I wanted an excuse to say I visited serverlessland. Read More.
