In Season 5, Episode 27, Karl and Jon are joined by Alam Ahmad, AWS Community Builder and content creator, to discuss four news stories: a new Secrets Manager integration with EventBridge, an AI-powered investigation feature for GuardDuty, a widely reported AWS billing display glitch, a domain expiry incident that locked a web developer out of his own account, and an update on the AWS data center in Bahrain. Along the way, the conversation somehow evolves from AWS billing errors into career advice on how missing a card payment can be an effective way to get people's attention.
05:33 - AWS Secrets Manager Now Publishes Secret Update Notifications to Amazon EventBridge
AWS Secrets Manager has been updated to publish secret update notifications directly to Amazon EventBridge, removing the need to rely solely on CloudTrail for visibility into secret changes. Previously, taking automated action on a secret update required complex workarounds involving CloudTrail logs, which Jon describes as requiring considerable effort to parse and act upon. Jon welcomes the change, noting it makes it far simpler to trigger downstream actions - such as validation workflows - when a secret is updated. Alam agrees the feature is common sense and highlights its particular value for large teams or organisations with high staff turnover, where credential visibility is critical. Jon also uses the opportunity to question the value proposition of Secrets Manager over SSM Parameter Store's secure string type, noting that at roughly 40 cents per secret, the cost differential is hard to justify given the functional similarity between the two services.
10:20 - AWS Guard Duty Investigation Agent - AI-Powered Threat Investigation
AWS has announced a preview of an investigation agent for Guard Duty that can automatically investigate findings, correlate related activity, and produce a report with actionable recommendations. Jon explains that Guard Duty ingests signals from multiple sources - including VPC Flow Logs, CloudTrail, and RDS login attempts - and generates findings, but that historically working out whether a finding is genuinely concerning required significant manual effort and environmental knowledge. The new agent can investigate a specific finding, a specific account, or up to 100 accounts across an organisation, reducing the legwork required from security teams. Alam notes it addresses the lower-level, high-volume work typical of a SOC tier-one role, and welcomes AWS's broader move towards automating security workflows. Jon flags uncertainty around pricing, noting that the DevOps Guru agent ties pricing to a support plan, which he finds an unfriendly model, and expresses interest in seeing what this agent will cost once out of preview. Karl also notes that unlike the DevOps Guru and Security agents, the investigation agent does not appear to carry title case branding, suggesting it may be a sub-feature of Guard Duty rather than a standalone named agent.
18:46 - AWS Billing Console Glitch Generates Trillion-Dollar Bills
A bug in the AWS billing console caused a number of customers to see wildly inflated estimated bills, with some figures reported as high as 55 trillion dollars - exceeding the GDP of the United States. Jon believes the root cause was a units-based calculation error, likely confusing bits and bytes, resulting in figures off by many orders of magnitude. The episode discusses the absurdity of the numbers themselves, with Alam noting the highest figure he encountered was around 7.8 trillion dollars. Jon's primary criticism is not the bug itself but AWS's public response, which described the issue as a "slight billing issue" - a characterisation he calls tone-deaf given that unexpected large bills can cause genuine distress and, in extreme cases, serious harm to individuals. Karl notes that in practice, no credit card would likely process a transaction of this size, offering some comfort, but agrees the communication was poorly handled.
25:27 - AWS Customer Loses All Client Websites After Account Suspended Over Expired Card
A web developer - running a digital firm called Taproom - lost access to all client websites and email accounts after his AWS account was suspended due to an expired payment card. Jon outlines the cascade of failures: domain management and DNS were both hosted within Route 53 in the same suspended account, the root account MFA relied on a software authenticator stored on a broken laptop, and the email backup MFA method used an address hosted on one of the suspended domains, making it impossible to log in and resolve the issue without a lengthy identity verification process with AWS support. Alam draws parallels with similar small-footprint issues seen in NOC work, such as DNS misconfigurations and failure to account for employees leaving, and acknowledges that the customer was bounced between departments when trying to resolve the situation. Jon's key takeaway is that critical recovery mechanisms - such as MFA methods and billing contact details - must never be hosted within the same account or service they are intended to recover. Karl notes that the team has seen similar issues within their own customer base, and Alam reflects that while many of the contributing factors were within the customer's control, hindsight makes it easy to identify mistakes that are common in practice.
32:56 - Iranian Cruise Missile Strike Reported Against AWS Data Centre in Bahrain
Iranian state media claimed that an AWS data centre in Bahrain was struck and destroyed by cruise missiles, reportedly in response to alleged US strikes on an under-construction nuclear facility. Jon urges caution about the specific claims, noting that state media from any country carries an inherent agenda and that a single missile would be unlikely to destroy a facility of that scale. Karl notes that AWS's status page for the Bahrain facility has not been updated since 30 April, when AWS was already advising customers to migrate workloads elsewhere. AWS reportedly stated that no customers were impacted and that workloads had been proactively moved to other sites. Alam raises the broader question of whether geopolitical risk must now form part of infrastructure planning decisions when selecting AWS regions, and Jon's conclusion is straightforward - workloads should not be running in this region at present.
