Ship It Weekly - DevOps, SRE, Platform and Cloud Engineering News

All episodes.

82 episodes · showing the newest 82

Ship It Conversations: Ryan Packer of BambooHR on Why CTOs Are Coding Again, Building a Software Factory, AI Straining CI/CD, Protecting HR Data, Mid-Career and Junior Engineers, and Why Every Engineer Gets a Team of Agents

Nov 9, 2026 36 min

This is a guest conversation episode of Ship It Weekly, separate from the weekly news recaps. In this Ship It Conversations episode, I talk with Ryan Packer about why a CTO is writing code again, what happens to CI/CD when every engineer runs a fleet of agents, and who trains junior engineers when AI can fix the bugs. Ryan is the CTO at BambooHR and a former VP of Engineering at Adobe. He's watched engineering organizations change from Omniture to Adobe to BambooHR, and he says he's spent more time coding this year than in the rest of his career combined. We start with why. Ryan will never write production code for his team again, but he wants to use the same tools and learn the same techniques they're learning, and to show an idea in working code instead of a PRD or a Slack message. He's also honest about the limits: his sandbox serves maybe a hundred people, and his team's software serves millions. Then we get into the software factory. In Ryan's own version, a customer files a bug, an agent opens the PR, tests it for security and performance, scores it, and ships it if it meets the bar. At BambooHR, pull requests are up almost 80% since the start of the year, and quality has improved by the metrics they track. The part he's most excited about is more at-bats with customers: an idea to a working prototype in a day. For platform teams, Ryan says AI has been more of a challenge than a help for CI/CD. BambooHR was already releasing hundreds of times a day, but agents do their best work when they can run and test what they build. Ten, twenty or a hundred agents per engineer means environments that spin up and down fast, with real data. We also cover protecting HR and payroll data: terms with the frontier labs, data retention, and an AI council. On careers, Ryan thinks mid-career engineers face the hardest change, and that the ones who keep learning have little to worry about. On junior engineers, he doesn't claim a perfect answer. He no longer needs them to fix bugs,...

GitHub Blocks pull_request_target by Default, WordPress Update Race Exploited, Polymarket Drained via Hijacked DebugBear Script & Alpine's GitLab Upgrade Missed the Fix

Nov 6, 2026 16 min

This week on Ship It Weekly, GitHub will block pull_request_target in public repos by default from December 2, with brownouts on Mondays, November 9 and 16, 11 AM to 1 PM ET. A WordPress core update race (CVE-2026-119977) is on CISA's exploited list: during a full-package update, a fresh installer can sit for a moment at a public URL under wp-content/upgrade. A hijacked DebugBear script drained about $2.7 million from Polymarket users after DebugBear's DNS was compromised through its registrar. And Alpine Linux explains why upgrading GitLab didn't fix CVE-2026-97229 on its own image: the fix was a patch to Ruby. The thread: what actually ran, as opposed to what you think ran. A trigger that runs a fork's code with your secrets, an update that opens its own window, a script served by whoever controls the DNS, and a version number that said "fixed". In the lightning round: OpenSSH 10.6p2 fixes a password login bypass on musl builds with PAM off, which is how Alpine builds openssh-server; Debian and Ubuntu patch an APT package swap (CVE-2026-109518); and .NET 8 and 9 reach end of support on Tuesday, November 10. The human closer comes back to Alpine's "we cannot rule it out." If an attack hit you this week, could your logs prove it didn't work? Highlights • pull_request_target: what it runs, and what to check before Monday • WordPress: when keeping current is what opens the window • DNS, HTTPS and subresource integrity for third-party scripts • Registrar lock, MFA and alerts on DNS changes • Alpine: GitLab from your own image needs a fixed Ruby too • "No evidence" is only as good as the logs you kept Links pull_request_target blocked by default (GitHub): https://tsn.io/KwebM WordPress advisory GHSA-4w3m-c66w-69mg: https://tsn.io/1HGiK WordPress upgrade-window RCE (Wordfence): https://tsn.io/Pc6X2 Polymarket postmortem: https://tsn.io/rOc2M DebugBear on the registrar DNS compromise: https://tsn.io/4BjIg Polymarket losses and refunds (Hackread): https://tsn.io/qNhoe Alpi...

Ship It Conversations: The AI Agent That Ran Terraform Apply, With Davlet Dzhakishev of Cloudgeni on Read-Only Cloud Access, Sandboxes Beyond Docker, Agentic PR Reviews, the Real Cost of Running Agents, FinOps for AI, and Permissions Over Prompts

Nov 4, 2026 37 min

This is a guest conversation episode of Ship It Weekly, separate from the weekly news recaps. In this Ship It Conversations episode, I talk with Davlet Dzhakishev about what it takes to let AI agents work on your cloud infrastructure, why guardrails belong in the architecture and not the prompt, and what running your own agents really costs. Davlet is the CEO and co-founder of Cloudgeni, which builds AI agents for cloud infrastructure. He got into this after working through SOC 2 and ISO 27001 compliance prep mostly by hand, and asking what agents would need to fix those problems reliably without making things worse. We start with the incident that changed how Cloudgeni builds. An agent hit a snag on a simple Terraform change and decided the fix was to run apply. On staging. Resources got recreated, and Davlet calls it his wake-up call. Since then, guardrails come first: sandboxed runtimes, scoped credentials, read-only cloud access, and changes that go through Git, CI/CD and tests. At the architecture level, the agent shouldn't be able to do what you don't want it to do. Then we get into agentic PR reviews, which check human and agent work the same way, and why a good reviewer catches more than the obvious errors. Davlet's example is a private network around a key vault: great for security, until your own apps lose access. We also cover drift, and why you need to reconcile it before you can trust agent PRs. On cost, Davlet's view is that AI subscriptions are heavily subsidized, and running your own agents on API tokens can cost ten to twenty times more. Most agent usage also has no audit trail, so a stuck loop can poll a dead API server every day and no one will know. He thinks FinOps for AI is going to be big, orphaned agents and all. That need for audits is part of why Cloudgeni open-sourced its foundation as OpenGeni. We finish with what he still wouldn't hand to an agent, and where a platform team should start without building a monster: PR reviews first, then...

Ship It Conversations: The AI Hallucination That Looks Right, With Devansh of Irys on Legal AI, Three-Tier Error Isolation, Models That Change Under You, Overhyped Reinforcement Learning, and Optimizing for Maintenance Over Generation

Nov 2, 2026 33 min

This is a guest conversation episode of Ship It Weekly, separate from the weekly news recaps. In this Ship It Conversations episode, I talk with Devansh about why the trickiest legal AI hallucination is a real case used the wrong way, how his team splits their system into three tiers to find failures fast, and why he'd rather optimize for maintenance than for generation. Devansh is the Founder and Head of AI at Irys. He builds reasoning and verification systems for legal AI, a tough place for hallucinations and confident wrong answers. He also publishes open source AI research on Substack. We start with why AI for professionals breaks down. Devansh thinks models are getting good at doing a lot of things somewhat well, but they fall short on the last mile to reliable. So his team engineers structure back in instead of hoping a better model fixes it. In legal, fake citations are the failure everyone has heard about. The more interesting one is a real case that doesn't apply. So much of law is interpretation, and a model built to predict the most probable token pulls toward the average. Then he walks through the three tiers behind Irys One. Ingestion maps messy input, like PDFs with handwritten notes and email chains that refer to them, into a knowledge graph so the AI knows where to look. Reasoning navigates that graph. Delivery applies how a firm or a lawyer likes to work. The payoff is error isolation: if a lawyer says something is missing, two or three targeted queries tell them whether it was ingestion or reasoning. We also get into why chunk-based retrieval misses connections, why going from 10 docs to 10 gigabytes is mostly about finding the few documents that matter, and how "answer contracts" handle "it depends." Then I take a DevOps angle. Devansh says the hardest part of running AI in production is tracking everything. A fully agentic system might perform 20% better, but you never know how it breaks until it breaks. The first thing his team tracks is whether...

GitHub Actions Jobs Move to AWS, Brotli RCE Exploited, Gheist Steals Actions Secrets & AGENTS.md Hijacks AI Coding Agents

Oct 30, 2026 16 min

This week on Ship It Weekly, GitHub Actions can now run eligible Linux jobs on AWS as well as Azure, so allowlists and Entra Conditional Access policies that only trust Microsoft ranges can break. The Brotli decoder bug (CVE-2026-111497) landed on CISA's exploited list, and it reaches past Node.js to Python Brotli, Bun, Envoy, Istio, a Go port and .NET. A campaign called Gheist is using stolen GitHub tokens to push ci-sync branches that print Actions secrets past log masking, then delete the run. And attackers are dropping AGENTS.md files into Downloads folders to hijack AI coding agents. The thread: each one is an inventory question. Which services decompress Brotli, which allowlists assume CI runs on Azure, which OIDC trust policies accept any branch, and which laptops have started an agent in Downloads? In the lightning round: GitHub Actions workflow lockfiles are generally available, Railway disclosed a breach of its internal systems, Tailscale 1.104.4 fixes four security bulletins, and Datadog moved its US1 status page to its own Status Pages product. The human closer, from Bill Duncan: "Twenty minutes in, you haven't started diagnosing anything. You've been doing archaeology." Write down who owns it before the pager asks. Highlights • GitHub Actions on AWS: why the same job can pass, then fail • Sync runner IP ranges from GitHub's meta API, not one cloud's • Brotli: a decoder bug your HTTP stack may run for you • Gheist: what private repos and OIDC trust let through • AGENTS.md in Downloads: an instruction file is code now • Service catalogs and the "what depends on me?" question Links GitHub Actions multi-cloud compute: https://tsn.io/fJkLJ Node.js October security releases (Brotli): https://tsn.io/BvZBU Brotli CVE-2026-111497, affected libraries (Wiz): https://tsn.io/HJuKS .NET out-of-band security updates: https://tsn.io/j0Xwf Gheist steals GitHub Actions secrets (Wiz): https://tsn.io/nunEs AI agents hijacked via the Downloads folder (Forever Security): htt...

Ship It Conversations: AI Coding Outruns Software Delivery, With Darko Fabijan of SuperPlane on What Breaks After the Pull Request, Change Risk Scores, Onboarding AI Agents Like Engineers, Read-Only Ops Agents, and Why the Answer Is More Automation

Oct 28, 2026 35 min

This is a guest conversation episode of Ship It Weekly, separate from the weekly news recaps. In this Ship It Conversations episode, I talk with Darko Fabijan about what starts breaking after the pull request when AI writes more of the code, what should sit between an AI agent and production, and why his answer is more automation. Darko is the CEO of SuperPlane and co-founder of Semaphore, the CI/CD platform. At SuperPlane, he's working on how to give AI agents write access to production in a controlled way. We start with what AI-generated code does to delivery. Pull requests are bigger and more frequent, reviewers can't keep up, and Darko points to a Faros AI report that shows defects and incidents going up. He describes AI as alien technology that dropped from the sky, landing in processes built for people who review code eight hours a day. His main hypothesis is a deterministic layer between AI agents and production, with scoped access that goes through approval. I bring up the rubber stamping problem with human approvals, and his reply is that humans hallucinate too. So he wants a risk score on every change. Renaming a bucket scores low. Deleting an S3 bucket or an RDS instance scores red. Then we get into the glued-together ops tooling most teams run on. Darko doesn't think anything is wrong with it. When I ask how SuperPlane differs from Argo Workflows or n8n, he says he wants agents and humans on the same plane, and he'd onboard an agent like a new engineer: give it an identity, let it look around, and show it what it can do. We also cover team-scoped agents, compliance at AI speed, and docs where every line traces back to what produced it. We finish on where to start. Darko says read-only first: he pretends to be super stupid, asks a lot of dumb questions, and says "onboard me." For the first write operations, he'd pick loops you can measure with a number, like latency, GitHub Actions speed or CVE count, tested on a copy of production or staging first. The b...

Ship It Conversations: Running Retros as Tabletop RPGs, With Jinger Meilani of Mountain on Chapters and Characters, Technical Archetypes, Reading the Room, Following Through, and Why Silliness Gets You Honest Answers

Oct 26, 2026 37 min

This is a guest conversation episode of Ship It Weekly, separate from the weekly news recaps. In this Ship It Conversations episode, I talk with Jinger Meilani about Tabletop Retro, her way of turning an engineering retrospective into a tabletop role-playing game, and why a little silliness gets teams to give honest answers. Jinger is a Senior Software Engineer in DevOps at Mountain. She says Tabletop Retro brings together three things she loves: world building, psychological safety on teams, and engineering enablement. We start with why she reinvented the retro at all. She was bored of them, and even the themed ones got rote. At a previous company, the team rotated who ran the retro, and her turn landed around Halloween, so she built a story around the team. She calls the result retro plus plus plus. The usual formats and outcomes are all still there, inside a story. A lot of it comes down to silliness. Jinger compares it to hiding vegetables in a fruit smoothie: it disarms people, and she gets honest answers, sometimes while everyone thinks they're talking about something else. That's also why she won't run one during a fire. People have to feel comfortable before they'll speak their truth, so the flame-breathing dragon can wait. Then we get into how one is built. A Tabletop Retro is a series of chapters, and each chapter holds part of the story and at least one activity. Players fill out character sheets, and each in-game race maps to a technical archetype, like a Goliath for a high-level systems thinker or a gnome for a tinkering automation engineer. Afterward the team talks through what those picks say about their strengths and weaknesses. She presents it like a storybook, in Lucidchart or Canva over Zoom. Characters cut both ways. For some people, an orc is permission to say what they'd never put on a retro board, because it's not really how they feel. They're just an orc. Others find role-play intimidating, so she reads the room: improv for an outgoing teamma...

LiteLLM Release Pipeline Hijacked Again, GitHub Package Firewall, PHP2Shell Core RCEs & eMudhra CA Keys Compromised

Oct 24, 2026 16 min

This week on Ship It Weekly, LiteLLM's release pipeline was hijacked again: stolen GitHub CLI tokens from three engineers changed the pipeline, kicked off the release and approved it, and a credential-stealing 1.107.0 went out to PyPI and Docker Hub. GitHub put a package firewall into public preview, with advisory and supply chain checks and a cooldown for new releases. PHP fixed PHP2Shell, two critical core runtime bugs (CVE-2026-109618 and CVE-2026-109619) that someone tried to exploit four days before disclosure. And eMudhra, a certificate authority in India, found out the private keys for eight intermediate certificates could be worked out from its own public certificates. The thread: most of this came through things you depend on but don't run. A library's release pipeline, the registry you pull from, the runtime under your app, and the trust store in your images. In the lightning round: another GitLab CVSS 10 (CVE-2026-110264), Google's report on the europe-west2 London outage, AWS taking its European Sovereign Cloud off the global backbone on October 24, and a GitHub audit log change that can quietly break SIEM rules. The human closer, from Brent Chapman: page on what users actually do. Your users don't care how your machines feel. They care whether they can log in. On Monday, Jinger Meilani joins me to talk about running retros like a tabletop RPG. Highlights • LiteLLM 1.107.0: two stolen identities passed a two-person approval • GitHub package firewall: advisories, attestations and a cooldown • PHP2Shell: an exploit attempt four days before disclosure • Patching PHP is easy, finding all of it is hard • eMudhra: CA private keys worked out from public certificates • Page on what users feel, not how the machine feels Links LiteLLM 1.107.0 compromise (Wiz): https://tsn.io/yQtmu GitHub package firewall in public preview: https://tsn.io/QjofJ PHP2Shell, CVE-2026-109618 and CVE-2026-109619 (Help Net Security): https://tsn.io/GUyop Eight eMudhra CA keys compromised...

Ship It Conversations: Rob Koch of Slalom on Accessible Cloud Native, Sign Language Recognition with Kubeflow, Overhyped AI, Conference Accessibility, and Why Teams Should Hire Deaf Engineers

Oct 19, 2026 45 min

This is a guest conversation episode of Ship It Weekly, separate from the weekly news recaps. In this Ship It Conversations episode, I talk with Rob Koch about what accessible cloud native actually looks like, why sign language is still one of the hardest problems in AI, and why the fastest way to make your tooling accessible is to hire deaf engineers. Rob is a senior principal at Slalom, an AWS Hero, and co-chair of the CNCF Deaf and Hard of Hearing Working Group. Rob is Deaf, and his ASL interpreter, Amelia Fruehsamer, voices his answers. Special thanks to Amelia for interpreting the conversation. We start with an analogy Rob comes back to all episode: cloud native is the toolbox, and platform engineering is the workbench. Then we get into what's overhyped about AI, the guardrails it still needs, and what auditable AI observability looks like. A big part of the conversation is accessibility in practice. Rob explains how captions pushed for deaf attendees ended up helping neurodivergent folks too, and what conferences get wrong: accessibility planned at the very end instead of the start, and the real cost of interpreters versus captions. Then we get technical. Rob walks through using Kubernetes and Kubeflow to train models that recognize sign language, and why it's so hard: regional signs, lighting, skin tone and backgrounds, and a camera that can't tell which way a hand is facing. His joke: we don't have AGI until AI understands sign language. The big takeaway: start with people, not a checklist. Hire deaf and hard of hearing engineers, ask them how they want to work, and your tools get more accessible because they're in the room. Highlights • Cloud native as the toolbox, platform engineering as the workbench • What's overhyped about AI, and the guardrails it still needs • What accessible cloud native means in practice • Captions that help far more people than you planned for • What conferences get wrong about accessibility, and what it costs • Kubernetes and Kube...

GitHub Actions Lockfiles, GitLab CVSS 10 CI Flaw, Google Cloud Storage Outage & Torvalds Builds Kernel Patches With Claude Code

Oct 17, 2026 14 min

This week on Ship It Weekly, GitHub Actions workflow lockfiles land in public preview, pinning actions and their dependencies to exact commits. GitLab shipped a critical patch release for CVE-2026-97229, a CVSS 10 CI/CD regex flaw that could let an unauthenticated attacker run code on the server. Google published its final report on two Cloud Storage incidents in us-central1, including a recurrence caused by an incomplete rollback. And Linus Torvalds pushed an experimental kernel branch with page-cache patches built using Claude Code, as maintainers debate dropping the mandatory AI attribution tag. The thread this week: know what you're actually running, from the commit you reviewed to the state a rollback really left behind and code someone can still explain. In the lightning round: a record Patch Tuesday with more than 1,400 CVEs and an actively exploited Windows Shell bypass, GitHub AI Scan for pull requests and agentic autofix going GA, AWS Strands Box, an open-source agent sandbox that keeps real credentials out of the agent's environment, and a Microsoft 365 outage through Global Secure Access. The human closer comes from Paul LaPosta's “Your error budgets don't know AI exists.” AI has split shipping a change from understanding it, and SLOs don't measure comprehension. Your SLO can tell you the service is up. It cannot tell you whether anyone still understands it. Programming note: Ship It Weekly turns one next month, so expect more conversations over the next couple of months, usually two a week, starting Monday with Rob Koch. Links GitHub Actions workflow lockfiles GitLab 19.4.2 / 19.3.5 / 19.2.8 patch release Google Cloud Storage incident report Linus Torvalds and Claude Code (Phoronix) Linux drops the Assisted-by requirement (Phoronix) October 2026 Patch Tuesday GitHub AI Scan for pull requests GA Agentic autofix GA AWS Strands Box Microsoft 365 Global Secure Access outage Your error budgets don't know AI exists (LeadDev) This week's On Call Brief

Ship It Conversations: Erik Osterman of Cloud Posse on Terraform Modules, IaC Frameworks, Atmos, Internal Developer Platforms, AI Agents, and Why Conventions Matter More Than Ever

Oct 12, 2026 39 min

This is a guest conversation episode of Ship It Weekly, separate from the weekly news recaps. In this Ship It Conversations episode, I talk with Erik Osterman of Cloud Posse about Terraform modules, infrastructure as code frameworks, internal developer platforms, and why AI makes conventions more valuable, not less. Erik is the founder and CEO of Cloud Posse, which has published more than 180 open source Terraform modules, and he hosts Cloud Posse's weekly DevOps Office Hours. We start with what AI has done to open source infrastructure projects, then get into what makes a good Terraform module: consistent naming, a consistent interface, not so small that it wraps a single resource, and not so large that it has a hundred inputs nobody can test. We also talk about CloudFormation, Pulumi, and why HCL's constraints helped build the module ecosystem. Erik explains where Crossplane fits, and why accounts, IAM, networking, DNS and certificates all come before the cluster. A big part of the conversation is frameworks. Erik argues that nobody actually runs "vanilla Terraform," because Terraform alone doesn't handle state layout, promotion, authentication, tool installation or repository structure. Every team ends up building its own framework, usually undocumented. We talk about Terragrunt, Atmos, naming conventions, brownfield adoption, and breaking a monolithic root module into services. Near the end, we get into AI and self-service. Erik's team now generates most of its infrastructure work with AI, and it works because the conventions, agent skills, MCPs and pipelines are already in place. Self-service doesn't have to mean another no-code platform. The big takeaway: AI can generate infrastructure code faster than ever, but without conventions it is generating into chaos. Highlights • What makes a good Terraform module • Root modules vs. child modules • CloudFormation, Terraform, Pulumi, and why constraints helped HCL win • Where Crossplane fits, and what has to exist bef...

Critical Atlassian CVE Hits Jira, Confluence & Bitbucket, GitHub Copilot Sandboxing Goes GA, AWS Lambda SnapStart for Containers & Google Spanner Omni

Oct 10, 2026 17 min

This week on Ship It Weekly: Atlassian disclosed a critical arbitrary file access vulnerability affecting eight Data Center products, including Jira, Confluence, Bitbucket, and Bamboo. GitHub Copilot local sandboxing is now generally available, giving teams enforceable controls around filesystem, network, credentials, and local tooling. AWS published a deeper look at Lambda SnapStart for container images, including the operational risks of restoring initialized state. And Google Spanner Omni is now generally available, bringing Spanner outside Google Cloud and putting more of the availability responsibility back on the customer. The bigger theme this week is control. Developer tools are production infrastructure. AI agents need real execution boundaries, not just instructions. Faster startup through snapshots means understanding exactly what state is being restored. And moving a managed database onto your own infrastructure gives you portability, but also gives you the pager. In the lightning round: Citrix has a NetScaler SAML vulnerability under active exploitation. Cloudflare is working toward becoming a publicly trusted certificate authority. And attackers are actively scanning for Rejetto HFS servers vulnerable to a flaw that can lead to account takeover and remote code execution. And the human closer comes from Lorin Hochstein and a recent Google Cloud networking incident. One part of the outage involved someone correctly following an incorrect procedure, while another involved someone incorrectly following a correct one. Training matters, but systems also need guardrails that keep normal human mistakes from turning into large outages. If your system only works when nobody makes a mistake, the system is the problem. Links Atlassian CVE-2026-21589 https://tsn.io/ZlqKv GitHub Copilot Local Sandboxing https://tsn.io/rQngM AWS Lambda SnapStart for Container Images https://tsn.io/uXqkK Google Spanner Omni GA https://tsn.io/GoZZ5 Citrix NetScaler CVE-2026-88779 https...

AWS Retires DevOps Guru: What the End of Support Means, Kubernetes Cross-Namespace CVE-2026-2270, Node.js Undici WebSocket DoS & Cloudflare’s New CLI for AI Agents

Oct 1, 2026 16 min

This week on Ship It Weekly: AWS is retiring Amazon DevOps Guru and pointing customers toward CloudWatch and the newer Amazon DevOps Agent. Kubernetes disclosed a vulnerability where StatefulSet and ControllerRevision permissions can allow cross-namespace pod creation under specific conditions. A vulnerability in Undici can let a malicious WebSocket server crash a Node.js process through compressed data. And Cloudflare launched a new CLI as AI agents grow from 25 percent to 48 percent of Wrangler usage. The bigger theme this week is how the systems around our infrastructure are changing. Managed cloud services still have lifecycles that eventually become migration work. Kubernetes authorization can depend on what controllers do with the resources users are allowed to manipulate. Applications acting as clients still process untrusted data. And infrastructure tooling is starting to treat AI agents as first-class users rather than humans who happen to automate commands. In the lightning round: another Kubernetes vulnerability affecting Windows nodes can expose NetNTLMv2 credentials through NTLM coercion. GitHub now supports custom runners for Dependabot version and security updates. And external systems like a CMDB or internal developer portal can push repository properties into GitHub while remaining the source of truth. And the human closer comes from Lorin Hochstein and SRE Weekly. Some availability risks are probably never going away. Resources are finite, networks fail, security controls can affect availability, and production systems have to change. Preventing individual failures still matters, but incident response is part of reliability engineering too. Sometimes improving reliability means getting better at handling the failures you cannot eliminate. Links Amazon DevOps Guru End of Support - https://tsn.io/GQHN8 Kubernetes CVE-2026-2270: Cross-Namespace Pod Creation - https://tsn.io/BsNs8 Undici CVE-2026-85024: WebSocket Denial of Service - https://tsn.io/Lnc...

AWS Puts Elastic Beanstalk on EKS, CrowdSec Supply-Chain Breach, Critical Next.js RCE, Microsoft Disrupts EvilTokens & Why Fixing the Initial Compromise Isn’t Enough

Sep 25, 2026 17 min

This week on Ship It Weekly: AWS introduced Elastic Beanstalk Cluster Mode, allowing multiple applications to run on shared EKS infrastructure while AWS handles much of the Kubernetes complexity. CrowdSec published how a software supply-chain compromise led to attackers copying roughly 170 private repositories using a stolen OAuth token. A critical Next.js vulnerability in ImageResponse can lead to remote code execution through attacker-controlled SVG data. And Microsoft disrupted EvilTokens, a cybercrime platform linked to more than 12,000 compromised inboxes across 10,000 organizations. The bigger theme this week is what happens after trust has been established. Elastic Beanstalk Cluster Mode puts more infrastructure behind a managed abstraction, but shared infrastructure still means understanding isolation and blast radius. CrowdSec shows how an initial compromise can become a credential problem long after the malicious code is gone. Next.js shows how something as ordinary as generating a social preview image can expose a server-side execution path. And EvilTokens shows how attackers can use valid access to move faster once inside an account. In the lightning round: F5 has a critical BIG-IP APM vulnerability under active exploitation. GitHub Enterprise Cloud can now export an inventory of credentials with enterprise access, including PATs, SSH keys, OAuth tokens, and GitHub App credentials. Zyxel patched a vulnerability affecting GS1900 switches. And Veeam Agent for Microsoft Windows has a privilege-escalation vulnerability that can lead to SYSTEM access. And the human closer comes back to CrowdSec. Removing the malicious package, patching the server, or reimaging the workstation does not necessarily end the incident. If an attacker already stole an OAuth token, cloud credential, SSH key, session, or registry credential, that access can survive long after the original compromise is gone. Containment means understanding not only how the attacker got in, but what t...

GitHub Actions Security, Cisco Email Gateway RCE, Helm 3 End-of-Life, Ubuntu 26.04 Runners & Why “Nothing Changed” Is Never the Whole Story

Sep 19, 2026 15 min

This week on Ship It Weekly: GitHub Actions workflow execution protections are now generally available, giving organizations more control over who and what can trigger individual workflows. Cisco is patching critical vulnerabilities in Secure Email Gateway, including an actively exploited issue that can lead to remote command execution as root. Helm 3 has reached its final minor release and is heading toward end-of-life in February 2027. And GitHub’s ubuntu-latest Actions runner is preparing to move from Ubuntu 24.04 to 26.04. The bigger theme this week is infrastructure that changes even when your code does not. GitHub is making CI execution permissions more explicit, Helm teams now have a defined migration deadline, and the ubuntu-latest transition is a good example of how a completely unchanged workflow can suddenly be running in a different environment. Pinning everything forever is not necessarily the answer. The important part is knowing which dependencies are allowed to move and testing those changes deliberately. In the lightning round: GitHub Actions checks, workflow runs, and statuses will begin following your configured retention period on October 1. GitHub Advanced Security can now enforce configurations from the enterprise level. GitHub added API support for tracking when self-hosted Actions runner versions lose support. And AI Scan for pull requests can now be used without requiring CodeQL default setup. And the human closer starts with a sentence almost every infrastructure engineer has heard during an incident: “But nothing changed.” Maybe nothing changed in the application, but the runner image changed, a dependency moved, a certificate expired, DNS changed, or an external service behaved differently. Latest tags, loose version constraints, external APIs, and even support windows are dependencies. The goal is not to freeze everything forever. It is to avoid accidental mutability, where something can change without the team realizing it was ever allo...

Amazon Linux 2027, GitHub Actions Cache Security, Secret-Scanning Merge Blocks, N-central CVSS 10 RCE, Karmada Graduation, ShieldCrash, CodeQL ARM64 & When Observability Fails Too

Sep 12, 2026 14 min

This week on Ship It Weekly: Amazon Linux 2027 enters public preview with kernel 7.1+, SELinux enforcing by default, DNF5, newer language runtimes, AWS-LC, and an x86-64-v3 baseline. GitHub Actions adds explicit cache permissions to reduce cache-poisoning risk. GitHub can now block pull requests from merging when they introduce exposed secrets. And N-able N-central has a critical pre-auth RCE that Huntress says is being actively exploited in the wild. The bigger theme this week is catching problems before they turn into incidents. Amazon Linux 2027 gives teams time to test AMIs, bootstrap scripts, agents, Terraform, CloudFormation, and CI/CD before the next platform generation becomes production reality. GitHub’s new cache controls make workflow trust boundaries explicit instead of leaving them implied. And secret-scanning rulesets move credential detection directly into the merge path, where developers can actually act on it. In the lightning round: Karmada graduates from the CNCF as multi-cluster and distributed AI scheduling grow, ShieldCrash research claims another Microsoft Defender patch bypass with SYSTEM-level access, CodeQL 2.27 adds native Linux ARM64 support, and Dependabot can now read private GitHub Packages without another personal access token. And the human closer is about what happens when observability shares the same failure domain as the thing it is watching. A full disk is bad enough. It gets worse when logs stop writing, monitoring data disappears, and the tools used to diagnose the outage start failing too. The takeaway is not that every monitoring component needs total isolation. It is that you should know what can blind you, and make sure at least one useful signal survives the failures you care about most. Links Amazon Linux 2027 Public Preview https://tsn.io/NHlEa Amazon Linux 2027 Overview and Preview Details https://tsn.io/izDYx Amazon Linux 2027 Known Issues and Preview Limitations https://tsn.io/tdugd GitHub Actions Cache Permissions...

AWS GWLB TCP Reset, Azure DevOps Live Migrations to GitHub, GitHub Runner Enforcement, Docker Root Risk, Lambda IAM Updates, PostgreSQL Upgrade Traps, SonicWall Zero-Days & Better Incident Reviews

Sep 4, 2026 17 min

This week on Ship It Weekly: AWS Gateway Load Balancer gets TCP Reset, giving applications a faster way to recover when firewalls or other inline appliances fail instead of waiting minutes for TCP retries to time out. Microsoft puts Enterprise Live Migrations into public preview for moving Azure DevOps repositories to GitHub Enterprise Cloud with data residency while developers keep working. GitHub is beginning enforcement against outdated self-hosted Actions runners. And Omarchy fixes a Docker configuration that effectively gave normal desktop processes a path to root. The bigger theme this week is failure modes hiding inside infrastructure we already trust. A dead network path can look like a slow application. A repository migration involves far more than copying Git history. A self-hosted runner can quietly become unsupported while it continues looking healthy. And giving a developer access to the Docker socket may sound like convenience until you remember that the Docker group is effectively a root-level privilege. In the lightning round: Lambda gets full IAM resource-based policies, AWS warns that circular PostgreSQL role memberships can stall major RDS and Aurora upgrades, a researcher releases the FalconFlank CrowdStrike privilege-escalation PoC while CrowdStrike investigates, and SonicWall patches two SMA1000 zero-days after confirming active exploitation. Links AWS Gateway Load Balancer TCP Reset https://www.tellerstech.com/go/s-d7e609ab/ Azure DevOps Enterprise Live Migrations Public Preview https://www.tellerstech.com/go/s-ea05aff9/ GitHub Actions Self-Hosted Runner Minimum Version Enforcement https://www.tellerstech.com/go/s-6e8540c4/ Omarchy: Any User Process Can Escalate to Root https://www.tellerstech.com/go/s-d22971c3/ AWS Lambda Full IAM Resource-Based Policies https://www.tellerstech.com/go/s-ff2a04b5/ Fix Circular Role Dependencies Before Upgrading RDS and Aurora PostgreSQL https://www.tellerstech.com/go/s-e4578f52/ FalconFlank CrowdStrike Privile...

Cloudflare Saves 100TB of RAM, AI Drives Server Prices Up, AWS Adds a Fourth London AZ, Route 53 DNS Self-Service, AKS eBPF Routing, Go 1.27, and the Danger of Hidden Infrastructure Assumptions

Aug 29, 2026 16 min

This week on Ship It Weekly: Cloudflare explains how five low-level optimizations to the cache behind 1.1.1.1 freed roughly 100 terabytes of RAM while also improving performance. OVHcloud is raising infrastructure prices as AI demand reshapes the memory supply chain. AWS adds a fourth Availability Zone to London, exposing automation that quietly assumed there would always be three. And Route 53 Global Resolver gets a cleaner cross-account model for DNS self-service. The bigger theme this week is assumptions. A few wasted bytes do not matter until you have 250 billion cache entries. A Region having three Availability Zones feels permanent until AWS adds a fourth. And centralized DNS governance works fine until every application team needs a networking ticket just to make a private zone resolvable. In the lightning round: new research looks at manipulating DRAM controller translation registers and the assumptions that creates for memory isolation, AKS eBPF Host Routing reaches general availability, CloudFront Functions can now put custom context directly into access logs, and Go 1.27 lands generic methods along with runtime, tooling, and standard-library improvements. And the human closer looks at an easy Kubernetes mistake: running kubectl against the wrong cluster. Because the active context belongs to the kubeconfig rather than a terminal tab, changing it in one shell can silently affect another. It is a good reminder that some friction is worth keeping around production, and that the safest guardrails live somewhere stronger than operator memory. Links Cloudflare: How We Saved 100 Terabytes of Memory by Optimizing 1.1.1.1’s DNS Cache https://www.tellerstech.com/go/s-9d6c2943/ OVHcloud Raises Prices as AI Memory Demand Reprices Non-AI Infrastructure https://tsn.io/tnaYj AWS Adds a Fourth Availability Zone to Europe (London) https://tsn.io/YfGxa Shared DNS Views with Amazon Route 53 Global Resolver https://tsn.io/WEyig DRAM Controller Register Manipulation Breaks CP...

Ship It Conversations: Justin Garrison of Sidero Labs on Kubernetes, Platform Engineering, AI, Golden Paths, and Knowing What to Say No To

Aug 24, 2026 41 min

This is a guest conversation episode of Ship It Weekly, separate from the weekly news recaps. In this Ship It Conversations episode, I talk with Justin Garrison of Sidero Labs about Kubernetes, platform engineering, bare metal, AI, golden paths, and why knowing what to say no to may be one of the most important skills a platform team can develop. Justin is Field CTO at Sidero Labs, the company behind Talos Linux, and co-host of Fork Around and Find Out. We start with the evolution of Kubernetes and how managed services like EKS and GKE made Kubernetes easier to consume while also pulling teams deeper into proprietary cloud ecosystems. Justin explains why on-prem and bare metal are getting renewed attention, especially as teams look at cloud costs, data sovereignty, and the operational overhead that comes with constantly optimizing cloud environments. We also get into where Kubernetes helps and where it becomes self-inflicted pain. Justin talks about abstraction, cognitive load, and why teams tend to use familiar tools for problems they were never really designed to solve. A big part of the conversation is platform engineering and golden paths. Justin argues that every organization needs its own path, but platforms become dangerous when they try to centralize everything. He shares why one of the best decisions his team made at Disney Plus was simply saying no to stateful workloads. We also talk about what really belongs in a platform: security controls, logging, monitoring, software supply chain visibility, and cost management. Justin explains why centralization can help in those areas, but can become a bottleneck when applied too broadly. Near the end, we get into AI, security, tooling dependency, and engineering culture. Justin makes the point that people have always formed strong attachments to tools, and AI is another version of that. The challenge is knowing where AI actually helps versus where it becomes another dependency teams stop questioning. The big takeaw...

GitHub Outage, PleaseFix Agentic Browser Vulnerability, AWS Certificate Manager Drops Email Validation, Cloudflare TypeScript CI Workflows, AI Observability Consolidation, and the Hidden Cost of “Simple” Platform Changes

Aug 21, 2026 17 min

This week on Ship It Weekly: GitHub suffers another widespread outage affecting the web interface, APIs, Actions, authentication, Copilot, and other critical developer workflows. Zenity Labs demonstrates PleaseFix attacks against agentic browsers, where malicious content can influence agents with access to authenticated sessions and privileged tools. AWS Certificate Manager is moving away from email validation, and Cloudflare is experimenting with CI pipelines defined as TypeScript instead of YAML. The bigger theme this week is dependencies and boundaries we tend to ignore until something breaks. GitHub is no longer just where the code lives. Agentic browsers are no longer just displaying webpages. Certificate renewal is not something you want depending on someone checking an inbox. And CI pipelines have become software systems of their own. Links GitHub Hit by Widespread Outage https://devops.com/github-hit-by-widespread-outage-halting-work-for-global-developers/ Zenity Labs: PleaseFix in Agentic Browsers https://zenity.io/company-overview/newsroom/company-news/zenity-labs-exposes-the-full-scope-of-pleasefix AWS Certificate Manager Ending Email Validation https://aws.amazon.com/blogs/security/aws-certificate-manager-will-discontinue-email-validation-to-prove-domain-validation-for-certificates/ Certificate Expiry Is Still Taking Down Major Platforms https://tokentimer.ch/blog/tls-certificate-expiry-outages Cloudflare Turns CI Pipelines into TypeScript Workflows https://www.infoq.com/news/2026/08/cloudflare-ci-code-workflows/ Dynatrace Acquires Arize https://devops.com/dynatrace-acquires-arize-as-ai-agents-deepen-the-observability-challenge/ AWS Open-Sources Dogwood https://www.infoq.com/news/2026/08/aws-dogwood-agent-policy/ Pulumi v3.258.0 https://github.com/pulumi/pulumi/releases/tag/v3.258.0 AWS Key Breach and Data-Transfer Signal https://assets.theregister.com/2026/08/13/20267/ Mario Saved the EU but Broke My System https://www.uptimelabs.io/articles/hamed-2012-...

Ship It Conversations: Ned Bellavance of Ned in the Cloud on DevOps Beyond the Buzzwords, Terraform, AI, the Future of Infrastructure as Code, and Why Fundamentals Still Matter

Aug 16, 2026 36 min

This is a guest conversation episode of Ship It Weekly, separate from the weekly news recaps. In this Ship It Conversations episode, I talk with Ned Bellavance of Ned in the Cloud about DevOps beyond the buzzwords, platform engineering, infrastructure as code, AI, and why fundamentals still matter even as the tools change. Ned is the founder of Ned in the Cloud and host of the Day 2 DevOps podcast, with more than 20 years in IT across systems administration, cloud, architecture, automation, and technical education. We start with a problem a lot of teams run into: adopting the ceremonies of DevOps without actually adopting the principles. Standups, sprints, pipelines, and tooling can make an organization look mature, but the real goal is better communication, faster feedback loops, and delivery tied to actual outcomes. We also talk about how teams decide what to prioritize next. Security, reliability, performance, FinOps, and platform work can all matter, but chasing whatever is newest does not help if the basics are still broken. A big part of the conversation is where infrastructure as code goes from here. We get into Terraform's state and scaling model, API rate limits, the Terraform/OpenTofu split, Terragrunt, and newer approaches like Swamp from System Initiative. AI is making infrastructure code cheaper to produce, but understanding the architecture behind that code is becoming more valuable. That leads into learning and career development. We talk about why networking, Linux, databases, security, cloud architecture, and troubleshooting still matter, even if an LLM writes most of the syntax. Build things, get them wrong in controlled environments, troubleshoot them, and learn what is happening underneath the abstraction. The big takeaway: tools will keep changing. Judgment, architecture, troubleshooting, and understanding the systems underneath them are much harder to automate away. Highlights • Why DevOps ceremony is not the same as DevOps principles • How tea...

Railway US East Outage, Stripe’s Graph-Based Database Recovery, Kata Containers Host Escape, DynamoDB Vector Search, AWS Network Firewall Proxy, Gateway API 1.6, and containerd 2.4

Aug 14, 2026 15 min

This week on Ship It Weekly: Railway explains how an upstream network problem turned into a much larger US East outage, including storage traffic falling back onto the management network and stale connections continuing to cause problems after routing recovered. Stripe shares how graph search and state machines helped cut database pager volume by about 30 percent. Kata Containers patches a critical guest-to-host escape, and DynamoDB adds native vector search. The bigger theme this week is what happens after the obvious failure. Fixing the route does not necessarily clear the connections created while it was broken. Automating recovery does not have to mean handing an AI agent unrestricted production access. And stronger isolation does not eliminate the components that still cross the guest-host boundary. In the lightning round: AWS brings explicit forward proxy functionality back through Network Firewall, Gateway API 1.6 moves TCPRoute and UDPRoute to stable, and containerd 2.4 enters beta with new functionality alongside breaking changes worth finding before your next runtime upgrade. Links Railway: July 2, 2026 US East Services Outage https://blog.railway.com/p/incident-report-july-2-2026-us-east-services-outage Stripe: How Stripe uses graph search and state machines to auto-remediate a global database fleet https://stripe.dev/blog/how-stripe-uses-graph-search-and-state-machines-to-auto-remediate-a-global-database-fleet Kata Containers: Guest-root to host-root escape via virtiofs https://github.com/kata-containers/kata-containers/security/advisories/GHSA-2gv2-cffp-j227 Amazon DynamoDB now supports real-time vector search https://aws.amazon.com/about-aws/whats-new/2026/08/amazon-dynamodb-vector-search/ AWS Network Firewall forward proxy preview https://aws.amazon.com/about-aws/whats-new/2026/08/aws-network-firewall-forward-proxy-preview/ Gateway API v1.6: TCPRoute and UDPRoute Graduate to Standard https://kubernetes.io/blog/2026/08/03/gateway-api-v1-6-release/ con...

AI Agents Target GitHub, Kubernetes 1.37 Deprecations, AWS Transit Gateway Policy Routing, IAM Identity Center Multi-Region, Cloudflare Meerkat, AMOS Mac Malware, and the Risk of Half-Migrated Production Systems

Aug 7, 2026 18 min

This week on Ship It Weekly: AI agents from Anthropic and OpenAI took unsanctioned actions on the real internet during UK government cyber testing, including an attempt to push malicious code into a real GitHub project. Kubernetes 1.37 starts retiring IPVS mode, pushes cgroup v1 closer to removal, and brings an SELinux volume change worth testing before upgrades. AWS Transit Gateway gets policy-based routing, and IAM Identity Center expands multi-Region support to organizations using AWS’s built-in directory. The bigger theme: access is not the same thing as authority, and availability is not just about whether your application is running. Agents need boundaries around the actions they can take. Routing policies need enough visibility to explain why traffic went where it did. And regional resilience does not help much if the people responding to the outage cannot authenticate. In the lightning round: Cloudflare’s Meerkat consensus system, AMOS macOS malware, N-able’s incomplete N-central fix, and an AWS CLI bug that disabled SSH host-key verification. Links AISI: Unsanctioned agent behaviour during cyber testing https://www.aisi.gov.uk/blog/incident-report-unsanctioned-agent-behaviour-during-cyber-testing Kubernetes v1.37 Sneak Peek https://kubernetes.io/blog/2026/07/31/kubernetes-v1-37-sneak-peek/ AWS Transit Gateway Policy-Based Routing https://aws.amazon.com/about-aws/whats-new/2026/07/aws-transit-gateway-policy-based-routing/ AWS IAM Identity Center multi-Region directory support https://aws.amazon.com/about-aws/whats-new/2026/07/aws-iam-identity-center-extends-multi-region-support-to-identity-center-directory Cloudflare: Introducing Meerkat https://blog.cloudflare.com/meerkat-introduction/ Atomic macOS / AMOS stealer infection https://isc.sans.edu/diary/rss/33208 N-able N-central exploitation after incomplete fix https://thehackernews.com/2026/08/n-able-says-attackers-take-over-n.html CVE-2026-18654: AWS CLI EMR SSH host-key verification https://aws.amazon.com/...

Telstra’s Time Sync Outage, DoorDash’s 1.5M RPS Cache, Stateless MCP, GitHub and PyPI Supply Chain Delays, and Why the Quietest Infrastructure Often Has the Biggest Blast Radius

Jul 31, 2026 16 min

This week on Ship It Weekly: Telstra’s mobile network jumped back to 2006 after a timing device restarted with the wrong date, disrupting calls, data sessions, and hundreds of emergency calls. DoorDash explains how Entity Cache, built with Envoy and Valkey, handles more than 1.5 million requests per second and uses stale-data policies, invalidation, and fallback behavior as a reliability layer. The latest MCP release candidate removes protocol-level sessions, making servers easier to scale behind ordinary load balancers while leaving teams responsible for authentication, tracing, retries, rate limits, and application state. GitHub and PyPI are also adding friction to package automation. Dependabot now delays routine updates by three days, while PyPI blocks new files from releases older than 14 days. Links Telstra outage https://www.telstra.com.au/exchange/our-mobile-network-outage-has-been-resolved-heres-what-happened DoorDash Entity Cache https://careersatdoordash.com/blog/high-performance-proxy-cache-for-doordash-services/ MCP specification release candidate https://blog.modelcontextprotocol.io/posts/2026-07-28-release-candidate/ Dependabot package cooldown https://github.blog/changelog/2026-07-14-dependabot-version-updates-introduce-default-package-cooldown/ PyPI release-file restrictions https://blog.pypi.org/posts/2026-07-22-releases-now-reject-new-files-after-14-days/ Amazon ECS Action Logs https://aws.amazon.com/about-aws/whats-new/2026/07/amazon-ecs-action-logs/ Network Load Balancer listener rules https://aws.amazon.com/about-aws/whats-new/2026/07/aws-network-load-balancer-supports-listener-rules/ Amazon Managed Prometheus limits https://aws.amazon.com/about-aws/whats-new/2026/07/amazon-managed-service-prometheus-1500m-metrics-workspace/ PixelSmash in FFmpeg https://jfrog.com/blog/pixelsmash-critical-ffmpeg-vulnerability-turns-media-files-into-weapons/ SRE Weekly Issue 527 https://sreweekly.com/sre-weekly-issue-527/ This week’s On Call Brief https://www.tel...

Ship It Conversations: Jay Lark of Hookbridge on Webhook Reliability, Retries, Idempotency, Replay, HMAC Security, Local Testing, and Operating Webhooks in Production

Jul 27, 2026 31 min

This is a guest conversation episode of Ship It Weekly, separate from the weekly news recaps. In this Ship It Conversations episode, I talk with Jay Lark of Hookbridge about webhook reliability, retries, idempotency, replay, security, local development, and what happens when a simple HTTP POST becomes production infrastructure. Jay is a Principal DevOps Engineer and the founder of Hookbridge, a service focused on making webhook delivery more reliable and easier to operate. We start with the basic model: one service sends an HTTP POST to another when something happens. The happy path is easy. The problems begin when an endpoint is unavailable, responds too slowly, receives duplicate events, or gets them out of order. Jay explains what “at least once delivery” means and why receivers must expect duplicates. We talk about idempotency, event IDs, retry behavior, availability during deployments, and why returning a 200 does not prove downstream processing succeeded. We also dig into observability and security. Teams need enough visibility to know whether a webhook arrived, whether signature verification passed, what response was returned, and where processing failed. Jay breaks down HMAC signatures, timestamps, replay protection, and why a valid signature still does not replace normal business-logic validation. Local development is another source of friction. External providers cannot send events directly to localhost, so developers often rely on temporary tunnels, staging deployments, copied payloads, or mocks. Jay explains how Hookbridge uses a fixed URL and local client to forward real webhook traffic to a developer’s machine. We also talk about n8n, self-hosted OpenClaw systems, Hookbridge pull endpoints, and when polling may be simpler or safer than exposing another inbound endpoint. The big takeaway: design the failure path before a webhook becomes business-critical. Verify the sender, expect duplicate and out-of-order events, build enough visibility to debug failu...

AWS CloudFormation Express Mode, Spark 4.2 Vector Search, AI Speeds Coding but Not Delivery, Platform Governance Developers Won’t Hate, and Why Could Is Not the Same as Should

Jul 24, 2026 18 min

This week on Ship It Weekly: AWS CloudFormation Express mode promises faster infrastructure feedback by reporting deployments complete before extended resource stabilization finishes. Apache Spark 4.2 adds native vector operations and nearest-neighbor joins, giving some teams a way to keep AI data workloads closer to the platforms they already run. GitLab’s latest research says AI is helping developers generate and commit code faster, but review, testing, governance, and deployment are not accelerating at the same pace. Then, a platform engineering case study from Sevdesk shows how minimum viable governance, useful feedback, and progressive enforcement can improve compliance without turning the platform team into another approval queue. The theme this week: making one part of the system faster does not automatically improve the whole system. In the lightning round: OpenShift 4.22.5 receives an important security update, outdated autoscaling thresholds disrupt GitHub Actions, Cloudflare experiences an incident where some POST requests fail to reach customer origins, and GitHub Code Quality becomes generally available—with automatic billing attached. The episode closes with Reid Savage reflecting on their first year managing Honeycomb’s SRE team and the difference between what a capable team could do and what it should do. Links CloudFormation Express mode https://aws.amazon.com/blogs/aws/accelerate-your-infrastructure-deployments-by-up-to-4x-with-aws-cloudformation-express-mode/ Apache Spark 4.2 https://spark.apache.org/releases/spark-release-4-2-0.html GitLab AI Accountability Report https://about.gitlab.com/resources/ai-accountability-survey-2026/ Platform governance at Sevdesk https://www.infoq.com/presentations/platform-engineering-team-compliance/ OpenShift 4.22.5 security update https://access.redhat.com/errata/RHSA-2026%3A37585 GitHub Actions incident https://github.com/orgs/community/discussions/201795 Cloudflare status https://www.cloudflarestatus.com/ GitHu...

Ship It Conversations: Mat Ryer of Grafana Labs on AI Observability, Agents, Evals, and Operating AI in Production

Jul 20, 2026 45 min

This is a guest conversation episode of Ship It Weekly, separate from the weekly news recaps. In this Ship It Conversations episode, I talk with Mat Ryer of Grafana Labs about AI observability, production agents, evals, telemetry cost, guardrails, and what changes once AI moves beyond demos and into systems teams actually depend on. Mat is Senior Director of AI at Grafana Labs, where he focuses on how AI fits into observability and production systems. We talk about Grafana Assistant, why AI observability is not just logs, latency, and HTTP 200s, and how teams can measure whether agents are actually helping. Mat gets into evals, LLM-as-judge patterns, traces as a way to think about conversations, user feedback, tool choice, model changes, and the cost of collecting new telemetry. We also dig into UX and trust. If an AI assistant gives you a wall of text, you still have to decide whether to believe it. If it can show the graph, deep link into Grafana, apply filters, and expose the source data, that becomes a much more useful operating experience. The big takeaway: start small, enhance workflows you already have, build feedback loops, and treat production AI like something you actually have to operate. Highlights • Why AI demos are easy, but production AI is harder • Why agents need observability, evals, and guardrails • How Grafana thinks about AI Assistant and AI observability • Where LLM-as-judge patterns, traces, tool calls, and feedback fit • Why telemetry cost problems may repeat with AI workloads • Why UX matters when operators need to trust the answer • Where AI can help SRE and platform teams today Links Mat Ryer on LinkedIn: https://www.linkedin.com/in/matryer/ Mat Ryer on GitHub: https://github.com/matryer Grafana Labs: https://grafana.com Grafana Assistant: https://grafana.com/products/cloud/ai-assistant/ Grafana AI Observability: https://grafana.com/docs/grafana-cloud/machine-learning/ai-observability/ Grafana Adaptive Telemetry: https://grafana.com/produc...

GitHub API Enumeration, Grok Build CLI Data Exposure, AWS Security Hub Network Scanning, AI-Powered Patch Pressure, and Why Visibility Is Not Ownership

Jul 18, 2026 19 min

This week on Ship It Weekly: Datadog tracked coordinated GitHub API enumeration, xAI’s Grok Build CLI reportedly uploaded repo data without redaction, AWS Security Hub added Network Scanning and exposure impact analysis, and Microsoft says AI-powered vulnerability discovery is changing patch pressure. The theme: visibility is not ownership. A GitHub API map does not revoke a token. An exposure finding does not close a port. A patch bulletin does not patch the fleet. And an AI coding tool reading your repo is still access. Brian covers GitHub as a production surface, AI coding tool data boundaries, cloud exposure based on reachability and blast radius, and why patching needs to look more like production operations than spreadsheet theater. Also, the Ship It Weekly shop is open at shop.tellerstech.com with Ship It Weekly t-shirt designs. Use coupon code SHIPTHESTORE for 20% off your order for the next few weeks. Links Datadog: Coordinated GitHub API enumeration https://securitylabs.datadoghq.com/articles/coordinated-github-api-enumeration/ The Verge: Grok Build CLI repository upload report https://www.theverge.com/ai-artificial-intelligence/965600/spacexai-grok-build-repository-upload AWS Security Hub Network Scanning https://aws.amazon.com/about-aws/whats-new/2026/07/aws-security-hub-network-scanning/ AWS Security Hub impact analysis for exposure findings https://aws.amazon.com/about-aws/whats-new/2026/07/impact-analysis-aws-security-hub/ Microsoft: Windows vulnerability management and AI-powered discovery https://blogs.windows.com/windowsexperience/2026/07/09/evolving-windows-vulnerability-management-to-meet-the-speed-of-ai-powered-discovery/ SRE Weekly Issue 525 https://sreweekly.com/sre-weekly-issue-525/ HalluSquatting / hallucinated package risk https://www.endorlabs.com/learn/slopsquatting-when-ai-agents-hallucinate-malicious-packages AWS Lambda Managed Instances for Java cold starts https://aws.amazon.com/blogs/compute/eliminating-java-cold-starts-with-aws-lamb...

EKS Rollbacks, GitHub Actions Supply Chain Attacks, AI Agentjacking, CloudWatch Log Alarms, and Why Safety Nets Don’t Replace Ownership

Jul 10, 2026 19 min

This week on Ship It Weekly: Amazon EKS added Kubernetes version rollbacks, Novee Security published Cordyceps research on GitHub Actions supply chain risk, Tenet Security showed how fake telemetry can hijack AI coding agents, and Amazon CloudWatch added alarms directly from log queries. The theme: safety nets are getting better, but the blast radius is getting wider. Rollback buttons, log alarms, zone-aware routing, secret scanning, and AI agent workflows all help, but they do not replace ownership. Brian covers why EKS rollbacks are useful but not a substitute for real upgrade discipline, why GitHub Actions YAML is production code with credentials, how fake Sentry telemetry can become hostile agent context, and why easier log-based alarms can also mean easier pager noise. In the lightning round: ECS Service Connect zone-aware routing, etcd 3.7, GitHub innersource advisories, secret scanning metadata improvements, and CloudWatch Application Signals service events. Links Amazon EKS Kubernetes version rollbacks https://aws.amazon.com/blogs/aws/upgrade-amazon-eks-clusters-with-confidence-using-kubernetes-version-rollbacks/ Novee Security: Cordyceps supply chain research https://novee.security/blog/cordyceps/ Tenet Security: Agentjacking through fake Sentry errors https://tenetsecurity.ai/blog/agentjacking-coding-agents-with-fake-sentry-errors/ Amazon CloudWatch log query alarms https://aws.amazon.com/about-aws/whats-new/2026/07/amazon-cloudwatch-log-alarms/ ECS Service Connect zone-aware routing https://aws.amazon.com/about-aws/whats-new/2026/07/ecs-service-connect-zone-aware/ etcd 3.7 announcement https://etcd.io/blog/2026/announcing-etcd-3.7/ GitHub innersource security advisories https://github.blog/changelog/2026-07-08-innersource-security-advisories-are-generally-available/ GitHub secret scanning extended metadata and multipart validation https://github.blog/changelog/2026-07-07-secret-scanning-extended-metadata-and-multipart-validation/ CloudWatch Application Si...

Ship It Conversations: Evan Phoenix of Miren on Deployment Pain, Terraform, Waypoint, and Better Defaults for Small Teams

Jul 6, 2026 41 min

This is a guest conversation episode of Ship It Weekly, separate from the weekly news recaps. In this Ship It: Conversations episode, I talk with Evan Phoenix of Miren about why deployment is still painful, what teams keep getting wrong when they try to simplify it, and why small teams may need better defaults more than more platform knobs. Evan is the CEO of Miren. He previously worked on Terraform Enterprise and Waypoint at HashiCorp, and he also built Puma and Rubinius. We talk about deployment as the “final boss” of software delivery. Not because teams do not know how to ship code, but because deployment is where everything collides: runtimes, registries, secrets, networking, cloud services, databases, rollbacks, and the internal platform nobody wants to touch anymore. A big theme is opinionated tooling. Engineers often say they want flexibility, but many teams are really asking for good defaults, a clear happy path, and fewer decisions to own. We also get into Terraform Enterprise, Terraform Cloud, Terragrunt, OpenTofu, Waypoint, Kubernetes, Heroku, ECS, container registries, and how AI changes the deployment conversation. AI can generate infrastructure code, but when that setup breaks, someone still has to understand it and be on call for it. Highlights • Why deployment is still painful after years of platforms and abstractions • What Evan learned from Terraform Enterprise and Waypoint • Why Terraform structure, state, modules, and repo layout remain hard • Why OpenTofu gained traction beyond the Terraform licensing change • Why Kubernetes can be too much surface area for some teams • What small teams actually need from deployment tooling • How AI changes infrastructure and deployment workflows • Why generated infrastructure still needs ownership and accountability Links • Miren: https://miren.dev • Miren on GitHub: https://github.com/mirendev • Evan Phoenix: https://evanphx.dev • Evan on Bluesky: https://bsky.app/profile/evanphx.dev • Evan on Linkedin: https:...

Amazon Q CVEs, Hijacked npm and Go Packages, AWS WAF HTTP/2 Issues, Lambda MicroVMs, and Why Execution Is the Boundary Now

Jul 3, 2026 18 min

This week on Ship It Weekly: Amazon Q Developer and the AWS language servers had a pair of trust-boundary CVEs, JFrog found hijacked npm and Go packages using hidden VS Code tasks to run malware when a workspace opens, AWS WAF had HTTP/2 request-body inspection issues, and AWS introduced Lambda MicroVMs for running user-generated and AI-generated code in isolated sandboxes. The bigger theme: execution is the boundary now. The repo, the IDE, the AI assistant, the WAF, and the sandbox all sit at the point where something gets to run, inspect, block, or decide. Before execution, trust is a policy. After execution, trust is a blast radius. In the lightning round, Brian covers GitHub’s record advisory volume, Git 2.55, Valkey 9.1 on Amazon ElastiCache, and a quick Fable 5 callback now that Anthropic’s Fable 5 is back online. Links AWS security bulletin: Amazon Q / AWS language server CVEs https://aws.amazon.com/security/security-bulletins/2026-047-aws/ JFrog: Hijacked npm packages using VS Code tasks https://research.jfrog.com/post/hijacked-npm-vscode-tasks-blockchain/ AWS security bulletin: AWS WAF HTTP/2 inspection issues https://aws.amazon.com/security/security-bulletins/2026-048-aws/ AWS Lambda MicroVMs https://aws.amazon.com/blogs/aws/run-isolated-sandboxes-with-full-lifecycle-control-aws-lambda-introduces-microvms/ GitHub Advisory Database record volume https://github.blog/security/supply-chain-security/inside-the-advisory-database-and-what-happens-when-vulnerability-volume-breaks-records/ Git 2.55 highlights https://github.blog/open-source/git/highlights-from-git-2-55/ Amazon ElastiCache Valkey 9.1 https://aws.amazon.com/blogs/database/announcing-valkey-9-1-for-amazon-elasticache/ Claude Fable 5 and Mythos 5 model docs https://platform.claude.com/docs/en/about-claude/models/introducing-claude-fable-5-and-claude-mythos-5 This week’s On Call Brief https://www.tellerstech.com/on-call-brief-news/2026-W27/ More episodes and full show notes https://shipitweekly.fm/

Ship It Conversations: Kat Traxler of Vectra AI on AI Security, the Zero-Day Clock, IAM, and Cloud Risk

Jun 28, 2026 42 min

This is a guest conversation episode of Ship It Weekly, separate from the weekly news recaps. In this Ship It: Conversations episode, I talk with Kat Traxler of Vectra AI about AI security, the zero-day clock, IAM, cloud risk, AI-assisted bug hunting, and why the scariest future security problems may still start with the boring fundamentals teams already struggle with today. Kat is a Principal Security Researcher at Vectra AI focused on abuse techniques and vulnerabilities in the public cloud, especially around the intersection of cloud security, AppSec, IAM, managed identities, and insecure-by-design flaws. We talk about the current AI security mood, from the excitement around faster research and bug hunting to the fear that AI could shrink the window between vulnerability disclosure and exploitation. Kat explains the “San Francisco Consensus,” why the zero-day clock is getting so much attention, and why she thinks the facts may be real while some of the conclusions are overextended. The bigger theme here is that AI is absolutely changing security work, but it does not erase the fundamentals. Attackers still take the lowest-friction path that works. For most teams, that still means credentials, IAM, misconfigurations, known vulnerabilities, and systems that were never threat-modeled as deeply as people assume. Highlights • Why AI security feels exciting and unsettling at the same time • What the “San Francisco Consensus” means and why people are talking about the zero-day clock • How AI may shrink the time between vulnerability disclosure and exploitation • Why Kat is skeptical of the full “zero-day apocalypse” narrative • Why credentials, IAM, misconfigurations, and known vulnerabilities still matter most for many teams • How AI helps narrow the search space in bug hunting and security research • Where AI is useful for code-level bugs, and where it still struggles with context and threat modeling • Why human expertise still matters when using AI for writing, resea...

containerd CRI Vulnerabilities, Datadog PostgreSQL HA on Kubernetes, AWS DevOps Agent with Datadog MCP Server, EKS Control Plane Egress, and Why Users Feel the Wait

Jun 26, 2026 19 min

This week on Ship It Weekly: containerd disclosed a batch of CRI plugin vulnerabilities, Datadog tested PostgreSQL high availability on Kubernetes and found that failover is not useful if it cannot happen safely, AWS DevOps Agent and Datadog MCP Server moved AI incident response closer to real production workflows, and Amazon EKS added customer-routed control-plane egress. The bigger theme: the control plane keeps getting wider. Runtimes, databases, incident agents, API-server egress, credentials, the cloud console, and object metadata are all becoming part of the production blast radius. And when something breaks, users do not experience your architecture diagram. They experience waiting. In the lightning round, Brian covers GitHub self-service credential revocation for incident response, AWS Management Console Private Access without internet connectivity, Vercel Connect and short-lived agent credentials, and Amazon S3 annotations. Links containerd CRI plugin vulnerabilities / AWS security bulletin https://aws.amazon.com/security/security-bulletins/2026-046-aws/ Datadog: PostgreSQL high availability on Kubernetes https://www.datadoghq.com/blog/engineering/postgresql-ha-kubernetes/ AWS DevOps Agent and Datadog MCP Server https://aws.amazon.com/blogs/devops/production-ready-autonomous-incident-resolution-with-aws-devops-agent-now-ga-and-datadog-mcp-server/ Amazon EKS customer-routed control-plane egress https://aws.amazon.com/blogs/containers/amazon-eks-now-supports-control-plane-egress-through-your-vpc/ GitHub self-service credential revocation for incident response https://github.blog/changelog/2026-06-24-self-service-credential-revocation-for-incident-response/ AWS Management Console Private Access https://aws.amazon.com/about-aws/whats-new/2026/06/aws-management-console-private/ Vercel Connect https://vercel.com/blog/introducing-vercel-connect Amazon S3 annotations https://aws.amazon.com/blogs/aws/amazon-s3-annotations-attach-rich-queryable-context-directly-to-yo...

Ship It Conversations: Guardsquare’s Joel DeStefano on Mobile App Security, Runtime Protection, App Hardening, and Why Scanning Isn’t Enough

Jun 21, 2026 35 min

This is a guest conversation episode of Ship It Weekly, separate from the weekly news recaps. In this Ship It: Conversations episode, I talk with Joel DeStefano from Guardsquare about mobile app security, why it is different from backend and cloud security, and why scanning alone is not enough once an app is shipped into the real world. We talk about the shift in trust model that happens with mobile apps. In backend and cloud systems, teams usually have more control over the runtime, infrastructure, policies, and monitoring. With mobile, the app becomes a public artifact running on someone else’s device, in an environment you do not fully control. The bigger theme here is that mobile security is not just “scan it before release.” Scanning matters, but teams also need to think about app hardening, obfuscation, runtime protection, monitoring, and whether the app connecting back to their APIs is genuine and uncompromised. Highlights • Why mobile changes the trust model compared to backend and cloud systems • What DevOps, SRE, and platform teams should understand about mobile app risk • Why scanning is useful, but not enough by itself • The danger of assuming app store approval means an app is secure • Why “we do not store sensitive data in the app” can be a misleading security argument • How attackers can reverse engineer apps, inspect workflows, and learn how the app talks to backend APIs • What code hardening and obfuscation actually help protect against • Why runtime checks matter for rooted devices, compromised environments, debuggers, hooking frameworks, overlays, and accessibility abuse • The difference between Android and iOS security assumptions • Why the OS is not responsible for protecting your app’s business logic • How mobile security should fit into CI/CD without destroying release velocity • What should block a release versus what should become tracked risk • Why testing, hardening, runtime protection, and monitoring should work together as one strategy •...

PeopleSoft Zero-Day Exploited, npm v12 Install Script Changes, GitHub Agentic Tokens, Anthropic Model Risk, and Default Trust Breaking

Jun 19, 2026 22 min

This episode of Ship It Weekly is about default trust getting punished. Brian covers Oracle’s emergency PeopleSoft advisory for CVE-2026-35273, npm v12 changing install-script defaults, GitHub Agentic Workflows moving away from long-lived personal access tokens, and Anthropic disabling Fable 5 and Mythos 5 after a U.S. export-control directive. The common thread: legacy ERP systems, package installs, CI/CD agents, and AI models all become production risks when teams trust the default without checking what that trust can actually do. In the lightning round, Brian covers Tekton CloudEvents moving to a dedicated events controller, NVIDIA Triton Inference Server 26.04 changing inference defaults, AWS Nitro Isolation Engine bringing formal verification to Graviton5-based isolation, and Homebrew 6.0 adding explicit trust for third-party taps. The bigger theme: production does not care why you trusted the default. It only cares what that default was allowed to do. The bigger theme: production does not care why you trusted the default. It only cares what that default was allowed to do. Links Oracle PeopleSoft CVE-2026-35273 advisory https://www.oracle.com/security-alerts/alert-cve-2026-35273.html npm v12 breaking changes https://github.blog/changelog/2026-06-09-upcoming-breaking-changes-for-npm-v12/ GitHub Agentic Workflows no longer need PATs https://github.blog/changelog/2026-06-11-agentic-workflows-no-longer-need-a-personal-access-token/ Anthropic Fable 5 / Mythos 5 access statement https://www.anthropic.com/news/fable-mythos-access Tekton Pipelines releases https://github.com/tektoncd/pipeline/releases NVIDIA Triton Inference Server 26.04 release notes https://docs.nvidia.com/deeplearning/triton-inference-server/release-notes/rel-26-04.html AWS Nitro Isolation Engine https://aws.amazon.com/blogs/compute/aws-nitro-isolation-engine-formally-verifying-the-hypervisor-in-the-aws-nitro-system/ Homebrew 6.0.0 https://brew.sh/2026/06/11/homebrew-6.0.0/ This week’s On Call Brief...

Ship It Conversations: Meta’s Francois Richard on AI Incident Response, SLOs, and Reliability at Scale

Jun 16, 2026 42 min

This is a guest conversation episode of Ship It Weekly, separate from the weekly news recaps. In this Ship It: Conversations episode, I talk with Francois Richard, Engineering Director at Meta, about reliability at scale, how AI is changing production risk, what teams actually learn from incidents, and why recovery practice matters just as much as prevention. We talk about the proactive and reactive sides of reliability, why SLOs should represent a promise to users instead of just another dashboard number, how incident reviews should drive real system improvements, and how teams can practice recovery before production forces the lesson on them. The bigger theme here is that reliability is not just about avoiding failure. It is about knowing what happens when prevention fails. That means practicing regional failure, understanding overload behavior, improving incident response, using AI carefully during investigation, and making reliability targets match the actual lifecycle and importance of the system. Highlights • Why reliability work starts with both prevention and recovery • The difference between reactive incident response and proactive reliability engineering • How Meta thinks about disaster recovery testing and regional failure practice • Why an SLO should be treated like a promise to users, not just a dashboard metric • How SLO trends help teams decide when to invest more in reliability or take more product risk • What engineers actually learn during the “pressure cooker” of an incident • Why incident reviews should produce follow-up work, not just a nicer explanation of what broke • The difference between finding the cause of an incident and improving the system • Where AI agents can help with incident investigation, telemetry, metrics, and query building • Why AI-generated code can increase change volume while reducing human context • How faster code generation changes the kinds of reliability problems teams should expect • Why recovery practice matters, es...

Coinbase Outage, Meta AI Account Recovery, AWS AgentCore Code Injection, Apigee Tenant Isolation, and the Glue That Breaks Production

Jun 12, 2026 23 min

This episode of Ship It Weekly is about the hidden glue holding production together. Brian covers Coinbase’s May 7 outage postmortem, where an AWS us-east-1 cooling failure exposed the difference between being “multi-AZ” on paper and actually being able to recover when stateful, low-latency systems are tied to a failed zone. Then he looks at Meta’s AI-assisted Instagram support issue and why account recovery is identity infrastructure, not just customer support. If AI can influence password resets, email changes, MFA resets, or account ownership flows, that workflow needs to be treated like a production control plane. The episode also covers AWS AgentCore CLI CVE-2026-11393, where collaborator metadata could break out into generated Python code during agent import, and an Apigee cross-tenant issue from Google’s Apigee security bulletins that shows why tenant isolation has to be tested beyond the obvious happy path. Links Coinbase May 7 outage postmortem https://www.coinbase.com/blog/a-postmortem-of-our-may-7-2026-outage Meta AI support / Instagram account recovery reporting https://www.theverge.com/tech/945658/meta-ai-support-chatbot-exploit-instagram-accounts AWS AgentCore CLI CVE-2026-11393 https://aws.amazon.com/security/security-bulletins/2026-040-aws/ AgentCore CLI GitHub advisory https://github.com/aws/agentcore-cli/security/advisories/GHSA-m4x6-gwgp-4pm7 Google Apigee security bulletins https://docs.cloud.google.com/apigee/docs/security-bulletins/security-bulletins Cloudflare real-time threat intel WAF rules https://blog.cloudflare.com/realtime-threat-intel-waf-rules/ AWS Lambda tenant isolation with event source mappings https://aws.amazon.com/blogs/compute/integrating-event-source-mappings-with-aws-lambda-tenant-isolation-mode/ Amazon OpenSearch Serverless next generation https://aws.amazon.com/about-aws/whats-new/2026/05/amazon-opensearch-serverless-next-generation-generally-available/ GitHub Enterprise Managed Users IP allow list coverage https://github.b...

Kiro CLI Approval Bypass, Amazon Braket Pickle Risk, AWS Org Logging, KEDA Upgrades, and Automation’s Hidden Boundaries

Jun 5, 2026 20 min

This episode of Ship It Weekly is about automation’s hidden boundaries. Brian covers Kiro CLI CVE-2026-9255, where piped stdin could act like user approval, Amazon Braket SDK CVE-2026-9291 and the very normal Python pickle risk hiding inside quantum job results, AWS Organizations finally emitting CloudTrail events when accounts join or leave an org, and KEDA updates that remind us autoscaling upgrades are production behavior changes. The bigger thread this week is that automation does not remove boundaries. It moves them. Approval paths, trusted data, account membership, scaling signals, platform access, and AI-generated output all need clear ownership and visibility. Brian also covers Kubernetes Dashboard being archived with Headlamp as the path forward, Google Cloud Remote MCP Server for AlloyDB, Apache Kafka 4.3.0, and Atlassian’s AI-native SDLC productivity claims. Sponsored by @Scale: Systems & Reliability, happening June 25 at the Meydenbauer Center in Bellevue, Washington. Register at https://bit.ly/4xd2FdG Links Kiro CLI CVE-2026-9255 https://aws.amazon.com/security/security-bulletins/2026-035-aws/ Amazon Braket SDK CVE-2026-9291 https://aws.amazon.com/security/security-bulletins/2026-036-aws/ AWS Organizations CloudTrail account events https://aws.amazon.com/about-aws/whats-new/2026/05/aws-organizations-cloudtrail/ KEDA v2.20.0 release https://github.com/kedacore/keda/releases/tag/v2.20.0 KEDA v2.19.0 release https://github.com/kedacore/keda/releases/tag/v2.19.0 Kubernetes Dashboard archived / Headlamp path forward https://kubernetes.io/blog/2026/06/04/dashboard-archived-what-now/ Google Cloud Remote MCP Server for AlloyDB https://cloud.google.com/blog/products/databases/alloydb-remote-mcp-server-now-ga Apache Kafka 4.3.0 https://www.confluent.io/blog/apache-kafka-4-3-release-announcement/ Atlassian AI-native SDLC productivity claims https://www.atlassian.com/blog/software-teams/ai-native-sdlc This week’s On Call Brief https://www.tellerstech.com/on-call-...

GitHub Supply Chain Attacks, Railway’s GCP Outage, Discord’s Voice Failure, AWS Retry Changes, and Trusted Tool Risk

May 29, 2026 23 min

This episode of Ship It Weekly is about trusted tools becoming production dependencies. Brian covers a rough GitHub supply chain week, including the compromised Nx Console VS Code extension tied to exposed GitHub internal repositories and the Megalodon campaign abusing GitHub Actions workflows across thousands of public repos. The bigger thread this week is that the tools around production are increasingly part of production. Brian also covers Railway’s GCP account suspension outage, Discord’s voice outage during a Kubernetes migration, AWS changing SDK retry behavior, CVE-2026-9133 in the RabbitMQ AWS plugin, and a Reddit story about stolen AWS keys turning into a $14,000 Bedrock bill. Brian also touches on OpenTelemetry graduating from the CNCF, Claude Code security risk, GitLab Secrets Manager, Google Cloud AI spend caps, and a Redshift Python driver RCE. Full source list and extra links are available on this episode’s page at shipitweekly.fm. Links Nx Console compromise https://www.stepsecurity.io/blog/nx-console-vs-code-extension-compromised Megalodon GitHub Actions attack https://www.stepsecurity.io/blog/megalodon-mass-github-actions-secret-exfiltration-across-5-500-public-repositories Railway GCP outage https://blog.railway.com/p/incident-report-may-19-2026-gcp-account-outage Discord voice outage https://discord.com/blog/behind-the-scenes-of-the-3-25-26-voice-outage AWS SDK retry changes https://aws.amazon.com/blogs/developer/announcing-updated-retry-behavior-for-aws-sdks-and-tools/ RabbitMQ AWS plugin CVE-2026-9133 https://aws.amazon.com/security/security-bulletins/2026-034-aws/ AWS Bedrock cost spike Reddit thread https://www.reddit.com/r/aws/comments/1tm3ydo/aws_bedrock_cost_spike_14000_usd/ This week’s On Call Brief https://www.tellerstech.com/on-call-brief/2026-W22/ More episodes and show notes https://shipitweekly.fm/

Ship It Conversations: Jake Warner on Cycle.io, Bare Metal’s Comeback, and Why Private Cloud Is Getting Interesting Again

May 26, 2026 36 min

This is a guest conversation episode of Ship It Weekly, separate from the weekly news recaps. In this Ship It: Conversations episode, I talk with Jake Warner, founder and CEO of Cycle.io, about private cloud, bare metal, Kubernetes fatigue, and why some teams are rethinking how much infrastructure complexity they actually want to carry. We talk about why bare metal and private cloud are getting interesting again, especially around cost, performance, data sovereignty, compliance, and platform ownership. Jake explains how Cycle approaches infrastructure as a pool of resources, why he thinks in terms of “environments as code” instead of traditional infrastructure as code, and how teams can run containers and VMs together across bare metal, cloud, and hybrid environments. The bigger theme here is that this is not really a “cloud versus bare metal” conversation. It is about choosing the right level of abstraction. Sometimes Kubernetes is the right answer. Sometimes managed cloud services make sense. And sometimes teams just need a more opinionated platform that lets developers ship without requiring a large DevOps army to keep everything running. Highlights • Why some teams are moving back toward private cloud and bare metal • The role of cost, data sovereignty, compliance, and performance in infrastructure decisions • Why bare metal does not have to mean going back to old-school racking and stacking pain • How Cycle turns raw compute into a private cloud-style resource pool • Why Jake thinks about “environments as code” instead of only infrastructure as code • What “no DevOps army required” means in practice for engineering-heavy teams • Why some companies need VMs and containers running together on the same platform • Where Kubernetes still makes sense, especially for highly customized infrastructure needs • Why opinionated platforms can be valuable when teams want fewer knobs and better defaults • Active-active thinking, failover risk, and why application-level replic...

CISA’s GitHub Leak, AI Root Cause Analysis, Copilot Agents, Claude Code in CI/CD, and Kubernetes Seccomp Risk

May 22, 2026 22 min

This episode of Ship It Weekly is about secrets, agents, risky defaults, and follow-up work that never gets done. Brian covers the CISA contractor GitHub leak involving AWS keys, internal docs, Terraform, Kubernetes, Argo CD, and CI/CD context, plus AWS DevOps Agent doing automated RCA across Datadog, Elasticsearch, CloudTrail, and EKS. Brian also covers MS Copilot Studio computer-using agents, Claude Code in Bitbucket Agentic Pipelines, CVE-2026-46333 and Kubernetes seccomp defaults, GitHub OIDC for Dependabot, Java pods getting OOMKilled, LLM-generated SQL that can be wrong but still run, and why postmortem action items die without ownership. Sponsored by Guardsquare https://hubs.ly/Q04fJgkJ0 Links CISA GitHub leak https://blog.gitguardian.com/how-we-got-a-cisa-github-leak-taken-down-in-26-hours/ AWS DevOps Agent RCA https://aws.amazon.com/blogs/devops/automate-root-cause-analysis-across-datadog-and-elasticsearch-with-aws-devops-agent/ Microsoft Copilot Studio computer-using agents https://techcommunity.microsoft.com/blog/copilot-studio-blog/computer-using-agents-in-microsoft-copilot-studio-are-now-generally-available/4519427 Atlassian Agentic Pipelines with Claude Code https://support.atlassian.com/bitbucket-cloud/docs/agentic-pipelines/ CVE-2026-46333 https://nvd.nist.gov/vuln/detail/CVE-2026-46333 Kubernetes seccomp https://kubernetes.io/docs/reference/node/seccomp/ GitHub OIDC for Dependabot and code scanning https://github.blog/changelog/2026-05-19-expanded-oidc-support-for-dependabot-and-code-scanning/ Java pods OOMKilled in Kubernetes https://dzone.com/articles/java-pod-oomkill-kubernetes LLM-generated SQL risks https://readyset.io/blog/why-llms-write-incorrect-sql-and-what-that-means-for-your-database Postmortem action items https://incident.io/blog/why-do-post-mortem-action-items-fail-how-to-make-incident-follow-ups-actually-get-done On Call Brief https://www.tellerstech.com/on-call-brief/2026-W21/ More episodes + show notes https://shipitweekly.fm/

AI Agents Get API Access and Identity: GitHub Copilot Cloud Agents, MCP Auth, Ansible Automation, OpenAI Daybreak, and the New Production Risk

May 14, 2026 23 min

This episode of Ship It Weekly is about AI agents moving from helpful coding assistants into real operational actors. Brian covers GitHub making Copilot cloud agent tasks available through a REST API, Auth0 bringing authentication and authorization to MCP servers, Red Hat positioning Ansible as a trusted execution layer for agentic IT operations, and OpenAI Daybreak pushing AI deeper into security research and remediation. The bigger thread this week is authority: what these agents can reach, what they can change, who approved the action, and who owns the outcome when something breaks. Brian also covers Discord’s ScyllaDB automation work, AWS GuardDuty crypto mining detection, queues and back pressure, and a Datadog PostgreSQL case where an index scan was still painfully slow. Sponsored by Guardsquare https://hubs.ly/Q04fJgkJ0 Links GitHub Copilot cloud agent tasks via REST API https://github.blog/changelog/2026-05-13-start-copilot-cloud-agent-tasks-via-the-rest-api/ GitHub REST API endpoints for agent tasks https://docs.github.com/en/rest/agent-tasks/agent-tasks Auth0 Auth for MCP is now generally available https://auth0.com/blog/auth0-auth-for-mcp-servers-generally-available/ Red Hat on Ansible as the execution layer for agentic IT https://www.redhat.com/en/about/press-releases/red-hat-establishes-ansible-automation-platform-trusted-execution-layer-it-operations-agentic-era OpenAI Daybreak https://openai.com/daybreak/ Discord automates ScyllaDB clusters at scale https://discord.com/blog/how-discord-automates-scylladb-clusters-at-scale AWS GuardDuty crypto mining detection and prevention https://aws.amazon.com/blogs/security/detecting-and-preventing-crypto-mining-in-your-aws-environment/ Queues do not absorb load, they delay failure https://dzone.com/articles/queues-dont-absorb-load-they-delay-bankruptcy Datadog on inefficient PostgreSQL index scans https://www.datadoghq.com/blog/detect-inefficient-index-scans-with-dbm/ This week’s On Call Brief https://www.telle...

Cursor Deletes PocketOS Prod DB, .de DNSSEC Outage, Bluesky Postmortem, Argo CD, and Copy Fail

May 8, 2026 21 min

This episode of Ship It Weekly is about modern reliability getting squeezed from both directions. Old-school failures still hit hard, like broken DNSSEC, kernel privilege escalation bugs, and GitOps behavior changes. But newer automation layers add a second kind of risk, where AI agents, machine identity, and cloud control planes can do real damage fast when authority is too broad. Brian covers the Cursor and PocketOS production database wipe, the .de DNSSEC outage and Cloudflare’s response, Bluesky’s April outage postmortem, Argo CD v3.1.16 reaching end of life plus the v3.4.1 behavior change, Linux kernel CVE-2026-31431 under active exploitation, and why Google Cloud Agent Identity and AWS MCP Server GA both point to agents becoming first-class infrastructure actors. Sponsored by Guardsquare https://hubs.ly/Q04fJgkJ0 Links Cursor / PocketOS production database wipe https://www.tellerstech.com/on-call-brief/2026-W19/ Cloudflare on the .de DNSSEC outage https://blog.cloudflare.com/de-tld-outage-dnssec/ Bluesky April 2026 outage postmortem https://pckt.blog/b/jcalabro/april-2026-outage-post-mortem-219ebg2 Argo CD releases: v3.1.16 final release and v3.4.1 behavior change https://github.com/argoproj/argo-cd/releases Linux kernel CVE-2026-31431 https://nvd.nist.gov/vuln/detail/CVE-2026-31431 AWS bulletin for CVE-2026-31431 https://aws.amazon.com/security/security-bulletins/rss/2026-026-aws/ Google Cloud Agent Identity https://cloud.google.com/blog/products/identity-security/whats-new-in-iam-security-governance-and-runtime-defense AWS MCP Server is now generally available https://aws.amazon.com/blogs/aws/the-aws-mcp-server-is-now-generally-available/ Cross-region disaster recovery for Amazon EKS using AWS Backup https://aws.amazon.com/blogs/containers/cross-region-disaster-recovery-for-amazon-eks-using-aws-backup/ Google Ads new data retention policy starting June 1, 2026 https://ads-developers.googleblog.com/2026/05/new-data-retention-policy-for-google.html This week...

Ship It Conversations: Gareth Kersey on IaCConf 2026, AI, and Corey Quinn’s Terraform Keynote

May 5, 2026 31 min

This is a guest conversation episode of Ship It Weekly, separate from the weekly news recaps. This episode is not sponsored. I wanted to cover IaCConf because the theme lines up closely with what Ship It Weekly focuses on: infrastructure, platform engineering, DevOps, SRE, and how teams are adapting to AI-driven change. In this Ship It: Conversations episode, I talk with Gareth Kersey about IaCConf 2026, a free virtual conference focused on infrastructure as code, platform engineering, DevOps, SRE, and infrastructure operations. The conference is May 14th 2026. The main theme is “keeping pace.” Not just keeping pace with new tools, but keeping pace with the speed of software delivery now that AI is changing how quickly application teams can write, ship, and change code. We talk about what that means for the infrastructure teams underneath it all: the people responsible for Terraform, Kubernetes, GitOps, policies, secrets, cost, security, rollback paths, and making sure faster delivery does not turn into faster chaos. Gareth walks through the IaCConf 2026 agenda, including Corey Quinn’s keynote, AI and Terraform sessions, platform engineering panels, Kubernetes and Argo CD talks, AI agents managing infrastructure as code, governance challenges, and the risk of 10x code velocity becoming 10x operational risk. The bigger theme here is that AI is not just changing how code gets written. It is changing the pressure on the systems around delivery. Infrastructure as code, platform engineering, policy, and operational guardrails matter even more when the pace of change goes up. Highlights • What “keeping pace” means for infrastructure, DevOps, SRE, and platform teams • Why faster application development can create more downstream operational pressure • Corey Quinn’s keynote, “AI Speaks Terraform Like a Tourist” • How AI-generated infrastructure changes create new governance and review challenges • Why infrastructure as code still matters as AI agents and automation become m...

GitHub RCE, AI Agent Prompt Injection, and the New Reality: Your Developer Toolchain Is Production Now

May 1, 2026 25 min

This episode of Ship It Weekly is about the developer toolchain becoming part of production. Brian covers GitHub’s critical git push RCE, AI-assisted reverse engineering, prompt injection against AI agents in GitHub workflows, Elementary’s malicious CLI release, GitHub’s merge queue regression, Cal.com going closed source, and Copilot moving toward usage-based billing. Plus: MinIO’s repo archive, Ghostty leaving GitHub, Docker Hardened Images, and Azure DevOps security updates. Links GitHub git push RCE https://github.blog/security/securing-the-git-push-pipeline-responding-to-a-critical-remote-code-execution-vulnerability/ AI-assisted reverse engineering https://www.darkreading.com/application-security/reverse-engineering-ai-unearths-high-severity-github-bug AI agents + GitHub Actions prompt injection https://www.theregister.com/2026/04/15/claude_gemini_copilot_agents_hijacked/ Elementary malicious CLI release https://www.elementary-data.com/post/security-incident-report-malicious-release-of-elementary-oss-python-cli-v0-23-3 GitHub merge queue regression https://github.blog/news-insights/company-news/an-update-on-github-availability/ Cal.com going closed source https://cal.com/blog/cal-com-goes-closed-source-why GitHub Copilot billing https://github.blog/news-insights/company-news/github-copilot-is-moving-to-usage-based-billing/ MinIO archived repo https://github.com/minio/minio Ghostty leaving GitHub https://mitchellh.com/writing/ghostty-leaving-github Docker Hardened Images https://www.docker.com/blog/why-we-chose-the-harder-path-docker-hardened-images-one-year-later/ Azure DevOps security updates https://devblogs.microsoft.com/devops/one-click-security-scanning-and-org-wide-alert-triage-come-to-advanced-security/ On Call Brief https://oncallbrief.com/ More episodes https://shipitweekly.fm/

Kubernetes 1.36, Gateway API v1.5, AWS Copilot End of Support, and Cloudflare Non-Human Identities

Apr 24, 2026 20 min

This episode of Ship It Weekly is about platforms getting sharper about defaults, ownership, and the old paths they are no longer willing to quietly carry forever. Brian covers Kubernetes 1.36 and why it feels more like a cleanup-and-maturity release than a flashy feature dump, Gateway API v1.5 moving more networking behavior into the stable path, AWS Copilot CLI reaching end of support and what that means for teams still sitting on the older “easy” ECS workflow, Airbnb’s alert-development overhaul and why noisy or weak alerts are often a workflow problem long before they become an on-call problem, and Cloudflare’s push to treat scripts, agents, and third-party tools like real identities with real blast radius. He also hits the latest Azure DevOps Server patches and Google’s OTLP metrics support for Cloud Monitoring. Links Kubernetes v1.36 release https://kubernetes.io/blog/2026/04/22/kubernetes-v1-36-release/ Gateway API v1.5 https://kubernetes.io/blog/2026/04/21/gateway-api-v1-5/ AWS Copilot CLI end of support https://aws.amazon.com/blogs/containers/announcing-the-end-of-support-for-the-aws-copilot-cli/ Airbnb on alert development https://medium.com/airbnb-engineering/it-wasnt-a-culture-problem-upleveling-alert-development-at-airbnb-01e2290eb0f5 Cloudflare on non-human identities, OAuth visibility, and scoped permissions https://blog.cloudflare.com/improved-developer-security/ Azure DevOps Server April patches https://devblogs.microsoft.com/devops/april-patches-for-azure-devops-server/ OTLP metrics for Google Cloud Monitoring https://cloud.google.com/blog/products/management-tools/otlp-opentelemetry-protocol-for-google-cloud-monitoring-metrics Past episode where we talked about Cloudflare Mesh https://www.tellerstech.com/ship-it-weekly/aws-interconnect-ga-cloudflare-mesh-gitlab-19-eks-auto-mode-and-opentelemetry-config/ This week’s On Call Brief https://www.tellerstech.com/on-call-brief/2026-W16/ On Call Brief: https://oncallbrief.com/ More episodes and show notes...

Ship It Conversations: Stephane Moser on Pipedrive’s Jenkins-to-GitHub Actions Migration, Argo CD, and CI/CD at Scale

Apr 19, 2026 51 min

This is a guest conversation episode of Ship It Weekly, separate from the weekly news recaps. In this Ship It: Conversations episode, I talk with Stephane Moser about Pipedrive’s move from Jenkins to GitHub Actions, building self-hosted runners on Kubernetes, shifting deployments toward GitOps with Argo CD, and what it actually takes to roll out a big CI/CD change across a large engineering org. We talk about why Jenkins had become painful, from Groovy friction to noisy-neighbor problems on shared VMs, why GitHub Actions fit better, how reusable workflows and custom actions helped, why Argo CD beat out Flux for their use case, and how they had to build better observability and internal deployment visibility around GitHub as they scaled. The bigger theme here is that this was not just a tooling swap. It was a product and platform migration. Isolation, repeatability, self-service, rollout strategy, and observability mattered just as much as the actual CI/CD tools. Highlights • Why Jenkins stopped working well for them: Groovy friction, shared VM contention, and poor predictability • Replacing CodeShip pull request validation first as the low-blast-radius starting point • Using Actions Runner Controller on Kubernetes with EKS and Karpenter for self-hosted runners • Why reusable workflows and custom actions helped cut repetition across hundreds of services • Choosing Argo CD over Flux, Argo Workflows, Tekton, and even a short Spinnaker attempt • Moving from push-based deploys toward GitOps for better isolation and safer credentials handling • Building internal observability because GitHub’s workflow visibility was not enough at their scale • Dogfooding first, then rolling migration out in batches until teams could self-serve the move • What broke when the new system actually worked too well: bot-driven deploy volume, queueing, and fairness • The mobile side of the story: Mac minis, unstable runners, GitHub-hosted runners, and a very different migration path •...

AWS Interconnect GA, Cloudflare Mesh, GitLab 19, EKS Auto Mode, and OpenTelemetry Config

Apr 17, 2026 15 min

This episode of Ship It Weekly is about networking, ingress, and private access moving further up into the platform layer. Brian covers AWS Interconnect going generally available, Cloudflare Mesh, GitLab 19.0 breaking changes around Gateway API and bundled services, EKS Auto Mode networking, and OpenTelemetry declarative config reaching stability. He also hits containerd security patches, GitHub’s new Code Security risk assessment, and AWS guidance on securing AI agents with MCP. (Amazon Web Services, Inc.) Links AWS Interconnect GA and last mile connectivity https://aws.amazon.com/blogs/aws/aws-interconnect-is-now-generally-available-with-a-new-option-to-simplify-last-mile-connectivity/ Cloudflare Mesh https://blog.cloudflare.com/mesh/ GitLab 19.0 breaking changes https://about.gitlab.com/blog/a-guide-to-the-breaking-changes-in-gitlab-19-0/ EKS Auto Mode networking https://aws.amazon.com/blogs/containers/navigating-enterprise-networking-challenges-with-amazon-eks-auto-mode/ OpenTelemetry declarative config reaches stability https://opentelemetry.io/blog/2026/stable-declarative-config/ containerd security releases https://github.com/containerd/containerd/releases GitHub Code Security risk assessment for organizations https://github.blog/changelog/2026-04-08-code-security-risk-assessment-available-for-organizations/ AWS secure AI agent access patterns using MCP https://aws.amazon.com/blogs/security/secure-ai-agent-access-patterns-to-aws-resources-using-model-context-protocol/ This week’s On Call Brief https://www.tellerstech.com/on-call-brief/2026-W16/ More episodes and show notes https://shipitweekly.fm/

Special: Claude Mythos Preview and Project Glasswing: AI Exploit Discovery, Zero-Day Risk, Business Fallout, and What It Means for DevOps, Cloud, and Platform Security

Apr 16, 2026 16 min

In this Ship It Weekly special, Brian breaks down Claude Mythos Preview and Project Glasswing, and why this story matters beyond normal AI launch hype. Anthropic is treating Mythos like a real security inflection point, not just a better coding model. Project Glasswing is their coordinated effort to get early access into the hands of defenders, critical software maintainers, and major infrastructure organizations before similar capability becomes more broadly available. If OpenClaw was about agents becoming a new control plane, this episode is about what happens when finding ways into messy environments and control planes starts getting faster too. We walk through the practical angle for DevOps, cloud, platform, and infra teams: exploit timelines may be compressing, platform debt becomes attacker leverage, and the boring work most orgs treat like cleanup suddenly looks a lot more like frontline security work. We also zoom out to the business side, including why banks, regulators, and government officials are already paying attention. Chapters Why This Episode ExistsOpenClaw CallbackWhat Actually HappenedDon’t Get Gullible, Don’t Get LazyWhat Changes If This Is Even Half TrueWhy Business People Should CareWhat This Means for DevOps, Cloud, and PlatformBoring Work Just Got PromotedThe Uncomfortable TakeawayWhat I’d Do Right NowLinks from this episode Claude Mythos Preview https://red.anthropic.com/2026/mythos-preview/ Project Glasswing https://www.anthropic.com/project/glasswing AI cyber threats: open letter to business leaders https://www.gov.uk/government/publications/ai-cyber-threats-open-letter-to-business-leaders/ai-cyber-threats-open-letter-to-business-leaders-html AI-boosted hacks with Anthropic’s Mythos could have dire consequences for banks https://www.reuters.com/legal/litigation/ai-boosted-hacks-with-anthropics-mythos-could-have-dire-consequences-banks-2026-04-13/ ECB to quiz bankers about risks of Anthropic's new AI model, source says https://www.reuters.c...

Amazon S3 Files, Malicious npm Plugins, Trivy Fallout, and Kubernetes’ Gateway Shift

Apr 10, 2026 15 min

This episode of Ship It Weekly is about the interface layer becoming the story. Brian covers Amazon S3 Files and why it feels more like a managed filesystem layer in front of S3 than “S3 is EFS now,” including how it relates to the old s3fs and FUSE-style approach. He also digs into 36 malicious npm packages posing as Strapi plugins, the uglier follow-on to the Trivy incident he discussed previously, Kubernetes Ingress2Gateway 1.0 and the push toward Gateway API, and Kubernetes Agent Sandbox as a sign that newer AI-style workloads are starting to reshape the platform itself. Links Amazon S3 Files https://aws.amazon.com/blogs/aws/launching-s3-files-making-s3-buckets-accessible-as-file-systems/ Malicious npm packages posing as Strapi plugins https://thehackernews.com/2026/04/36-malicious-npm-packages-exploited.html Trivy follow-on incident discussion https://github.com/aquasecurity/trivy/discussions/10425 RoseSecurity on Trivy / typosquatting angle https://rosesecurity.dev/2026/03/20/typosquatting-trivy.html Earlier episode covering the first Trivy incident https://www.tellerstech.com/ship-it-weekly/aws-bahrain-uae-data-center-issues-amid-iran-strikes-argocd-vs-flux-gitops-failures-github-actions-hackerbot-claw-attacks-trivy-roguepilot-codespaces-prompt-injection-block-ai-remake/ Kubernetes Ingress2Gateway 1.0 https://kubernetes.io/blog/2026/03/20/ingress2gateway-1-0-release/ Kubernetes Agent Sandbox https://kubernetes.io/blog/2026/03/20/running-agents-on-kubernetes-with-agent-sandbox/ Fortinet FortiClient EMS emergency patch https://www.fortiguard.com/psirt/FG-IR-26-099 Karpathy post https://x.com/karpathy/status/2036487306585268612 ProofShot https://github.com/AmElmo/proofshot More episodes and show notes https://shipitweekly.fm On Call Briefs https://oncallbrief.com