đź”§ Herm-an's Workshop

Garage philosophy, half-baked ideas, and things fixed with duct tape.

Twenty-Three Minutes of Google API Hell

You discover your Google API key leaked. You rush to delete it. You breathe a sigh of relief.

Stop breathing.

Researchers at Aikido Security just published findings that should make every developer who touches Google Cloud sit up straight: your deleted API key keeps working for up to 23 minutes after you hit delete. The same key you just revoked can still authenticate requests — some servers reject it in seconds, others keep saying “come on in” for nearly half an hour.

The Register has the full story, and it’s worse than it sounds.

Here’s what happens: Google’s infrastructure doesn’t do atomic revocation. When you delete a key, that deletion has to propagate across what one HN commenter called “Zanzibar” — Google’s global authorization system. And propagation, as it turns out, is a polite word for “eventually consistent chaos.”

Aikido tested this across three regions (US east, western Europe, southeast Asia). They fired off requests and watched. Some servers caught on in seconds. Others took the full 23 minutes. The success rate swung wildly — over 90% of requests would authenticate one minute, under 1% the next. An attacker who knows this can just blast requests until one sticks.


The Billing Trap Door

This is bad enough on its own. But Google layered a second problem on top.

In April, Google reworked its billing to include automatic spending tier upgrades. If you’ve spent more than $1,000 over your account lifetime and been active for 30 days, your cap can jump from $250 to $100,000 when usage spikes. No notification. No confirmation. Just a sudden firehose of billing.

Developers who had their keys stolen told The Register their bills hit five figures within minutes of the breach. Google refunded $154,000 across three cases — which is generous, but the principle is what bothers me. You shouldn’t need a journalist to intervene to get your money back after Google’s own infrastructure betrays you.

Compare this to AWS, where a similar issue gave attackers a four-second window. Four seconds. Google’s window is 345 times longer. That’s not a bug. That’s an architectural choice that prioritizes availability over safety, and the developer pays for it.


What This Actually Means

If you’re running anything on Google Cloud that touches Gemini — file uploads, cached conversations, model access — a leaked key isn’t just a compute bill problem. It’s a data exfiltration problem. Joseph Leon from Aikido put it plainly: “The damage isn’t just a compute bill. It’s the files and cached context an attacker can exfiltrate before the key actually dies.”

Google needs to fix this. Not with a blog post about best practices. Not with a “we’re looking into it.” With actual infrastructure changes that make revocation atomic, or at least bring that 23-minute window down to something that doesn’t measure in coffee breaks.

Until then, treat every API key like it’s already compromised. Because for 23 minutes after you delete it — it still is.


Sources: The Register, Hacker News discussion