Lessons learned: Testing new Azure features

It’s a rainy morning here in Helsinki, and I’m thankful for being able to spend a few hours at a small café close to a customer site before a workshop.

For the past weeks, I’ve been dealing with a slight issue – or let’s call it a huge problem – in one of my Azure environments. To understand the case, I’ll share a brief background first.

In May this year, Microsoft announced a few new capabilities in Azure that immediately became available. Like the rest of the Nordics, I was busily preparing for the coming holiday season, and I didn’t have time to dig further into that announcement. Once I returned to work mode in August, I wanted to take one of these for a spin – the Azure DNS Private Resolver.

It’s a service that allows you to query private DNS zones (in Azure) from your on-premises infrastructure. Essentially, a personal DNS service without needing to provision virtual machines. See here for more details on the service.

Provisioning is slightly awkward as you need to track the VNET allocations, IP ranges, what goes in and out, and so forth. I spent a few hours setting it up but realized I’d need more time to conduct proper tests. At this time, I checked how much I would estimate the service to cost me.

Okay, it’s certainly not free. That would be about $360/month for me, as I had an inbound and outbound endpoint provisioned. Usually, my Azure consumption is around 5-10 €/day. I don’t run virtual machines or anything super heavy in these test/Pay-As-You-Go tenants.

I habitually check my Azure cost a few times a week. Suddenly, I caught this about a week after I left the Azure DNS Private Resolver, waiting for more time to work with:

Suddenly, my total Azure consumption spiked to more than 300 € in just one day! What’s going on?

I drill further down to Azure Cost Management, and I find this (it’s monthly view):

Hold on! Is that almost 30,000 € for Azure DNS!? Yes, yes, it is.

I end up spending the whole weekend troubleshooting this. I woke up on a Monday morning with Microsoft trying to charge my PAYG credit card the outstanding invoice. Thankfully, my bank rejected this.

This is how it looked in Azure Cost Management:

That’s some severe DNS’ing overnight! I don’t host too many DNS zones in Azure, and they usually cost me 5 cents a month.

Turns out this was a billing issue within Azure. The Azure Private DNS Resolver – still in preview then – started billing the monthly cost each hour. It accumulates pretty fast.

I lost a bit of sleep over this. I’m pretty resilient to most things in life, and even if I knew the cost was warrantless, I still fretted over it. I spent a few weeks with Azure Support going back and forth on this. We have meetings, I share my screen, and we debate how this happened. In Microsoft’s defense, the support was top-notch.

I play scenarios in my head. Do I have to sell my house? Will the kids be sleeping with me under the bridge? I’m a fan of dark humor, but I’m still stressing about this.

Finally, I received word from Azure Support that the billing glitch has been resolved. It still takes a few more weeks for the Cost Management to reflect this, and I’m now back to everyday consumption. Also, I don’t have Azure DNS Private Resolver instances.

Lessons learned

I did learn some hard lessons here. Perhaps they are apparent now that you’ve read through my predicament, but let me list them.

First, when you provision something ‘just for quickly testing,‘ always have a logic to clean up after you! I frequently use tags to automatically remove stuff I don’t need, but this time I didn’t add those tags for some reason. My excuse is that I was busy and left unsupervised 😁.

Second, have a budget. I thought I had, but as I shuffled through multiple subscriptions, this one subscription didn’t have an active budget any longer. This is due to other issues that are perhaps out of the scope of this blog post, but the budgeting also didn’t work:

Third, always take screenshots! I use the fantastic Greenshot to capture a lot of stuff for these blog posts and other uses, and they provide me with a great journal on how things went down. Evidence and/or a clear recollection of what was done allows you to sleep better.