3 posts tagged with "devops"

There is a certain category of test that feels good to write but does not actually test what you think it does. Retry logic sits squarely in that category.

The usual pattern is this: inject a fake HttpMessageHandler, make it return a 429 or 503 on the first N calls, assert that the code retried and eventually succeeded. The test passes. You ship with confidence. Then, in production, a real throttling event triggers a path through the Azure SDK that your mock never covered, and the retry policy does not behave the way the test implied.

The issue is not that the mock is wrong. It is that the mock bypasses the entire SDK transport layer. When you return a 429 from a fake handler, you are testing whether your own retry wrapper handles it correctly. You are not testing whether Azure.Core's built-in retry pipeline fires, whether the Retry-After header is respected, or whether the SDK's own exception hierarchy propagates through your application code the way you assumed. That is a different bar entirely.

For a while, local Azure development in my projects looked like this: Azurite for Blob and Queue Storage, the Microsoft Service Bus Emulator for messaging, the Cosmos DB Emulator for document storage, and nothing at all for Key Vault, Container Registry, or Entra. The last three were either skipped in local runs, mocked, or simply hit against a real Azure subscription when I needed them.

That worked, to a point. But the setup cost was real. Each emulator has its own Docker image, its own port range, its own quirks, and its own certificate story. Compose files grew. CI pipelines grew to match. And the services that had no emulator at all stayed untested locally, which is exactly the category where surprises tend to show up in production.

I wanted to see whether a single tool could cover the whole stack without making too many compromises.

Every team that tests against real Azure services in CI eventually hits the same four problems. You need credentials for the pipeline. Those credentials need to be stored somewhere, rotated, and audited, and if they leak they affect a real environment. The tests themselves become flaky because Azure provisioning has variable latency and your Service Bus namespace occasionally takes three minutes to appear. And if you run on private agents, you need to add networking complexity on top of all of that.

These are not Azure-specific problems. They show up in any pipeline that depends on external cloud services. But the fix for them often is Azure-specific, and the usual answers (use a dedicated test subscription, use Managed Identity, sanitize your test fixtures) treat the symptoms without changing the fundamental structure of the problem.

I wanted to see whether running tests against a local Azure emulator in CI could close those gaps cleanly. The short version is: it can, and the job runs in 38 seconds.