How Did We Discover the ACR Purge Task Failure?

While working on a massive infrastructure modernization project for a fast-growing FinTech platform, we needed to optimize cloud storage costs. The platform’s robust CI/CD pipeline was generating dozens of container images daily. Over time, the Azure Container Registry (ACR) accumulated thousands of outdated images, increasing both our Azure bill and our potential security attack surface.

To automate the cleanup, we utilized Terraform to provision an Azure Container Registry scheduled task. The goal was simple: periodically purge old images based on a regex filter and retention duration. Applying the Terraform configuration went flawlessly and the task was successfully created in Azure.

However, when we tested the deployment by manually triggering the scheduled task via the Azure CLI, the execution failed immediately with a cryptic error: Unable to find image acr:latest locally. This failure halted our storage optimization efforts and highlighted a nuanced discrepancy between how Azure CLI executes commands locally versus how ACR Tasks parse and execute YAML definitions in the cloud. This challenge inspired this article so other engineering teams can avoid the same infrastructure-as-code pitfalls.

Why Do We Need Automated ACR Cleanup in FinTech Architecture?

In high-throughput environments like FinTech applications, microservices are updated and deployed constantly. Every commit merged to the main branch triggers a build, pushing a new image tag to the Azure Container Registry. Without a lifecycle management policy, these environments quickly hit storage limits and incur unnecessary costs.

Automating this via Terraform using the azurerm_container_registry_task resource allows infrastructure teams to maintain configuration as code. By defining a purge task directly within our registry configuration, we ensure that every new environment provisioned automatically includes garbage collection for outdated artifacts. Organizations often hire cloud developers for scalable architecture precisely to ensure that these operational efficiencies are baked into the foundational code rather than treated as afterthoughts.

Why Did the Terraform ACR Task Fail with an Image Not Found Error?

Once the Terraform apply was complete, we validated the setup by running the task manually. We expected the task to execute the purge command and delete images older than 14 days matching our regex filter. Instead, the task logs revealed an abrupt stop.

The logs showed that the execution engine was attempting to pull a container image named acr:latest and failing. This was perplexing because running the equivalent standard Azure CLI command directly from our local terminals executed perfectly and deleted the intended images.

We noticed another anomaly: whenever there was a syntax error in the command string, the ACR Task engine would default to throwing this exact acr:latest image missing error. The error messaging was misleading, masking the true root cause, which lay buried in the encoded YAML step defined within our Terraform resource.

How Did We Diagnose and Evaluate Solutions for the ACR Purge Failure?

Our diagnostic process began by stripping away the Terraform abstraction. We took the exact YAML definition generated by our Terraform code and attempted to run it directly against the registry using the Azure CLI. The manual YAML execution threw the exact same error, proving the issue was not Terraform’s base64 encoding, but the YAML payload itself.

What Alternative Solutions Did We Consider?

During our troubleshooting, we considered and tested several different approaches:

• Bypassing Terraform Encoding: We tested the Terraform resource with and without the base64 encoding function, as the provider documentation suggests encoding is optional. This did not change the outcome, ruling out string encoding corruption.
• Running Azure CLI in a Custom Container: We considered writing a custom Docker image that contained the Azure CLI, deploying it and having the ACR task run a shell script. While this would work, it adds unnecessary maintenance overhead for a custom image.
• Using Lifecycle Management Policies: We evaluated native ACR lifecycle management, but it lacked the granular regex-based filtering required for our specific tag naming conventions.

Finally, we audited the exact syntax of the command: cmd: acr purge –filter ‘.*:^[0..9]*$’ –ago 14d. We discovered two distinct issues. First, the ACR Task runtime executes commands by spinning up containers. When it sees cmd: acr, it looks for a container image named acr. We needed to specify the fully qualified Microsoft ACR CLI image path. Second, our regex filter syntax [0..9] was invalid for the Go-based regex engine used by the purge tool; it fails to match a range of numbers.

What is the Correct Terraform Configuration for Azure ACR Purge Tasks?

To resolve the issue, we rewrote the encoded step in our Terraform configuration. We replaced the implicit command with the explicit Microsoft container registry image for the ACR CLI. We also corrected the regex syntax from [0..9] to standard digit matching [d] (or [0-9]).

Here is the sanitized, corrected implementation for deploying a reliable ACR purge task:

resource "azurerm_container_registry_task" "automated_purge" {
  name                  = "weekly-image-purge"
  container_registry_id = azurerm_container_registry.enterprise_acr.id
  
  platform { 
    os = "Linux" 
  }
  
  timer_trigger { 
    name     = "WeeklySchedule"
    schedule = "0 2 * * 0" # Runs every Sunday at 2 AM
  }
  
  encoded_step {
    # Using the fully qualified image path for acr-cli prevents the 'acr:latest' error
    # Fixing the regex range from [0..9] to [d] or [0-9] ensures proper matching
    task_content = base64encode(<<-CONTENT
      version: v1.1.0
      steps:
        - cmd: mcr.microsoft.com/acr/acr-cli:0.5 purge --filter '.*:^[d]*$' --ago 14d
    CONTENT
    )
  }
  
  identity {
    type = "SystemAssigned"
  }
}
# Ensure the task's managed identity has permissions to delete images
resource "azurerm_role_assignment" "acr_purge_contributor" {
  scope                = azurerm_container_registry.enterprise_acr.id
  role_definition_name = "AcrPush"
  principal_id         = azurerm_container_registry_task.automated_purge.identity[0].principal_id
}

By defining the image as mcr.microsoft.com/acr/acr-cli:0.5, the ACR Task engine successfully pulls the correct utility container to execute the purge command. The role assignment ensures the SystemAssigned identity has the right scope to perform the deletion.

What Are the Key Takeaways for DevOps and Cloud Engineering Teams?

Troubleshooting this issue yielded several critical insights for infrastructure automation:

• Beware of Misleading Errors: In cloud automation tools, syntax errors are often swallowed or masked by generic execution errors. The missing image error was a byproduct of bad YAML command syntax.
• Understand Task Execution Contexts: ACR Tasks are essentially headless container runners. Always supply fully qualified image paths when invoking command-line tools inside YAML steps.
• Validate Regex Contextually: Different tools utilize different regex engines (e.g., PCRE, Go regex, POSIX). What works in a Bash script might fail in a YAML task parser. Test regex against the specific language engine the tool uses.
• Isolate the Abstraction: When Terraform deployments fail on execution rather than provisioning, extract the raw payload and run it directly via CLI or API to isolate the fault domain.
• Manage Managed Identities: A purge task will fail silently if the SystemAssigned identity lacks the AcrPush or equivalent Contributor roles required to modify the registry catalog.

This level of troubleshooting is a prime example of why enterprises hire software developer teams with deep infrastructure experience. Anticipating how different cloud services interact is crucial for maintaining resilient pipelines.

How Can You Prevent Similar Infrastructure as Code Failures?

Automated cleanup of container registries is a fundamental practice for managing cloud spend and security. However, implementing it requires understanding the distinct behaviors of execution environments like Azure ACR Tasks. By explicitly defining container images in task steps and rigorously testing regex filters outside of Terraform, teams can ensure their automated maintenance tasks run reliably.

If your organization plans to hire terraform developers for infrastructure automation or needs to scale complex cloud architectures reliably, contact us. Our pre-vetted engineering teams deliver robust, maintainable infrastructure solutions tailored to your enterprise.

Social Hashtags

#Terraform #MicrosoftAzure #AzureACR #AzureContainerRegistry #DevOps #InfrastructureAsCode #CloudAutomation #AzureDevOps #ContainerSecurity #CloudCostOptimization #FinOps #CICD #TerraformTips #CloudEngineering #DevOpsTroubleshooting