Azure DevOps On-Premise Deployment: Fix CI/CD Agent Bloat

How Did We Discover the CI/CD Agent Bloat in Azure DevOps?

While working on a massive infrastructure modernization initiative for a global FinTech platform, we encountered a severe bottleneck in our CI/CD pipelines. The client was migrating over twenty independent microservices and legacy monolithic applications into a more structured release process. These applications were destined for a highly secure, on-premise farm consisting of four primary Windows Server machines.

During the initial phases of setting up the Azure DevOps release pipelines, the engineering team logically decided to use the native Environment resource feature. For each new project added to Azure DevOps, they navigated to the Environment tab, added a Virtual Machine resource and ran the generated configuration script on the target servers. This worked flawlessly for the first three projects.

However, as we began integrating the fourth project, a glaring architectural flaw surfaced. Because Azure DevOps Environments are strictly scoped to individual projects and cannot be shared across project boundaries, the team was forced to install a new, dedicated agent for every single project on the same four servers. We realized that if we continued down this path for all twenty projects, each on-premise Windows server would be burdened with twenty distinct agent processes running as Windows Services. This unnecessary overhead caused CPU spikes, memory exhaustion and deployment timeouts, ultimately threatening the stability of the entire on-premise environment. This specific challenge inspired this article, providing a technical blueprint for utilizing shared release infrastructure and azure devops deployment groups to avoid similar resource exhaustion in enterprise environments.

What Was the Business Context and On-Premise Architecture?

The client operates a core transaction routing system that requires high availability and strict data governance, making full public cloud migration unfeasible at this stage. Instead, they operate a robust on-premise data center. The architecture involves multiple independent development teams working across disparate Azure DevOps projects for security and repository isolation.

Deployments needed to occur simultaneously across a cluster of four Windows Server instances acting in an active-active load-balanced configuration. The CI/CD pipelines were responsible for building the .NET applications, running unit tests and pushing the compiled artifacts to the IIS servers hosting the applications.

The business required rapid, reliable deployments without compromising server performance. When you hire dotnet developers for enterprise modernization, one of the primary expectations is that the release pipelines will streamline delivery, not introduce new infrastructure constraints. The agent bloat was actively degrading the application hosting environment, making a scalable deployment strategy an immediate priority.

Why Did the Windows Servers Suffer from Agent Exhaustion?

The core issue stemmed from a misunderstanding of how Azure DevOps isolates resources between projects and how different deployment strategies execute on target machines.

When you use the modern YAML-based Environments feature, Azure DevOps treats each environment as a project-level entity. Installing the agent via the Environment setup script registers that specific machine to that specific project’s environment. Consequently, the agent continuously polls Azure DevOps for jobs related exclusively to that project.

Furthermore, standard Agent Pools could not solve this directly. While an Agent Pool can be shared across projects at the Organization level, a deployment job routed to an Agent Pool will only execute on a single available agent. If you need to deploy an application to four servers simultaneously, a standard Agent Pool job will simply pick one server, deploy there and finish, leaving the other three untouched.

The team was trapped between two flawed options: install twenty project-specific agents on each server to use YAML Environments or manually write looping scripts to push code from a single build agent to the target servers. The logs revealed overlapping file locks, excessive memory consumption by the agent background processes and chaotic deployment state management.

How Did We Approach Azure DevOps Deployment Groups and Alternatives?

To resolve this architectural bottleneck, we paused the CI/CD rollout and evaluated several structural approaches to multi-server, multi-project deployments. We considered these solutions as well:

Could We Centralize All Deployments in a Single Azure DevOps Project?

One approach was to create a centralized DevOps project solely for release management. All environments would live in this single project, requiring only one agent per server. Other development projects would publish their build artifacts and cross-project pipeline triggers would initiate the deployment in the central hub. While this solves the agent bloat, it violates the client’s strict RBAC and security isolation requirements, as developers would lose visibility into their specific release pipelines.

Could We Use a Single Build Agent with PowerShell Remoting?

We explored utilizing a standard, shared Agent Pool containing a single build agent (separate from the web servers). The pipeline would execute on this build agent and use WinRM (Windows Remote Management) or PowerShell deployment tasks to push the compiled code out to the four target web servers. This is a highly effective pattern for modern YAML pipelines, but it requires maintaining complex deployment scripts, managing WinRM certificates and handling concurrent deployment locking manually.

Are Azure DevOps Deployment Groups the Right Fit?

We revisited azure devops deployment groups. Often dismissed by newer teams as a legacy feature tied to Classic Release Pipelines, deployment groups actually provide the exact cross-project multi-machine capability required here. At the backend, a Deployment Group is backed by a Deployment Pool, which exists at the Organization level. This means you can install a single agent on a Windows server, register it to an Organization-wide Deployment Pool and then reference that pool from multiple different Azure DevOps projects using project-scoped Deployment Groups. This completely eliminates agent duplication while natively supporting parallel deployments to multiple servers.

What is the Final Implementation for Shared Deployment Architecture?

We determined that a hybrid approach leveraging Organization-wide Deployment Pools combined with modern YAML deployment jobs offered the cleanest, most scalable solution without sacrificing server health.

Here is the implementation path we took:

1. Creating the Organization-Level Deployment Pool

Instead of creating project-level environments, we navigated to the Azure DevOps Organization Settings and created a shared Deployment Pool. We generated the registration script once and executed it on the four target Windows Servers. This resulted in exactly one Azure Pipelines agent running as a Windows Service per server, drastically reducing memory and CPU overhead.

2. Mapping Deployment Groups in Individual Projects

Inside each of the twenty independent Azure DevOps projects, we navigated to the Pipelines section and created a Deployment Group. Instead of creating a new pool, we selected the existing Organization-level Deployment Pool. This effectively created a secure, project-scoped pointer to the shared infrastructure.

3. Configuring the YAML Pipeline

While Deployment Groups are heavily associated with Classic Releases, they can still be invoked within YAML pipelines using the deployment job strategy. We updated the deployment stage of our YAML templates to target the deployment group:

stages:
- stage: DeployToOnPrem
  jobs:
  - deployment: DeployWeb
    displayName: Deploy to IIS Web Farm
    pool:
      name: Default
    environment: 
      name: production
      resourceType: VirtualMachine
    strategy:
      runOnce:
        deploy:
          steps:
          - task: IISWebAppManagementOnMachineGroup
            inputs:
              IISDeploymentType: 'IISWebsite'
              ActionIISWebsite: 'CreateOrUpdateWebsite'
              WebsiteName: 'FinTechCoreApp'
              PhysicalPath: '%SystemDrive%inetpubwwwrootFinTechCoreApp'

Wait, we noticed a critical nuance during implementation. Modern YAML environments map cleanly to VirtualMachine resources, but if you want to use the classic Deployment Group agents in YAML without reverting to UI-based releases, you must use a standard job with deployment group tasks or adopt the WinRM push model.

The Refined Modern YAML Approach:

To fully adhere to modern YAML standards while avoiding agent bloat, we ultimately pivoted to the single shared Agent Pool with a push-based deployment strategy using WinRM. We set up a single dedicated Deployment Agent (a separate virtual machine). The YAML pipeline executes on this dedicated agent, downloads the artifact and pushes it to the four servers using the IIS Web App Deployment via WinRM task.

steps:
- task: IISWebAppDeploymentOnMachineGroup
  displayName: 'Deploy to Web Farm via WinRM'
  inputs:
    WebSiteName: 'FinTechApp'
    Package: '$(Pipeline.Workspace)/drop/*.zip'
    MachineNames: 'SERVER01, SERVER02, SERVER03, SERVER04'
    AdminUserName: '$(AdminUser)'
    AdminPassword: '$(AdminPassword)'
    Protocol: 'HTTPS'

This implementation achieved all requirements: zero agent bloat on the web servers, native multi-machine deployment, isolated project pipelines and full modern YAML adoption.

What Are the Core Lessons for DevOps Engineering Teams?

Scaling deployment infrastructure requires foresight. When companies hire devops developers for CI/CD automation, they expect architectural maturity that anticipates infrastructure limits. Here are the key takeaways from this optimization:

• Understand Scope Boundaries: Azure DevOps Environments are strictly project-scoped. Do not use them as shared infrastructure across disparate projects unless you are prepared to manage the resulting agent duplication.
• Decouple Build from Target Environments: Do not install pipeline agents directly on application hosting servers unless absolutely necessary. Treat application servers as deployment targets, not CI/CD runners.
• Leverage Push-Based Deployments for On-Premise: For multi-machine on-premise clusters, utilizing a dedicated deployment runner that pushes artifacts via WinRM or SSH is far cleaner than pulling artifacts from multiple localized agents.
• Re-evaluate Legacy Features: While azure devops deployment groups are considered an older pattern, their underlying Organization-level shared pool mechanism is highly effective. Understand how these map to modern pipelines before discarding them.
• Monitor CI/CD Overhead: Regularly audit your on-premise servers for rogue background services. A poorly architected CI/CD pipeline can degrade application performance just as easily as bad application code.

How Can We Help Optimize Your Release Engineering?

Balancing infrastructure health, security boundaries and rapid release cycles requires deep architectural experience. If you are struggling with pipeline bloat, legacy release management or scaling complex enterprise architectures, it might be time to hire software developer teams who understand the nuances of production-grade CI/CD. Whether you need to streamline Azure DevOps, modernize .NET infrastructure or build resilient deployment automation, our vetted remote engineering teams deliver structured, accountable results. Ready to optimize your engineering operations? contact us today to discuss your technical challenges.

Social Hashtags

#AzureDevOps #DevOps #CICD #AzurePipelines #YAML #DevOpsEngineering #CloudEngineering #WindowsServer #WinRM #IIS #DotNet #InfrastructureAutomation #ReleaseEngineering #SoftwareDevelopment