Polly v8 Retry & Circuit Breaker: Why Retries Stop Early

INTRODUCTION: Why Did Our Resilience Pipeline Fail Prematurely?

While working on an enterprise ERP integration for a global retail logistics platform, we were tasked with stabilizing an unreliable third-party inventory API. To handle transient faults and prevent cascading system failures, we implemented a sophisticated resilience pipeline using the Polly library in .NET. The architecture relied on a combination of Retry, Circuit Breaker and Timeout strategies.

The goal was straightforward: if the downstream service struggled, the pipeline would retry up to 100 times over a prolonged period while a Circuit Breaker protected the network from being flooded. However, during load testing, we discovered a baffling anomaly. The pipeline would aggressively retry while the circuit was open, but it consistently gave up around the 80th attempt, completely ignoring the configured maximum of 100 attempts.

In distributed systems, unpredictable pipeline behavior can lead to dropped messages, corrupted states and SLA breaches. Resolving these obscure edge cases is a core competency we emphasize when businesses partner with us and hire software developer teams for mission-critical applications. This challenge inspired this article so other engineering teams can avoid the hidden pitfalls of mismatched resilience policies.

PROBLEM CONTEXT: Where Did the Polly Circuit Breaker Issue Surface?

The issue surfaced in a background synchronization worker responsible for pushing high-volume inventory updates to an external system. We used the newer Polly v8 AddResilienceHandler extension for our HttpClient. The pipeline was structured to execute strategies from the outside in: Retry as the outermost shell, Circuit Breaker in the middle and a strict Timeout on the inner HTTP execution.

Our intent was simple:

• If the downstream API fails or times out, Retry.
• If a certain threshold of failures is met, the Circuit Breaker opens for 5 seconds.
• While open, execution attempts immediately throw a BrokenCircuitException.
• The Retry strategy catches this exception and waits 100ms before trying again, exhausting retries until the circuit half-opens and a real request is permitted.

We expected the pipeline to keep pushing until it reached 100 retries or successfully obtained an HTTP 200 OK. Instead, our logs showed the retries abruptly halting mid-cycle. Understanding these exact nuances is why organizations hire dotnet developers for enterprise modernization—superficial implementations often fail under real-world load.

WHAT WENT WRONG: Why Did the Retry Abort Before Max Attempts?

To diagnose the issue, we analyzed the detailed application logs. Here is a sanitized snapshot of the sequence we observed:

[08:21:39 INF] Retry 37, after 100.00ms, due to: BrokenCircuitException 
[08:21:40 INF] Circuit half opened now
[08:21:40 INF] Opening circuit now
[08:21:40 ERR] Server returned a 500
... (retries 38 through 81 continue failing fast with BrokenCircuitException) ...
[08:21:45 INF] Retry 82, after 100.00ms, due to: BrokenCircuitException 
[08:21:45 INF] Circuit half opened now
[08:21:45 INF] Opening circuit now
[08:21:45 ERR] Server returned a 500

At attempt 82, the retries simply stopped. No timeout exception was thrown and the application moved on, assuming the operation was complete.

The root cause was hidden in how Polly passes results between inner and outer policies. When the Circuit Breaker transitioned to the Half-Open state at 08:21:45, it allowed exactly one request to pass through to the network. That request resulted in an HTTP 500 Internal Server Error.

The Circuit Breaker’s ShouldHandle predicate was configured to recognize HTTP 500 as a failure. It tripped back to the Open state. However, it did not throw an exception; it simply returned the HttpResponseMessage (containing the 500 status) back up the chain to the outer Retry strategy.

Here was the fatal flaw: the Retry strategy’s ShouldHandle predicate was only configured to handle HttpRequestException and BrokenCircuitException. It was not configured to handle a failed HttpResponseMessage. When the Retry strategy received the HTTP 500 response, it evaluated the result, decided it wasn’t a handled failure condition, treated it as a “success” and exited the pipeline.

HOW WE APPROACHED THE SOLUTION: What Diagnostics and Tradeoffs Did We Consider?

When tracking down premature pipeline termination, we explored several potential culprits before identifying the predicate mismatch. Whether you hire python developers for scalable data systems or .NET architects, diagnosing pipeline failures requires a methodical elimination of variables.

Did We Consider Global Timeout Adjustments?

Our first assumption was that an overarching HttpClient.Timeout or a cancellation token was aborting the entire request. By default, HttpClient has a 100-second timeout. Since our logs indicated the process aborted in merely 6 seconds (from 08:21:39 to 08:21:45), we quickly ruled out global connection timeouts.

Did We Look Into Task Cancellation Tokens?

We verified the upstream caller to ensure no scoped cancellation tokens were firing. We confirmed that the thread remained alive and no TaskCanceledException was logged. This indicated the pipeline was intentionally returning control back to the caller.

Did We Evaluate Aligning Predicates (The Breakthrough)?

Once we audited the policy definitions side-by-side, the inconsistency became glaring. We realized that in a nested pipeline, the outermost policy dictates the final completion criteria. If an inner policy (Circuit Breaker) handles a specific response but the outer policy (Retry) does not, the unhandled result will slip through and terminate the retry loop.

FINAL IMPLEMENTATION: How Do We Properly Configure Polly Retry and Circuit Breaker?

To fix the issue, we had to ensure that the Retry strategy’s ShouldHandle predicate encompassed all conditions handled by the Circuit Breaker. We updated the Retry policy to also handle the specific HTTP status codes that indicate a transient failure.

Here is the corrected implementation:

services.AddHttpClient<InventoryAppClient>()
    .ConfigureHttpClient(c => c.BaseAddress = new Uri("https://internal-api.gateway/"))
    .AddResilienceHandler("app-pipeline", pipelinebuilder =>
    {
        // Define a shared predicate to ensure consistency across policies
        var transientErrorPredicate = new PredicateBuilder<HttpResponseMessage>()
            .Handle<HttpRequestException>()
            .Handle<BrokenCircuitException>()
            .HandleResult(r => r.StatusCode == System.Net.HttpStatusCode.InternalServerError || 
                               r.StatusCode == System.Net.HttpStatusCode.ServiceUnavailable);
        // Outer Policy: Retry
        pipelinebuilder.AddRetry(new HttpRetryStrategyOptions
        {
            BackoffType = DelayBackoffType.Constant,
            Delay = TimeSpan.FromMilliseconds(100),
            MaxRetryAttempts = 100,
            OnRetry = static args =>
            {
                Serilog.Log.Logger.Information($"Retry {args.AttemptNumber} triggered.");
                return default;
            },
            ShouldHandle = transientErrorPredicate, // Fixed: Now handles HTTP 500s
            UseJitter = false
        });
        // Inner Policy: Circuit Breaker
        pipelinebuilder.AddCircuitBreaker(new HttpCircuitBreakerStrategyOptions
        {
            BreakDuration = TimeSpan.FromSeconds(5),
            FailureRatio = 0.1,
            MinimumThroughput = 2,
            OnClosed = static args => { /* Logging */ return default; },
            OnOpened = static args => { /* Logging */ return default; },
            OnHalfOpened = static args => { /* Logging */ return default; },
            SamplingDuration = TimeSpan.FromSeconds(2),
            ShouldHandle = transientErrorPredicate // Fixed: Uses the same shared predicate
        });
        // Innermost Policy: Timeout per execution
        pipelinebuilder.AddTimeout(TimeSpan.FromSeconds(1));
    });

Validation Steps:

• We triggered controlled failures returning HTTP 500.
• The Circuit Breaker tripped, throwing BrokenCircuitException.
• The Retry strategy successfully handled both the BrokenCircuitException during the Open state and the HTTP 500 during the Half-Open state.
• The pipeline reliably exhausted all 100 retry attempts as initially intended.

LESSONS FOR ENGINEERING TEAMS: What Can You Apply to Your Resilience Strategy?

Complex architectures require deep structural validation. If you plan to hire ai developers for production deployment or backend architects for system reliability, ensure your teams follow these core principles when building resilience pipelines:

• Align Your Predicates: When nesting resilience strategies, the outer strategies must handle all the fault conditions (exceptions and results) that the inner strategies are meant to protect against. Using a shared PredicateBuilder prevents silent leaks.
• Understand Policy Execution Order: In Polly v8 (ResiliencePipelineBuilder), strategies are executed in the order they are added. The first added is the outermost wrapper and the last added is the innermost handler directly surrounding the delegate.
• Circuit Breakers Return Results: Remember that Circuit Breakers do not always throw exceptions. If a Circuit Breaker trips because of an HTTP result (like a 500 error), it will return that HTTP result. Your outer Retry strategy must be prepared to catch it.
• Log Contextually: Log not just that a retry occurred, but why. Our resolution was heavily dependent on seeing Server returned a 500 interwoven with the BrokenCircuitException logs.
• Test State Transitions: Do not just test “Open” and “Closed” states. The “Half-Open” state is where the most complex interactions between nested policies occur.

WRAP UP: How Can You Ensure Fault-Tolerant System Architecture?

Transient faults are inevitable in distributed networks. Our deep dive into Polly’s pipeline behavior underscores the importance of fully understanding how nested resilience policies interact. A minor oversight in a predicate configuration can completely neutralize an otherwise robust Circuit Breaker, leading to unhandled failures in production. By aligning your fault-handling conditions across all pipeline layers, you ensure true application resiliency.

Building fault-tolerant enterprise applications requires teams that understand the fine technical details beneath the surface. Whether you need to optimize a backend microservice or hire app developer to create a mobile app that handles offline gracefully, our engineering pods are equipped to deliver. If you are looking to scale your engineering capabilities with vetted experts, contact us.

Social Hashtags

#Polly #DotNET #CSharp #SoftwareEngineering #Resilience #CircuitBreaker #Microservices #DotNETCore #BackendDevelopment #DistributedSystems #CloudNative #SystemDesign #DevOps #Programming #SoftwareArchitecture #APIDevelopment #DeveloperTips #TechBlog