Fix n8n Merge Issues & Execution Loops in Workflows

INTRODUCTION

While working on an automated media generation pipeline for a SaaS platform, we encountered a fascinating challenge with workflow orchestration. The system was designed to take a batch of user requests via a messaging webhook, cross-reference them against a historical database to check for previous generations, trigger an AI generation API for net-new requests, and finally organize the results into newly provisioned cloud storage folders.

At first glance, this is a standard ETL and orchestration flow. However, during integration testing, we realized the workflow was exhibiting erratic behavior. Specifically, our data arrays were inflating inexplicably after merging historical and new data, and our cloud storage module was creating duplicate folders instead of a single designated directory per run.

When engineering teams build complex workflow automations, understanding the underlying execution engine’s default behaviors is crucial. Issues like array inflation or unintended loop executions can lead to severe API rate-limiting, corrupted data states, and inflated cloud costs. This scenario inspired this article to help other engineering teams avoid similar pitfalls when managing list-based execution engines like n8n.

PROBLEM CONTEXT

The business use case required processing multiple incoming requests simultaneously. For instance, a user might request four new digital assets. The architecture dictated that we first query a cloud spreadsheet to see if any of these four assets had been generated previously. If they existed, we would skip generation; if not, we would proceed to generate them.

Within our n8n architecture, this logic surfaced two distinct operational bottlenecks:

• The Merge Array Inflation: We routed two inputs into a single Code node. Input 1 contained 2 historical items, and Input 2 contained 4 new requested items. Instead of outputting a mapped array of 4 items with a boolean flag (e.g., generatedBefore: true/false), the workflow produced an inflated output of 8 items, breaking downstream logic.
• The Duplicate Execution Loop: Further down the flow, a node responsible for creating a destination folder in our cloud storage was executing multiple times. We wanted one folder created per workflow trigger, but the system was provisioning a new folder for every individual item in the array.

WHAT WENT WRONG

To resolve this, we had to dig into the internal execution mechanics of the automation engine.

1. The Cartesian Product Problem in Code Nodes

In many node-based automation tools, when you connect multiple inputs into a single processing node, the engine attempts to pair them. Because we were passing an array of 2 items and an array of 4 items into the Code node, the engine implicitly multiplied the context, creating a Cartesian product. Even when configuring the node to “Run Once for All Items,” the presence of two distinct upstream connections confused the context execution, resulting in 8 overlapping items.

2. Item-Based Execution Contexts

The second issue—duplicate folder creation—stems from how downstream nodes process arrays. By default, most action nodes in n8n run once per item in the incoming data array. Because our array contained 4 items, the “Create Folder” API was triggered 4 times. While applying an “Execute Once” parameter or using an Aggregate node before the folder creation stops the duplicate calls, it inadvertently strips the array down to a single item. This means subsequent nodes (like the file upload node) only receive one item to process, completely breaking the batch upload requirement.

When organizations hire software developers to build out robust enterprise automations, overcoming these nuanced context-execution hurdles is what separates fragile scripts from production-ready architecture.

HOW WE APPROACHED THE SOLUTION

Our diagnostic process focused on decoupling the data fetching from the primary execution loop and restructuring the workflow’s branching logic.

Fixing the Merge Issue

Instead of physically wiring two inputs into the Code node, we realized we only needed to pass the primary data stream (the new requests) into the node. For the secondary data (the historical records), we could programmatically query the upstream node’s output directly using the engine’s built-in methods. This completely bypasses the engine’s implicit multi-input array multiplication.

Fixing the Downstream Execution Loop

For the duplicate folder creation, we needed to satisfy two conflicting requirements: execute the folder creation exactly once, and preserve the full array of items for the subsequent upload node. We considered three tradeoffs:

• Option A: Use “Execute Once” on the folder node and accept data loss. (Rejected)
• Option B: Write custom code to interact with the cloud storage API, handling the array manually. (Rejected to preserve visual workflow maintainability)
• Option C: Use parallel branching. Route the execution to a parallel branch that creates the folder once, saves the resulting Folder ID, and then merges it back with the main item array. (Selected)

This architectural thinking is heavily applied when we hire Python developers for scalable data systems, as decoupling state modification from data transformation is a foundational principle of stable pipelines.

FINAL IMPLEMENTATION

Here is how we refactored the pipeline to ensure clean execution.

Refactored Code Node for Clean Merging

We disconnected the historical data node from the Code node’s input. The Code node now only receives the new requests. We then used the $('NodeName').all() syntax to fetch the historical data programmatically.

// Set Node to: "Run Once for All Items"
// 1. Fetch the primary input (New Requests)
const newRequests = $input.all().map(item => item.json);
// 2. Fetch the historical data explicitly from the isolated upstream node
const historicalData = $('Fetch Existing Records').all().map(item => item.json);
// 3. Map the arrays cleanly without engine-induced inflation
const processedItems = newRequests.map((request) => {
  const matchFound = historicalData.find(
    (historyItem) => historyItem.assetName === request.name
  );
  
  return {
    name: request.name,
    generatedBefore: !!matchFound,
    prompt: `Generate isolated high-quality asset for ${request.name}`
  };
});
// 4. Return formatted data for downstream processing
return processedItems.map(item => ({ json: item }));

Refactored Folder Creation Logic

To solve the duplicate folder creation without losing our item array, we utilized a “Merge” node configured to combine the data streams properly.

1. Branch 1 (Data): The array of 4 generated assets waits at a Merge node.
2. Branch 2 (Action): A single execution path triggers the “Create Folder” node once. The output is a single JSON object containing the new folderId.
3. The Merge: We configure the Merge node to “Combine” or “Append” the folderId to every item in Branch 1.

Now, the downstream upload node receives 4 items, each correctly tagged with the identical destination folderId, and the folder creation API is only hit once. Whether you hire AI developers for production deployment or automation engineers, this pattern—isolating side-effects from data streams—is essential.

LESSONS FOR ENGINEERING TEAMS

Based on this implementation, here are actionable insights engineering teams should apply when orchestrating complex workflows:

• Avoid Implicit Multi-Input Wiring: In list-based workflow engines, physically connecting multiple data streams into a single code node often results in Cartesian products. Query secondary data programmatically whenever possible.
• Decouple State Changes from Data Pipelines: Actions that change external state (like creating a directory) should be isolated from nodes processing item arrays to prevent duplicate executions.
• Understand the Engine’s Looping Mechanics: Always verify whether a node executes per-item or per-run. Relying on default configurations will cause unexpected API spam.
• Use Merge Nodes for Enrichment: When you need a single global variable (like a new Folder ID) applied to a batch of items, use parallel processing and a Merge node to append the data, rather than relying on sequential pass-through.
• Implement Idempotency: Whenever possible, design downstream systems to be idempotent. If the folder creation API was strictly idempotent based on a unique timestamped key, duplicate calls would fail gracefully rather than cluttering storage.
• Scale Testing: Always test workflows with arrays of varying lengths (e.g., 1 item, 0 items, 50 items) to expose implicit looping bugs before they hit production.

WRAP UP

Debugging inflated arrays and duplicate API calls in workflow automation reveals a core truth about visual node-based architectures: they require the exact same rigor, decoupled design, and algorithmic thinking as traditional software development. By explicitly managing data fetching and branching our execution paths, we transformed an erratic automation script into a resilient, production-grade pipeline. When businesses aim to modernize their operations, ensuring they have the right architectural expertise is critical—whether they are building from scratch or looking to hire dotnet developers for enterprise modernization.

Social Hashtags

#n8n #WorkflowAutomation #NoCode #AutomationEngineering #DevOps #APIIntegration #LowCode #ProcessAutomation #SaaSDevelopment #BackendDevelopment #CloudAutomation #EngineeringTips

If your team is struggling with scaling automation workflows, complex integrations, or building reliable engineering pipelines, contact us to see how our dedicated engineering teams can help you deliver robust solutions.