Automations fail silently and create duplicates for the same underlying reason: they assume every event arrives exactly once and every step succeeds. In practice, events can arrive twice, late, or not at all, and API calls time out halfway. A dependable automation expects this. It recognizes repeats, retries safely, checks its own results on a schedule, and tells a person when something needs attention.
This article explains the specific causes behind duplicate records and missing data, and what to change whether your automations run in Zapier, Make, n8n, or custom code.
What the problem looks like day to day
- The same customer appears twice in your CRM, with slightly different details.
- An order was created twice in the ERP, or a customer received two confirmation emails.
- A lead from your website never reached sales, and nobody knew until the prospect followed up.
- An automation "has been off for two weeks" and was noticed only through a customer complaint.
- Staff run a weekly cleanup to merge duplicates and fill gaps.
The weekly cleanup is a sign that the automation has become a source of work instead of a way to remove it.
Why duplicates happen
At-least-once delivery
Many platforms deliver webhooks at least once, which means sometimes more than once. If the receiving side does not respond in time, the sender retries even though the first attempt succeeded. Shopify, for example, advises using the X-Shopify-Webhook-Id header to ignore duplicate deliveries in its webhook best practices.
Retries without idempotency
A step creates a record, then the connection drops before the confirmation comes back. The tool retries and creates a second record. APIs that support idempotency keys, such as Stripe's, let a client retry safely: the server returns the original result instead of repeating the action. Many business APIs do not offer this, so the automation has to check before it creates.
Deduplication that is narrower than you think
Workflow tools do deduplicate, but within limits. Zapier explains in its help documentation that it compares each item's unique ID with IDs a Zap has seen before, and that deduplication only applies within the same Zap. Two Zaps using the same trigger will both run.
"Create" where "find or create" was needed
Many duplicates come from a step that always creates a contact or company instead of first searching for an existing one by a stable key, such as email or an external ID.
Two automations doing the same job
Over time, different people build overlapping automations: one in the CRM, one in Zapier, one in the ecommerce platform. Each works, and together they create everything twice.
Why failures stay silent
- Errors go to a log nobody reads. Most tools record failures. Few teams route those failures to a person.
- Partial success looks like success. Step one creates the order, step two fails to add the line items. The run shows as completed with an error in the middle.
- The trigger stops firing. An expired connection, a changed field name, or a disabled webhook means nothing runs, so nothing errors.
- Filters swallow records. A filter excludes records that should have passed, and there is no count of what was skipped.
The silent-trigger case is the most dangerous. An automation with no runs looks the same as an automation with no work to do.
If your team is already cleaning up duplicates or discovering gaps weeks later, list your most important automations and what depends on each. We can go through them on a Systems Discovery Call.
How to make automations dependable
1. Give every record a stable key
Store the source system's ID on the destination record, for example the Shopify order ID on the ERP order or the form submission ID on the CRM lead. Before creating anything, look it up by that key. This single change prevents most duplicates.
2. Record what you have processed
Keep a small table of processed event IDs. When an event arrives, check the table first. If it is there, stop.
3. Retry with backoff, and make retries safe
Retry temporary failures with increasing delays. Use idempotency keys where the API supports them, and the lookup step from point 1 where it does not.
4. Reconcile on a schedule
Once a day, or more often for critical flows, compare the source and destination: every paid order should exist in the ERP, every form lead in the CRM. Report or fix the differences. Shopify's guidance recommends this kind of reconciliation job because webhooks can be missed.
5. Alert on silence as well as on errors
Alert when a flow that normally runs fifty times a day has run zero times since morning. This catches expired connections and disabled triggers.
6. Make failures replayable
Failed records should wait in a queue with the reason, so after a fix someone can replay them. Zapier, for example, supports replaying errored runs, and n8n lets you attach an error workflow that starts with an Error Trigger node and can send alerts.
7. Keep one owner per flow
Document which automation moves which data. Before building a new one, check the list.
A quick audit you can run
| Question | Healthy answer |
|---|---|
| Does each destination record store the source ID? | Yes, and it is checked before creating |
| Who is notified when a run fails? | A named person or channel that is read daily |
| Would we notice if the flow stopped running? | Yes, through a volume or heartbeat alert |
| Can failed records be replayed? | Yes, without re-entering data |
| Is anything reconciled on a schedule? | Yes, for business-critical flows |
| Is there a list of all automations and their owners? | Yes, and it is current |
Any "no" on a critical flow is worth fixing before adding new automations.
When to move a flow into custom code
Workflow tools can implement most of the above with care. A flow is a candidate for custom code when the fixes above require many extra steps, when volume makes usage pricing expensive, or when a single failure has real financial impact, such as orders, payments, or inventory. We compare these options in Native connector, middleware, or custom API integration and n8n or custom software.
Common questions
Can we clean up existing duplicates automatically?
Partly. Records that share a stable key can be merged automatically. Records that differ slightly need rules, such as matching on normalized email or phone, and a review step for uncertain matches. Fix the cause first, or the cleanup will need repeating.
Is this only a problem for no-code tools?
No. Custom code has the same failure modes if it is written without idempotency, reconciliation, and alerts. The difference is that in code you can build these in exactly where they are needed.
Next step
If duplicates, gaps, or silent failures are creating cleanup work, book a Systems Discovery Call. It is 45 minutes on your tools, workflows, and where data goes missing. Bring a list of your critical automations and a few examples of duplicates or gaps. You leave with a clear picture of what to fix first, whether you hire us or not.
Farooq Ch
Oct 2026