When an integration fails silently, records stop moving but nothing reports an error, so the gap shows up days or weeks later in a customer complaint or month-end reconciliation. You prevent it with structured logs, retries with backoff, a dead-letter queue, alerts on errors and on silence (say, zero records in 24 hours), and a written runbook.
What happens when an API integration fails silently, and how do you monitor and recover from it?
What a silent failure actually looks like
An integration fails loudly when it crashes and someone gets an email. It fails silently when it keeps running but stops doing its job, and every screen still looks normal. The usual causes:
- An OAuth token expired, and the job writes
401 Unauthorizedto a log nobody reads. - The API returned
429 Too Many Requestsand the code dropped the record instead of trying again. - Someone renamed or deleted a field upstream, so the mapping writes blanks or skips the record.
- A webhook arrived before the record it belongs to existed, and the handler returned success anyway.
- Half a batch succeeded, half failed, and nothing recorded which half.
- A no-code platform paused the workflow after repeated errors and notified one person, who has since left.
The outage itself is rarely the expensive part. The cleanup is: working out which records are missing, which were created twice, and which customers were invoiced wrong, usually by comparing exports by hand.
The six pieces that make failures visible and recoverable
Structured logging
Every run writes one line per record: source ID, target ID, action, outcome, error message and timestamp. Keep 30 to 90 days. The test is simple: can you answer "what happened to invoice 1042?" in under a minute without an engineer.
Retries with exponential backoff
Transient errors (timeouts, 5xx responses, 429) are retried with growing waits, for example 1, 2, 4, 8 and 16 seconds with a little random jitter, then stop after five attempts. If the API sends a Retry-After header, wait that long instead. Permanent errors such as a 400 validation failure are not retried, because the same request will fail the same way.
Retries need idempotency so a second attempt cannot create a second record. That means storing the source system’s ID on the target record and checking for it before creating, or using idempotency keys where the API supports them, as Stripe does.
A dead-letter queue
When a record exhausts its retries, it goes to a holding table with its full payload and the last error. Nothing is dropped. Once the cause is fixed, one command replays the queue in order.
Alerting on errors and on silence
Alert when the dead-letter queue is not empty, when the error rate on a run passes a threshold, and when nothing has succeeded in a set window. The silence alert, a heartbeat, catches the failures that throw no error at all. Send alerts to a shared Slack channel or email group, never to one person.
Re-authentication
Tokens get revoked when the connecting user leaves, changes a password or loses a licence, or when an admin removes the app. Connect through a dedicated integration user, refresh tokens before they expire, and on the first 401 pause the queue and alert with reconnect instructions. Pausing matters: records wait instead of failing one by one.
Rate limits
HubSpot, Salesforce and QuickBooks Online all cap API calls per time window, and the limits vary by plan. Throttle below the cap, use batch endpoints, and schedule backfills off-hours so they do not starve the live sync. Our Fundz connector framework handles rate limits per tenant across HubSpot, Salesforce and Gmail, because one busy customer must not block the rest.
Worked example: HubSpot closed-won to QuickBooks to Stripe
A deal moves to closed-won in HubSpot. The integration creates a QuickBooks Online customer and a draft invoice, and when the Stripe payment webhook arrives it marks the invoice paid and updates the deal. Here is where each step breaks and what the build does about it.
| Step | What goes wrong | How it is caught | How it recovers |
|---|---|---|---|
| HubSpot closed-won webhook | Delivery missed | Nightly check compares closed-won deals to QuickBooks customers | Missing deals are queued automatically |
| Create QuickBooks customer | Retry creates a duplicate | Lookup by stored HubSpot ID before create | Retry finds the existing customer |
| Create draft invoice | QuickBooks token expired | First 401 alerts the channel; queue pauses | Reconnect, then the queue resumes in order |
| Stripe payment webhook | Arrives before the invoice exists | Event parked as parent missing | Retried every 15 minutes for 24 hours, then alerts |
| Update HubSpot deal | 429 rate limit | Logged; Retry-After honoured | Backoff retries, then dead-letter queue |
This is the difference between a recipe and an integration you can leave alone. A Stripe-to-Salesforce and Google Sheets finance sync we built on these patterns has run for two years with minimal maintenance.
On cost: our typical two-system integration runs $1,500–$6,000. Field mapping and error handling move a build within that range, and a payments flow like this one, with reconciliation and a dead-letter queue, sits toward the top. These are Smplfy’s figures at $85 an hour.
What a runbook contains
A runbook is the short document that lets someone other than the original developer keep the integration healthy. Ours cover:
- What the integration does, which systems it touches, and who owns each one.
- Where the logs live and how to look up a single record.
- Every alert, what it means and what to do first.
- How to reconnect each system: which integration user, which admin screen, which permissions.
- How to inspect and replay the dead-letter queue.
- How to backfill a date range after an outage without creating duplicates.
- Known failure modes and their fixes.
- Who holds access to what, and who to call.
Write it so an operations lead can handle a token expiry and a replay without an engineer. If only the developer can recover the integration, it has a single point of failure.
When to build this, and when not to
- Stay on Zapier or Make when volume is low, a missed record would be noticed the same day, and the platform’s built-in run history and replay cover you. Point its error notifications at a shared inbox.
- Build it in code when the integration moves money, invoices or customer records, when steps depend on each other, or when one bad week costs more hours of cleanup than the build.
- Do not hire anyone yet if a few dozen records a month move and one person can check them weekly against a report. A manual check is a valid monitor.
If you already have an integration in code without these pieces, adding logging, a dead-letter queue and alerts is usually cheaper than rebuilding it.
Frequently asked questions
How do I know if my integration is failing silently right now?
Compare counts. Pull last month’s records from the source system (closed-won deals, new orders, payments) and the matching records in the target system. If the numbers differ or duplicates appear, something is failing without telling you.
What is the difference between a retry and a dead-letter queue?
A retry tries the same request again after a short wait, which fixes timeouts and rate limits. A dead-letter queue holds records that still failed after every retry, with the payload and error, so nothing is lost. Once the cause is fixed, the queue is replayed.
How long should integration logs be kept?
Keep per-record logs for 30 to 90 days, long enough to cover a month-end reconciliation and the questions that follow it. Finance or regulated data may need longer retention, set by your own policy. Store logs where a non-engineer can search them by record ID.
Can Zapier or Make handle retries and alerts?
Partly. Both keep a run history and let you replay failed runs, and both can send error notifications. What they do not give you by default is a heartbeat alert when nothing runs, duplicate prevention across retries, or a reconciliation check, which is why money-moving flows usually end up in code.
What does it cost to add monitoring and error handling to an integration?
At Smplfy, a typical two-system integration costs $1,500–$6,000, and error handling is one of the main things that moves it toward the top. Adding logging, retries and alerts to an existing integration is billed at $85 an hour and depends on how the current code is structured.
Related answers
Want to know where your integration could fail silently?
We will review your setup and send a short written scope, or walk through it on a free 20-minute call. Have ready the systems involved, roughly how many records move each month, and what you check by hand today. Start a conversation.