All case studies

Building Software That Stays Reliable When Your Partners' Systems Fail

A small business owner on the phone while working on a laptop in her studio

Your customers never see your payment gateway. They never see your courier's tracking system, your supplier's stock feed or the accounting package that issues your invoices. They see your website, your app and your name on the receipt.

So when one of those partners has a bad day, nobody blames the partner. They blame you.

Modern businesses run on these connections. APIs, the channels that let one system talk to another, now make up 57% of the dynamic internet traffic Cloudflare handles. And partners do fail. The Uptime Institute has found that roughly two-thirds of publicly reported outages come from third-party providers such as cloud, telecoms and hosting companies, not from the businesses that rely on them.

This article explains what it takes to build software that stays reliable when the systems around it don't, using real examples from a US online gaming platform we support. Every one of those examples was caused by an outside provider, and every one was handled without a customer being charged or paid twice.

A small business owner on the phone while working on a laptop in her studio
When a partner's system goes wrong, the phone call comes to you, not to them.

How Partners Let You Down

Outages are only one kind of failure, and usually not the most expensive one. In practice, partner systems let you down in quieter ways:

  • They go silent mid-transaction. You ask the payment gateway to charge a card, and the connection drops before the answer comes back. Did the charge go through or not?
  • They send the same message twice, or late. A courier confirms a delivery, then confirms it again an hour later. A bank notification arrives after your system has already given up on it.
  • They send messages in the wrong order. A "cancel this" instruction arrives before the original order it is meant to cancel.
  • They change their rules overnight. A provider lowers a transaction limit, or starts offering a new option your system has never seen.
  • They go into maintenance, sometimes with notice and sometimes without.

None of this is unusual. It is simply what happens when several independent systems, run by different companies, try to agree about money and orders in real time.

What Happens Without a Plan

When software isn't built for these moments, the damage lands in three places.

First, money gets stuck. A customer pays, but the order never confirms. A supplier is paid, but your books still say "pending". Funds sit in limbo while everyone works out who has what.

Second, customers lose trust. A double charge, a missing refund or a broken checkout page feels personal to the person on the other end. PwC's customer research found that one in three customers would leave a brand they love after a single bad experience.

Third, your staff become detectives. Someone has to compare your records against the partner's, line by line, and decide what really happened to each transaction. That can take days, and it pulls your best people away from the work that grows the business.

A customer tapping a blank bank card on a card payment machine
A single payment can pass through several companies' systems before it settles. Any one of them can go quiet halfway.

What "Resilient Integration" Means in Plain Terms

A resilient integration is a connection to a partner system that is built on the assumption that the partner will, sooner or later, misbehave. In everyday terms, it comes down to four promises.

1. Every transaction can be retried safely

If your system asks a partner to do something and gets no answer, the natural thing is to ask again. The danger is that the first request actually worked, and asking again charges the customer twice.

The fix is to give every transaction its own unique reference and send that same reference every time you ask. The receiving system recognises a repeat and returns the original answer instead of acting again. Engineers call this "idempotency"; you can think of it as an order number that can never be processed twice. Large payment platforms build their own systems around exactly this idea, and we apply it in both directions: to the requests we send out, and to the messages partners send back to us.

Diagram: a payment request with order reference ORDER-1042 is sent, the reply is lost, the same request is sent again with the same reference, and the provider returns the original result so the customer is charged once
A lost reply is harmless when every request carries a reference the other side recognises.

On the gaming platform, this mattered more than usual. Game providers report wins and losses to the platform's wallet every time a round ends. Some of them, after a player's connection drops and they reopen a game, report the result of the earlier round under a brand-new session. A naive system would either reject that result, leaving a winning player unpaid, or treat it as new money and pay it twice. We built the wallet to match each result to the original bet by round, player, currency and game, check that the bet was funded exactly once, and then pay it exactly once, no matter which session it arrives under.

The same thinking covers messages that arrive in the wrong order. If a provider sends a "cancel" before the transaction it refers to has been recorded, the system writes down a permanent note that this transaction was cancelled. If the original then turns up late, it finds that note and cannot charge the player. Nothing has to be cleaned up by hand.

2. Stuck items are detected and reconciled automatically

Even with safe retries, some transactions will end up half-finished because a partner stopped responding at exactly the wrong moment. The question is whether you find out from a regular automated check or from an angry customer.

Reconciliation simply means comparing your records with your partner's and making them agree. A resilient system does this on a schedule, settles what it safely can using the partner's confirmed result, and hands anything unclear to a person with the evidence already gathered.

Diagram of a four-step cycle: compare records, spot the gaps, settle automatically, flag the rest to staff with evidence attached
Reconciliation works best as a routine the system runs, not a project your staff take on after something goes wrong.

During the platform's launch week, transactions got stuck at four separate external game providers, each for its own reasons on the provider's side. In every case we traced the affected rounds against the provider's own records, rehearsed each correction in a trial run that changed nothing, applied it in one controlled step, and then confirmed that a repeat of the provider's original message could not pay out again. Every case was resolved, and no customer was paid twice.

We use AI throughout this kind of work: to sift through large volumes of logs and partner records, to spot patterns across thousands of transactions, and to draft the investigation reports. A person on our team reviews every finding and approves every change to money.

Customer support staff wearing headsets in a bright office
Every stuck transaction the system settles on its own is one fewer call your support team has to take.

3. Customers see a clear message, not a broken page

Sometimes the right answer is to stop using a partner for a while. If a provider announces maintenance, or is clearly struggling, the worst outcome is a customer starting a payment or a booking that is bound to fail.

On the gaming platform, staff can schedule a maintenance window for any outside provider from the back office. During that window, customers who open an affected game see a clear maintenance screen instead of a spinning wheel or an error. Everything else on the platform keeps working, and the game comes back on its own when the window ends.

For other businesses, the same idea applies to a courier that cannot take bookings, a payment method that is temporarily down, or a booking system under maintenance. Your customers see an honest message and, where possible, an alternative, and your brand keeps its dignity.

A parcel being handed from one person to another
Couriers, booking systems and suppliers all have off days. A clear message keeps customers informed while the partner recovers.

4. Rule changes are handled through a setting, not a rebuild

Partners change their terms with little warning. At one point the gaming platform's payout provider cut its payout limit from USD 40,000 to USD 5,000, with almost no notice. Because payout limits are settings that staff manage in the back office rather than values buried in code, operations carried on. No emergency rebuild was needed. Each limit change is recorded with what it was before and after, so there is always a clear history of who changed what.

The same provider later offered customers a way to receive payouts that the business did not support. We built the platform to check each payout notification against what the customer actually confirmed on the provider's page, not just what they originally asked for. A payout only goes out through a confirmed and approved method. Anything unfamiliar is held for review instead of being paid, so an unexpected change on the partner's side can only ever stop a payment, never send one the wrong way.

What This Means for Your Business

You may not run a gaming platform, but if your business takes payments online, books couriers, syncs orders with an accounting or ERP system, or takes reservations through a booking platform, you face the same risks. Building your integrations to be resilient gives you:

  • Protected revenue. Orders don't get lost when a partner goes quiet, and customers are never charged twice, so you avoid refunds, chargebacks and disputes.
  • Time back for your team. Automatic reconciliation replaces days of manual checking with a short list of exceptions, each with the evidence attached.
  • Customer trust that survives a partner's bad day. A clear maintenance message and a correct balance make you look in control, even when someone else's system has failed.
  • Freedom to change partners and terms. When limits and rules live in settings, a partner's overnight change becomes a quick adjustment rather than an urgent software project.
  • A clear audit trail. Every retry, cancellation and correction is recorded, which makes conversations with partners, auditors and regulators far easier.
Four colleagues smiling while reviewing work together on a laptop
Resilient integrations free your team to focus on customers and growth, not on untangling partner problems.

Questions to Ask About Your Own Systems

If you are not sure how your current software would cope, these questions are a good place to start:

  • If our payment provider stopped answering in the middle of a payment, could a customer be charged twice?
  • How would we find out about a transaction that is stuck between us and a partner? Would we spot it, or would the customer have to tell us?
  • What does a customer see when one of our partners is down?
  • If a partner changed a limit or a rule tonight, would we need a developer to keep trading tomorrow?

If any of these answers make you uneasy, you are not alone. Most businesses only discover how their integrations behave on the day a partner fails.

How Dev3 Studio Can Help

Dev3 Studio designs and builds integrations with payment gateways, couriers, accounting and ERP systems, booking platforms and other partner APIs. We build safe retries, automatic reconciliation, clear maintenance handling and adjustable limits in from the start. We can also review your existing integrations and strengthen the ones that carry the most risk. We work with businesses across South Africa and internationally, and we stay on after launch to keep those connections healthy as your partners change.

If your business depends on systems you don't control, let's talk about making it resilient.

A few more answers.

Ask us about your setup
What is a resilient integration?

It is a connection between your software and a partner's system, such as a payment gateway, courier or accounting package, that is built to expect the partner to fail sometimes. It retries safely without charging anyone twice, finds and settles stuck transactions on its own, shows customers a clear message when the partner is down, and lets staff adjust limits and rules without new code.

How do you stop customers being charged twice when a payment provider times out?

Every transaction gets its own unique reference, and the same reference is sent every time the request is repeated. The receiving system recognises a repeat and returns the original result instead of acting again, so a lost reply can be retried safely.

What is automatic reconciliation?

It is a scheduled check that compares your records with your partner's, spots anything stuck, missing or mismatched, settles what it safely can using the partner's confirmed result, and passes anything unclear to your staff with the evidence already gathered.

Can you make our existing integrations more reliable without rebuilding everything?

Usually, yes. We start by reviewing the integrations that carry the most money or customer impact, then add safe retries, reconciliation checks, maintenance handling and adjustable limits where they matter most.

Which kinds of systems do you integrate with?

We integrate with payment gateways and payout providers, couriers, accounting and ERP systems, booking platforms and other partner APIs, for businesses in South Africa and internationally.

We use cookies from Google Analytics and Google Ads to see how this site is used and whether our ads are working. They stay off unless you click "Accept". How we use cookies