Built for good networks, shipped to bad ones

Masdary is our B2B ordering app for shops in Aden, Yemen. In August 2026 production told us something no code review had: catalog sync stalled, checkout spun, and some shops saw “No Internet” while our backend was reachable.

The problem

I ran an audit of the whole app. Its verdict fit in one line: the app was correct on a reliable network and structurally unprepared for an unreliable one. Five things compounded.

• We could not see the cause. Crash reporting filtered out network errors, so we saw side effects, not the failing links.
• Sync asked an expensive question. To check whether its catalog was fresh, the phone uploaded its whole local fingerprint map, on links where the uplink is the narrowest part.
• Half-finished syncs looked finished. An interrupted pass could still be stamped fresh, and the offline cache could answer a “server” read with stale data.
• Invisible changes caused full refetches. Product fingerprints changed for fields no customer sees, so devices did the right thing with the wrong input: they refetched the whole catalog, again and again.
• Checkout waited forever. Timeouts had no upper bound, and the order’s idempotency key lived in memory. A cart change or an app restart created a new key, so a retry after a timeout was not guaranteed to be recognised as the same order.

The keys already existed. The flaw was how long a key lived.

The decision: change the protocol, not the timeouts

The tempting fix was to raise timeouts and patch each bug. We did patch the worst bugs first. Then I made the case for replacing the sync itself: the old mechanism uploaded the client’s state to ask a question, and no timeout fixes that.

Four choices shaped the rest:

• Ship the server gate first. Fixing fingerprints in an app release would itself have triggered the fleet-wide refetch we were trying to stop. The server now skips the write when nothing visible changed. That needed no release and helps users who never update.
• No forced upgrades. Old versions of the app keep working.
• No fallback to the old path. If the new sync fails, the app waits and retries it. Falling back would turn every network failure into the most expensive request we have.
• Fix the origin, not the timeout. A field run in Aden showed slow responses from the origin whatever the link did, so full snapshots moved behind a CDN, chosen on measurements from Aden.

The design

Every call

1. Call class
2. Total deadline
3. Retry with jitter
4. Breaker per host

Catalog

1. My version
2. Change feed
3. One local transaction

Order

1. Key minted once, stored
2. Attempt
3. Timeout
4. Same key
5. Same order

Push

1. ERP change
2. Visible to customers?
3. New version
4. Devices sync

• Every call has a class and a deadline. One network policy sorts each call by who is waiting and what giving up costs them. Each call gets a total deadline, not only a timeout that restarts with every chunk.
• The app knows how good the link is. Each call’s outcome feeds a rolling verdict: healthy, degraded, unusable or offline. On a degraded link, deadlines stretch, optional downloads wait, and the banner says “very slow” instead of “no internet”. The thresholds are remote config, so we tune them without a release.
• Retries sit behind circuit breakers. Backoff works per call, per sync pass and per host, with bounded jitter. Each host has a breaker that fails fast, then lets one probe through. Cooldowns ignore the device clock, because device clocks in this population are often wrong.
• The catalog is a change feed. The phone sends its version and gets back only what changed. The rows and the new version are saved in one local transaction: they move together, or neither does. Long syncs pause and resume, and an unknown version is never treated as current. Full snapshots come from the CDN and resume where they stopped. A push message tells devices that a new version exists.
• One key per order. The idempotency key is created once per order, stored across restarts and reused after an unclear timeout. It stays fixed while an attempt is in flight, even if the cart changes, and the server replays the first result. The app never retries a write that has no key: a key minted per attempt is worse than no key.
• Measure in the field. An in-app diagnostics tool runs one probe at a time, records a failed probe as a data point, and always finishes its run.

What I did as the lead

I ran the audit and wrote the plan. Then I wrote the first implementation myself: the first client fixes, the diagnostics tool, the network policy, the key pinning, the server side of the change feed, and the field-results write-up with the CDN comparison. I opened the pull request on 24 Aug 2026 and it merged on 27 Aug.

The team carried it from there: cart and checkout hardening, the tests, the explainer for the new catalog sync and the handover.

What changed

The redesign merged into the app in August 2026. On a device run, the app opened without the connectivity gate, the change feed ran end to end, and stale cache no longer overwrote local data. A control run in Aden confirmed the premises: the upload we removed was a real cost, the slow origin did not depend on the link, which justified the CDN, and we retuned a threshold remotely without shipping a release.

What I cannot show yet is a before-and-after measurement on a lossy link. That run is the next step, and its result is what I want to publish here.

Carried into the super app

HSA Group’s super app is designed on the same rules, moved into a shared SDK: one retry policy on the client, load shedding on the backend, and an idempotency key that stays with every mutating call from end to end. Infrastructure faults become unknowns, never wrong answers. Its catalog is local-first, and in Oct 2026 I measured from Aden to choose where it is hosted.