LDledgerd/change review/#482
review state synced
Pull request #482

Add jittered exponential backoff to payment webhook retries

Fixed retry intervals made failed payment webhooks arrive in synchronized waves, amplifying load on already struggling merchant endpoints. This change spreads attempts over time, gives each endpoint its own retry budget, and stops cleanly after six tries.

sol/retry-jitter→mainsol (agent)
01 / behavior

The retry path, now with breathing room.

Delivery stays immediate on the happy path. Failures enter a bounded, randomized schedule instead of hammering the endpoint on a fixed cadence.

Payment webhook retry flow A webhook is delivered immediately. Success returns a 2xx response. A failed attempt is checked, scheduled with jitter, retried up to six times, then sent to a dead-letter queue. EVENT / 00webhook receivedsigned payload ATTEMPT / Ndeliversame idempotency key CLASSIFYinspect response2xx / 429 / 5xx FAILUREretry eligible?never ordinary 4xx SCHEDULEbackoff + jitter0.5× to 1.5× AFTER 6DLQ 1 5xx yes N < 6

Before / synchronized

Every failure retried at 10 seconds. A merchant outage turned one burst into a repeating burst with no hard stop.

After / distributed

Delays grow from 5 seconds to 5 minutes, each randomized by 0.5× to 1.5×, with dead-lettering after attempt six.

02 / annotated diff

Read the decision, not just the syntax.

Three focused hunks show the algorithm, the delivery policy, and the proof. Margin notes call out intent, risk, and coverage.

Annotation lanes

RationaleRiskTest coverage
GOinternal/retry/backoff.go
+52−8
@@ -12,11 +12,38 @@ package retry
1212 const maxAttempts = 6
13−const retryDelay = 10 * time.Second
13+var retrySchedule = [...]time.Duration{
14+ 5 * time.Second, 20 * time.Second, 60 * time.Second,
15+ 2 * time.Minute, 5 * time.Minute,
16+}
1418
19+func Delay(attempt int, rng *rand.Rand) time.Duration {
20+ base := retrySchedule[min(attempt-1, len(retrySchedule)-1)]
21+ factor := 0.5 + rng.Float64()
22+ return time.Duration(float64(base) * factor)
23+}
GOinternal/webhook/deliver.go
+61−30
@@ -84,16 +103,29 @@ func (d *Deliverer) HandleFailure(ctx context.Context, job Job, res *http.Response)
84−if res.StatusCode >= 400 {
85− d.queue.PushAfter(job, retryDelay)
86−}
103+retryable := res.StatusCode == http.StatusTooManyRequests ||
104+ res.StatusCode >= http.StatusInternalServerError
105+if !retryable {
106+ d.metrics.PermanentFailure(job.EndpointID, res.StatusCode)
107+ return
108+}
92110
111+if job.Attempt >= maxAttempts {
112+ d.deadLetter.Write(ctx, job, "retry_budget_exhausted")
113+ d.alerts.NotifyDeadLetter(ctx, job.EndpointID)
114+ return
115+}
116+d.queue.PushAfter(job.NextAttempt(), retry.Delay(job.Attempt, d.rng))
GOinternal/webhook/deliver_test.go
+29−0
@@ -141,6 +141,35 @@ func TestRetryPolicy(t *testing.T)
141+func TestDelayStaysWithinJitterBounds(t *testing.T) {
142+ rng := rand.New(rand.NewSource(42))
143+ for attempt, base := range retrySchedule {
144+ for sample := 0; sample < 500; sample++ {
145+ got := Delay(attempt+1, rng)
146+ require.GreaterOrEqual(t, got, base/2)
147+ require.LessOrEqual(t, got, base+base/2)
148+ }
149+ }
150+}
03 / your review

Leave a decision the agent can act on.

This is living review state. Every native control persists server-side and returns when you do, even from another machine.

Reviewer checklist

0 / 6complete

Review verdict

Saved as you type. Synced to this page.
Your place is durable. Close now and reopen later. The checklist, verdict, and notes will be exactly where you left them.agent-readable in later sessions →
04 / risk + rollout

Bounded change, observable exit.

The algorithm ships behind a service flag. Delivery latency and dead-letter volume gate each rollout step.

webhook_retry_jitter_v2

Four gates over 24 hours

Advance only while p95 delivery latency stays under 90 seconds and dead-letter rate stays below 0.05%.

1%30 min · internal
10%2 hr · watch queue
50%6 hr · compare
100%24 hr · settle