• State Verified Answer
  • Replies 5 replies
  • Answers 1 answer
  • Subscribers 2 subscribers
  • Views 2 views
  • Users 0 members are here
Options
Related

Webhook retry behavior on failed deliveries

Carlos Vega

Seeing inconsistent delivery on our webhook endpoint. Some events arrive 3-4 times. Need to understand:

  • What's the retry policy? Exponential backoff or fixed intervals?
  • How many attempts before FieldPulse gives up?
  • Is there a dead letter queue or visibility into failed deliveries?

Currently returning 500 on purpose for testing — seeing retries come in at weird intervals. No docs on this.

Also: what's the idempotency key situation? Need to dedupe on our end.

Parents
  • 0

    Carlos — yeah, the retry behavior isn't super well documented yet (we're fixing that). Here's what's happening:

    Current retry policy:

    • Exponential backoff: 1s, 2s, 4s, 8s, 16s, then every 60s
    • Max attempts: 24 hours worth of retries, then the event is dropped
    • No dead letter queue today — we just log and move on

    The "weird intervals" you're seeing are probably the exponential backoff plus some jitter we add to prevent thundering herd. The jitter is ±20% of the calculated delay.

    Idempotency: Each delivery attempt has a unique X-FieldPulse-Delivery-Id header, but that's per-attempt, not per-event. The event itself has event.id in the payload — that's what you want to dedupe on.

    Example payload structure:

    {
  "event": {
    "id": "evt_abc123",
    "type": "work_order.status_changed",
    "created_at": "2025-08-18T10:20:00Z"
  },
  "data": { ... }
}

    Store evt_abc123 with a TTL and ignore duplicates. We don't guarantee exactly-once delivery.

    Worth noting: if your endpoint is consistently failing (5xx or timeouts > 30s), we do start backing off the connection pool for your endpoint, which can delay initial delivery attempts. So that 500 test might be affecting observed behavior.

    We're tracking a proper delivery dashboard internally — no ETA but it's on the roadmap. For now, email me your webhook ID if you want me to pull recent delivery attempts from our logs.

Reply
  • 0

    Carlos — yeah, the retry behavior isn't super well documented yet (we're fixing that). Here's what's happening:

    Current retry policy:

    • Exponential backoff: 1s, 2s, 4s, 8s, 16s, then every 60s
    • Max attempts: 24 hours worth of retries, then the event is dropped
    • No dead letter queue today — we just log and move on

    The "weird intervals" you're seeing are probably the exponential backoff plus some jitter we add to prevent thundering herd. The jitter is ±20% of the calculated delay.

    Idempotency: Each delivery attempt has a unique X-FieldPulse-Delivery-Id header, but that's per-attempt, not per-event. The event itself has event.id in the payload — that's what you want to dedupe on.

    Example payload structure:

    {
  "event": {
    "id": "evt_abc123",
    "type": "work_order.status_changed",
    "created_at": "2025-08-18T10:20:00Z"
  },
  "data": { ... }
}

    Store evt_abc123 with a TTL and ignore duplicates. We don't guarantee exactly-once delivery.

    Worth noting: if your endpoint is consistently failing (5xx or timeouts > 30s), we do start backing off the connection pool for your endpoint, which can delay initial delivery attempts. So that 500 test might be affecting observed behavior.

    We're tracking a proper delivery dashboard internally — no ETA but it's on the roadmap. For now, email me your webhook ID if you want me to pull recent delivery attempts from our logs.

Children
No Data