• State Verified Answer
  • Replies 5 replies
  • Answers 2 answers
  • Subscribers 2 subscribers
  • Views 3 views
  • Users 0 members are here
Options
Related

Batch updates vs single record API calls

Omar Hassan

The current API documentation suggests that all write operations (POST, PATCH, DELETE) must be performed on individual records. However, for large-scale data synchronization scenarios — e.g., updating status on 10,000+ work orders nightly from an external ERP — this presents significant performance concerns.

Specifically, I am seeking clarification on the following:

  1. Does the FieldPulse API expose any batch or bulk endpoints (e.g., /batch, /bulk, or JSON:API-style compound documents) that would allow multiple records to be modified in a single request?
  2. If not, what is the recommended architecture for high-volume operations? I.e., should clients implement exponential backoff with jitter per RFC 7234, or are there higher rate limits available for authenticated integration partners?
  3. Are there any idempotency guarantees (e.g., Idempotency-Key headers per RFC 7230) to prevent duplicate records in the event of network partition during a long-running batch process?

I have reviewed API Rate Limits and Best Practices, but it does not address the batch semantics question directly. Any guidance on the intended pattern here would be appreciated.

Parents
  • 0

    Hey Omar — yeah, this is a gap in the docs that we're actively working to address. Short version: no batch endpoints today, but there are some patterns that'll keep you out of trouble.

    Right now every write is single-record. For your 10K nightly sync scenario, I'd recommend:

    // Rough pattern we see working at scale
const BATCH_SIZE = 100;        // concurrent in-flight
const RATE_LIMIT_RPS = 10;     // conservative, burst to 20 if needed
const RETRIES = 3;

// implement token bucket + exponential backoff
// expect ~15-20 min for 10K records with this setup

    The rate limit is per API key, not per account — so you can shard across multiple keys if you really need to move faster, though honestly I'd push back on whether you need all 10K in one batch. Incremental sync (only changed since last run) usually drops that by 90%+.

    Re: idempotency — no Idempotency-Key header yet, but most POSTs return the record ID which you can store and use for PATCH deduplication. For creates specifically, we're looking at adding UUID client generation in Q4; there's an internal ticket you can reference if you need to track it (ENG-4421).

    I'll flag this thread when the batch RFC goes public. It's been discussed but not prioritized — your use case helps make that case.

Reply
  • 0

    Hey Omar — yeah, this is a gap in the docs that we're actively working to address. Short version: no batch endpoints today, but there are some patterns that'll keep you out of trouble.

    Right now every write is single-record. For your 10K nightly sync scenario, I'd recommend:

    // Rough pattern we see working at scale
const BATCH_SIZE = 100;        // concurrent in-flight
const RATE_LIMIT_RPS = 10;     // conservative, burst to 20 if needed
const RETRIES = 3;

// implement token bucket + exponential backoff
// expect ~15-20 min for 10K records with this setup

    The rate limit is per API key, not per account — so you can shard across multiple keys if you really need to move faster, though honestly I'd push back on whether you need all 10K in one batch. Incremental sync (only changed since last run) usually drops that by 90%+.

    Re: idempotency — no Idempotency-Key header yet, but most POSTs return the record ID which you can store and use for PATCH deduplication. For creates specifically, we're looking at adding UUID client generation in Q4; there's an internal ticket you can reference if you need to track it (ENG-4421).

    I'll flag this thread when the batch RFC goes public. It's been discussed but not prioritized — your use case helps make that case.

Children
No Data