API Rate Limiting Causing Incomplete Data Exports

If you're pulling large datasets via the API and noticing your exports are mysteriously truncating — yeah, this one tripped me up too when I first looked at it. The symptoms look like a successful request (you get a 200, maybe even a full-looking JSON payload), but then you realize you're missing thousands of records.

Summary

API rate limiting is causing large data exports to terminate early, returning partial datasets without clear error signaling.

Symptoms

• Export scripts report "success" but row counts don't match expected totals
• Last page of paginated results returns fewer records than page_size, with no next URL
• Response headers show X-RateLimit-Remaining: 0 but status code remains 200
• Same export run twice produces different record counts

Affected Versions

FieldPulse API v2.0 and v2.1. Particularly impacts endpoints with high data density:

• GET /work_orders with expanded customer data
• GET /customers with custom fields included
• GET /time_entries across date ranges > 90 days

What's Actually Happening

Here's the gotcha: the API has a secondary limit on "complexity points" that isn't surfaced in the standard rate limit headers. When you request heavy expansions (nested customer objects, line items, form responses), you're burning through this budget faster than the standard 1000 req/min limit suggests.

Once you hit the complexity ceiling, the API quietly truncates your response rather than erroring. Not ideal — we're pushing to get this changed to a proper 429 with a Retry-After header, but that's still in review.

Immediate Workaround

Until the fix ships, you'll want to:

# Add explicit throttling between paginated requests
import time

def fetch_all_work_orders():
    results = []
    url = "https://api.fieldpulse.com/v2/work_orders?page_size=50"
    # ^ smaller page size, avoid expansions in initial fetch
    
    while url:
        resp = requests.get(url, headers=auth_headers)
        data = resp.json()
        results.extend(data['results'])
        
        # Force 200ms delay regardless of rate limit headers
        time.sleep(0.2)
        
        # Check for truncation signal: fewer results than page_size
        if len(data['results']) 'next'):
            print(f"Warning: possible truncation at {url}")
            time.sleep(2)  # back off aggressively
            continue  # retry same page
            
        url = data.get('next')
    
    return results

Then fetch related data (customer details, etc.) in a second pass using batch endpoints where possible. Yeah, it's more requests, but each one stays under the complexity radar.

Better Pattern (Recommended)

If you're doing regular large exports, use the API pagination workaround pattern which includes adaptive backoff based on response size. Also consider:

• Webhook-based sync instead of bulk export for ongoing replication
• Incremental exports using modified_since parameter
• Our new bulk export endpoint (beta, /exports/jobs) — async, no rate limit issues, CSV or JSON output

Status

Engineering is tracking this as API-2847. Target behavior: return 429 with Retry-After and a clear error message when complexity limits are hit. No firm release date yet — I'll update this article when I know more.

Hit me up in the comments if you're seeing different behavior or found another pattern that works.