Export all work order data to data warehouse

Question

From a governance perspective, our organization requires comprehensive historical work order data for compliance reporting and business intelligence initiatives. We are evaluating options for establishing an automated pipeline from FieldPulse to our Snowflake data warehouse.

It is worth noting that our current volume is approximately 50,000 work orders annually across 12 regional branches, with projected 40% year-over-year growth. We require:

Full historical export (5+ years retention)
Incremental daily sync capability
Preservation of audit timestamps and user attribution
Custom field inclusion without truncation

We have considered the native CSV export functionality documented in Exporting Data to CSV, but this appears unsuitable for automated pipelines at our scale.

Specifically, we seek clarification on:

Does the FieldPulse API support cursor-based pagination for reliable incremental extraction without missed records during high-activity periods?
Are there documented rate limits that would constrain a daily full-table synchronization approach?
Is there a recommended pattern for handling deleted work orders (soft-delete visibility versus hard-delete propagation)?
Does FieldPulse offer a direct data warehouse connector, or must all integration route through the REST API?

I have confirmed that our technical team has provisioned API credentials with appropriate scoping. We are prepared to implement a custom ETL process if necessary, but prefer to align with established patterns before committing engineering resources.

Reference: Our security review indicates that API Rate Limits and Best Practices provides partial guidance, but does not address data warehouse-specific considerations.

Eli Torres (Staff) · Accepted Answer

Hi Fatima — great to see this level of planning going into the pipeline. Let me address each point with what we've seen work at similar scale.

Cursor-based pagination

We don't currently support cursor-based pagination (I know, I know — it's on the roadmap). What we do have is offset-based pagination with a stable sort on updated_at. The pattern that works for incremental sync is:

GET /v1/work_orders?updated_since={last_sync_timestamp}&sort=updated_at&order=asc&limit=500

The key gotcha: if a record gets updated during your pagination walk, it can appear twice (once on the current page, once on a later page). We recommend deduplicating on id in your staging layer.

Rate limits

100 req/min for standard keys, 500 req/min for enterprise. At 50k records with 500/page, you're looking at ~100 requests for a full sync — well within limits. The pain point is historical backfill: 5 years × 50k = 250k records = 500 requests, which is fine, but you'll want to throttle to avoid hammering the API.

Deleted records

We soft-delete (status = archived). Hard deletes are rare and require support intervention. For your warehouse, I'd recommend treating archived records as "deleted" in your BI layer but keeping them in cold storage for compliance.

Warehouse connectors

No native connector today. API-only, though we're in early talks with Fivetran about a partner-built connector — nothing committed.

Happy to share a Python sample that implements the incremental pattern with backoff/retry if that would help.

Devon Marsh · Answer

YMMV on the offset pagination — we tried exactly what Eli described and hit edge cases during our fiscal year-end when dispatchers were bulk-updating hundreds of records. The stable sort on updated_at held, but we saw some records with identical timestamps get reordered across pages. Our workaround: we added a secondary sort on id and dedupe aggressively in staging. Heads up that the API doesn't document this clearly, but sort=updated_at,id does work. Also worth noting: custom fields come back as a nested object in the JSON, not flat columns. Your ETL will need to handle that flattening unless you want variant schemas in Snowflake. "custom_fields": {
 "cf_123": {"label": "Equipment Type", "value": "HVAC-R"},
 "cf_456": {"label": "Warranty Expiry", "value": "2026-03-15"}
} We ended up building a type-2 slowly-changing dimension table for the custom fields specifically since they change more often than the base work order data.