Data Exports: do delta exports overlap, and what is the recommended loading strategy?

Hello,

We are designing an ingestion pipeline that loads Data Exports into our data warehouse, and we would like to confirm how delta exports should be handled before we finalize it.

Specifically:

  1. What time window does each delta export cover? Does it include only the changes since the previous delta, or a longer period?
  2. We are evaluating the paid tier, where deltas run every 30 minutes. If each delta covers a longer window than that, consecutive deltas would contain overlapping records. Is that the expected behavior?
  3. Given the above, what loading strategy do you recommend: appending each delta, or merging it into the existing tables?

Thank you in advance.

Hi! I hope you are doing well!

Good catch, they do overlap. Each delta includes changes from up to 8 hours before the run, on both tiers. On the paid tier only the frequency changes, so consecutive 30-minute deltas cover a lot of the same window.

Our recommendation is to load them with an upsert by primary key rather than an append. That way a record that shows up in several deltas keeps its latest version instead of being duplicated.

More info here: Data Exports | ShipHero GraphQL API

Let us know how it goes!

Best regards,
Luciano