The watermark
Each row has anupdated_at timestamp that advances whenever the row changes (a delivery receipt revises a message’s status; a charge is refunded). Pass a conservative UTC checkpoint recorded before the previous successful scan, with some overlap:
- Messages:
updated_since(ISO-8601, timezone-aware — includeZor an offset) - Billing records:
updatedSince(same format)
updated_at > watermark, sorted oldest-change-first, and keyset-paginated on (updated_at, id). A row that changes while you page may reappear later or require the next scan. Replay with overlap and dedupe by id as described under Idempotent upserts below.
Billing records stay newest-first regardless of the watermark: passing updatedSince filters rows to updated_at >= watermark (inclusive) without changing the sort order, and pagination is a descending keyset cursor on (updated_at, id).
Rows that predate the
updated_at field are not in the change feed. Do one initial full sync (list without a watermark) to capture the existing state, then switch to incremental. The full list is ordered by message timestamp, so its largest observed updated_at is not a safe checkpoint.Cursor pagination
Both endpoints return apagination object. Follow next_cursor until has_more is false:
cursor query param. Do not construct or parse it.
The sync loop
- Before the first page, record the current UTC time minus an overlap as the next checkpoint.
- Start from your stored watermark (or omit it for the initial full scan). Keep
updated_sinceon every cursor page. - Upsert each row by
id, retaining the row with the newerupdated_at. A later receipt can revisedeliveredtofailed. - Follow
next_cursoruntilhas_moreisfalse. - After every page and upsert succeeds, persist the pre-scan checkpoint, not the largest row timestamp. On failure, keep the prior checkpoint and retry with overlap.
Python
Run reconciliation as a safety net in addition to webhooks. It can recover changes missed while a webhook receiver is down; monitor failed scans and retry them from the prior checkpoint.
Idempotent upserts
Because a status can be revised and overlapping scans replay rows, your upsert must be idempotent: key on the rowid, and for messages retain the row with the newer updated_at. Never append — always upsert.
Related
List messages
The message change feed and its filters.
Billing records
The billing change feed for charges and refunds.