Backfill

History is worth more than live traffic at the start of an integration: it is what gives sinio.ai the baselines it needs before it can tell you anything useful. Loading it uses the same endpoint as live ingestion, with one field changed.

One field

{
  "mode": "backfill",
  "events": [
    {
      "source_event_id": "evt_10001",
      "type": "badge_in",
      "occurred_at": "2025-01-06T07:52:31+01:00",
      "roles": { "device": "BORNE-A12", "subject": "P-4582" },
      "payload": { "badge": "04A2B9C1" }
    }
  ]
}

await sinio.backfillEvents(history)

Same endpoint, same limits, same idempotency, same response shape — and the same field on a webhook URL, which honours mode too. There is no window restriction: an occurred_at from three years ago is accepted exactly like one from this morning.

What backfill changes

Episodes, habits and findings are still produced. Only alerting is switched off. A year of history should teach the platform what normal looks like; it should not wake anyone up at 3am for something that happened last March.

Two independent guards decide it:

The second guard exists because the expiry job runs long after ingestion and knows nothing about the mode a batch was sent with. Without it, importing twelve months of badge-ins with no matching badge-outs would produce thousands of alerted missing_step findings weeks later.

Why derivation is replayed, not run on arrival

A backfill does not derive during ingestion. Events are persisted, and once a batch has arrived the project's processes replay the history in a canonical order. Replays still waiting their turn fold into one, so sending a file in many batches does not queue as many replays. A project with no process yet has nothing to form: the first process published replays the history then.

The reason is ordering. Deriving on arrival exposes the result to the order of your file: a badge_out read before its badge_in produces an out_of_order episode with no duration — and a run of those produces an empty baseline, which is the one thing a backfill exists to build.

The replay walks events in an order composed entirely of source fields:

ORDER BY occurred_at, source_id, source_event_id, revision

None of them come from insertion. That is what makes the same logical event land in the same position no matter what order it was ingested in — the internal id is deliberately excluded, being monotonic with arrival rather than with meaning.

We test this by ingesting the same events sorted, then shuffled, including closing steps before their opening ones: the episodes and habits come out identical to the hash. Deriving on arrival over the shuffled stream instead produced 12 closed episodes out of 44 expected, 17 orphan closes, and a third of the habit rows with no median duration.

Practical advice

• Backfill before you go live. Load history first, let it settle, then switch the same integration to live webhook mode.
• Keep the batch_id of each response. sinio.ai assigns it, one per call — the SDKs collect them in batchIds (batch_ids in Python). Quote it when something needs to be looked into.
• Let the SDK chunk. Hand it the whole file; it will split at 500 and merge the counters.
• Replays are free. If an import is interrupted, run it again from the start. Idempotency means everything already stored comes back as duplicates.
• Expect pending_classification. Old exports often contain types nobody has classified yet. They are stored, not lost, and derived once the type is given a meaning.