You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: services/libs/tinybird/bucketing-architecture.md
+10-4Lines changed: 10 additions & 4 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -252,11 +252,17 @@ WHERE cityHash64(segmentId) % 10 = {bucket_number}
252
252
253
253
### Bootstrap Procedure
254
254
255
+
The snapshot pipes use **append COPY MODE**. This means running a snapshot pipe will add rows to the existing datasource rather than replacing it. To recreate the buckets from scratch, follow these steps for each bucket `#` (0–9):
256
+
255
257
1.**Prepare**: Ensure all bucket datasources exist
256
-
2.**Execute Snapshots**: Run all 10 snapshot pipes manually
257
-
3.**Verify**: Check each bucket has ~10% of total records
258
-
4.**Enable Enrichment**: Allow scheduled copy pipes to run
259
-
5.**Monitor**: Watch for snapshotId updates in cleaned buckets
258
+
2.**Pause** the `activityRelations_bucket_clean_enrich_copy_pipe_#` pipe
259
+
3.**Truncate**`activityRelations_bucket_MV_ds_#`
260
+
4.**Run** the `activityRelations_bucket_MV_snapshot_#` pipe
261
+
5.**Confirm** that `activityRelations_bucket_MV_ds_#` was populated and has 10% of total records
262
+
6.**Resume** the `activityRelations_bucket_clean_enrich_copy_pipe_#` pipe
263
+
7.**Monitor**: Watch for snapshotId updates in cleaned buckets
264
+
265
+
> **Why pause the enrichment pipe first?** The enrichment copy pipe reads from `activityRelations_bucket_MV_ds_#`. Truncating while it runs could cause it to produce empty or partial results. Pausing ensures a clean window for the snapshot to populate the datasource.
0 commit comments