Skip to content

Commit 50eb125

Browse files
authored
chore: update octolens mentions sink frequency (#3896)
1 parent 24de661 commit 50eb125

1 file changed

Lines changed: 5 additions & 5 deletions

File tree

services/libs/tinybird/pipes/octolens_mentions_sink.pipe

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,8 @@
11
DESCRIPTION >
22
Sink pipe to export Octolens mentions data to Kafka for Snowflake ingestion.
33
Reads from mentions datasource using FINAL to get deduplicated data.
4-
Runs daily at 00:30 UTC, exporting only data created since the last run.
5-
Uses createdAt filter to get rows added the previous day (incremental sync).
4+
Runs hourly at :40 UTC, exporting only data created since the last run.
5+
Uses createdAt filter to get rows added in the previous hour (incremental sync).
66

77
NODE octolens_mentions_select_fields
88
SQL >
@@ -31,14 +31,14 @@ SQL >
3131
arrayStringConcat(keywords, ',') as keywords,
3232
authorFollowerCount,
3333
arrayStringConcat(tags, ',') as tags,
34-
toStartOfDay(now()) as date
34+
now() as date
3535
FROM mentions FINAL
36-
WHERE createdAt >= toStartOfDay(now()) - INTERVAL 1 DAY AND createdAt < toStartOfDay(now())
36+
WHERE createdAt >= toStartOfHour(now()) - INTERVAL 1 HOUR AND createdAt < toStartOfHour(now())
3737

3838
TYPE SINK
3939
EXPORT_SERVICE kafka
4040
EXPORT_CONNECTION_NAME lfx-oracle-kafka-streaming
41-
EXPORT_SCHEDULE 30 0 * * *
41+
EXPORT_SCHEDULE 40 * * * *
4242
EXPORT_FORMAT csv
4343
EXPORT_STRATEGY @new
4444
EXPORT_KAFKA_TOPIC octolens_mentions_sink

0 commit comments

Comments
 (0)