Currently tsdb blocks can only be converted to a parquet file after a full 24 hour day. However, we could potentially add a flag to enable rolling conversions for the current day.
This would allow all incoming blocks for the current day to be converted to parquet as soon as they are uploaded to object storage. When new blocks are discovered by the converter, it could then just do the conversion again for all blocks of the current day, overwriting the current partial parquet file with the most recent data. We could keep track of this by just updating the parquet meta file maxt for the most recent block of the day that has been discovered/converted.
Currently tsdb blocks can only be converted to a parquet file after a full 24 hour day. However, we could potentially add a flag to enable rolling conversions for the current day.
This would allow all incoming blocks for the current day to be converted to parquet as soon as they are uploaded to object storage. When new blocks are discovered by the converter, it could then just do the conversion again for all blocks of the current day, overwriting the current partial parquet file with the most recent data. We could keep track of this by just updating the parquet meta file maxt for the most recent block of the day that has been discovered/converted.