.*: Add support for StartAfter option in ListObjects

[waltherlee]

Thanos compactors have to walk-through and list all metadata files (or block folders depending on the discovery strategy) in S3 before starting compactions. For buckets with dozens of millions of blocks, that sync can take hours, and it has to be started all over every time the compactors are restarted, even if the first sync had finished and compactions had already started.

I’m working on a checkpoint implementation in Thanos to be able to recover a meta sync where it was left after a restart, but for this to work, objstore needs to support StartAfter, StartOffset and Marker as an option for the ListObjects Iter.

I tried to cover as many providers as possible. I checked the documentation for each one to make sure that they take object prefixes as value for the offset.

• I added CHANGELOG entry for this change.
• Change is not relevant to the end user.

Changes

• Added a WithStartAfter() option in objstore.go.
• Implemented support in 8 providers:
  • S3 (ListObjectsOptions.StartAfter), GCS (Query.StartOffset), BOS/COS/OSS/OBS (Marker), OCI (Start), Swift (ObjectsOpts.Marker).
  • afaict all these options are used to skip objects lexicographically lower than the value, so the implementations are similar.
• Added StartAfter handling in PrefixedBucket to prepend the bucket prefix to the option value.
• Added StartAfter support in InMemBucket for testing.
• Not supported (no option in the native API): Azure, Filesystem (maybe?).

Verification

• Added StartAfter non-recursive and recursive cases in the AcceptanceTest.
• Added InMemBucket unit tests covering recursive, non-recursive, and unsupported-provider validation.
• Updated the number of expected iter counts in TestMetricBucket_Close.

[GiedriusS]

👋 thanks for your contribution. I would suggest writing a small RFC first and opening a PR with it in the Thanos repo so that we could discuss your feature before implementing it because it sounds to me like there could be lots of edge cases

[waltherlee]

@GiedriusS Sounds good! Will do. But yes, there's a lot of states we can have between multiple compactor shards and/or time windows.

I will be testing a first version in the following weeks in a fork, so hopefully I can also show some prod data and metrics in the Thanos PR 🤞

[joe-elliott]

we use the thanos obj store clients and are interested in this feature. are maintainers open to merging this even if its not directly used by thanos?

thx @waltherlee for working on this 🙏

[waltherlee]

I will be sending the RFC I mentioned above this week.

It took me a little longer to make it work, but I have a version that has been working as expected for weeks. That should help with the review in this PR as well 🙂

[waltherlee]

🤞 Proposal sent: thanos-io/thanos#8765