Skip to content

[lake] Order lake snapshot metadata by commit_timestamp#3324

Open
Kaixuan-Duan wants to merge 1 commit into
apache:mainfrom
Kaixuan-Duan:issue-2625-latest-snapshot
Open

[lake] Order lake snapshot metadata by commit_timestamp#3324
Kaixuan-Duan wants to merge 1 commit into
apache:mainfrom
Kaixuan-Duan:issue-2625-latest-snapshot

Conversation

@Kaixuan-Duan
Copy link
Copy Markdown
Contributor

Purpose

Linked issue: close #2625

When tiering commits both a readable snapshot and a tiered snapshot in the same run, the coordinator records two lake snapshot metadata entries. The physical list order does not guarantee chronological ordering, so "latest" may incorrectly point to a stale readable snapshot, breaking the latest-snapshot semantics.

Brief change log

  • Add a commitTimestamp field to LakeSnapshotMetadata to record server-side commit time for each snapshot entry.
  • In LakeTableHelper.registerLakeTableSnapshotV2(), stamp each new entry with a strictly monotonic timestamp via max(now, prevMax + 1) to guarantee correct ordering even under clock skew.
  • Update LakeTable.getLatestLakeSnapshotMetadata() to pick the entry with the largest commitTimestamp instead of blindly returning the last list element.
  • Update LakeTable.getOrReadLatestReadableTableSnapshot() to traverse entries in commitTimestamp-descending order, so a stale readable entry at the physical tail does not mask a more recent one.
  • Update LakeTableJsonSerde to serialize/deserialize the optional commit_timestamp field. Legacy znodes without this field fall back to UNKNOWN_COMMIT_TIMESTAMP (0), preserving backward compatibility.

Tests

./mvnw -pl fluss-server -am test \
-DfailIfNoTests=false \
-Dtest='LakeTableTest,\
LakeTableJsonSerdeTest,\
LakeTableHelperTest,\
LakeTableSnapshotLegacyJsonSerdeTest,\
LakeTableTieringManagerTest,\
NotifyReplicaLakeTableOffsetTest'

API and Format

  • ZK lake_snapshots V2 JSON format adds an optional commit_timestamp field per snapshot entry. The field is omitted for legacy entries (backward compatible). No RPC or client API changes.

Documentation

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

TieringCommitOperator: Non-atomic dual commit can expose wrong “latest” snapshot for visibility

1 participant