Tiering support commit by time #2185

luoyuxia · 2025-12-16T09:23:52Z

Purpose

Linked issue: close #1893

Brief change log

Tests

API and Format

Documentation

Copilot

Pull request overview

This PR implements tiering timeout support to prevent long-running table tiering operations from blocking other tables. The feature introduces a maximum duration for tiering a single table, after which it will be force-completed or skipped.

Key Changes

Added forceIgnore flag to TieringSplit and its subclasses to mark splits that should be skipped due to timeout
Implemented periodic timeout checking in TieringSourceEnumerator with configurable max duration and detection interval
Introduced TieringReachMaxDurationEvent to notify readers when a table reaches max tiering duration
Updated split handling in TieringSplitReader to force-complete in-progress log splits and skip new splits when timeout occurs

Reviewed changes

Copilot reviewed 16 out of 16 changed files in this pull request and generated 18 comments.

Show a summary per file

File	Description
TieringSplit.java	Added `forceIgnore` field and methods to mark splits for skipping
TieringSnapshotSplit.java	Added constructors supporting `forceIgnore` parameter
TieringLogSplit.java	Added constructors supporting `forceIgnore` parameter
TieringSplitSerializer.java	Updated serialization to include `forceIgnore` field
TieringSourceEnumerator.java	Implemented periodic timeout checking, deadline tracking, and timeout event broadcasting
TieringSplitReader.java	Added logic to force-complete log splits and skip splits marked with `forceIgnore`
TieringSourceReader.java	Integrated timeout event handling and custom fetcher manager
TieringSourceFetcherManager.java	New class to manage timeout notifications to split readers
TieringReachMaxDurationEvent.java	New event class to signal table timeout to readers
TieringSourceOptions.java	Added configuration options for max duration and detection interval
TieringSource.java	Updated builders to support new timeout configuration parameters
LakeTieringJobBuilder.java	Wired up timeout configuration from Fluss config
TieringSplitGenerator.java	Removed hardcoded `numberOfSplits` parameter, changed log levels to DEBUG
TieringSplitSerializerTest.java	Added tests for `forceIgnore` serialization and updated string representations
TieringSourceEnumeratorTest.java	Added timeout test, refactored assertions to use `containsExactlyInAnyOrderElementsOf`, added helper methods
TieringSourceReaderTest.java	New test file for testing timeout event handling at reader level

Comments suppressed due to low confidence (2)

fluss-flink/fluss-flink-common/src/main/java/org/apache/fluss/flink/tiering/source/split/TieringSplitGenerator.java:339

The log level was changed from INFO to DEBUG for this message about skipping splits when offset conditions are met. This is appropriate as it reduces log verbosity for expected behavior. However, ensure this doesn't make it harder to diagnose why certain tables aren't being tiered, as this condition might occur frequently during normal operation and could be useful for troubleshooting.

                LOG.debug(
                        "The lastCommittedBucketOffset {} is equals or bigger than latestBucketOffset {}, skip generating split for bucket {}",
                        lastCommittedBucketOffset,
                        latestBucketOffset,
                        tableBucket);
                return Optional.empty();
            }
        }
    }

    private Optional<TieringSplit> generateSplitForLogTableBucket(
            TablePath tablePath,
            TableBucket tableBucket,
            @Nullable String partitionName,
            @Nullable Long lastCommittedBucketOffset,
            long latestBucketOffset) {
        if (latestBucketOffset <= 0) {
            LOG.debug(
                    "The latestBucketOffset {} is equals or less than 0, skip generating split for bucket {}",
                    latestBucketOffset,
                    tableBucket);
            return Optional.empty();
        }

        // the bucket is never been tiered
        if (lastCommittedBucketOffset == null) {
            // the bucket is never been tiered, scan fluss log from the earliest offset
            return Optional.of(
                    new TieringLogSplit(
                            tablePath,
                            tableBucket,
                            partitionName,
                            EARLIEST_OFFSET,
                            latestBucketOffset));
        } else {
            // the bucket has been tiered, scan remain fluss log
            if (lastCommittedBucketOffset < latestBucketOffset) {
                return Optional.of(
                        new TieringLogSplit(
                                tablePath,
                                tableBucket,
                                partitionName,
                                lastCommittedBucketOffset,
                                latestBucketOffset));
            }
        }
        LOG.debug(
                "The lastCommittedBucketOffset {} is equals or bigger than latestBucketOffset {}, skip generating split for bucket {}",
                lastCommittedBucketOffset,
                latestBucketOffset,
                tableBucket);

fluss-flink/fluss-flink-common/src/main/java/org/apache/fluss/flink/tiering/source/TieringSplitReader.java:168

When a table reaches the max tiering duration, the timeout check only forces completion for log splits (lines 159-161 and 292). However, snapshot splits can also be in progress. If a table times out while processing snapshot splits, they won't be force-completed, leading to potential hangs or inconsistent behavior. Consider extending the timeout handling to snapshot splits as well.

        // may read snapshot firstly
        if (currentSnapshotSplitReader != null) {
            // for snapshot split, we don't force to complete it
            // since we rely on the log offset for the snapshot to
            // do next tiering, if force to complete, we can't get the log offset
            CloseableIterator<RecordAndPos> recordIterator = currentSnapshotSplitReader.readBatch();
            if (recordIterator == null) {
                LOG.info("Split {} is finished", currentSnapshotSplit.splitId());
                return finishCurrentSnapshotSplit();
            } else {
                return forSnapshotSplitRecords(
                        currentSnapshotSplit.getTableBucket(), recordIterator);
            }
        } else {
            if (currentLogScanner != null) {
                if (timeoutTables.contains(currentTableId)) {
                    return forceCompleteTieringLogRecords();
                }
                ScanRecords scanRecords = currentLogScanner.poll(pollTimeout);
                // force to complete records
                return forLogRecords(scanRecords);
            } else {
                return emptyTableBucketWriteResultWithSplitIds();
            }
        }

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2025-12-17T12:02:27Z

.../src/main/java/org/apache/fluss/flink/tiering/source/enumerator/TieringSourceEnumerator.java

+            for (int reader : readers) {
+                context.sendEventToSourceReader(
+                        reader, new TieringReachMaxDurationEvent(reachMaxDurationTable));
+            }


The deadline is removed from tieringTablesDeadline when a table finishes or fails (lines 240 and 259), but it's never removed when the timeout event is sent at line 323. This means a timed-out table will continue triggering timeout checks and events on every periodic check interval, potentially sending duplicate TieringReachMaxDurationEvent messages to readers. Consider removing the deadline after sending the timeout event.

Suggested change

}

}

// remove the deadline once timeout is handled to avoid duplicate timeout events

tieringTablesDeadline.remove(reachMaxDurationTable);

Copilot · 2025-12-17T12:02:28Z

.../src/main/java/org/apache/fluss/flink/tiering/source/enumerator/TieringSourceEnumerator.java

+            for (TieringSplit tieringSplit : pendingSplits) {
+                if (tieringSplit.getTableBucket().getTableId() == reachMaxDurationTable) {
+                    // force ignore this tiering split since the tiering for this table is timeout,
+                    // we have to force to set to ignore the tiering split so that the
+                    // tiering source reader can ignore them directly
+                    tieringSplit.forceIgnore();
+                } else {
+                    // we can break directly, if found any one split's table id is not equal to the
+                    // timeout
+                    // table, the following split must be not equal to the table id
+                    break;


The pendingSplits list is accessed both from the periodic timeout check (line 301) and the assignSplits method (line 350), which can be called from different threads via handleSplitRequest and addSplitsBack. While assignSplits synchronizes on readersAwaitingSplit, the timeout check doesn't use any synchronization, which could lead to a ConcurrentModificationException or inconsistent state when iterating and modifying pendingSplits concurrently.

Suggested change

for (TieringSplit tieringSplit : pendingSplits) {

if (tieringSplit.getTableBucket().getTableId() == reachMaxDurationTable) {

// force ignore this tiering split since the tiering for this table is timeout,

// we have to force to set to ignore the tiering split so that the

// tiering source reader can ignore them directly

tieringSplit.forceIgnore();

} else {

// we can break directly, if found any one split's table id is not equal to the

// timeout

// table, the following split must be not equal to the table id

break;

// Access to pendingSplits must be synchronized consistently with other paths

// (e.g., assignSplits) to avoid concurrent modification.

synchronized (readersAwaitingSplit) {

for (TieringSplit tieringSplit : pendingSplits) {

if (tieringSplit.getTableBucket().getTableId() == reachMaxDurationTable) {

// force ignore this tiering split since the tiering for this table is

// timeout, we have to force to set to ignore the tiering split so that the

// tiering source reader can ignore them directly

tieringSplit.forceIgnore();

} else {

// we can break directly, if found any one split's table id is not equal to

// the timeout table, the following split must be not equal to the table id

break;

}

Copilot · 2025-12-17T12:02:28Z

.../src/main/java/org/apache/fluss/flink/tiering/source/enumerator/TieringSourceEnumerator.java


    private void assignSplits() {
        /* This method may be called from both addSplitsBack and handleSplitRequest, make it thread safe. */
+        // todo: do we need to add lock?


The TODO comment at line 340 asks "do we need to add lock?" but this question is now more critical given that the timeout checking code (lines 301-312) accesses pendingSplits without synchronization while assignSplits() synchronizes on readersAwaitingSplit. This creates a potential race condition. The TODO should either be resolved by adding proper synchronization or be updated to reflect the new concurrency concerns.

Copilot · 2025-12-17T12:02:28Z

...link-common/src/test/java/org/apache/fluss/flink/tiering/source/TieringSourceReaderTest.java

+class TieringSourceReaderTest extends FlinkTestBase {
+
+    @Test
+    void testHandleTieringReachMaxDurationEvent1() throws Exception {


The test method name 'testHandleTieringReachMaxDurationEvent1' has a '1' suffix, which typically indicates there should be multiple similar tests or that this is a placeholder name. Consider using a more descriptive name like 'testHandleTieringReachMaxDurationEventWithForceIgnore' or 'testMaxDurationEventWithEmptyAndNonEmptySplits' to better describe what the test verifies.

Suggested change

void testHandleTieringReachMaxDurationEvent1() throws Exception {

void testHandleTieringReachMaxDurationEventWithForceIgnore() throws Exception {

Copilot · 2025-12-17T12:02:29Z

...uss-flink-common/src/main/java/org/apache/fluss/flink/tiering/source/TieringSplitReader.java

+    private static final long UNKNOW_BUCKET_OFFSET = -1;
+


Typo in constant name: 'UNKNOW_BUCKET_OFFSET' should be 'UNKNOWN_BUCKET_OFFSET' to be consistent with UNKNOWN_BUCKET_TIMESTAMP.

Suggested change

private static final long UNKNOW_BUCKET_OFFSET = -1;

// unknown bucket offset for empty split or snapshot split

private static final long UNKNOWN_BUCKET_OFFSET = -1;

/**

* @deprecated Use {@link #UNKNOWN_BUCKET_OFFSET} instead.

*/

@Deprecated

private static final long UNKNOW_BUCKET_OFFSET = UNKNOWN_BUCKET_OFFSET;

Copilot · 2025-12-17T12:02:32Z

...uss-flink-common/src/main/java/org/apache/fluss/flink/tiering/source/TieringSplitReader.java

+                        currentTableTieredOffsetAndTimestamp.get(bucket);
+                long logEndOffset =
+                        logOffsetAndTimestamp == null
+                                ? UNKNOW_BUCKET_OFFSET


Typo in variable reference: 'UNKNOW_BUCKET_OFFSET' should be 'UNKNOWN_BUCKET_OFFSET'.

Suggested change

? UNKNOW_BUCKET_OFFSET

? UNKNOWN_BUCKET_OFFSET

Copilot · 2025-12-17T12:02:32Z

...ss-flink-common/src/main/java/org/apache/fluss/flink/tiering/source/TieringSourceReader.java

    public TieringSourceReader(
+            FutureCompletingBlockingQueue<RecordsWithSplitIds<TableBucketWriteResult<WriteResult>>>
+                    elementsQueue,
            SourceReaderContext context,
            Connection connection,
            LakeTieringFactory<WriteResult, ?> lakeTieringFactory) {
+        this(elementsQueue, context, connection, lakeTieringFactory, DEFAULT_POLL_TIMEOUT);
+    }


The constructor now requires an elementsQueue parameter, changing from creating the queue internally to accepting it externally. However, the non-test constructor at line 53 doesn't expose a way to customize the queue, which limits flexibility for testing or customization. While this is addressed by the @VisibleForTesting constructor, consider whether the production code path should also allow queue configuration or if it should remain internal.

Copilot · 2025-12-17T12:02:32Z

...common/src/main/java/org/apache/fluss/flink/tiering/source/split/TieringSplitSerializer.java

            out.writeBoolean(false);
        }

+        // write force ignore
+        out.writeBoolean(split.isForceIgnore());


The serialization format has changed by adding a new forceIgnore boolean field (line 78), but the version number remains at VERSION_0. This breaks backward compatibility: older deserializers will fail when reading data serialized by the new version, and the new deserializer will fail when reading old serialized data that doesn't have the forceIgnore field. Consider bumping the version to VERSION_1 and handling both old and new formats in the deserialize method for backward compatibility.

Copilot · 2025-12-17T12:02:33Z

...uss-flink-common/src/main/java/org/apache/fluss/flink/tiering/source/TieringSplitReader.java

+                            tieringSplit.getPartitionName(),
                            null,
-                            logSplit.getStoppingOffset(),
+                            UNKNOW_BUCKET_OFFSET,


Typo in variable reference: 'UNKNOW_BUCKET_OFFSET' should be 'UNKNOWN_BUCKET_OFFSET'.

Suggested change

UNKNOW_BUCKET_OFFSET,

UNKNOWN_BUCKET_OFFSET,

Copilot · 2025-12-17T12:02:33Z

...-common/src/main/java/org/apache/fluss/flink/tiering/event/TieringReachMaxDurationEvent.java

+ * SourceEvent used to notify TieringSourceReader that a table has reached the maximum tiering
+ * duration and should be force completed.
+ */
+public class TieringReachMaxDurationEvent implements SourceEvent {


Class TieringReachMaxDurationEvent overrides hashCode but not equals.

luoyuxia added 3 commits December 16, 2025 11:43

tiering support commit by time

e542d6d

[lake] support lake tiering service force to commit due to timeout

06fcd47

[lake] support lake tiering service force to commit due to timeout

46c293e

luoyuxia force-pushed the tiering-support-commit-by-time branch from 38395e1 to 46c293e Compare December 17, 2025 11:42

luoyuxia requested a review from Copilot December 17, 2025 11:46

Copilot started reviewing on behalf of luoyuxia December 17, 2025 11:46 View session

Copilot AI reviewed Dec 17, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Tiering support commit by time #2185

Tiering support commit by time #2185

Uh oh!

luoyuxia commented Dec 16, 2025 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Dec 17, 2025

Uh oh!

Copilot AI Dec 17, 2025

Uh oh!

Copilot AI Dec 17, 2025

Uh oh!

Copilot AI Dec 17, 2025

Uh oh!

Copilot AI Dec 17, 2025

Uh oh!

Copilot AI Dec 17, 2025

Uh oh!

Copilot AI Dec 17, 2025

Uh oh!

Copilot AI Dec 17, 2025

Uh oh!

Copilot AI Dec 17, 2025

Uh oh!

Copilot AI Dec 17, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

-            }
+            }
+            // remove the deadline once timeout is handled to avoid duplicate timeout events
+            tieringTablesDeadline.remove(reachMaxDurationTable);

	void testHandleTieringReachMaxDurationEvent1() throws Exception {
	void testHandleTieringReachMaxDurationEventWithForceIgnore() throws Exception {

-    private static final long UNKNOW_BUCKET_OFFSET = -1;
+    // unknown bucket offset for empty split or snapshot split
+    private static final long UNKNOWN_BUCKET_OFFSET = -1;
+    /**
+     * @deprecated Use {@link #UNKNOWN_BUCKET_OFFSET} instead.
+     */
+    @Deprecated
+    private static final long UNKNOW_BUCKET_OFFSET = UNKNOWN_BUCKET_OFFSET;

Tiering support commit by time #2185

Are you sure you want to change the base?

Tiering support commit by time #2185

Uh oh!

Conversation

luoyuxia commented Dec 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Brief change log

Tests

API and Format

Documentation

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Key Changes

Reviewed changes

Uh oh!

Copilot AI Dec 17, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 17, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 17, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 17, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 17, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 17, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 17, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 17, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 17, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 17, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

luoyuxia commented Dec 16, 2025 •

edited

Loading