[SPARK-38101][CORE] Fix executors failing fetching map statuses with INTERNAL_ERROR_BROADCAST#54987
Open
azmatsiddique wants to merge 6 commits intoapache:masterfrom
Open
[SPARK-38101][CORE] Fix executors failing fetching map statuses with INTERNAL_ERROR_BROADCAST#54987azmatsiddique wants to merge 6 commits intoapache:masterfrom
azmatsiddique wants to merge 6 commits intoapache:masterfrom
Conversation
…n last CSV column ### What changes were proposed in this pull request? This PR fixes an issue where the CSV reader inconsistently parses empty quoted strings (`""`) when the `escape` option is set to an empty string (`""`). Previously, mid-line empty quoted strings correctly resolved to null/empty, but the last column resolved to a literal `"` character due to univocity parser behavior. ### Why are the changes needed? To ensure consistent parsing of CSV data regardless of column position. ### Does this PR introduce _any_ user-facing change? Yes, it fixes a bug where users were receiving incorrect data (a literal quote instead of an empty/null value) for the last column in a row under specific CSV configurations. ### How was this patch tested? Added a new regression test in `CSVSuite` that verifies consistent parsing of both mid-line and end-of-line empty quoted fields.
…INTERNAL_ERROR_BROADCAST
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This PR introduces a retry mechanism in MapOutputTrackerWorker.getStatuses to mitigate executor failures during shuffle map status fetching. Specifically, it wraps the RPC fetch and broadcast deserialization in a bounded retry loop (up to 3 attempts with a 100ms delay). If MapOutputTracker.deserializeOutputStatuses fails due to the broadcast variable being concurrently invalidated on the driver (marked by a SparkException with "Unable to deserialize"), the worker will now retry the request to obtain a fresh broadcast or the map statuses directly.
Why are the changes needed?
Executors can fail with [INTERNAL_ERROR_BROADCAST] Failed to get broadcast... if the driver invalidates a cached map status broadcast (via updateMapOutput) while an executor is in the process of fetching or deserializing it. This race condition, while rare, causes MetadataFetchFailedException and task retries. By handling this specifically at the MapOutputTrackerWorker level, we can recover from these transient invalidations without failing the task.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
Unit Test: Added a new test case SPARK-38101: concurrent updateMapOutput not interfering with getStatuses in
MapOutputTrackerSuite.scala
. This test simulates aggressive concurrent map status updates on the driver while multiple executor threads fetch statuses, verifying that the retry logic successfully masks the invalidation errors.
Regression Testing: Ran the full MapOutputTrackerSuite (34 tests) and all passed.