Skip to content

[WIP] Dynamically adjust partition write parallelism #3260

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 4 commits into
base: main
Choose a base branch
from

Conversation

ErikFang
Copy link
Contributor

@ErikFang ErikFang commented May 14, 2025

What changes were proposed in this pull request?

enhance celeborn to

  1. allow shuffle client to write to one partition with multiple partition locations parallelly
  2. allow lifecycle manager to detect parallel level for each partition by partition splitting frequency

as discussed with @RexXiong offline, CIP document will be drafted later

Why are the changes needed?

TBD

Does this PR introduce any user-facing change?

TBD

How was this patch tested?

TBD

@SteNicholas SteNicholas force-pushed the main branch 2 times, most recently from 5590ef0 to 0dffcf6 Compare May 26, 2025 09:57
Copy link

This PR is stale because it has been open 20 days with no activity. Remove stale label or comment or this will be closed in 10 days.

@github-actions github-actions bot added the stale label Jun 22, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants