Can we just set a larger value for celeborn.worker.graceful.shutdown.timeout?

Can we just set a larger value for celeborn.worker.graceful.shutdown.timeout?

For graceful shutdown, we can do this and let other user know by adding in the config description.

But we still need to handle this for decommission flow, so this config can be general / default way to provide some time buffer without requiring config tuning from users.

Can we just set a larger value for celeborn.worker.graceful.shutdown.timeout?

For graceful shutdown, we can do this and let other user know by adding in the config description.

But we still need to handle this for decommission flow, so this config can be general / default way to provide some time buffer without requiring config tuning from users.

The config celeborn.worker.graceful.shutdown.timeout is Celeborn worker's config, which means that the users should not tune it because Celeborn Cluster should not be exposed to the users. In what scenario will the user tune the configs for the Celeborn cluster?

@FMX By users i meant the Celeborn admins, not the client.

For worker decommission, the force exit timeout is 600s. But in some cases the shutdown hook will not fully execute, lets say sendWorkerDecommissionToMaster() took 2s. Due to which we will miss out on the information on unreleased shuffle which gets print at the end of decommission shutdown hook.

Similarly, for worker graceful shutdown, current default timeout is 600s, which accounts for check slots finished timeout (480s) and partition sorter timeout (120s). So either we can increase the graceful shutdown default value slightly or ask the celeborn admins to increase the timeout slightly grater than (check slots finished timeout + partition sorter timeout).

Both of the cases can be handled by adding small cooldown time.

@FMX By users i meant the Celeborn admins, not the client.

For worker decommission, the force exit timeout is 600s. But in some cases the shutdown hook will not fully execute, lets say sendWorkerDecommissionToMaster() took 2s. Due to which we will miss out on the information on unreleased shuffle which gets print at the end of decommission shutdown hook.

Similarly, for worker graceful shutdown, current default timeout is 600s, which accounts for check slots finished timeout (480s) and partition sorter timeout (120s). So either we can increase the graceful shutdown default value slightly or ask the celeborn admins to increase the timeout slightly grater than (check slots finished timeout + partition sorter timeout).

Both of the cases can be handled by adding small cooldown time.

In this scenario, why not increase the graceful shutdown time slightly? Looks like in both approaches, the cluster admin will need to change the config file.

Increasing the timeout approach will only solve for graceful shutdown, decommission flow will still have this problem.