[Bugfix] Flush TunableOp results before worker processes are destroyed.

[naromero77amd]

This PR is a bug fix for users that run TunableOp on multi-GPU vLLM workloads. This is mostly ROCm users.

Currently, TunableOp with worker processes enabled will only write the results for GPU 0 from the main process. Normally, the TunableOp results are flushed to the file system when the TunableOp C++ destructor is called. The worker processes appear to be destroyed before TunableOp destructor is called. What users are experiencing is that the main process ends up with TunableOp results, while the other worker processes that are driving the other GPU end up with an empty TunableOp results file. We now force a flush of the TunableOp results before the worker processes are terminated.

cc: @hongxiayang

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

[hongxiayang]

Suggested change

	if ((tunable.is_enabled() is True) and
	(tunable.tuning_is_enabled() is True) and
	(tunable.record_untuned_is_enabled() is False)):
	tunable.write_file()
	if tunable.is_enabled() and tunable.tuning_is_enabled() and
	not tunable.record_untuned_is_enabled():
	tunable.write_file()

[hongxiayang]

Suggested change

	if (tunable.is_enabled() and tunable.tuning_is_enabled() and
	not tunable.record_untuned_is_enabled()):
	if tunable.is_enabled() and tunable.tuning_is_enabled() and
	not tunable.record_untuned_is_enabled():

[naromero77amd]

Actually, I tried without the outermost () and I got a syntax error. Maybe because it is split across two lines?

[hongxiayang]

yes, it needs \ to break long if statement into multiple lines, or bracket.

[naromero77amd]

One more thing, in case it matters, the tunable.record_untuned_is_enabled() is only available in PyTorch 2.6 or later. Not sure what is the vLLM practice for backwards compatibility.

[DarkLight1337]

Please merge from main and fix the pre-commit errors.

[hongxiayang]

One more thing, in case it matters, the tunable.record_untuned_is_enabled() is only available in PyTorch 2.6 or later. Not sure what is the vLLM practice for backwards compatibility.

Can you please put a comment in the code about this?