CA-411319: Concurrent VM.assert_can_migrate failure

[BengangY]

When the customers open "Migrate VM Wizard" on XenCenter, XenCenter will call
VM.assert_can_migrate to check each host in each pool connected to XenCenter
if the VM can be migrated to it. The API VM.assert_can_migrate then calls
VM.export_metadata. VM.export_metadata will lock VM. During this time, other
VM.export_metadata requests will fail as they can't get VM lock.

The solution is to add retry when failing to lock VM.

[Vincent-lau]

As you said export_metadata does not support concurrency, so what would happen if it was called concurrently? Reporting a failure? If it was a genuine failure do we still want to retry?

[BengangY]

As you said export_metadata does not support concurrency, so what would happen if it was called concurrently? Reporting a failure? If it was a genuine failure do we still want to retry?

I said concurrently from XC view. It's misleading. I updated the description.

[edwintorok]

@GabrielBuica also noticed that we're missing locking around quite a few allowed operation updates, which can lead to more race conditions.

Eventually we'll probably need a more reliable solution, where you can't misuse the allowed ops APIs, for now this fixes a particular bug that has been observed.

[Vincent-lau]

This still sounds a bit strange to me... Why would export_metadata fail if it cannot acqurie the VM lock? Shouldn't it just block waiting for the VM lock?

Also I am a bit confused which line of code is acquiring the lock here? Is it add_to_current_operations?

[BengangY]

This still sounds a bit strange to me... Why would export_metadata fail if it cannot acqurie the VM lock? Shouldn't it just block waiting for the VM lock?

lock_vm would fail if another operation is in progress (in assert_operation_valid).

Also I am a bit confused which line of code is acquiring the lock here? Is it add_to_current_operations?

Yes.

[robhoes]

There is something wrong with the static analysis test?

[psafont]

Error: ref 'refs/heads/gh-readonly-queue/master/pr-6476-4f2f18582cee0ed39f87af8bc5e70057b33545c1' not found in this repository - https://docs.github.com/rest

[lindig]

I recently saw a failing CI step as well. Re-ran that CI test and it passed.