CA-411766: Detach VBDs right after VM Halted

[BengangY]

In XSI-1915, MCS shutdowned a VM and tried to destroy VBD right after MCS received the event which came from power_state's change and failed. The failure reason is below:

1. The update for VM's power_state and the update for VBDs is not a transaction, so the client may receive the event from the update for power_state and operate VBDs before the update for VBDs.
2. The VM's running on supporter. The DB operation needs to send RPC to the coordinator. This needs time.
3. Between the update for VM's power_state and the update for VBD, xapi also updates the field pending_guildencs which needs at least 8 DB operation. This also delays the update for VBDs.

It's not straightforward to add transactions for these DB operations. The workaround is to move the update for pending_guildencs to the end of the DB operation of VBDs, VIFs, GPUs, etc. So that VBD will be updated after the update for VM's power_state immediately.

[lindig]

I like the explanation

[edwintorok]

They should probably wait for the task's completion instead of acting on power state changes. There is always going to be a race, even with this change (this just reduces the race window)

[edwintorok]

The correct way would be to watch for events on VBD allowed operations. But apparently we don't expose 'destroy' as an allowed op, we probably should

[stormi]

What is MCS, in the context of XAPI?

[BengangY]

What is MCS, in the context of XAPI?

MCS means Machine Creation Services. It's a component of CVAD (Citrix Virtual Apps and Desktops).

[lindig]

MCS is a service that is part of another a Citrix product to deploy VMs. XenServer is only one of its supported backends.

[edwintorok]

The correct way would be to watch for events on VBD allowed operations. But apparently we don't expose 'destroy' as an allowed op, we probably should

Discussed that watching VBD.is_currently_attached to become false would work better here. Then there would be no race.
Some documentation here on how the client should behave would be good, otherwise all they can do is guess (and we've seen that they'd often guess wrong)

[stormi]

Thanks. The commit message clearly describes the problem, but for someone external to XenServer it would be even better if the introduction would not directly refer to XSI and to an internal acronym.

Example of how I'd put it:

Beginning of commit message:

An internal tool at XenServer tried to destroy VBD right after it received the event which came from power_state's change and failed. The failure reason is below:

Then at the end of the commit message, additional useful details for XenServer devs:

This is related to XSI-1915 where internal deploy tool MCS triggered the issue.

It feels more inclusive to the rest of the open source community involved in XAPI, in my opinion.

That was my nitpicking of the day, just sharing my impressions from outside XenServer :)

[BengangY]

Thanks. The commit message clearly describes the problem, but for someone external to XenServer it would be even better if the introduction would not directly refer to XSI and to an internal acronym.

Example of how I'd put it:

Beginning of commit message:

An internal tool at XenServer tried to destroy VBD right after it received the event which came from power_state's change and failed. The failure reason is below:

Then at the end of the commit message, additional useful details for XenServer devs:

This is related to XSI-1915 where internal deploy tool MCS triggered the issue.

It feels more inclusive to the rest of the open source community involved in XAPI, in my opinion.

That was my nitpicking of the day, just sharing my impressions from outside XenServer :)

@stormi Thanks for your suggestion! I will update the description.