[BUG] volume-backup-tester.py leaking backup and volume resources

[mbuechse]

I run the test against a cloud that's missing cinder-backup. The test creates a backup resource, which is in permanent error state because of the missing service, and I cannot delete it. This might be called a cinder bug, but the test triggered it. We cannot use a test that leaks resources.

What's more: the test doesn't delete the volume either because of the uncaught timeout exception...

My shell journey (abridged):

$ Tests/iaas/volume-backup/volume-backup-tester.py --os-cloud poc-wgcloud --debug

Performing cleanup for resources with the 'scs-test-' prefix ...
Creating volume ...
↳ waiting for volume with ID 'ffc4136f-df3f-4d00-8ad7-78a0dd2c04ad' to reach status 'available' ...
Create empty volume: PASS
Creating backup from volume ...
Traceback (most recent call last):
  File "/home/matthiasbuechse/opensrc/scs/standards/Tests/iaas/volume-backup/volume-backup-tester.py", line 283, in <module>
    main()
  File "/home/matthiasbuechse/opensrc/scs/standards/Tests/iaas/volume-backup/volume-backup-tester.py", line 278, in main
    test_backup(conn, prefix=args.prefix, timeout=args.timeout)
  File "/home/matthiasbuechse/opensrc/scs/standards/Tests/iaas/volume-backup/volume-backup-tester.py", line 115, in test_backup
    assert seconds_waited < timeout, (
           ^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: Timeout reached while waiting for backup to reach status 'available' (backup id: 86cc7f49-e91d-47b6-9668-959759820136) after 60 seconds

$ OS_CLOUD=poc-wgcloud .cargo/bin/osc block-storage backup show 86cc7f49-e91d-47b6-9668-959759820136
┌───────────────────────┬────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Attribute             ┆ Value                                                                                                                                                                                                                                                                                                      │
╞═══════════════════════╪════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╡
│ created_at            ┆ 2024-10-30T19:55:23.000000                                                                                                                                                                                                                                                                                 │
│ data_timestamp        ┆ 2024-10-30T19:55:23.000000                                                                                                                                                                                                                                                                                 │
│ fail_reason           ┆ Service not found for creating backup.                                                                                                                                                                                                                                                                     │
│ has_dependent_backups ┆ false                                                                                                                                                                                                                                                                                                      │
│ id                    ┆ 86cc7f49-e91d-47b6-9668-959759820136                                                                                                                                                                                                                                                                       │
│ is_incremental        ┆ false                                                                                                                                                                                                                                                                                                      │
│ links                 ┆ [{"href":"https://block-storage.l1a.cloudandheat.com/v3/8ef56c4ec0a046f98ec3d95c7a677a6e/backups/86cc7f49-e91d-47b6-9668-959759820136","rel":"self"},{"href":"https://block-storage.l1a.cloudandheat.com/8ef56c4ec0a046f98ec3d95c7a677a6e/backups/86cc7f49-e91d-47b6-9668-959759820136","rel":"bookmark"}] │
│ metadata              ┆ {}                                                                                                                                                                                                                                                                                                         │
│ name                  ┆ scs-test-volume-backup                                                                                                                                                                                                                                                                                     │
│ object_count          ┆ 0                                                                                                                                                                                                                                                                                                          │
│ size                  ┆ 1                                                                                                                                                                                                                                                                                                          │
│ status                ┆ error                                                                                                                                                                                                                                                                                                      │
│ updated_at            ┆ 2024-10-30T19:55:23.000000                                                                                                                                                                                                                                                                                 │
│ volume_id             ┆ ffc4136f-df3f-4d00-8ad7-78a0dd2c04ad                                                                                                                                                                                                                                                                       │
└───────────────────────┴────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘

$ OS_CLOUD=poc-wgcloud .cargo/bin/osc block-storage backup force-delete 86cc7f49-e91d-47b6-9668-959759820136
Error: 
   0: openstack server error:
      	uri: `https://block-storage.l1a.cloudandheat.com/v3/8ef56c4ec0a046f98ec3d95c7a677a6e/backups/86cc7f49-e91d-47b6-9668-959759820136/action?`
      	status: `403 Forbidden`
      	data: `{"forbidden":{"code":403,"message":"Policy doesn't allow volume_extension:backup_admin_actions:force_delete to be performed."}}`

Location:
   openstack_cli/src/bin/osc.rs:31

[mbuechse]

definition of done

• if possible, detect missing service without leaking resources
• also (try to) delete resources with other states besides available (such as error)
• don't use assert if the exception essential for the program flow (asserts will be stripped with python3 -O, and then the loop will become an endless loop)
• don't abort cleanup upon an error (i.e., remove the volume even if the backup cannot be removed)

[markus-hentsch]

Turns out that Cinder is terrible at handling changes to the backup service at runtime, especially when using volume service set [--disable/--enable] while the other Cinder services keep running.
Cinder exhibits different behavior depending on the presence of the backup service, its service registration entry as well as the current state of other Cinder services.

Furthermore, Cinder always defaults to the Swift object storage backup driver if no configuration option for the backup service is set and will automatically discover the Swift endpoint from the service catalog if it is not specified in the configuration.
The only determining factor is the volume service entry in Cinder's database as well as the presence of an actual cinder-backup process.
This can lead to the backup service to be either present or absent in non-obvious manners.

Permutations of Cinder behavior with regard to the backup service (click to expand ...)

A) backup service disabled but running

State:

• openstack volume service set --disable devstack cinder-backup
• systemctl start devstack@c-bak

Result:

• cinder-backup shows up as disabled in openstack volume service list
• backup creation succeeds, backup eventually enters available state
• backup restore succeeds in the backend but shows error in the client
• backup deletion fails
• volume deletion works

stack@devstack:~/devstack$ openstack volume create --size 1 --image test-cirros first-volume
stack@devstack:~/devstack$ openstack volume backup create --name first-volume-backup first-volume
+-----------+--------------------------------------+
| Field     | Value                                |
+-----------+--------------------------------------+
| id        | 50e43443-0b80-4b96-8540-c89cda032a48 |
| name      | first-volume-backup                  |
| volume_id | 49655279-56f9-4887-b46b-a0eab860aa23 |
+-----------+--------------------------------------+
stack@devstack:~/devstack$ openstack volume backup list
+--------------------------------------+---------------------+-------------+-----------+------+-------------+
| ID                                   | Name                | Description | Status    | Size | Incremental |
+--------------------------------------+---------------------+-------------+-----------+------+-------------+
| 50e43443-0b80-4b96-8540-c89cda032a48 | first-volume-backup | None        | available |    1 | False       |
+--------------------------------------+---------------------+-------------+-----------+------+-------------+

stack@devstack:~/devstack$ openstack volume backup restore first-volume-backup second-volume
ResourceNotFound: 404: Client Error for url: http://192.168.144.111/volume/v3/a4f577b38eb54c2f929d6a80290b635e/backups/50e43443-0b80-4b96-8540-c89cda032a48/restore, Service cinder-backup could not be found.

# it says that it failed, but it actually succeeded:
stack@devstack:~/devstack$ openstack volume list
+--------------------------------------+---------------+-----------+------+-------------+
| ID                                   | Name          | Status    | Size | Attached to |
+--------------------------------------+---------------+-----------+------+-------------+
| 5c76c9ad-6c43-459e-9b7c-09bba78efeae | second-volume | available |    1 |             |
| 49655279-56f9-4887-b46b-a0eab860aa23 | first-volume  | available |    1 |             |
+--------------------------------------+---------------+-----------+------+-------------+

stack@devstack:~/devstack$ openstack volume delete first-volume second-volume
# (success)

stack@devstack:~/devstack$ openstack volume backup delete first-volume-backup
Failed to delete backup with name or ID 'first-volume-backup': ResourceNotFound: 404: Client Error for url: http://192.168.144.111/volume/v3/a4f577b38eb54c2f929d6a80290b635e/backups/50e43443-0b80-4b96-8540-c89cda032a48, Service cinder-backup could not be found.
1 of 1 backups failed to delete.

stack@devstack:~/devstack$ openstack volume backup list
+--------------------------------------+---------------------+-------------+-----------+------+-------------+
| ID                                   | Name                | Description | Status    | Size | Incremental |
+--------------------------------------+---------------------+-------------+-----------+------+-------------+
| 50e43443-0b80-4b96-8540-c89cda032a48 | first-volume-backup | None        | available |    1 | False       |
+--------------------------------------+---------------------+-------------+-----------+------+-------------+

Note: the backup can be deleted once openstack volume service set --enable devstack cinder-backup is used.

B) backup service disabled and stopped

State:

• openstack volume service set --disable devstack cinder-backup
• systemctl stop devstack@c-bak

Result:

• cinder-backup shows up as disabled in openstack volume service list
• backup creation command is accepted and returns
• backup stuck in creating state forever
• volume stuck in backing-up state forever
• neither can be deleted

stack@devstack:~/devstack$ openstack volume create --size 1 --image test-cirros first-volume
stack@devstack:~/devstack$ openstack volume backup create --name first-volume-backup first-volume
+-----------+--------------------------------------+
| Field     | Value                                |
+-----------+--------------------------------------+
| id        | 3f396ca9-86ea-4008-9041-63a656e03a4b |
| name      | first-volume-backup                  |
| volume_id | f96c3f3d-6c4e-4179-82f0-b38ccbc6ff78 |
+-----------+--------------------------------------+
stack@devstack:~/devstack$ openstack volume backup list
+--------------------------------------+---------------------+-------------+----------+------+-------------+
| ID                                   | Name                | Description | Status   | Size | Incremental |
+--------------------------------------+---------------------+-------------+----------+------+-------------+
| 3f396ca9-86ea-4008-9041-63a656e03a4b | first-volume-backup | None        | creating |    1 | False       |
+--------------------------------------+---------------------+-------------+----------+------+-------------+
stack@devstack:~/devstack$ openstack volume list
+--------------------------------------+--------------+------------+------+-------------+
| ID                                   | Name         | Status     | Size | Attached to |
+--------------------------------------+--------------+------------+------+-------------+
| f96c3f3d-6c4e-4179-82f0-b38ccbc6ff78 | first-volume | backing-up |    1 |             |
+--------------------------------------+--------------+------------+------+-------------+

C) backup service disabled and stopped + other services restarted

This case effectively represents the backup service not being configured in the first place and hence reproduces the original bug report.

State:

• openstack volume service set --disable devstack cinder-backup
• systemctl stop devstack@c-bak
• systemctl restart devstack@c-api devstack@c-vol devstack@c-sch

Result:

• backup service shows up as "disabled" and "down" in openstack volume service list
• backup creation command returns
• backup enters error state
• backup deletion fails
• volume deletion works

stack@devstack:~/devstack$ openstack volume create --size 1 --image test-cirros first-volume
stack@devstack:~/devstack$ openstack volume backup create --name first-volume-backup first-volume
+-----------+--------------------------------------+
| Field     | Value                                |
+-----------+--------------------------------------+
| id        | 87806d02-cdd3-40a5-8b48-e17b1fb7ada6 |
| name      | first-volume-backup                  |
| volume_id | 21fdadb9-5567-4ec7-82d5-965c15554f80 |
+-----------+--------------------------------------+
stack@devstack:~/devstack$ openstack volume backup list
+--------------------------------------+---------------------+-------------+--------+------+-------------+
| ID                                   | Name                | Description | Status | Size | Incremental |
+--------------------------------------+---------------------+-------------+--------+------+-------------+
| 87806d02-cdd3-40a5-8b48-e17b1fb7ada6 | first-volume-backup | None        | error  |    1 | False       |
+--------------------------------------+---------------------+-------------+--------+------+-------------+

stack@devstack:~/devstack$ openstack volume backup delete first-volume-backup
Failed to delete backup with name or ID 'first-volume-backup': ResourceNotFound: 404: Client Error for url: http://192.168.144.111/volume/v3/a4f577b38eb54c2f929d6a80290b635e/backups/87806d02-cdd3-40a5-8b48-e17b1fb7ada6, Service cinder-backup could not be found.
1 of 1 backups failed to delete.

stack@devstack:~/devstack$ openstack volume delete first-volume
# (success)

D) backup service enabled but stopped

State:

• openstack volume service set --enable devstack cinder-backup
• systemctl stop devstack@c-bak
• systemctl restart devstack@c-api devstack@c-vol devstack@c-sch

Result:

• backup service shows up as "enabled" but "down" in openstack volume service list
• backup creation command returns
• backup enters error state
• volume deletion works

stack@devstack:~/devstack$ openstack volume create --size 1 --image test-cirros first-volume
stack@devstack:~/devstack$ openstack volume backup create --name first-volume-backup first-volume
+-----------+--------------------------------------+
| Field     | Value                                |
+-----------+--------------------------------------+
| id        | c8b57468-5d5b-4f31-be98-6ed063bd7f1b |
| name      | first-volume-backup                  |
| volume_id | 205a5458-e111-4214-ab86-180c88a36ef7 |
+-----------+--------------------------------------+
stack@devstack:~/devstack$ openstack volume backup list
+--------------------------------------+---------------------+-------------+--------+------+-------------+
| ID                                   | Name                | Description | Status | Size | Incremental |
+--------------------------------------+---------------------+-------------+--------+------+-------------+
| c8b57468-5d5b-4f31-be98-6ed063bd7f1b | first-volume-backup | None        | error  |    1 | False       |
+--------------------------------------+---------------------+-------------+--------+------+-------------+

stack@devstack:~/devstack$ openstack volume backup delete first-volume-backup
Failed to delete backup with name or ID 'first-volume-backup': ResourceNotFound: 404: Client Error for url: http://192.168.144.111/volume/v3/a4f577b38eb54c2f929d6a80290b635e/backups/c8b57468-5d5b-4f31-be98-6ed063bd7f1b, Service cinder-backup could not be found.
1 of 1 backups failed to delete.

stack@devstack:~/devstack$ openstack volume delete first-volume
# (success)

Case A is especially bizarre as it allows some actions against a disabled service that do have implications.
Case C seems to be the closest to what was reported above and should resemble setups where the volume backup service was never deployed.

The most reliable precaution would be to check volume service list (i.e. the corresponding API) for the presence of any backup service and if it is both "enabled" and "up".
Sadly, this API (volume_extension:services:index) is per default admin-only and can't be queried as a regular user.

It is pretty bad that Cinder allows creation of a volume backup resource regardless of the backup service's presence and state but on the other hand does not allow deletion of such resource if anything is wrong with the service or if it is absent.

[mbuechse]

@markus-hentsch Impressive research work! So there's nothing we can do really? :(

[berendt]

@mbuechse Can you try if a volume backup list fails when backups are not possible? That might be a workaround if you try that first. If this does not work then no backups are usable.

[markus-hentsch]

@mbuechse Can you try if a volume backup list fails when backups are not possible? That might be a workaround if you try that first. If this does not work then no backups are usable.

Everything in openstack volume backup [create|list|show] succeeds* even if the backup service is disabled and down. These things happen mostly at database/API level and seem only asynchronously consumed by the backup service. Only openstack volume backup delete fails as it actually tries to talk to the backup service in a synchronous manner.

* volume backup create succeeds so far as creating a backup resource that is visible via volume backup list, it will be in "error" state however if the backup service is absent.

That seems to me like a pretty severe architectural design flaw in Cinder from a user's perspective as the presence of the backup service cannot be checked until creation/deletion attempts.

[markus-hentsch]

@markus-hentsch Impressive research work! So there's nothing we can do really? :(

I can address the other valid points you raised about the testing script.

However, in regard to the undeletable leftover backup resource I currently have no idea how to tackle. Any attempts to figure out the presence of the backup service as a non-admin user have failed due to it being obfuscated by the API layer (see comment above).
I'm currently trying to get in touch with the Cinder community¹ but so far I'm out of ideas.

Footnotes

1. https://meetings.opendev.org/irclogs/%23openstack-cinder/%23openstack-cinder.2024-11-04.log.html#t2024-11-04T14:08:48 ↩

[markus-hentsch]

FTR, the backup service is not mandatory. The official documentation even states:

Optionally, install and configure the backup service. [...]

As such, I think this is an actual flaw/bug in Cinder in general.