Revisit the need for individual queues for each supported Android API level

[ivanpovazan]

Description

.NET 9 supports Android API levels ranging from 21 as the minimum version to 34.

To properly verify the full support we have to periodically run tests on every API level, as described in: dotnet/runtime#107868 and a PR for example: dotnet/runtime#111858
For this purpose we have 14 queues:

• Ubuntu.2204.Amd64.Android.21.Open
• Ubuntu.2204.Amd64.Android.22.Open
• Ubuntu.2204.Amd64.Android.23.Open
• Ubuntu.2204.Amd64.Android.24.Open
• Ubuntu.2204.Amd64.Android.25.Open
• Ubuntu.2204.Amd64.Android.26.Open
• Ubuntu.2204.Amd64.Android.27.Open
• Ubuntu.2204.Amd64.Android.28.Open
• Ubuntu.2204.Amd64.Android.29.Open
• Ubuntu.2204.Amd64.Android.30.Open
• Ubuntu.2204.Amd64.Android.31.Open
• Ubuntu.2204.Amd64.Android.32.Open
• Ubuntu.2204.Amd64.Android.33.Open
• Ubuntu.2204.Amd64.Android.34.Open

While on daily basis, we run all our testing on API level 29 using:

• Ubuntu.2204.Amd64.Android.29.Open

Proposal

It seems redundant to have all the queue available especially since we run tests on all supported API versions only a few times a year. Additionally, by reducing the number of queues we would improve maintenance and cost.

One idea would be to:

• Remove all queues Ubuntu.2204.Amd64.Android.*.Open except Ubuntu.2204.Amd64.Android.29.Open
• Preserve Ubuntu.2204.Amd64.Android.29.Open as is.
• Introduce new queue Ubuntu.2204.Amd64.Android.AllVersions.Open which will instead have different emulator images installed for every supported API level, and will be used only for our periodical/manual runs
• Adapt our infrastructure to utilize this change
  • NOTE: This is TBD for now

[ivanpovazan]

@vitek-karas @steveisok @matouskozak @akoeplinger @ilyas1974
I opened this issue up for discussion.

Please add your thoughts on potential problems with this approach.

[akoeplinger]

I'd call the mega queue something like Ubuntu.2204.Amd64.Android.AllVersions.Open so we don't need to change the name if we add/remove versions.

One potential issue could be the size of the image, we need to see how large it'd be.

The script in helix-machines would also need to be adapted to not run the emulator service at startup and somehow pass in the requested API level since I assume we can't run emulators for all versions at the same time due to cpu/memory.

[matouskozak]

I think we would need to explore how much of the infrastructure (helix, xharness, etc.) would need to adapt so that we can send a job to a queue which would essentially duplicate itself to be run on all (or selected) emulators installed on the VM.

One thing to note, the current queues are VMs so there shouldn't be any running cost on them if they are not in use but there surely is a maintenance cost on the helix side to keep them compliant.

[ilyas1974]

A few questions:

• As the plan is to consolidate queues, is there any reason why we can't also use this opportunity to update to Ubuntu.2204?
  • I know Ubuntu.24.04 queues for android do not exist currently, I'm just throwing out the idea since we'll need to do work in this space.
  • This will also allow you to do testing in queues that are completely separate from the ones that are currently being used for "production".
• I thought (perhaps my information is wrong) there were problems with running multiple emulator versions on a single box - hence the reason for the separate queues per API version.
• I will try to create a queue with the name you indicated above, but it may be too long for some of our dependencies.
  • Does Ubuntu.2204.amd64.androidapis.com work as an alternative?
• What type of urgency/priority does this work have?
  • I want to make sure I'm able to get funding for this work within the time frame you are needing.

[matouskozak]

I know Ubuntu.24.04 queues for android do not exist currently, I'm just throwing out the idea since we'll need to do work in this space.

If the emulators for needed API levels are available on Ubuntu 24.04, I don't think we would have a problem with using that.

[ivanpovazan]

Here is the summary of our internal meeting and identified action items:

No action

Keep using Ubuntu.2204.Amd64.Android.29.Open for all testing (including servicing branches)

Action

• Introduce two new queues for testing API levels: 35 and 36, with 22.02 Ubuntu, emulators should be configured as our current offerings, queues should be named: Ubuntu.2204.Amd64.Android.35.Open and Ubuntu.2204.Amd64.Android.36.Open
• Introduce a new queue for API level 29, with 24.02 Ubuntu, emulators should be configured as our current 22.02 offering, queue should be named: Ubuntu.2404.Amd64.Android.29.Open
• Introduce a new queue for all API levels, with 24.02 Ubuntu, configured with 26 emulator images for API 21-36:
  10x x86 (API 30 is the last which has x86 images) and 16x x64 abi variants, queue should be named: Ubuntu.2404.Amd64.Android.AllVersions.Open

Follow up work

Start testing new queues and figure out integration changes required to work with AllVersion queue

---

@ilyas1974 as we discussed, there are 3 action items I listed above.

Please let me know if you want to have separate GH issues created for them or you will handle it, thank you.

[ivanpovazan]

/cc: @steveisok

[ilyas1974]

One last question in this space. can we try Azure Linux 3.0 as the base for these new queues? And I will take care of getting all the GH issues created for this work.

[ivanpovazan]

can we try Azure Linux 3.0 as the base for these new queues?

@akoeplinger ?

[akoeplinger]

I quickly tried using the azure linux 3.0 docker image and was able to install sdkmanager and emulator but it didn't start because of nested virtualization issues in the docker. It looks good to me so far though so on a real VM it probably works.

[ilyas1974]

Thank you, Alex. I will go with the plan of making the new queues Azure Linux based. We do use a different SKU for the docker workloads with nested virtualization enabled. As a test, if I spin up an Azure Linux 3.0, install sdkmanager and the emulator, all I need to verify is that things start, correct?

[akoeplinger]

yes that should be enough. I noticed that I needed to install the e2fsprogs package because creating the emulator avd calls out to the chattr tool and that's contained in that package. Not sure if that's specific to the minimalistic AL3.0 docker image I used though

[ilyas1974]

Tagging this as part of issue #3593.