Hash names of all the packaged DSOs

[grendello]

This PR adds a comment section to libxamarin-app.so the branch
and commit hash of Xamarin.Android which produced the shared library.
The information can be displayed using llvm-objdump or objdump
as follows (parameters for both commands are the same):

$ llvm-objdump -sj .comment libxamarin-app.so 

libxamarin-app.so:    file format elf64-littleaarch64
Contents of section .comment:
 0000 58616d61 72696e2e 416e6472 6f696420  Xamarin.Android 
 0010 68617368 2d6d6f72 65204020 65366465  hash-more @ e6de
 0020 33343934 37333661 64306634 61613933  3494736ad0f4aa93
 0030 39393833 63336532 61313566 31623066  9983c3e2a15f1b0f
 0040 31643132 00                          1d12.

Use the hash to map any form of DSO name sent to us by Mono (we generate
hashes for several mutations) to find a cache entry for that DSO. Cache
includes the actual DSO name, whether it should be ignored when
requested (true for empty AOT DSOs) and the cached module handle.

This removes a bit of string processing from both startup and the
application run time.

This commit started with the goal to implement the above, but during
implementation of the DSO cache I hit known limitations of the native
assembler generator code which, until this commit weren't a problem
(mostly related to hard-coded structure and array alignment). So now
it includes a new implementation of the said generator, this time it
fully implements the structure and symbol alignment calculation for
all the supported architectures. Also added is the ability to generate
assembler code from managed structures, using reflection, without the need
of manually written code. It also fixes previously unnoticed issue which
made typemap structures not aligned properly, which may have made them
slightly slower than necessary

The implemented changes improve startup of both plain XA and MAUI applications,
measured on Pixel 3 XL running Android 12:

Plain XA application

AOT, Displayed time

Before	After	Δ	Notes
252.400	245.000	-2.93% ✓	defaults; profiled AOT; 32-bit build
253.500	247.000	-2.56% ✓	defaults; profiled AOT; 64-bit build
251.000	247.100	-1.55% ✓	defaults; full AOT; 64-bit build
255.700	252.500	-1.25% ✓	defaults; full AOT; 32-bit build
257.000	256.400	-0.23% ✓	defaults; full AOT+LLVM; 64-bit build
254.300	258.000	+1.43% ✗	defaults; full AOT+LLVM; 32-bit build

AOT, Total Initialization Time

Before	After	Δ	Notes
41.684	38.677	-7.21% ✓	defaults; profiled AOT; 64-bit build
41.357	38.727	-6.36% ✓	defaults; profiled AOT; 32-bit build
47.356	43.911	-7.27% ✓	defaults; full AOT; 32-bit build
47.256	44.356	-6.14% ✓	defaults; full AOT; 64-bit build
49.223	45.813	-6.93% ✓	defaults; full AOT+LLVM; 64-bit build
48.755	46.098	-5.45% ✓	defaults; full AOT+LLVM; 32-bit build

No AOT, Displayed time

Before	After	Δ	Notes
309.900	305.900	-1.29% ✓	preload disabled; no compression; 32-bit build
317.900	309.900	-2.52% ✓	preload disabled; no compression; 64-bit build
311.500	311.600	+0.03% ✗	preload enabled; 64-bit build
314.800	311.600	-1.02% ✓	preload disabled; 64-bit build
309.200	312.500	+1.06% ✗	preload enabled; no compression; 64-bit build
312.600	313.300	+0.22% ✗	preload disabled; 32-bit build
314.100	315.300	+0.38% ✗	preload enabled; 32-bit build
312.300	315.700	+1.08% ✗	preload enabled; no compression; 32-bit build

No AOT, Total Init Time

Before	After	Δ	Notes
61.433	60.357	-1.75% ✓	preload disabled; no compression; 32-bit build
60.245	60.521	+0.46% ✗	preload enabled; no compression; 32-bit build
60.216	60.801	+0.96% ✗	preload disabled; no compression; 64-bit build
60.261	61.141	+1.44% ✗	preload enabled; no compression; 64-bit build
62.366	61.636	-1.17% ✓	preload enabled; 64-bit build
61.924	61.776	-0.24% ✓	preload disabled; 64-bit build
61.754	62.007	+0.41% ✗	preload enabled; 32-bit build
62.529	62.792	+0.42% ✗	preload disabled; 32-bit build

MAUI Hello World

AOT, Displayed time

Before	After	Δ	Notes
868.900	831.700	-4.28% ✓	defaults; profiled AOT; 64-bit build
868.900	839.200	-3.42% ✓	defaults; profiled AOT; 32-bit build
901.900	868.800	-3.67% ✓	defaults; full AOT; 64-bit build
914.300	876.600	-4.12% ✓	defaults; full AOT+LLVM; 64-bit build
904.300	877.500	-2.96% ✓	defaults; full AOT; 32-bit build
913.400	879.000	-3.77% ✓	defaults; full AOT+LLVM; 32-bit build

AOT, Total Init Time

Before	After	Δ	Notes
114.953	95.573	-16.86% ✓	defaults; profiled AOT; 32-bit build
115.280	95.971	-16.75% ✓	defaults; profiled AOT; 64-bit build
205.505	172.190	-16.21% ✓	defaults; full AOT; 64-bit build
206.151	172.892	-16.13% ✓	defaults; full AOT; 32-bit build
215.617	181.895	-15.64% ✓	defaults; full AOT+LLVM; 64-bit build
216.313	181.980	-15.87% ✓	defaults; full AOT+LLVM; 32-bit build

No AOT, Displayed time

Before	After	Δ	Notes
1152.800	1145.200	-0.66% ✓	preload enabled; 64-bit build; no compression
1164.000	1153.100	-0.94% ✓	preload enabled; 64-bit build
1141.500	1155.200	+1.19% ✗	preload disabled; 64-bit build; no compression
1156.300	1159.500	+0.28% ✗	preload disabled; 64-bit build
1230.500	1231.900	+0.11% ✗	preload enabled; 32-bit build; no compression
1235.600	1238.300	+0.22% ✗	preload disabled; 32-bit build; no compression
1241.400	1241.900	+0.04% ✗	preload enabled; 32-bit build
1240.500	1253.100	+1.01% ✗	preload disabled; 32-bit build

No AOT, Total Init Time

Before	After	Δ	Notes
73.570	73.827	+0.35% ✗	preload disabled; 64-bit build; no compression
73.512	73.962	+0.61% ✗	preload enabled; 64-bit build; no compression
76.792	76.771	-0.03% ✓	preload enabled; 64-bit build
77.382	77.109	-0.35% ✓	preload disabled; 64-bit build
84.462	84.566	+0.12% ✗	preload disabled; 32-bit build; no compression
84.051	84.725	+0.80% ✗	preload enabled; 32-bit build; no compression
88.051	88.015	-0.04% ✓	preload enabled; 32-bit build
87.596	88.445	+0.96% ✗	preload disabled; 32-bit build

[grendello]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[jonpryor]

Why have any DSOCacheEntry values in the "stub" build? Why not just a zero-length array? (Or does something require non-zero length arrays?)

[grendello]

I'm using disassembly of the stub to make sure our managed generator does the right job, for that I need it to have "valid" content.

[jonpryor]

Why do we need to start emitting ident "Xamarin.Android 12.2.99"?

What do we start doing when .NET 7+ is our primary distribution vehicle and not Classic Xamarin.Android? Presumably we'll stop bumping or otherwise remove $(ProductVersion) eventually, and then use $(AndroidPackVersion) instead. (Presumably?). Should use use $(ProductVersion) for both .NET 6+ & Classic now, $(AndroidPackVersion) for both now, or selectively use one depending on which is being used?

[grendello]

I don't really care which version we use, I wanted to have Xamarin.Android ${VERSION} in there, it might be helpful in diagnosing issues to know which XA built the binary. We can put git commit hash there, whatever you think makes the most sense.

[jonpryor]

I kinda like the xamarin/xamarin-android@COMMIT-HASH idea, actually… Especially since it's "product name agnostic", and thus avoids the entire question in the first place.

[grendello]

I think it still should have some "product" name, not just the hash. Xamarin.Android [HASH]?

[jonpryor]

A "product name" of xamarin/xamarin-android@ avoids the need (responsibility) to pick a name, and since I'm being incredibly wishy-washy here, abdicating that responsibility sounds awesome.

…or we just stick with Xamarin.Android. :-).

The hash remains useful, and should be extremely helpful for future issue investigations.

[jonpryor]

Another one for the naming $DEITYs, "write empty buffer"? I'm not at all sure what the semantics are based on just looking at this line.

I wonder if BeginBuffer(…) would be better? Unless this is asserting that the buffer is in fact empty, but why would we ever have known empty buffers? (I have not yet read what WriteEmptyBuffer() does. I'm not sure I'll understand what it does when I get there.)

[grendello]

I'm really not sure what you're asking for here. If one doesn't know what the buffer does in this context, then I don't want them to touch the code. This is the lowest level generator for a specific set of specific structures that describe the entirety of application config, and modifying the code requires understanding what the structures are on the native side. Once that is understood, the terms used will make sense. The buffer is in fact empty, because it just allocates space in the output DSO to be filled at the run time. In this case, the code is used when assembly stores are off and we read the bundled assemblies from the apk, filling in this structure:

struct XamarinAndroidBundledAssembly final
{
        int32_t  apk_fd;
        uint32_t data_offset;
        uint32_t data_size;
        uint8_t *data;
        uint32_t name_length;
        char    *name;
};

the buffer is for the assembly name (as suggested by BundledAssemblyNameWidth in the method invocation) and has to be there because name has to point to something, we don't allocate data directly. All that information is there, in source code, and I do expect anyone modifying this code to familiarize themselves with the native side...

I'm really not sure how do you want to explain all that in a method name?

[jonpryor]

What is an "inline" width?

[grendello]

Get maximum width of data buffer allocated inline (that is not pointed to)

[jonpryor]

Where do these values come from?

[grendello]

From assembly generated by g++ and clang, somewhat described here https://sourceware.org/binutils/docs-2.37/as/ARM-Directives.html#ARM-Directives
Not all of them apply to data-only DSOs, but I want to match output of the compilers.

[jonpryor]

Where'd these values come from?

[grendello]

https://sourceware.org/binutils/docs-2.37/as/Section.html#Section

[jonpryor]

Should this be log_warn() and not log_info()?

[jonpryor]

Proposed commit message:

[monodroid] Hash the names of all packaged DSOs (#6522)

Context: https://github.com/xamarin/xamarin-android/pull/6522
Context: https://sourceware.org/binutils/docs-2.37/as/index.html
Context: https://sourceware.org/binutils/docs-2.37/as/AArch64-Directives.html#AArch64-Directives
Context: https://sourceware.org/binutils/docs-2.37/as/ARM-Directives.html#ARM-Directives
Context: https://sourceware.org/binutils/docs-2.37/as/Quad.html#Quad

Commit 000cf5a8 introduced hashing of native library names so that
we would only try to load specific Mono components which were
configured at app build time.

Expand this concept so that we support hashing any form of native
library name (DSO name) sent to us by Mono, and look for a cached
entry for that DSO.  The cache includes the actual native library
name, whether it should be ignored when requested (e.g. an empty AOT
`.so` file; see db161ae7 & df667b07), and the cached `dlopen()` value:

	struct DSOCacheEntry {
	    uint64_t    hash;
	    bool        ignore;
	    const char *name;
	    void       *handle;
	};
	DSOCacheEntry dso_cache[] = {…};

The `dso_cache` is computed at App build time, and stored in
`libxamarin-app.so`, and contains values sorted on
`DSOCacheEntry::hash`.

The use of `dso_cache` removes a bit of string processing from both
startup and the application run time, reducing app startup times in
some scenarios:

| Scenario                                      | Before ms |  After ms |        Δ |
| --------------------------------------------- | --------: | --------: | -------: |
| Plain Xamarin.Android (JIT, 64-bit)           |   311.500 |   311.600 | +0.03% ✗ |
| Plain Xamarin.Android (Profiled AOT, 64-bit)  |   253.500 |   247.000 | -2.56% ✓ |
| .NET MAUI Hello World (JIT, 64-bit)           |  1156.300 |  1159.500 | +0.28% ✗ |
| .NET MAUI Hello World (Profiled AOT, 64-bit)  |   868.900 |   831.700 | -4.28% ✓ |

(Times measured on a Pixel 3 XL.)

Above table is a subset of values from xamarin/xamarin-android#6522;
see original PR for complete table information.  We believe that the
occasional increases are within the margin of error.

While implementing the above described `dso_cache` idea, I hit known
limitations of the native assembler generator code which, until this
commit, weren't a problem (mostly related to hard-coded structure and
array alignment).  Address these limitations by rewriting the assembly
generator.  It now fully implements the structure and symbol alignment
calculation for all the supported architectures.  Also added is the
ability to generate assembler code from managed structures, using
reflection, without the need of manually written code.  It also fixes
a previously unnoticed issue which made typemap structures not aligned
properly, which may have made them slightly slower than necessary.

[grendello]

Regarding the TODOs above, the differences in both mentioned cases are so small that I chalked it up to measurement error (especially for Displayed times which is inherently unreliable), but JIT may be affected by external factors on device as well. The tests are executed on a "live" device which does a lot in the background, any form of I/O may affect the tests performance. Over the time I've been working on performance, I learned that differences of around 1% can largely be ignored and accepted as "no change"