Clean up the documentation for the library testing (#59401)

MaximLipnin · web-flow · commit b9a2c48aa84b · 2021-11-08T12:11:54.000+03:00
diff --git a/docs/workflow/testing/libraries/testing-android.md b/docs/workflow/testing/libraries/testing-android.md
@@ -1,18 +1,23 @@
 # Testing Libraries on Android
 
+## Prerequisites
+
 The following dependencies should be installed in order to be able to run tests:
 
+- OpenJDK
 - Android NDK
 - Android SDK
-- OpenJDK
-- OpenSSL
+
+To manage the dependencies, you can install them via terminal or using Android Studio.
+
+### Using a terminal
 
 OpenJDK can be installed on Linux (Ubuntu) using `apt-get`:
 ```bash
 sudo apt-get install openjdk-8 zip unzip
 ```
 
-Android SDK, NDK and OpenSSL can be automatically installed via the following script:
+Android SDK and NDK can be automatically installed via the following script:
 ```bash
 #!/usr/bin/env bash
 set -e
@@ -21,7 +26,6 @@ NDK_VER=r21b
 SDK_VER=6200805_latest
 SDK_API_LEVEL=29
 SDK_BUILD_TOOLS=29.0.3
-OPENSSL_VER=1.1.1g-alpha-1
 
 if [[ "$OSTYPE" == "darwin"* ]]; then
     HOST_OS=darwin
@@ -47,8 +51,21 @@ yes | ${ANDROID_SDK_ROOT}/cmdline-tools/tools/bin/sdkmanager --sdk_root=${ANDROI
 ${ANDROID_SDK_ROOT}/cmdline-tools/tools/bin/sdkmanager --sdk_root=${ANDROID_SDK_ROOT} "platform-tools" "platforms;android-${SDK_API_LEVEL}" "build-tools;${SDK_BUILD_TOOLS}"
 ```
 
+### Using Android Studio
+
+Android Studio offers a convenient UI:
+- to install all the dependencies;
+- to manage android virtual devices;
+- to make easy use of adb logs.
+
 ## Building Libs and Tests for Android
 
+Before running a build you might want to set the Android SDK and NDK environment variables:
+```
+export ANDROID_SDK_ROOT=<PATH-TO-ANDROID-SDK>
+export ANDROID_NDK_ROOT=<PATH-TO-ANDROID-NDK>  
+```
+
 Now we're ready to build everything for Android:
 ```
 ./build.sh mono+libs -os Android -arch x64
@@ -66,16 +83,43 @@ The following shows how to run tests for a specific library
 ./dotnet.sh build /t:Test src/libraries/System.Numerics.Vectors/tests /p:TargetOS=Android /p:TargetArchitecture=x64
 ```
 
+### Running the functional tests
+
+There are [functional tests](https://github.com/dotnet/runtime/tree/main/src/tests/FunctionalTests/) which aim to test some specific features/configurations/modes on a target mobile platform.
+
+A functional test can be run the same way as any library test suite, e.g.:
+```
+./dotnet.sh build /t:Test -c Release /p:TargetOS=Android /p:TargetArchitecture=x64 src/tests/FunctionalTests/Android/Device_Emulator/PInvoke/Android.Device_Emulator.PInvoke.Test.csproj
+```
+
+Currently functional tests are expected to return `42` as a success code so please be careful when adding a new one.
+
+### Testing various configurations
+
+It's possible to test various configurations by setting a combination of additional MSBuild properties such as `RunAOTCompilation`,`MonoForceInterpreter`, and some more.
+
+1. AOT
+
+To build for AOT only mode, add `/p:RunAOTCompilation=true /p:MonoForceInterpreter=false` to a build command.
+
+2. AOT-LLVM
+
+To build for AOT-LLVM mode, add `/p:RunAOTCompilation=true /p:MonoForceInterpreter=false /p:MonoEnableLLVM=true` to a build command.
+
+3. Interpreter
+
+To build for Interpreter mode, add `/p:RunAOTCompilation=false /p:MonoForceInterpreter=true` to a build command.
+
 ### Test App Design
-Android app is basically a [Java Instrumentation](https://github.com/dotnet/runtime/blob/main/src/mono/msbuild/AndroidAppBuilder/Templates/MonoRunner.java) and a simple Activity that inits the Mono Runtime via JNI. This Mono Runtime starts a simple xunit test
+Android app is basically a [Java Instrumentation](https://github.com/dotnet/runtime/blob/main/src/tasks/AndroidAppBuilder/Templates/MonoRunner.java) and a simple Activity that inits the Mono Runtime via JNI. This Mono Runtime starts a simple xunit test
 runner called XHarness.TestRunner (see https://github.com/dotnet/xharness) which runs tests for all `*.Tests.dll` libs in the bundle. There is also XHarness.CLI tool with ADB embedded to deploy `*.apk` to a target (device or emulator) and obtain logs once tests are completed.
 
 ### Obtaining the logs
 XHarness for Android doesn't talk much and only saves test results to a file. However, you can also subscribe to live logs via the following command:
 ```
 adb logcat -s "DOTNET"
 ```
-Or simply open `logcat` window in Android Studio or Visual Stuido.
+Or simply open `logcat` window in Android Studio or Visual Studio.
 
 ### AVD Manager
 If Android Studio is installed, [AVD Manager](https://developer.android.com/studio/run/managing-avds) can be used from the IDE to create and start Android virtual devices. Otherwise, the Android SDK provides the [`avdmanager` command line tool](https://developer.android.com/studio/command-line/avdmanager).
diff --git a/docs/workflow/testing/libraries/testing-apple.md b/docs/workflow/testing/libraries/testing-apple.md
@@ -1,34 +1,117 @@
-# Testing Libraries on iOS and tvOS
+# Testing Libraries on iOS, tvOS, and MacCatalyst
 
-In order to build libraries and tests for iOS or tvOS you need recent version of XCode installed (e.g. 11.3 or higher).
+## Prerequisites
 
-Build Libraries for iOS Simulator:
+- XCode 11.3 or higher
+- a certificate and provisioning profile if using a device
+- a simulator with a proper device type and OS version.
+Go `XCode > Window > Devices and Simulators` to revise the list of the available simulators and then `"+" button on bottom left > OS Version dropdown selection > Download more simulator runtimes` in case you need to download more simulators.
+
+## Building Libs and Tests
+
+You can build and run the library tests:
+- on a simulator;
+- on a device.
+
+Run the following command in a terminal:
+```
+./build.sh mono+libs -os <TARGET_OS> -arch <TARGET_ARCHITECTURE>
+```
+where `<TARGET_OS>` is one of the following:
+- iOSSimulator
+- tvOSSimulator
+- MacCatalyst
+- iOS
+- tvOS
+
+and `<TARGET_ARCHITECTURE>` is one of the following:
+- x64
+- arm64 (for device)
+
+e.g., to build for an iOS simulator, run:
 ```
 ./build.sh mono+libs -os iOSSimulator -arch x64
 ```
+
 Run tests one by one for each test suite on a simulator:
 ```
 ./build.sh libs.tests -os iOSSimulator -arch x64 -test
 ```
+
+### Building for a device
+
 In order to run the tests on a device:
-- Set the os to `iOS` instead of `iOSSimulator`
-- Specify `DevTeamProvisioning` (see [developer.apple.com/account/#/membership](https://developer.apple.com/account/#/membership), scroll down to `Team ID`):
+- Set the `-os` parameter to a device-related value (see above)
+- Specify `DevTeamProvisioning` (see [developer.apple.com/account/#/membership](https://developer.apple.com/account/#/membership), scroll down to `Team ID`).
+
+For example:
 ```
 ./build.sh libs.tests -os iOS -arch x64 -test /p:DevTeamProvisioning=H1A2B3C4D5
 ```
-[AppleAppBuilder](https://github.com/dotnet/runtime/blob/main/src/mono/msbuild/AppleAppBuilder/AppleAppBuilder.cs) generates temp Xcode projects you can manually open and resolve provisioning issues there using native UI and deploy to your devices.
+[AppleAppBuilder](https://github.com/dotnet/runtime/blob/main/src/tasks/AppleAppBuilder/AppleAppBuilder.cs) generates temp Xcode projects you can manually open and resolve provisioning issues there using native UI and deploy to your devices.
 
 ### Running individual test suites
+
 - The following shows how to run tests for a specific library:
 ```
 ./dotnet.sh build src/libraries/System.Numerics.Vectors/tests /t:Test /p:TargetOS=iOS /p:TargetArchitecture=x64
 ```
 
+Also you can run the built test app through Xcode by opening the corresponding `.xcodeproj` and setting up the right scheme, app, and even signing if using a local device.
+
+There's also an option to run a `.app` through `xcrun`, which is simulator specific. Consider the following shell script:
+
+```
+    xcrun simctl shutdown <IOSSIMULATOR_NAME>
+    xcrun simctl boot <IOSSIMULATOR_NAME>
+    open -a Simulator
+    xcrun simctl install <IOSSIMULATOR_NAME> <APP_BUNDLE_PATH>
+    xcrun simctl launch --console booted <IOS_APP_NAME>
+```
+
+where
+`<IOSSIMULATOR_NAME>` is a name of the simulator to start, e.g. `"iPhone 11"`,
+`<APP_BUNDLE_PATH>` is a path to the iOS test app bundle,
+`<IOS_APP_NAME>` is a name of the iOS test app
+
+### Running the functional tests
+
+There are [functional tests](https://github.com/dotnet/runtime/tree/main/src/tests/FunctionalTests/) which aim to test some specific features/configurations/modes on a target mobile platform.
+
+A functional test can be run the same way as any library test suite, e.g.:
+```
+./dotnet.sh build /t:Test -c Release /p:TargetOS=iOSSimulator /p:TargetArchitecture=x64 src/tests/FunctionalTests/iOS/Simulator/PInvoke/iOS.Simulator.PInvoke.Test.csproj
+```
+
+Currently functional tests are expected to return `42` as a success code so please be careful when adding a new one.
+
+### Viewing logs
+- see the logs generated by the XHarness tool
+- use the `Console` app on macOS:
+`Command + Space`, type in `Console`, search the appropriate process (System.Buffers.Tests for example).
+
+### Testing various configurations
+
+It's possible to test various configurations by setting a combination of additional MSBuild properties such as `RunAOTCompilation`,`MonoEnableInterpreter`, and some more.
+
+1. Interpreter Only
+
+This configuration is necessary for hot reload scenarios.
+
+To enable the interpreter, add `/p:RunAOTCompilation=true /p:MonoEnableInterpreter=true` to a build command.
+
+2. AOT only
+
+To build for AOT only mode, add `/p:RunAOTCompilation=true /p:MonoEnableInterpreter=false` to a build command.
+
+3. AOT-LLVM
+
+To build for AOT-LLVM mode, add `/p:RunAOTCompilation=true /p:MonoEnableInterpreter=false /p:MonoEnableLLVM=true` to a build command.
+
 ### Test App Design
-iOS/tvOS `*.app` (or `*.ipa`) is basically a simple [ObjC app](https://github.com/dotnet/runtime/blob/main/src/mono/msbuild/AppleAppBuilder/Templates/main-console.m) that inits the Mono Runtime. This Mono Runtime starts a simple xunit test
+iOS/tvOS `*.app` (or `*.ipa`) is basically a simple [ObjC app](https://github.com/dotnet/runtime/blob/main/src/tasks/AppleAppBuilder/Templates/main-console.m) that inits the Mono Runtime. This Mono Runtime starts a simple xunit test
 runner called XHarness.TestRunner (see https://github.com/dotnet/xharness) which runs tests for all `*.Tests.dll` libs in the bundle. There is also XHarness.CLI tool to deploy `*.app` and `*.ipa` to a target (device or simulator) and listens for logs via network sockets.
 
 ### Existing Limitations
-- Most of the test suites crash on devices due to #35674
 - Simulator uses JIT mode only at the moment (to be extended with FullAOT and Interpreter)
 - Interpreter is not enabled yet.
diff --git a/docs/workflow/testing/mono/testing.md b/docs/workflow/testing/mono/testing.md
@@ -93,7 +93,20 @@ For example, the following command is for running System.Runtime tests:
 make run-tests-corefx-System.Runtime
 ```
 ### Mobile targets and WebAssembly
-Build and run library tests against Webassembly, Android or iOS. See instructions located in [Library testing document folder](../libraries/)
+Build and run library tests against WebAssembly, Android or iOS. See instructions located in [Library testing document folder](../libraries/)
+
+## Running the functional tests
+
+There are the [functional tests](https://github.com/dotnet/runtime/tree/main/src/tests/FunctionalTests/) which aim to test some specific features/configurations/modes on Android, iOS-like platforms (iOS/tvOS + simulators, MacCatalyst), and WebAssembly.
+
+A functional test can be run the same way as any library test suite, e.g.:
+```
+./dotnet.sh build /t:Test -c Release /p:TargetOS=Android /p:TargetArchitecture=x64 src/tests/FunctionalTests/Android/Device_Emulator/PInvoke/Android.Device_Emulator.PInvoke.Test.csproj
+```
+
+Currently the functional tests are expected to return `42` as a success code so please be careful when adding a new one.
+
+For more details, see instructions located in [Library testing document folder](../libraries/).
 
 # Running the Mono samples
 There are a few convenient samples located in `$(REPO_ROOT)/src/mono/sample`, which could help you test your program easily with different flavors of Mono or do a sanity check on the build. The samples are set up to work with a specific configuration; please refer to the relevant Makefile for specifics. If you would like to work with a different configuration, you can edit the Makefile.
