[Xamarin.Android.Build.Tasks] per-RID assemblies & typemaps (#8164)

Context: 929e701 Context: ce2bc68 Context: #7473 Context: #8155 The managed linker can produce assemblies optimized for the target `$(RuntimeIdentifier)` (RID), which means that they will differ between different RIDs. Our "favorite" example of this is `IntPtr.Size`, which is inlined by the linker into `4` or `8` when targeting 32-bit or 64-bit platforms. (See also #7473 and 929e701.) Another platform difference may come in the shape of CPU intrinsics which will change the JIT-generated native code in ways that will crash the application if the assembler instructions generated for the intrinsics aren't supported by the underlying processor. In addition, the per-RID assemblies will have different [MVID][0]s and **may** have different type and method metadata token IDs, which is important because typemaps *use* type and metadata token IDs; see also ce2bc68. All of this taken together invalidates our previous assumption that all the managed assemblies are identical. "Simply" using `IntPtr.Size` in an assembly that contains `Java.Lang.Object` subclasses will break things. This in turn could cause "mysterious" behavior or crashes in Release applications; see also Issue #8155. Prevent the potential problems by processing each per-RID assembly separately and output correct per-RID LLVM IR assembly using the appropriate per-RID information. Additionally, during testing I found that for our use of Cecil within `<GenerateJavaStubs/>` doesn't consistently remove the fields, delegates, and methods we remove in `MarshalMethodsAssemblyRewriter` when marshal methods are enabled, or it generates subtly broken assemblies which cause **some** applications to segfault at run time like so: I monodroid-gc: 1 outstanding GREFs. Performing a full GC! F libc : Fatal signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0x8 in tid 12379 (t6.helloandroid), pid 12379 (t6.helloandroid) F DEBUG : *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** F DEBUG : Build fingerprint: 'google/raven_beta/raven:14/UPB3.230519.014/10284690:user/release-keys' F DEBUG : Revision: 'MP1.0' F DEBUG : ABI: 'arm64' F DEBUG : Timestamp: 2023-07-04 22:09:58.762982002+0200 F DEBUG : Process uptime: 1s F DEBUG : Cmdline: com.microsoft.net6.helloandroid F DEBUG : pid: 12379, tid: 12379, name: t6.helloandroid >>> com.microsoft.net6.helloandroid <<< F DEBUG : uid: 10288 F DEBUG : tagged_addr_ctrl: 0000000000000001 (PR_TAGGED_ADDR_ENABLE) F DEBUG : signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0x0000000000000008 F DEBUG : Cause: null pointer dereference F DEBUG : x0 0000000000000000 x1 0000007ba1401af0 x2 00000000000000fa x3 0000000000000001 F DEBUG : x4 0000007ba1401b38 x5 0000007b9f2a8360 x6 0000000000000000 x7 0000000000000000 F DEBUG : x8 ffffffffffc00000 x9 0000007b9f800000 x10 0000000000000000 x11 0000007ba1400000 F DEBUG : x12 0000000000000000 x13 0000007ba374ad58 x14 0000000000000000 x15 00000013ead77d66 F DEBUG : x16 0000007ba372f210 x17 0000007ebdaa4a80 x18 0000007edf612000 x19 000000000000001f F DEBUG : x20 0000000000000000 x21 0000007b9f2a8320 x22 0000007b9fb02000 x23 0000000000000018 F DEBUG : x24 0000007ba374ad08 x25 0000000000000004 x26 0000007b9f2a4618 x27 0000000000000000 F DEBUG : x28 ffffffffffffffff x29 0000007fc592a780 F DEBUG : lr 0000007ba3701f44 sp 0000007fc592a730 pc 0000007ba3701e0c pst 0000000080001000 F DEBUG : 8 total frames F DEBUG : backtrace: F DEBUG : #00 pc 00000000002d4e0c /data/app/~~Av24J15xbf20XdrY3X2_wA==/com.microsoft.net6.helloandroid-4DusuNWIAkz1Ssi7fWVF-g==/lib/arm64/libmonosgen-2.0.so (BuildId: 761134f2369377582cc3a8e25ecccb43a2e0a877) F DEBUG : #1 pc 00000000002c29e8 /data/app/~~Av24J15xbf20XdrY3X2_wA==/com.microsoft.net6.helloandroid-4DusuNWIAkz1Ssi7fWVF-g==/lib/arm64/libmonosgen-2.0.so (BuildId: 761134f2369377582cc3a8e25ecccb43a2e0a877) F DEBUG : #2 pc 00000000002c34bc /data/app/~~Av24J15xbf20XdrY3X2_wA==/com.microsoft.net6.helloandroid-4DusuNWIAkz1Ssi7fWVF-g==/lib/arm64/libmonosgen-2.0.so (BuildId: 761134f2369377582cc3a8e25ecccb43a2e0a877) F DEBUG : #3 pc 00000000002c2254 /data/app/~~Av24J15xbf20XdrY3X2_wA==/com.microsoft.net6.helloandroid-4DusuNWIAkz1Ssi7fWVF-g==/lib/arm64/libmonosgen-2.0.so (BuildId: 761134f2369377582cc3a8e25ecccb43a2e0a877) F DEBUG : #4 pc 00000000002be0bc /data/app/~~Av24J15xbf20XdrY3X2_wA==/com.microsoft.net6.helloandroid-4DusuNWIAkz1Ssi7fWVF-g==/lib/arm64/libmonosgen-2.0.so (BuildId: 761134f2369377582cc3a8e25ecccb43a2e0a877) F DEBUG : #5 pc 00000000002bf050 /data/app/~~Av24J15xbf20XdrY3X2_wA==/com.microsoft.net6.helloandroid-4DusuNWIAkz1Ssi7fWVF-g==/lib/arm64/libmonosgen-2.0.so (BuildId: 761134f2369377582cc3a8e25ecccb43a2e0a877) F DEBUG : #6 pc 00000000002a53a4 /data/app/~~Av24J15xbf20XdrY3X2_wA==/com.microsoft.net6.helloandroid-4DusuNWIAkz1Ssi7fWVF-g==/lib/arm64/libmonosgen-2.0.so (mono_gc_collect+44) (BuildId: 761134f2369377582cc3a8e25ecccb43a2e0a877) F DEBUG : #7 pc 000000000000513c <anonymous:7ec716b000> This is because we generate Java Callable Wrappers over a set of original (linked or not) assemblies, then we scan them for classes derived from `Java.Lang.Object` and use that set as input to the marshal methods rewriter, which makes the changes (generates wrapper methods, decorates wrapped methods with `[UnmanagedCallersOnly]`, removes the old delegate methods as well as delegate backing fields) to all the `Java.Lang.Object` subclasses, then writes the modified assembly to a `new/<assembly.dll>` location (efa14e2), followed by copying the newly written assemblies back to the original location. At this point, we have the results returned by the subclass scanner in memory and **new** versions of those types on disk, but they are out of sync, since the types in memory refer to the **old** assemblies, but AOT is ran on the **new** assemblies which have a different layout, changed MVIDs and, potentially, different type and method token IDs (because we added some methods, removed others etc) and thus it causes the crashes at the run time. The now invalid set of "old" types is passed to the typemap generator. This only worked by accident, because we (incorrectly) used only the first linked assembly which happened to be the same one passed to the JLO scanner and AOT - so everything was fine at the execution time. Address this by *disabling* LLVM Marshal Methods (8bc7a3e) for .NET 8, setting `$(AndroidEnableMarshalMethods)`=False by default. We'll attempt to fix these issues for .NET 9. [0]: https://learn.microsoft.com/dotnet/api/system.reflection.module.moduleversionid?view=net-7.0
dotnet · Jul 13, 2023 · 6836818 · 6836818
1 parent f21e10d
commit 6836818
Show file tree

Hide file tree

Showing 16 changed files with 1,123 additions and 354 deletions.
diff --git a/src/Xamarin.Android.Build.Tasks/Tasks/GenerateJavaStubs.cs b/src/Xamarin.Android.Build.Tasks/Tasks/GenerateJavaStubs.cs
diff --git a/src/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/BuildTest.cs b/src/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/BuildTest.cs
@@ -1326,10 +1326,10 @@ public void Dispose ()
 			using (var builder = CreateApkBuilder (Path.Combine ("temp", TestContext.CurrentContext.Test.Name))) {
 				builder.ThrowOnBuildFailure = false;
 				Assert.IsFalse (builder.Build (proj), "Build should have failed with XA4212.");
-				StringAssertEx.Contains ($"error XA4", builder.LastBuildOutput, "Error should be XA4212");
+				StringAssertEx.Contains ($"error : XA4", builder.LastBuildOutput, "Error should be XA4212");
 				StringAssertEx.Contains ($"Type `UnnamedProject.MyBadJavaObject` implements `Android.Runtime.IJavaObject`", builder.LastBuildOutput, "Error should mention MyBadJavaObject");
 				Assert.IsTrue (builder.Build (proj, parameters: new [] { "AndroidErrorOnCustomJavaObject=False" }), "Build should have succeeded.");
-				StringAssertEx.Contains ($"warning XA4", builder.LastBuildOutput, "warning XA4212");
+				StringAssertEx.Contains ($"warning : XA4", builder.LastBuildOutput, "warning XA4212");
 			}
 		}
 

diff --git a/src/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/BuildTest2.cs b/src/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/BuildTest2.cs
@@ -26,7 +26,7 @@ public partial class BuildTest2 : BaseTest
 			new object[] {
 				/* isClassic */              false,
 				/* isRelease */              true,
-				/* marshalMethodsEnabled */  true,
+				/* marshalMethodsEnabled */  false,
 			},
 			new object[] {
 				/* isClassic */              false,

diff --git a/src/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/Tasks/LinkerTests.cs b/src/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/Tasks/LinkerTests.cs
@@ -1,9 +1,12 @@
 using System;
 using System.IO;
 using System.Linq;
+using System.Text;
 using Java.Interop.Tools.Cecil;
 using Mono.Cecil;
+using Mono.Cecil.Cil;
 using Mono.Linker;
+using Mono.Tuner;
 using MonoDroid.Tuner;
 using NUnit.Framework;
 using Xamarin.ProjectTools;
@@ -514,7 +517,7 @@ public void AndroidUseNegotiateAuthentication ([Values (true, false, null)] bool
 		}
 
 		[Test]
-		public void DoNotErrorOnPerArchJavaTypeDuplicates ()
+		public void DoNotErrorOnPerArchJavaTypeDuplicates ([Values(true, false)] bool enableMarshalMethods)
 		{
 			if (!Builder.UseDotNet)
 				Assert.Ignore ("Test only valid on .NET");
@@ -525,26 +528,73 @@ public void DoNotErrorOnPerArchJavaTypeDuplicates ()
 			lib.Sources.Add (new BuildItem.Source ("Library1.cs") {
 				TextContent = () => @"
 namespace Lib1;
-public class Library1 : Java.Lang.Object {
+public class Library1 : Com.Example.Androidlib.MyRunner {
 	private static bool Is64Bits = IntPtr.Size >= 8;
 
 	public static bool Is64 () {
 		return Is64Bits;
 	}
+
+	public override void Run () => Console.WriteLine (Is64Bits);
 }",
+			});
+			lib.Sources.Add (new BuildItem ("AndroidJavaSource", "MyRunner.java") {
+				Encoding = new UTF8Encoding (encoderShouldEmitUTF8Identifier: false),
+				TextContent = () => @"
+package com.example.androidlib;
+
+public abstract class MyRunner {
+	public abstract void run();
+}"
 			});
 			var proj = new XamarinAndroidApplicationProject { IsRelease = true, ProjectName = "App1" };
 			proj.References.Add(new BuildItem.ProjectReference (Path.Combine ("..", "Lib1", "Lib1.csproj"), "Lib1"));
 			proj.MainActivity = proj.DefaultMainActivity.Replace (
 				"base.OnCreate (bundle);",
 				"base.OnCreate (bundle);\n" +
 				"if (Lib1.Library1.Is64 ()) Console.WriteLine (\"Hello World!\");");
+			proj.SetProperty ("AndroidEnableMarshalMethods", enableMarshalMethods.ToString ());
 
 
 			using var lb = CreateDllBuilder (Path.Combine (path, "Lib1"));
 			using var b = CreateApkBuilder (Path.Combine (path, "App1"));
 			Assert.IsTrue (lb.Build (lib), "build should have succeeded.");
 			Assert.IsTrue (b.Build (proj), "build should have succeeded.");
+
+			var intermediate = Path.Combine (Root, b.ProjectDirectory, proj.IntermediateOutputPath);
+			var dll = $"{lib.ProjectName}.dll";
+			Assert64Bit ("android-arm", expected64: false);
+			Assert64Bit ("android-arm64", expected64: true);
+			Assert64Bit ("android-x86", expected64: false);
+			Assert64Bit ("android-x64", expected64: true);
+
+			void Assert64Bit(string rid, bool expected64)
+			{
+				var assembly = AssemblyDefinition.ReadAssembly (Path.Combine (intermediate, rid, "linked", "shrunk", dll));
+				var type = assembly.MainModule.FindType ("Lib1.Library1");
+				Assert.NotNull (type, "Should find Lib1.Library1!");
+				var cctor = type.GetTypeConstructor ();
+				Assert.NotNull (type, "Should find Lib1.Library1.cctor!");
+				Assert.AreNotEqual (0, cctor.Body.Instructions.Count);
+
+				/*
+				 * IL snippet
+				 * .method private hidebysig specialname rtspecialname static 
+				 * void .cctor () cil managed 
+				 * {
+				 *   // Is64Bits = 4 >= 8;
+				 *   IL_0000: ldc.i4 4
+				 *   IL_0005: ldc.i4.8
+				 *   ...
+				 */
+				var instruction = cctor.Body.Instructions [0];
+				Assert.AreEqual (OpCodes.Ldc_I4, instruction.OpCode);
+				if (expected64) {
+					Assert.AreEqual (8, instruction.Operand, $"Expected 64-bit: {expected64}");
+				} else {
+					Assert.AreEqual (4, instruction.Operand, $"Expected 64-bit: {expected64}");
+				}
+			}
 		}
 	}
 }
diff --git a/...ild.Tasks/Tests/Xamarin.ProjectTools/Resources/Base/BuildReleaseArm64SimpleDotNet.apkdesc b/...ild.Tasks/Tests/Xamarin.ProjectTools/Resources/Base/BuildReleaseArm64SimpleDotNet.apkdesc
@@ -2,64 +2,64 @@
   "Comment": null,
   "Entries": {
     "AndroidManifest.xml": {
-      "Size": 3032
+      "Size": 3036
     },
     "assemblies/_Microsoft.Android.Resource.Designer.dll": {
       "Size": 1024
     },
     "assemblies/Java.Interop.dll": {
-      "Size": 58703
+      "Size": 58990
     },
     "assemblies/Mono.Android.dll": {
-      "Size": 86588
+      "Size": 88074
     },
     "assemblies/Mono.Android.Runtime.dll": {
-      "Size": 5798
+      "Size": 5819
     },
     "assemblies/rc.bin": {
       "Size": 1235
     },
     "assemblies/System.Console.dll": {
-      "Size": 6442
+      "Size": 6448
     },
     "assemblies/System.Linq.dll": {
-      "Size": 9123
+      "Size": 9135
     },
     "assemblies/System.Private.CoreLib.dll": {
-      "Size": 536436
+      "Size": 537441
     },
     "assemblies/System.Runtime.dll": {
-      "Size": 2623
+      "Size": 2629
     },
     "assemblies/System.Runtime.InteropServices.dll": {
-      "Size": 3752
+      "Size": 3768
     },
     "assemblies/UnnamedProject.dll": {
-      "Size": 3349
+      "Size": 3222
     },
     "classes.dex": {
-      "Size": 19748
+      "Size": 377064
     },
     "lib/arm64-v8a/libmono-component-marshal-ilgen.so": {
-      "Size": 93552
+      "Size": 97392
     },
     "lib/arm64-v8a/libmonodroid.so": {
-      "Size": 380832
+      "Size": 380704
     },
     "lib/arm64-v8a/libmonosgen-2.0.so": {
-      "Size": 3160360
+      "Size": 3177168
     },
     "lib/arm64-v8a/libSystem.IO.Compression.Native.so": {
       "Size": 723560
     },
     "lib/arm64-v8a/libSystem.Native.so": {
-      "Size": 94392
+      "Size": 94424
     },
     "lib/arm64-v8a/libSystem.Security.Cryptography.Native.Android.so": {
       "Size": 154904
     },
     "lib/arm64-v8a/libxamarin-app.so": {
-      "Size": 16624
+      "Size": 11080
     },
     "META-INF/BNDLTOOL.RSA": {
       "Size": 1213
@@ -95,5 +95,5 @@
       "Size": 1904
     }
   },
-  "PackageSize": 2685258
+  "PackageSize": 2771274
 }