upgrade to net6.0 version + add explicit apphost loading for specific platforms #224

YuliiaKovalova · 2023-07-31T11:26:09Z

Problem

MSBuildLocator expects that hostfxr lib is already loaded in the process - that is true for Windows load library, but not for all platforms (macOS, for example).
Some of our customers face an issue like:


07/17/2023 15:48:47 Greenwich Mean Time: Error : 9543 :!!Resolve NET Core SDK: Failed
System.DllNotFoundException: Unable to load shared library 'hostfxr' or one of its dependencies. In order to help diagnose loading problems, consider setting the DYLD_PRINT_LIBRARIES environment variable: 
dlopen(/usr/local/share/dotnet/shared/Microsoft.NETCore.App/7.0.9/hostfxr.dylib, 1): image not found
dlopen(/Users/timjroberts/.vscode/extensions/ms-dotnettools.csdevkit-0.2.100-darwin-x64/components/CPS/platforms/darwin-x64/node_modules/@microsoft/visualstudio-projectsystem-buildhost.darwin-x64/hostfxr.dylib, 1): image not found
dlopen(hostfxr.dylib, 1): image not found
dlopen(/usr/local/share/dotnet/shared/Microsoft.NETCore.App/7.0.9/libhostfxr.dylib, 1): image not found
dlopen(/Users/timjroberts/.vscode/extensions/ms-dotnettools.csdevkit-0.2.100-darwin-x64/components/CPS/platforms/darwin-x64/node_modules/@microsoft/visualstudio-projectsystem-buildhost.darwin-x64/libhostfxr.dylib, 1): image not found
dlopen(libhostfxr.dylib, 1): image not found
dlopen(/usr/local/share/dotnet/shared/Microsoft.NETCore.App/7.0.9/hostfxr, 1): image not found
dlopen(/Users/timjroberts/.vscode/extensions/ms-dotnettools.csdevkit-0.2.100-darwin-x64/components/CPS/platforms/darwin-x64/node_modules/@microsoft/visualstudio-projectsystem-buildhost.darwin-x64/hostfxr, 1): image not found
dlopen(hostfxr, 1): image not found
dlopen(/usr/local/share/dotnet/shared/Microsoft.NETCore.App/7.0.9/libhostfxr, 1): image not found
dlopen(/Users/timjroberts/.vscode/extensions/ms-dotnettools.csdevkit-0.2.100-darwin-x64/components/CPS/platforms/darwin-x64/node_modules/@microsoft/visualstudio-projectsystem-buildhost.darwin-x64/libhostfxr, 1): image not found
dlopen(libhostfxr, 1): image not found

Solution

Microsoft.NETCore.DotNetAppHost is a metadata package that loads a platform-specific assembly in runtime.
Specifying reference to it explicitly
<PackageReference Include="runtime.$(RuntimeIdentifier).Microsoft.NETCore.DotNetAppHost" Version="7.0.9" />
helps to copy the content in project/bin folder , so missed library can be resolved.

…latforms.

ladipro · 2023-07-31T11:58:29Z

src/MSBuildLocator/Microsoft.Build.Locator.csproj

@@ -11,6 +11,7 @@
    <SignAssembly>true</SignAssembly>
    <AssemblyOriginatorKeyFile>key.snk</AssemblyOriginatorKeyFile>
    <EnableDefaultNoneItems>false</EnableDefaultNoneItems>
+	<RuntimeIdentifier Condition="'$(RuntimeIdentifier)' == ''">$(NETCoreSdkRuntimeIdentifier)</RuntimeIdentifier>


This makes it work when building locally (NETCoreSdkRuntimeIdentifier is what the SDK is running on) but it's probably not enough to build a package that can be used everywhere.

Can we easily enumerate the problematic RIDs and package up the native dependency for all of them?

I wasn't able to enumerate them easily, but listing runtime assemblies works as expected

baronfel · 2023-07-31T14:42:57Z

src/MSBuildLocator/Microsoft.Build.Locator.csproj

@@ -25,6 +25,11 @@
    <PackageReference Include="Microsoft.VisualStudio.SDK.EmbedInteropTypes" Version="15.0.36" PrivateAssets="all" />
  </ItemGroup>

+  <ItemGroup Condition="'$(TargetFramework)'=='net6.0'">
+	 <!-- These package references help to resolve assemblies in runtime for platforms, where these are not loaded to AppContext by default (e.g MacOS). -->
+    <PackageReference Include="Microsoft.NETCore.DotNetAppHost" Version="7.0.9" />


MSBuildLocator targeting net6.0 suggests to me that we should be using the latest 6.0.x version of this package - how do consumers on different TFMs need to resolve this package? Is the DotNetAppHost package something that needs to be 'linked' to the final caller's TFM, so that net8.0 maps to 8.0 packages, net7.0 to 7.0.x packages, etc?

I just wanted to avoid adding this package for Framework target.
Probably either @rainersigwald or @ladipro can answer these questions.

Presumably the latest hosting library works with older versions of the runtime, as I think for us it is a requirement to be able to locate a .NET 7 MSBuild using MSBuildLocator built against .NET 6. @vitek-karas, can you please confirm?

Thet nethost (which is the one thing used from this package) is definitely fully backward compatible. There's obviously no 100% forward compatibility forever, but in general we will try really hard to keep it forward compatible for as long as possible.
So in that sense, use the latest version of the package regardless of what TFM you target (just like it's a good idea to use the latest SDK to build your app regardless of the TFM) - and upgrade this often (once 8 ships, upgrade to that and so on).

src/MSBuildLocator/build/TraverseRuntimeIdentifiers.targets

YuliiaKovalova · 2023-08-01T09:16:54Z

@ladipro , @rainersigwald is it ok that we have started to generate this folder in output?

runtime content

How will it be shipped?

ladipro · 2023-08-02T07:44:01Z

@YuliiaKovalova this looks expected, the native libraries should not be taking too much extra space.

How will it be shipped?

It should become part of the Microsoft.Build.Locator package.

ladipro

Can you please check that the hostfxr P/Invoke works when MSBuildLocator is used via a PackageReference. i.e. that the newly added native dependencies are correctly packed?

Also, is it possible to cover the P/Invoke with a test? Something that would have previously failed on Mac or Linux.

ladipro · 2023-08-02T08:12:32Z

Also, is it possible to cover the P/Invoke with a test? Something that would have previously failed on Mac or Linux.

Actually, it should still be failing because we're currently not making the call to nethost!get_hostfxr_path. Or are the current changes enough to fix the issue? I hope I'm not misunderstanding the changes.

YuliiaKovalova · 2023-08-02T08:16:28Z

Also, is it possible to cover the P/Invoke with a test? Something that would have previously failed on Mac or Linux.

Actually, it should still be failing because we're currently not making the call to nethost!get_hostfxr_path. Or are the current changes enough to fix the issue? I hope I'm not misunderstanding the changes.

It works as expected now because the source issue with the missed lib is resolved. Previously, it was failing here https://github.com/YuliiaKovalova/MSBuildLocator/blob/a7e81c92e079900d3403e71542c59d6cc767daaf/src/MSBuildLocator/DotNetSdkLocationHelper.cs#L116C36-L116C56 with

07/17/2023 15:48:47 Greenwich Mean Time: Error : 9543 :!!Resolve NET Core SDK: Failed System.DllNotFoundException: Unable to load shared library 'hostfxr' or one of its dependencies. In order to help diagnose loading problems, consider setting the DYLD_PRINT_LIBRARIES environment variable: dlopen(/usr/local/share/dotnet/shared/Microsoft.NETCore.App/7.0.9/hostfxr.dylib, 1): image not found dlopen(/Users/timjroberts/.vscode/extensions/ms-dotnettools.csdevkit-0.2.100-darwin-x64/components/CPS/platforms/darwin-x64/node_modules/@microsoft/visualstudio-projectsystem-buildhost.darwin-x64/hostfxr.dylib, 1): image not found dlopen(hostfxr.dylib, 1): image not found dlopen(/usr/local/share/dotnet/shared/Microsoft.NETCore.App/7.0.9/libhostfxr.dylib, 1): image not found dlopen(/Users/timjroberts/.vscode/extensions/ms-dotnettools.csdevkit-0.2.100-darwin-x64/components/CPS/platforms/darwin-x64/node_modules/@microsoft/visualstudio-projectsystem-buildhost.darwin-x64/libhostfxr.dylib, 1): image not found dlopen(libhostfxr.dylib, 1): image not found dlopen(/usr/local/share/dotnet/shared/Microsoft.NETCore.App/7.0.9/hostfxr, 1): image not found dlopen(/Users/timjroberts/.vscode/extensions/ms-dotnettools.csdevkit-0.2.100-darwin-x64/components/CPS/platforms/darwin-x64/node_modules/@microsoft/visualstudio-projectsystem-buildhost.darwin-x64/hostfxr, 1): image not found dlopen(hostfxr, 1): image not found dlopen(/usr/local/share/dotnet/shared/Microsoft.NETCore.App/7.0.9/libhostfxr, 1): image not found dlopen(/Users/timjroberts/.vscode/extensions/ms-dotnettools.csdevkit-0.2.100-darwin-x64/components/CPS/platforms/darwin-x64/node_modules/@microsoft/visualstudio-projectsystem-buildhost.darwin-x64/libhostfxr, 1): image not found dlopen(libhostfxr, 1): image not found

There is no need to request the path.

ladipro · 2023-08-02T08:28:48Z

That's odd, it looks like we're not even loading the nethost library so it's not clear how it makes the call to hostfxr succeed 🤔

YuliiaKovalova added 2 commits July 31, 2023 13:15

upgrade netcore version + add explicit apphost loading for specific p…

60cb8cc

…latforms.

downgrade sdk version

1166b2a

YuliiaKovalova changed the title ~~upgrade netcore version + add explicit apphost loading for specific platforms~~ upgrade to net6.0 version + add explicit apphost loading for specific platforms Jul 31, 2023

YuliiaKovalova mentioned this pull request Jul 31, 2023

DRAFT: Fix Unable to load shared library 'hostfxr' #209

Closed

upgrate sdk version for tests

9db5fb6

YuliiaKovalova mentioned this pull request Jul 31, 2023

MSBuildLocator.QueryVisualStudioInstances Fails on Mac and Linux from Version 1.5.3 #210

Closed

YuliiaKovalova requested review from ladipro and rainersigwald July 31, 2023 11:45

fix formatting

57665c4

ladipro reviewed Jul 31, 2023

View reviewed changes

fix review comment

07c1274

baronfel reviewed Jul 31, 2023

View reviewed changes

src/MSBuildLocator/build/TraverseRuntimeIdentifiers.targets Outdated Show resolved Hide resolved

rainersigwald reviewed Jul 31, 2023

View reviewed changes

src/MSBuildLocator/build/TraverseRuntimeIdentifiers.targets Outdated Show resolved Hide resolved

YuliiaKovalova added 2 commits August 1, 2023 11:06

fix review comments

a76e660

fix invalid package name

a7e81c9

ladipro reviewed Aug 2, 2023

View reviewed changes

YuliiaKovalova marked this pull request as draft August 3, 2023 11:07

mogocat mentioned this pull request Aug 5, 2023

[Bug] Unable to load shared library 'hostfxr' or one of its dependencies dotnet/docfx#9047

Closed

YuliiaKovalova closed this Aug 10, 2023

YuliiaKovalova deleted the dev/ykovalova/fix_macos_sdk_resolution branch August 10, 2023 10:31

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

upgrade to net6.0 version + add explicit apphost loading for specific platforms #224

upgrade to net6.0 version + add explicit apphost loading for specific platforms #224

YuliiaKovalova commented Jul 31, 2023 •

edited

Loading

ladipro Jul 31, 2023

YuliiaKovalova Aug 1, 2023

baronfel Jul 31, 2023

YuliiaKovalova Jul 31, 2023

ladipro Aug 2, 2023

vitek-karas Aug 2, 2023

YuliiaKovalova commented Aug 1, 2023 •

edited

Loading

ladipro commented Aug 2, 2023

ladipro left a comment

ladipro commented Aug 2, 2023 •

edited

Loading

YuliiaKovalova commented Aug 2, 2023 •

edited

Loading

ladipro commented Aug 2, 2023

upgrade to net6.0 version + add explicit apphost loading for specific platforms #224

upgrade to net6.0 version + add explicit apphost loading for specific platforms #224

Conversation

YuliiaKovalova commented Jul 31, 2023 • edited Loading

Problem

Solution

ladipro Jul 31, 2023

Choose a reason for hiding this comment

YuliiaKovalova Aug 1, 2023

Choose a reason for hiding this comment

baronfel Jul 31, 2023

Choose a reason for hiding this comment

YuliiaKovalova Jul 31, 2023

Choose a reason for hiding this comment

ladipro Aug 2, 2023

Choose a reason for hiding this comment

vitek-karas Aug 2, 2023

Choose a reason for hiding this comment

YuliiaKovalova commented Aug 1, 2023 • edited Loading

ladipro commented Aug 2, 2023

ladipro left a comment

Choose a reason for hiding this comment

ladipro commented Aug 2, 2023 • edited Loading

YuliiaKovalova commented Aug 2, 2023 • edited Loading

ladipro commented Aug 2, 2023

YuliiaKovalova commented Jul 31, 2023 •

edited

Loading

YuliiaKovalova commented Aug 1, 2023 •

edited

Loading

ladipro commented Aug 2, 2023 •

edited

Loading

YuliiaKovalova commented Aug 2, 2023 •

edited

Loading