This document describes the infrastructure provided by the Arcade SDK for publishing build assets.
In order to use the new publishing mechanism, the easiest way to start is by turning your existing build pipeline into an AzDO YAML stage, and then making use of a YAML template (eng/common/templates/post-build/post-build.yml) provided by Arcade to use the default publishing stages. The process is explained below step by step.
-
Update the Arcade SDK version used by the repository to
1.0.0-beta.19360.8or newer. -
Disable asset publishing during the build. There are two common situations here. Some build definitions make use of the
jobs.ymltemplate and others make use of thejob.yml(singular). The former is a wrapper around a few things, among them thejob.ymlandpublish-build-assets.ymltemplates. If your build definition doesn't usejobs.ymlyou'll need to directly pass thePublishUsingPipelinesparameter to the included templates. See examples below.-
If the build job uses the
eng\common\templates\jobs\jobs.ymltemplate, set the parameterenablePublishUsingPipelinestotrue. See example below:jobs: - template: /eng/common/templates/jobs/jobs.yml parameters: enablePublishUsingPipelines: true
-
If the build job makes direct use of
eng\common\templates\job\job.ymlyou will have to do the following changes.-
Set the
enablePublishUsingPipelinesparameter totruewhen instantiatingjob.yml:jobs: ... - template: /eng/common/templates/job/job.yml parameters: ... enablePublishUsingPipelines: true ...
-
Make sure that you use the template
eng\common\templates\job\publish-build-assets.ymlto inform Maestro++ that all build jobs have finished executing. Also, make sure that you are setting the template parameterenablePublishUsingPipelinestotrue:jobs: ... - ${{ if and(ne(variables['System.TeamProject'], 'public'), notin(variables['Build.Reason'], 'PullRequest')) }}: - template: /eng/common/templates/job/publish-build-assets.yml parameters: ... publishUsingPipelines: true ...
-
-
-
You'll also need to pass the below MSBuild property to the Arcade build scripts.
| Name | Value |
|---|---|
| /p:DotNetPublishUsingPipelines | true |
For example, if the repo has the following configuration for invoking cibuild.cmd:
- _InternalBuildArgs: /p:DotNetSignType=$(_SignType)
/p:TeamName=$(_TeamName)
/p:DotNetPublishBlobFeedKey=$(dotnetfeed-storage-access-key-1)
/p:DotNetPublishBlobFeedUrl=$(_PublishBlobFeedUrl)
/p:DotNetPublishToBlobFeed=$(_DotNetPublishToBlobFeed)
/p:DotNetSymbolServerTokenMsdl=$(microsoft-symbol-server-pat)
/p:DotNetSymbolServerTokenSymWeb=$(symweb-symbol-server-pat)
/p:OfficialBuildId=$(BUILD.BUILDNUMBER)
- script: eng\common\cibuild.cmd
-configuration $(_BuildConfig)
-prepareMachine
$(_InternalBuildArgs)after setting the needed MSBuild properties it should looks like this:
- _InternalBuildArgs: /p:DotNetSignType=$(_SignType)
/p:TeamName=$(_TeamName)
/p:DotNetPublishBlobFeedKey=$(dotnetfeed-storage-access-key-1)
/p:DotNetPublishBlobFeedUrl=$(_PublishBlobFeedUrl)
/p:DotNetPublishToBlobFeed=$(_DotNetPublishToBlobFeed)
/p:DotNetSymbolServerTokenMsdl=$(microsoft-symbol-server-pat)
/p:DotNetSymbolServerTokenSymWeb=$(symweb-symbol-server-pat)
/p:OfficialBuildId=$(BUILD.BUILDNUMBER)
/p:DotNetPublishUsingPipelines=$(_PublishUsingPipelines)
- script: eng\common\cibuild.cmd
-configuration $(_BuildConfig)
-prepareMachine
$(_InternalBuildArgs)-
Transform your existing build-definition to a single stage. Do that by nesting the current job definition(s) under the
stageskeyword. For instance, this example build definition with a single job definition:jobs: - template: /eng/common/templates/jobs/jobs.yml parameters: enablePublishUsingPipelines: true ...
should be changed to:
stages: - stage: build displayName: Build jobs: - template: /eng/common/templates/jobs/jobs.yml parameters: enablePublishUsingPipelines: true ...
We suggest you to use the stage name build and have only one build stage. However, that's not a requirement. If you choose to use a different stage name or need to use multiple build stages you'll need to pass the name of the stage(s) to the
post-build.ymltemplate (see table on next section). -
Import the new
eng\common\templates\post-build\post-build.ymlArcade template at the end of the build definition. This will import all default test, validate and publishing stages provided by Arcade. The bottom part of your build definition will look like this:- ${{ if and(ne(variables['System.TeamProject'], 'public'), notin(variables['Build.Reason'], 'PullRequest')) }}: - template: eng\common\templates\post-build\post-build.yml parameters: enableSourceLinkValidation: false ...
The
post-build.ymltemplate accepts the following parameters:Name Type Description Default Value enableSourceLinkValidation bool Run SourceLink validation during the post-build stage. false enableSigningValidation bool Run signing validation during the post-build stage. true enableNugetValidation bool Run NuGet package validation tool during the post build stage. true symbolPublishingAdditionalParameters string Additional arguments for the PublishToSymbolServers sdk task. '' artifactsPublishingAdditionalParameters string Additional arguments for the PublishArtifactsInManifest sdk task. '' signingValidationAdditionalParameters string Additional arguments for the SigningValidation sdk task. '' publishInstallersAndChecksums bool Publish installers packages and checksums from the build artifacts to the dotnetcli storage account. Documentation for opting in to automatic checksum generation can be found in the Checksum section of this document. false SDLValidationParameters object Parameters for the SDL job template, as documented in the SDL template documentation -- validateDependsOn [array] Which stage(s) should the validation stage depend on. build publishDependsOn [array] Which stage(s) should the publishing stage(s) depend on. Validate After these changes the build job(s) will publish the build assets to Azure DevOps build artifacts instead of immediately publishing them to a feed or storage location. Once the post-build template is added, a repo's official build will include a series of stages that will publish the assets to different locations, depending on the Maestro++ default channel(s) that the build is assigned to.
Examples of the use of the basic onboarding scenario can be found in the following repos:
The pipeline for a build with stages enabled will look like the one shown below. In this example the build was assigned to the .Net Core 5 Dev channel but not to .Net Eng - Latest one.
NOTE: You need to have the AzDO Multi-stage pipelines preview feature enabled to see an UI like the one below. Take a look here to see how to enable preview features in Azure DevOps.
Since the post-build stages will only trigger during builds that run in the internal project (i.e., they won't show up on public builds), there are some additional steps that need to be performed in order to test that the changes to the pipeline are correct, and that publishing works as expected.
-
Create a branch on the Azure DevOps internal mirror of the repo that includes the pipeline changes.
-
Set up the "General Testing Channel" as a default channel for the internal repo + branch combination using Darc.
darc add-default-channel --channel "General Testing" --branch "<my_new_branch>" --repo "https://dev.azure.com/dnceng/internal/_git/<repo_name>"
-
Queue a build for your test branch
-
Once the Build and Validate stages complete, the General Testing stage should execute and publish the packages to the feed during the
Publish Assetsjob.
Arcade also includes support for automatically generating checksum files. To opt in to this feature, in each project that generates an asset for which you want to generate a checksum, add an Item named GenerateChecksumItems to the project file, which includes the output path of the original asset, and a metadata element named DestinationPath which represents the desired output path of the checksum file.
Example:
```XML
<ItemGroup>
<GenerateChecksumItems Include="@(OutputFile)">
<DestinationPath>%(FullPath).Sha512</OutputPath>
</GenerateChecksumItems>
</ItemGroup>
```
You will also need to pass publishInstallersAndChecksums=true to the post-build.yml template.
Repositories that make direct use of tasks in Tasks.Feed to publish assets during their build jobs should move away from doing so, as they would likely end up publishing to incorrect feeds for servicing builds.
However, if for some reason the infra in the default publishing stages don't meet you requirements you can create additional stages and make them dependent on the default ones. That way, it will at least be clear that the build does custom operations.
Note: We strongly suggest that you discuss with the .Net Engineering team the intended use case for this before starting your work. We might be able to give other options.
If you use the legacy PushToBlobFeed task from the Microsoft.DotNet.Build.Tasks.Feed package, you should change your code to use a new task called PushToAzureDevOpsArtifacts. This new task is also in the Tasks.Feed package and should act as a drop-in replacement for the previous one.
PushToAzureDevOpsArtifacts generates an appropriately populated Build Asset Manifest and registers the build assets as Azure DevOps build artifacts. This will guarantee that the default publishing stages will be able to access the build assets.
A conversion to PushToAzureDevOpsArtifacts for repos that are using the PushToBlobFeed task inside their build would look like this:
-
Replace the
PushToBlobFeedtask withPushToAzureDevOpsArtifacts:<PropertyGroup> <AssetManifestFileName>ManifestFileName.xml</AssetManifestFileName> <AssetManifestPath>$(ArtifactsLogDir)AssetManifest\$(AssetManifestFileName)</AssetManifestPath> </PropertyGroup> <PushToBlobFeed ExpectedFeedUrl="$(FeedURL)" AccountKey="$(FeedKey)" ItemsToPush="@(ItemsToPush)" ManifestBuildData="Location=$(FeedURL)" ManifestRepoUri="$(BUILD_REPOSITORY_URI)" ManifestBranch="$(BUILD_SOURCEBRANCH)" ManifestBuildId="$(BUILD_BUILDNUMBER)" ManifestCommit="$(BUILD_SOURCEVERSION)" AssetManifestPath="$(AssetManifestPath)" PublishFlatContainer="$(PublishFlatContainer)" />
becomes
<PropertyGroup> <AssetManifestFileName>ManifestFileName</AssetManifestFileName> <AssetManifestPath>$(ArtifactsLogDir)AssetManifest\$(SdkAssetManifestFileName)</AssetManifestPath> <!-- Create a temporary directory to store the generated asset manifest by the task --> <TempWorkingDirectory>$(ArtifactsDir)\..\AssetsTmpDir\$([System.Guid]::NewGuid())</TempWorkingDirectory> </PropertyGroup> <MakeDir Directories="$(TempWorkingDirectory)"/> <!-- Generate the asset manifest using the PushToAzureDevOpsArtifacts task --> <PushToAzureDevOpsArtifacts ItemsToPush="@(ItemsToPush)" ManifestBuildData="Location=$(FeedURL)" ManifestRepoUri="$(BUILD_REPOSITORY_URI)" ManifestBranch="$(BUILD_SOURCEBRANCH)" ManifestBuildId="$(BUILD_BUILDNUMBER)" ManifestCommit="$(BUILD_SOURCEVERSION)" PublishFlatContainer="$(PublishFlatContainer)" AssetManifestPath="$(AssetManifestPath)" AssetsTemporaryDirectory="$(TempWorkingDirectory)" /> <!-- Copy the generated manifest to the build's artifacts --> <Copy SourceFiles="$(AssetManifestPath)" DestinationFolder="$(TempWorkingDirectory)\$(AssetManifestFileName)" /> <Message Text="##vso[artifact.upload containerfolder=AssetManifests;artifactname=AssetManifests]$(TempWorkingDirectory)/$(AssetManifestFileName)" Importance="high" />
This will do something similar to what the SDK does for its default publishing pipeline, as seen in publish.proj.
Note: the usage of a temporary directory for placing the assets while uploading them is needed to guarantee that nothing interferes with the upload since it occurs asynchronously. See this issue for context.
Starting with Arcade SDK version 5.0.0-beta.20120.2 there is not support anymore for the old publishing infrastructure where the Arcade SDK handled publishing of all assets during the build stage. That means, that if:
-
The build definition sets
/p:DotNetPublishusingPipelines=true: Arcade will handle the control of assets publishing to Maestro++ and also that the build definition doesn't need to inform any of the following properties to the build scripts CIBuild.cmd/sh :Property DotNetPublishBlobFeedKey DotNetPublishBlobFeedUrl DotNetPublishToBlobFeed DotNetSymbolServerTokenMsdl DotNetSymbolServerTokenSymWeb -
The build definition doesn't set
/p:DotNetPublishingUsingPipelinesor set it to false: only symbols will be published and they will be controlled by the Arcade SDK. The build definition still needs to inform theDotNetSymbolServerToken[Msdl/SymWeb]properties, but the following properties aren't required anymore:Property DotNetPublishBlobFeedKey DotNetPublishBlobFeedUrl DotNetPublishToBlobFeed
Furthermore, starting with Arcade SDK version 5.0.0-beta.20120.2 the default value for the DotNetArtifactsCategory property is .NETCore, therefore you don't need to set that property anymore if you were setting it to .NETCore.
-
Controlled by Maestro++ Channels: The locations where packages are published to are determined based on which Maestro++ channel the build is assigned to. Look here for more info about channels.
-
Publishing is decoupled from the build job: Publishing is managed by the Arcade SDK entirely and assets are not published to any external storage during the build job. They are instead registered as Azure DevOps artifacts and only published to external locations after the build job finishes.
-
Single view for building and publishing: The new infrastructure doesn't use Release Pipelines - the previous one did. Instead, the concept of Stages is used. See below section about stages.
Stages are a concept introduced by Azure DevOps to organize the jobs in a pipeline. Just as Jobs are a collection of Steps, Stages can be thought of as a collection of Jobs, where for example, the same pipeline can be split into Build, Test and, Publishing stages.
Stages are the way that Azure DevOps is bringing build and release pipelines together, and are going to be the replacement for the RM UI based release pipelines. The official documentation for YAML-based Stages can be found here.
Using stages for publishing seeks to unify the Arcade SDK build artifact publishing mechanisms into a single solution that brings together the advantages of both the in-build synchronous publishing and the previous release pipeline based asynchronous publishing approaches. Other benefits are:
- Clearly separate the concepts of build, test, publish and validate.
- Support publishing and validation errors to be reported in the build page UI.
- Stages can depend on each other, which provides a natural way to extend default Arcade publishing infra with custom (repo or branch specific) publishing steps.
Each Maestro++ channel is configured (currently via YAML) to use three Azure DevOps feeds:
- A transport feed: used for publishing packages intended for use internally in the .Net Core stack.
- A shipping feed: used for publishing packages that will be directly used by end users.
- A symbols feed: symbol packages (
.symbols.nupkg) are published to this feed as well as to symbol servers.
The target feed will be public/private depending on whether the Maestro++ channel is public/private. For public channels the packages/blobs are also published to the legacy dotnetfeed/dotnet-core feed - You can override this and publish to a another custom feed, see description in a further section.
Each stable builds (i.e., Release Official Builds) publish to a different set of target feeds. This is because these builds always produce the same package version and overriding packages in the feeds is usually something not supported. Whenever a branch receive a PR from Maestro++, that contains packages that were published to a dynamically created feed, it will add the new feed to the repository root NuGet.Config file as a package source feed. Note that Maestro++ currently doesn't update NuGet.Config with the static feeds.
There are a few benefits, but the bottom line is: you can rely on Arcade SDK and Maestro++ to determine the correct place to publish the build assets. This is specially important for servicing and/or private builds where assets must not go to public locations before further validations. The new infrastructure also performs Signing validation, SDL validation and NuGet packages metadata validation.
This happens because the publishing job will only execute in stage(s) representing a channel(s) that is configured as a Default Channel for the build in Maestro++. All other stages will only execute the Setup Maestro Vars job.
Each stage represents a different Maestro++ channel. Therefore, as the number of channels in Maestro increase the number of stages also increase. We are considering to change this representation so that it doesn't clutter the build UI.
Currently Azure DevOps does not support communicating "YAML variables" across stages. The recommended workaround to do this is to use an AzDO artifact to persist the variables. The Setup Maestro Vars job is used to read one of such artifacts and set stage-scope variables based on the file content.
Symbol publishing to MSDL and SymWeb will be done as a regular part of publishing the build assets. The symbol packages (i.e., symbols.nupkg files) are also published to a feed as a form of backup.
Yes, that's possible. You need to use Darc to do that.
Most frequent cause of this is that there is no Default Channel configured for the build. Take a look here to see how to check that.
The DotNetPublishUsingPipelines is a flag that Arcade SDK uses to determine if the repo wants Maestro++ to control all aspects of publishing. If that parameter is not set (not advisable) Arcade SDK will only publish symbols produced by the build; publishing of other assets should be taken care of by the repo build definition.
- PackageArtifacts: contains all NuGet (.nupkg) packages to be published.
- BlobArtifacts: contains all blob artifacts (usually .symbols.nupkg) to be published.
- PdbArtifacts: contains all PDB artifacts to be published to symbol servers - SymWeb & MSDL.
- ReleaseConfigs: contains configuration files used by the post-build stages. In particular it should contain a file called
ReleaseConfigs.txtthat stores the BAR BuildId, the list of default channels IDs of the build and whether the current build is stable or not, respectively.
Note: only packages and blobs described in at least one build manifest will be published.
The publishing logs are stored inside an Azure DevOps artifacts container named PostBuildLogs. Each activated post-build channel/stage will have a subfolder under PostBuildLogs. Each job in a publishing channel/stage will have .binlogs in the container.