Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ubuntu.2004.ArmArch exists in different regions between HelixImages and HelixPRImages #4156

Open
3 tasks
chcosta opened this issue Sep 23, 2024 · 4 comments
Open
3 tasks
Labels
dotnet-helix-machines Ops - Service Maintenance Used to track issues related to maintaining the services .NET Eng Supports Proposed-for-DncDevOps Issues or epics which may represent operational tasks, for consideration

Comments

@chcosta
Copy link
Member

chcosta commented Sep 23, 2024

ubuntu.2004.armarch image is in westus2 in the 'HelixImages' Azure Compute Gallery, but in westus in 'HelixPRImages'. We likely got into this state because the compute hash, as it currently exists, skips a lot of deployment during staging because it only computes such a narrow set of definition values. ubuntu.2004.armarch needs to be in westus2 for both galleries. Currently, if you accidentally deploy ubuntu.2004.armarch during a staging ci job (by changing one of the deployment values defined in definitions/shared/linux.yaml which it uses for the hash), you'll encounter an error like this:

                     ##[error]D:\a\_work\1\s\DeployQueues.dll(,): error : Failed to delete existing VM in pr-ubuntu.2004.armarch.open-dev-chcosta-upgradepol-a-scaleset: "The gallery image /subscriptions/84a65c9a-787d-45da-b10a-3a1cefce8060/resourceGroups/HelixPRImages/providers/Microsoft.Compute/galleries/HelixPRImages/images/ubuntu.2004.armarch/versions/2024.0917.232437 is not available in westus2 region. Please contact image owner to replicate to this region, or change your requested region."
                     Status: 404
                     ErrorCode: GalleryImageNotFound

Release Note Category

  • Feature changes/additions
  • Bug fixes
  • Internal Infrastructure Improvements

Release Note Description

@ilyas1974 ilyas1974 added the Ops - Service Maintenance Used to track issues related to maintaining the services .NET Eng Supports label Sep 23, 2024
@dougbu
Copy link
Member

dougbu commented Oct 1, 2024

the Region: westus2 property in the ubuntu.2004.armarch definition YAML should control the deployment region regardless of the environment (PR, staging, prod). where is that being overridden for deployments from PR builds❓ that is, how does this image get created in westus at all❓

separately I agree including the region in the hash might be useful. I'm not sure that would actually move the image between regions as you expect however. is this 🤞

@ilyas1974 ilyas1974 added the Proposed-for-DncDevOps Issues or epics which may represent operational tasks, for consideration label Oct 17, 2024
@dougbu
Copy link
Member

dougbu commented Oct 18, 2024

is this definitely an Ops issue @chcosta and @ilyas1974❓ just wondering if it needs triage

@ilyas1974
Copy link
Contributor

I think we have two issues here. The first is to correct the issue where we have images in different regions (ops), the second is the prevention\mitigation of how this happened and how to prevent it from happening again. I think that separate issue is something that can be discussed in triage.

@dougbu
Copy link
Member

dougbu commented Oct 21, 2024

broke this into #4324 and #4325. marked second as Needs triage

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
dotnet-helix-machines Ops - Service Maintenance Used to track issues related to maintaining the services .NET Eng Supports Proposed-for-DncDevOps Issues or epics which may represent operational tasks, for consideration
Projects
None yet
Development

No branches or pull requests

3 participants