Tart Push to GitLab Registry Failing

[yangyi625]

• System Information

  • Tart version: 2.18.4
  • Tart version that works: 2.14.0
  • Operating System: macOS 14.5
  • Hardware: Apple M1
  • GitLab version: v16.11.10

• Problem Description

  • I'm encountering an issue when attempting to push a Tart VM image to my GitLab registry. The push operation consistently fails with Tart version 2.18.4, whereas it works fine with the older version 2.14.0.

[fkorotkov]

How do you authenticate with the registry?

[yangyi625]

I authenticate with the registry using GitLab CI's token. The specific login command I use is:

echo "$CI_JOB_TOKEN" | tart login $CI_REGISTRY --username $CI_REGISTRY_USER --password-stdin

I discovered that commenting out the following line in the tart project's source allowed me to successfully push to the gitlab registry:

tart/Sources/tart/OCI/Layerizer/DiskV2.swift

Line 33 in c0b2093

if try await !registry.blobExists(compressedDataDigest) {

This workaround resolved the issue I was experiencing. However, I haven't yet identified the specific cause of this issue.

[edigaryev]

I've managed to successfully push a cloned ghcr.io/cirruslabs/ubuntu@sha256:7809213cbd1847a066e7f2589862f417bec2bb277438154e78e65f3a8e6394ea VM image to GitLab Enterprise Edition 17.5.0-pre (the one that's currently runs on gitlab.com).

Could you please try pushing the same image to your GitLab server using your own credentials (and not a temporary CI ones)?

[tvtamas]

Hi,

I'm getting the same error as @yangyi625:

pushing config...
pushing disk... this will take a while...
21%
Error: cancelled
Attempting to re-try...
Error: cancelled
Attempting to re-try...
Error: cancelled
Attempting to re-try...
Error: AuthFailed(why: "received unexpected HTTP status code 400 while retrieving an authentication token", details: "Bad Request")

Using Tart 2.18.5, self-hosted GitLab v17.4.1 and GitLab's fork of the registry version: v4.9.0-gitlab.

The push is done by using a Personal Access Token still valid for a couple of months.

After retrying the command several tries the upload percentage does increase.

[tvtamas]

Downgrading to Tart version 2.16.0 fixes the push issue with GitLab.

To downgrade:

curl -O https://raw.githubusercontent.com/cirruslabs/homebrew-cli/b516c00303c4ad55f636ac3fdd689dd04ff5e46e/tart.rb
brew install --build-from-source tart.rb

[yangyi625]

Hi @edigaryev ,

Using a personal token works fine, but I still prefer to use a CI token.

I tried using Mitmproxy to see which part was causing the error. The results are as follows:

It looks more like it's getting a 401 error during upload. The response is as follows:

I can't quite understand why I'm encountering an authentication issue here

[hblockx]

try to use --concurrent 1. For us the same issue appeared and got fixed by the concurrent 1. Also I tried adding a "retry" for the blobtransmission and it also helped with pulling and pushing images. Maybe I should contribute.