Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

No progress information while copying images to ECR #6681

Open
hannes-ucsc opened this issue Nov 2, 2024 · 7 comments
Open

No progress information while copying images to ECR #6681

hannes-ucsc opened this issue Nov 2, 2024 · 7 comments
Assignees
@hannes-ucsc
Labels
needs design [process] Solution to issue has yet to be devised orange [process] Done by the Azul team spike:1 [process] Spike estimate of one point

Comments

@hannes-ucsc
Copy link
Member

hannes-ucsc commented Nov 2, 2024 •
edited
Loading

When @achave11-ucsc and I diagnosed a problem with deploying the shared component from his computer, we noticed that Skopeo was eerily quiet during the copy. It appeared to be hung or not make any significant progress. Either way, we need to ensure that it obvious to the user what progress Skopeo is making, or if it is making any at all.
@hannes-ucsc hannes-ucsc added the orange [process] Done by the Azul team label Nov 2, 2024
@hannes-ucsc
Copy link
Member Author

Spike to reproduce on a different computer.

@achave11-ucsc
Copy link
Member

@hannes-ucsc: "@dsotirho-ucsc confirms this is reproducible (Skopeo lacks progress info). @dsotirho-ucsc to investigate if Skopeo can be coaxed into providing progress information."

@achave11-ucsc achave11-ucsc added needs design [process] Solution to issue has yet to be devised spike:1 [process] Spike estimate of one point labels Nov 4, 2024
@dsotirho-ucsc
Copy link
Contributor

@dsotirho-ucsc to investigate if Skopeo can be coaxed into providing progress information.

This is a requested feature since May 2019

Please provide a basic progress / transfer speed output ("CI compatible") on copy
https://www.github.com/containers/skopeo/issues/658

@hannes-ucsc
Copy link
Member Author

Please consider --debug as a workaround.

@dsotirho-ucsc
Copy link
Contributor

dsotirho-ucsc commented Nov 5, 2024 •
edited
Loading

Without --debug

  • 26 lines of output (for null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w)
Terraform will perform the following actions:

  # null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w is tainted, so must be replaced
-/+ resource "null_resource" "image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w" {
      ~ id       = "6334824724549167320" -> (known after apply)
        # (1 unchanged attribute hidden)
    }

Plan: 1 to add, 0 to change, 1 to destroy.

Warnings:

- Argument is deprecated
  on shared.tf.json line 95 (and 3 more)

To see the full warning notes, run Terraform without -compact-warnings.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w: Destroying... [id=6334824724549167320]
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w: Destruction complete after 0s
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w: Creating...
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w: Provisioning with 'local-exec'...
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): Executing: ["/bin/sh" "-c" "python /Users/daniel/repo/azul1/scripts/manage_images.py --copy docker.io/bitnami/kibana:7.10.2"]
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): 2024-11-05 13:43:46,983   DEBUG MainThread __main__: Creating or opening semaphore '/manage_images.tempdev'
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): 2024-11-05 13:43:46,983    INFO MainThread __main__: Acquiring semaphore '/manage_images.tempdev'
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): 2024-11-05 13:43:46,983    INFO MainThread __main__: Acquired semaphore '/manage_images.tempdev'
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): 2024-11-05 13:43:46,983    INFO MainThread __main__: Copying image TagImageRef(registry='docker.io', username='bitnami', repository=('kibana',), tag='7.10.2') for platform Platform(os='linux', arch='amd64', variant=None)
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): Getting image source signatures
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): Copying blob sha256:b8fc884dd91a599ae724fda23284dd85f56909be1f8d5c165490014d05c4362d
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): Copying blob sha256:0e7355dd88d515484520f6eafe991cac9bf485b27294104b8b829314390b88c7
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): Copying blob sha256:6575deaf42b75e48d277f543ddf2ab60dd00068556fdf3420eb4f82d80ba9ac3
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): Copying blob sha256:459bfeb2c0756e6f3ab51be2c0dc307a5e9931c74cf094e9c04b43c831a87dce
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): Copying blob sha256:38b52af9d09f760abfd31dab3a82f1241ae47f3233889ecbc51717f863e91f47
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): Copying blob sha256:be4dd8fa80cd87920dbed821d9a1fd075569e9fa2f08d4d83b53d5fc50ee5e48
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): Copying blob sha256:80885efcfd989383e3a387b9ee19da4a55b58b1262a984b743419ecd27bb85ad
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): Copying blob sha256:a67866c074909a07ee3ed39d98d690c32e820a03af9909177bdeedf6b4280075
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): Copying blob sha256:eb8791bb028a2dadc5492d15620a1396d0f755fce5007664136e559efd1b6ff0
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): Copying blob sha256:cee68e0ab7a3c36e0e46a2f887feefaad3aff7618406a463ef97999072d53e12
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): Copying config sha256:5afa1090aae4d2df1754c9d071ccdd23c13cafe65ab1963d3223c8be8ee83629
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): Writing manifest to image destination
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): 2024-11-05 13:43:49,696    INFO MainThread botocore.credentials: Found credentials in shared credentials file: ~/.aws/credentials
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): 2024-11-05 13:43:49,735    INFO MainThread azul.deployment: Allocated new Boto3 client for 'ecr' with ID 4369692528
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): 2024-11-05 13:43:50,241    INFO MainThread __main__: Released semaphore '/manage_images.tempdev'
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w: Creation complete after 3s [id=5577006791947779410]

Apply complete! Resources: 1 added, 0 changed, 1 destroyed.

With --debug

  • 110 lines of output (for null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w)
Terraform will perform the following actions:

  # null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w is tainted, so must be replaced
-/+ resource "null_resource" "image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w" {
      ~ id       = "8674665223082153551" -> (known after apply)
        # (1 unchanged attribute hidden)
    }

Plan: 1 to add, 0 to change, 1 to destroy.

Warnings:

- Argument is deprecated
  on shared.tf.json line 95 (and 3 more)

To see the full warning notes, run Terraform without -compact-warnings.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w: Destroying... [id=8674665223082153551]
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w: Destruction complete after 0s
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w: Creating...
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w: Provisioning with 'local-exec'...
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): Executing: ["/bin/sh" "-c" "python /Users/daniel/repo/azul1/scripts/manage_images.py --copy docker.io/bitnami/kibana:7.10.2"]
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): 2024-11-05 13:47:23,786   DEBUG MainThread __main__: Creating or opening semaphore '/manage_images.tempdev'
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): 2024-11-05 13:47:23,786    INFO MainThread __main__: Acquiring semaphore '/manage_images.tempdev'
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): 2024-11-05 13:47:23,786    INFO MainThread __main__: Acquired semaphore '/manage_images.tempdev'
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): 2024-11-05 13:47:23,786    INFO MainThread __main__: Copying image TagImageRef(registry='docker.io', username='bitnami', repository=('kibana',), tag='7.10.2') for platform Platform(os='linux', arch='amd64', variant=None)
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:23-08:00" level=debug msg="Using registries.d directory /opt/homebrew/etc/containers/registries.d"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:23-08:00" level=debug msg="Loading registries configuration \"/opt/homebrew/etc/containers/registries.conf\""
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:23-08:00" level=debug msg="Found credentials for 654654270592.dkr.ecr.us-east-1.amazonaws.com/docker.io/bitnami/kibana in credential helper containers-auth.json in file /var/folders/d1/1lnk2yfj2tj93pyr9_0mx7w00000gn/T/tmpvlvr_37g"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:23-08:00" level=debug msg=" No signature storage configuration found for 654654270592.dkr.ecr.us-east-1.amazonaws.com/docker.io/bitnami/kibana:7.10.2, using built-in default file:///Users/daniel/.local/share/containers/sigstore"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:23-08:00" level=debug msg="Looking for TLS certificates and private keys in /etc/docker/certs.d/654654270592.dkr.ecr.us-east-1.amazonaws.com"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:23-08:00" level=debug msg="Using registries.d directory /opt/homebrew/etc/containers/registries.d"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:23-08:00" level=debug msg="Trying to access \"docker.io/bitnami/kibana@sha256:6e96a761f087cb0e3917998e6ed439ebc5a5bdb910f5477c1c39eb7bcbe54210\""
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:23-08:00" level=debug msg="Found credentials for docker.io/bitnami/kibana in credential helper containers-auth.json in file /var/folders/d1/1lnk2yfj2tj93pyr9_0mx7w00000gn/T/tmpvlvr_37g"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:23-08:00" level=debug msg=" No signature storage configuration found for docker.io/bitnami/kibana@sha256:6e96a761f087cb0e3917998e6ed439ebc5a5bdb910f5477c1c39eb7bcbe54210, using built-in default file:///Users/daniel/.local/share/containers/sigstore"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:23-08:00" level=debug msg="Looking for TLS certificates and private keys in /etc/docker/certs.d/docker.io"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:23-08:00" level=debug msg="GET https://registry-1.docker.io/v2/"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:24-08:00" level=debug msg="Ping https://registry-1.docker.io/v2/ status 401"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:24-08:00" level=debug msg="GET https://auth.docker.io/token?account=dsotirhoucsc&scope=repository%3Abitnami%2Fkibana%3Apull&service=registry.docker.io"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:24-08:00" level=debug msg="GET https://registry-1.docker.io/v2/bitnami/kibana/manifests/sha256:6e96a761f087cb0e3917998e6ed439ebc5a5bdb910f5477c1c39eb7bcbe54210"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:24-08:00" level=debug msg="Content-Type from manifest GET is \"application/vnd.docker.distribution.manifest.v2+json\""
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:24-08:00" level=debug msg="Using SQLite blob info cache at /Users/daniel/.local/share/containers/cache/blob-info-cache-v1.sqlite"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:24-08:00" level=debug msg="IsRunningImageAllowed for image docker:docker.io/bitnami/kibana@sha256:6e96a761f087cb0e3917998e6ed439ebc5a5bdb910f5477c1c39eb7bcbe54210"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:24-08:00" level=debug msg=" Using default policy section"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:24-08:00" level=debug msg=" Requirement 0: allowed"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:24-08:00" level=debug msg="Overall: allowed"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): Getting image source signatures
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:24-08:00" level=debug msg="Reading /Users/daniel/.local/share/containers/sigstore/bitnami/kibana@sha256=6e96a761f087cb0e3917998e6ed439ebc5a5bdb910f5477c1c39eb7bcbe54210/signature-1"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:24-08:00" level=debug msg="Not looking for sigstore attachments: disabled by configuration"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:24-08:00" level=debug msg="We can't modify the manifest, hoping for the best..."
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): Copying blob sha256:b8fc884dd91a599ae724fda23284dd85f56909be1f8d5c165490014d05c4362d
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): Copying blob sha256:0e7355dd88d515484520f6eafe991cac9bf485b27294104b8b829314390b88c7
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): Copying blob sha256:6575deaf42b75e48d277f543ddf2ab60dd00068556fdf3420eb4f82d80ba9ac3
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:24-08:00" level=debug msg="Checking if we can reuse blob sha256:6575deaf42b75e48d277f543ddf2ab60dd00068556fdf3420eb4f82d80ba9ac3: general substitution = false, compression for MIME type \"application/vnd.docker.image.rootfs.diff.tar.gzip\" = true"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): Copying blob sha256:459bfeb2c0756e6f3ab51be2c0dc307a5e9931c74cf094e9c04b43c831a87dce
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:24-08:00" level=debug msg="Checking if we can reuse blob sha256:0e7355dd88d515484520f6eafe991cac9bf485b27294104b8b829314390b88c7: general substitution = false, compression for MIME type \"application/vnd.docker.image.rootfs.diff.tar.gzip\" = true"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): Copying blob sha256:38b52af9d09f760abfd31dab3a82f1241ae47f3233889ecbc51717f863e91f47
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:24-08:00" level=debug msg="Checking if we can reuse blob sha256:b8fc884dd91a599ae724fda23284dd85f56909be1f8d5c165490014d05c4362d: general substitution = false, compression for MIME type \"application/vnd.docker.image.rootfs.diff.tar.gzip\" = true"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:24-08:00" level=debug msg="Checking /v2/docker.io/bitnami/kibana/blobs/sha256:b8fc884dd91a599ae724fda23284dd85f56909be1f8d5c165490014d05c4362d"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:24-08:00" level=debug msg="GET https://654654270592.dkr.ecr.us-east-1.amazonaws.com/v2/"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:24-08:00" level=debug msg="Checking /v2/docker.io/bitnami/kibana/blobs/sha256:6575deaf42b75e48d277f543ddf2ab60dd00068556fdf3420eb4f82d80ba9ac3"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): Copying blob sha256:be4dd8fa80cd87920dbed821d9a1fd075569e9fa2f08d4d83b53d5fc50ee5e48
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:24-08:00" level=debug msg="Checking if we can reuse blob sha256:be4dd8fa80cd87920dbed821d9a1fd075569e9fa2f08d4d83b53d5fc50ee5e48: general substitution = false, compression for MIME type \"application/vnd.docker.image.rootfs.diff.tar.gzip\" = true"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:24-08:00" level=debug msg="Checking /v2/docker.io/bitnami/kibana/blobs/sha256:be4dd8fa80cd87920dbed821d9a1fd075569e9fa2f08d4d83b53d5fc50ee5e48"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:24-08:00" level=debug msg="Checking if we can reuse blob sha256:459bfeb2c0756e6f3ab51be2c0dc307a5e9931c74cf094e9c04b43c831a87dce: general substitution = false, compression for MIME type \"application/vnd.docker.image.rootfs.diff.tar.gzip\" = true"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:24-08:00" level=debug msg="Checking /v2/docker.io/bitnami/kibana/blobs/sha256:459bfeb2c0756e6f3ab51be2c0dc307a5e9931c74cf094e9c04b43c831a87dce"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:24-08:00" level=debug msg="Checking if we can reuse blob sha256:38b52af9d09f760abfd31dab3a82f1241ae47f3233889ecbc51717f863e91f47: general substitution = false, compression for MIME type \"application/vnd.docker.image.rootfs.diff.tar.gzip\" = true"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:24-08:00" level=debug msg="Checking /v2/docker.io/bitnami/kibana/blobs/sha256:38b52af9d09f760abfd31dab3a82f1241ae47f3233889ecbc51717f863e91f47"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:24-08:00" level=debug msg="Checking /v2/docker.io/bitnami/kibana/blobs/sha256:0e7355dd88d515484520f6eafe991cac9bf485b27294104b8b829314390b88c7"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:25-08:00" level=debug msg="Ping https://654654270592.dkr.ecr.us-east-1.amazonaws.com/v2/ status 401"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:25-08:00" level=debug msg="HEAD https://654654270592.dkr.ecr.us-east-1.amazonaws.com/v2/docker.io/bitnami/kibana/blobs/sha256:b8fc884dd91a599ae724fda23284dd85f56909be1f8d5c165490014d05c4362d"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:25-08:00" level=debug msg="HEAD https://654654270592.dkr.ecr.us-east-1.amazonaws.com/v2/docker.io/bitnami/kibana/blobs/sha256:6575deaf42b75e48d277f543ddf2ab60dd00068556fdf3420eb4f82d80ba9ac3"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:25-08:00" level=debug msg="HEAD https://654654270592.dkr.ecr.us-east-1.amazonaws.com/v2/docker.io/bitnami/kibana/blobs/sha256:459bfeb2c0756e6f3ab51be2c0dc307a5e9931c74cf094e9c04b43c831a87dce"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:25-08:00" level=debug msg="HEAD https://654654270592.dkr.ecr.us-east-1.amazonaws.com/v2/docker.io/bitnami/kibana/blobs/sha256:be4dd8fa80cd87920dbed821d9a1fd075569e9fa2f08d4d83b53d5fc50ee5e48"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:25-08:00" level=debug msg="HEAD https://654654270592.dkr.ecr.us-east-1.amazonaws.com/v2/docker.io/bitnami/kibana/blobs/sha256:38b52af9d09f760abfd31dab3a82f1241ae47f3233889ecbc51717f863e91f47"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:25-08:00" level=debug msg="HEAD https://654654270592.dkr.ecr.us-east-1.amazonaws.com/v2/docker.io/bitnami/kibana/blobs/sha256:0e7355dd88d515484520f6eafe991cac9bf485b27294104b8b829314390b88c7"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:25-08:00" level=debug msg="... already exists"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:25-08:00" level=debug msg="Skipping blob sha256:459bfeb2c0756e6f3ab51be2c0dc307a5e9931c74cf094e9c04b43c831a87dce (already present):"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): Copying blob sha256:80885efcfd989383e3a387b9ee19da4a55b58b1262a984b743419ecd27bb85ad
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:25-08:00" level=debug msg="Checking if we can reuse blob sha256:80885efcfd989383e3a387b9ee19da4a55b58b1262a984b743419ecd27bb85ad: general substitution = false, compression for MIME type \"application/vnd.docker.image.rootfs.diff.tar.gzip\" = true"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:25-08:00" level=debug msg="Checking /v2/docker.io/bitnami/kibana/blobs/sha256:80885efcfd989383e3a387b9ee19da4a55b58b1262a984b743419ecd27bb85ad"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:25-08:00" level=debug msg="HEAD https://654654270592.dkr.ecr.us-east-1.amazonaws.com/v2/docker.io/bitnami/kibana/blobs/sha256:80885efcfd989383e3a387b9ee19da4a55b58b1262a984b743419ecd27bb85ad"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:25-08:00" level=debug msg="... already exists"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:25-08:00" level=debug msg="Skipping blob sha256:38b52af9d09f760abfd31dab3a82f1241ae47f3233889ecbc51717f863e91f47 (already present):"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): Copying blob sha256:a67866c074909a07ee3ed39d98d690c32e820a03af9909177bdeedf6b4280075
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:25-08:00" level=debug msg="Checking if we can reuse blob sha256:a67866c074909a07ee3ed39d98d690c32e820a03af9909177bdeedf6b4280075: general substitution = false, compression for MIME type \"application/vnd.docker.image.rootfs.diff.tar.gzip\" = true"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:25-08:00" level=debug msg="Checking /v2/docker.io/bitnami/kibana/blobs/sha256:a67866c074909a07ee3ed39d98d690c32e820a03af9909177bdeedf6b4280075"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:25-08:00" level=debug msg="HEAD https://654654270592.dkr.ecr.us-east-1.amazonaws.com/v2/docker.io/bitnami/kibana/blobs/sha256:a67866c074909a07ee3ed39d98d690c32e820a03af9909177bdeedf6b4280075"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:25-08:00" level=debug msg="... already exists"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:25-08:00" level=debug msg="... already exists"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:25-08:00" level=debug msg="Skipping blob sha256:b8fc884dd91a599ae724fda23284dd85f56909be1f8d5c165490014d05c4362d (already present):"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): Copying blob sha256:eb8791bb028a2dadc5492d15620a1396d0f755fce5007664136e559efd1b6ff0
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:25-08:00" level=debug msg="Checking if we can reuse blob sha256:eb8791bb028a2dadc5492d15620a1396d0f755fce5007664136e559efd1b6ff0: general substitution = false, compression for MIME type \"application/vnd.docker.image.rootfs.diff.tar.gzip\" = true"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:25-08:00" level=debug msg="Checking /v2/docker.io/bitnami/kibana/blobs/sha256:eb8791bb028a2dadc5492d15620a1396d0f755fce5007664136e559efd1b6ff0"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:25-08:00" level=debug msg="HEAD https://654654270592.dkr.ecr.us-east-1.amazonaws.com/v2/docker.io/bitnami/kibana/blobs/sha256:eb8791bb028a2dadc5492d15620a1396d0f755fce5007664136e559efd1b6ff0"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:25-08:00" level=debug msg="... already exists"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:25-08:00" level=debug msg="Skipping blob sha256:0e7355dd88d515484520f6eafe991cac9bf485b27294104b8b829314390b88c7 (already present):"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): Copying blob sha256:cee68e0ab7a3c36e0e46a2f887feefaad3aff7618406a463ef97999072d53e12
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:25-08:00" level=debug msg="Checking if we can reuse blob sha256:cee68e0ab7a3c36e0e46a2f887feefaad3aff7618406a463ef97999072d53e12: general substitution = false, compression for MIME type \"application/vnd.docker.image.rootfs.diff.tar.gzip\" = true"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:25-08:00" level=debug msg="Checking /v2/docker.io/bitnami/kibana/blobs/sha256:cee68e0ab7a3c36e0e46a2f887feefaad3aff7618406a463ef97999072d53e12"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:25-08:00" level=debug msg="HEAD https://654654270592.dkr.ecr.us-east-1.amazonaws.com/v2/docker.io/bitnami/kibana/blobs/sha256:cee68e0ab7a3c36e0e46a2f887feefaad3aff7618406a463ef97999072d53e12"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:25-08:00" level=debug msg="... already exists"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:25-08:00" level=debug msg="Skipping blob sha256:6575deaf42b75e48d277f543ddf2ab60dd00068556fdf3420eb4f82d80ba9ac3 (already present):"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:25-08:00" level=debug msg="Skipping blob sha256:be4dd8fa80cd87920dbed821d9a1fd075569e9fa2f08d4d83b53d5fc50ee5e48 (already present):"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:25-08:00" level=debug msg="... already exists"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:25-08:00" level=debug msg="... already exists"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:25-08:00" level=debug msg="Skipping blob sha256:80885efcfd989383e3a387b9ee19da4a55b58b1262a984b743419ecd27bb85ad (already present):"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:25-08:00" level=debug msg="... already exists"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:25-08:00" level=debug msg="Skipping blob sha256:a67866c074909a07ee3ed39d98d690c32e820a03af9909177bdeedf6b4280075 (already present):"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:25-08:00" level=debug msg="Skipping blob sha256:eb8791bb028a2dadc5492d15620a1396d0f755fce5007664136e559efd1b6ff0 (already present):"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:25-08:00" level=debug msg="... already exists"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:25-08:00" level=debug msg="Skipping blob sha256:cee68e0ab7a3c36e0e46a2f887feefaad3aff7618406a463ef97999072d53e12 (already present):"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): Copying config sha256:5afa1090aae4d2df1754c9d071ccdd23c13cafe65ab1963d3223c8be8ee83629
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:25-08:00" level=debug msg="Downloading /v2/bitnami/kibana/blobs/sha256:5afa1090aae4d2df1754c9d071ccdd23c13cafe65ab1963d3223c8be8ee83629"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:25-08:00" level=debug msg="GET https://registry-1.docker.io/v2/bitnami/kibana/blobs/sha256:5afa1090aae4d2df1754c9d071ccdd23c13cafe65ab1963d3223c8be8ee83629"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:26-08:00" level=debug msg="No compression detected"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:26-08:00" level=debug msg="Compression change for blob sha256:5afa1090aae4d2df1754c9d071ccdd23c13cafe65ab1963d3223c8be8ee83629 (\"application/vnd.docker.container.image.v1+json\") not supported"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:26-08:00" level=debug msg="Using original blob without modification"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:26-08:00" level=debug msg="Checking /v2/docker.io/bitnami/kibana/blobs/sha256:5afa1090aae4d2df1754c9d071ccdd23c13cafe65ab1963d3223c8be8ee83629"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:26-08:00" level=debug msg="HEAD https://654654270592.dkr.ecr.us-east-1.amazonaws.com/v2/docker.io/bitnami/kibana/blobs/sha256:5afa1090aae4d2df1754c9d071ccdd23c13cafe65ab1963d3223c8be8ee83629"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:26-08:00" level=debug msg="... already exists"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): Writing manifest to image destination
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:26-08:00" level=debug msg="PUT https://654654270592.dkr.ecr.us-east-1.amazonaws.com/v2/docker.io/bitnami/kibana/manifests/7.10.2"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): 2024-11-05 13:47:26,476    INFO MainThread botocore.credentials: Found credentials in shared credentials file: ~/.aws/credentials
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): 2024-11-05 13:47:26,514    INFO MainThread azul.deployment: Allocated new Boto3 client for 'ecr' with ID 4324740896
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): 2024-11-05 13:47:26,974    INFO MainThread __main__: Released semaphore '/manage_images.tempdev'
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w: Creation complete after 3s [id=5577006791947779410]

Apply complete! Resources: 1 added, 0 changed, 1 destroyed.

@achave11-ucsc
Copy link
Member

@hannes-ucsc: "It appears that skopeo detected the destination blobs to already exists, which raises doubts as to whether the deletion from ECR was effective. If it wasn't, the above terminal transcript might not correctly represent what debug output would look like when blobs are actually copied."

@dsotirho-ucsc
Copy link
Contributor

The --debug is not a suitable workaround for the lack of progress bars or status during image uploads. The status lines such as

null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): Copying blob sha256:6575deaf42b75e48d277f543ddf2ab60dd00068556fdf3420eb4f82d80ba9ac3

are printed the same and in the same frequency with or without --debug.

The additional debug lines that are printed mostly relate to internal processes between image uploads (e.g. connection setup, checking if the destination already exists).

There is a potentially useful debug line mentioning the completion of a blob's upload, however a single image can consist of many blobs (10 for the kibana image), and the increased number of log lines printed with debug enabled, 34 vs 242 for one (kibana) image, make finding all the blobs relating to one image not easy to obtain.

Example: logs relating to one blob's upload (note last line "Upload of layer ... complete")

null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): Copying blob sha256:459bfeb2c0756e6f3ab51be2c0dc307a5e9931c74cf094e9c04b43c831a87dce
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-06T16:25:08-08:00" level=debug msg="Checking if we can reuse blob sha256:459bfeb2c0756e6f3ab51be2c0dc307a5e9931c74cf094e9c04b43c831a87dce: general substitution = false, compression for MIME type \"application/vnd.docker.image.rootfs.diff.tar.gzip\" = true"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-06T16:25:08-08:00" level=debug msg="Checking /v2/docker.io/bitnami/kibana/blobs/sha256:459bfeb2c0756e6f3ab51be2c0dc307a5e9931c74cf094e9c04b43c831a87dce"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-06T16:25:09-08:00" level=debug msg="HEAD https://654654270592.dkr.ecr.us-east-1.amazonaws.com/v2/docker.io/bitnami/kibana/blobs/sha256:459bfeb2c0756e6f3ab51be2c0dc307a5e9931c74cf094e9c04b43c831a87dce"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-06T16:25:09-08:00" level=debug msg="Ignoring BlobInfoCache record of digest \"sha256:459bfeb2c0756e6f3ab51be2c0dc307a5e9931c74cf094e9c04b43c831a87dce\" with unknown compression"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-06T16:25:09-08:00" level=debug msg="Downloading /v2/bitnami/kibana/blobs/sha256:459bfeb2c0756e6f3ab51be2c0dc307a5e9931c74cf094e9c04b43c831a87dce"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-06T16:25:09-08:00" level=debug msg="GET https://registry-1.docker.io/v2/bitnami/kibana/blobs/sha256:459bfeb2c0756e6f3ab51be2c0dc307a5e9931c74cf094e9c04b43c831a87dce"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-06T16:25:09-08:00" level=debug msg="... not present"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-06T16:25:10-08:00" level=debug msg="Checking /v2/docker.io/bitnami/kibana/blobs/sha256:459bfeb2c0756e6f3ab51be2c0dc307a5e9931c74cf094e9c04b43c831a87dce"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-06T16:25:10-08:00" level=debug msg="HEAD https://654654270592.dkr.ecr.us-east-1.amazonaws.com/v2/docker.io/bitnami/kibana/blobs/sha256:459bfeb2c0756e6f3ab51be2c0dc307a5e9931c74cf094e9c04b43c831a87dce"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-06T16:25:17-08:00" level=debug msg="PUT https://654654270592.dkr.ecr.us-east-1.amazonaws.com/v2/docker.io/bitnami/kibana/blobs/uploads/57f73247-1f70-3cf5-abdc-72b09c4c3e93?digest=sha256%3A459bfeb2c0756e6f3ab51be2c0dc307a5e9931c74cf094e9c04b43c831a87dce"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-06T16:25:17-08:00" level=debug msg="Upload of layer sha256:459bfeb2c0756e6f3ab51be2c0dc307a5e9931c74cf094e9c04b43c831a87dce complete"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@hannes-ucsc hannes-ucsc

Labels
needs design [process] Solution to issue has yet to be devised orange [process] Done by the Azul team spike:1 [process] Spike estimate of one point
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@hannes-ucsc @achave11-ucsc @dsotirho-ucsc