How can i download all the libraries stored in the PROXY?

[midestefanis]

I am already using the Proxy in my company, but I don't see much reduction in build times. I thought of using the proxy information as an index and downloading all the libraries on our CI/CD node. Is there an easy way to download all the libraries from Athens?

[midestefanis]

I'm considering setting up a process that examines the /catalog, creates a go.mod file with the relevant information, and then executes a go get ./.. command. My understanding is that if I use go mod download, it will fetch only the latest version of a library.

[matt0x6F]

Hey @midestefanis! Athens by default stores files on disk, so you could theoretically rsync them down or something. You could also loop through the catalog and just download all the files periodically: https://gomods.io/intro/protocol/

That said, are you sure that your CI pipelines are talking to Athens? You should see 200's in Athens corresponding to the packages being requested. If Athens is misconfigured to fetch from upstream each time then you won't see any improvement in download times.

[midestefanis]

Hi @matt0x6F, thanks for the response. I'm using an S3 bucket as storage. I haven't tested the times using the disk. Does the speed improve a lot?

Regarding the other topic, yes, I have created a solution that loops the catalog and downloads the libraries. The problem is that I didn't find a way to pre-compile the libraries because of problems like this:

1. cmd/go: precompile dependencies for docker image cache golang/go#45474
2. cmd/go: unclear how to cache transitive dependencies in a Docker image golang/go#27719