Develop a lazy reposync service

[agraul]

Project Title:

Develop a lazy reposync service

Description:

Uyuni is an open source systems management solution that can be used to manage various Linux distributions using a powerful web UI and API.

Uyuni uses a component, reposync, to mirror linux package repositories for deb and rpm packages. reposync in its current form first downloads all packages, stores them in the filesystem, extracts metadata (e.g. name, version, dependencies, etc.) from packages, and then populates the database with the extracted data.

This project idea is to create a new, lazy, reposync. Instead of downloading all packages first and using them as the metadata-source, the new reposync uses the repository metadata (Packages for deb, primary.xml for rpm) to populate the database. At a later time, the actual mirroring/downloading of the packages takes place.

Deliverable:

There are two parts to this project. It would also be possible to only do the first part for now.

1. A new component that reads repository metadata, parses it, and populates the Uyuni database
2. A daemon that downloads packages and stores them in the filesystem

Mentor:

@agraul, @cbosdo

Skills:

• Good Python or Go knowledge, the tool should be implemented in either language
• Basic SQL knowledge, Uyuni uses a postgresql database to store its data
• Basic knowledge of containerized applications would be nice, but is not required

Skill Level:

• Medium/Hard

Project Size:

• Only part 1: Medium Sized Project (175 hours)
• Full: Large Sized Project (350 hours)

Get started:

• Join GitHub Discussions and chatrooms: uyuni-project/devel uyuni-project/users
• Get familiar with uyuni, especially with the current reposync
• Set up a development Uyuni server VM, e.g. with sumaform

[rxbryan]

Hi @agraul,

I'm Bryan Elee. I believe you're mentoring this project.

I would like to know if there are any tests or issues that need to be addressed before someone can start working on this project, aside from the "Get Started" instructions.

[cbosdo]

I would like to know if there are any tests or issues that need to be addressed before someone can start working on this project, aside from the "Get Started" instructions.

Getting a development Uyuni server is a must as you will need it later to test your work and experiment. Other than that we would like to see some code of yours with the application. This can be a fix for a simple issue like those listed in https://github.com/uyuni-project/uyuni/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22 but if you feel like getting started with a prototype for your project that is fine too.

[cbosdo]

Note that we are moving the server to a big container: I just documented how to setup sumaform for this in uyuni-project/sumaform#1518 This is the recommended way to go.

[waterflow80]

@cbosdo,
Seems like the link for creating the default storage pool is no longer valid.

I've tried to create a storage pool this way:

$ mkdir -p /uyuni/pools/default
$ virsh pool-define-as --name default --type dir --target /uyuni/pools/default
$ virsh pool-autostart default
$ virsh pool-start default
$ virsh pool-list

However, when I run terraform apply (after init and validate), I still get this error message:

I assume there's another way of defining the storage pool for libvirt and uyuni.
Operating system: openSUSE Tumbleweed

[cbosdo]

@cbosdo, Seems like the link for creating the default storage pool is no longer valid.

Nice catch. I just updated this link and the one for the network.

I've tried to create a storage pool this way:

$ mkdir -p /uyuni/pools/default
$ virsh pool-define-as --name default --type dir --target /uyuni/pools/default
$ virsh pool-autostart default
$ virsh pool-start default
$ virsh pool-list

This looks good to me.

However, when I run terraform apply (after init and validate), I still get this error message:

I wonder if that could be an issue with the libvirt connection.
I am exporting LIBVIRT_DEFAULT_URI=qemu:///system in my shell configuration and added my user to libvirt and kvm groups. This way all the virsh commands I run with my user are targeting the system instance and not the user one.

You can verify this by running virsh -c qemu:///system pool-list. If the pool doesn't show up in that output then you only created it for your user libvirtd connection and you need to run the commands again with -c qemu:///system parameter.

[waterflow80]

Thanks @cbosdo, that solved the error, I appreciate it.

But now I got another error, after running terraform apply again, I kept getting these logs:

module.base.module.base_backend.libvirt_volume.volumes["opensuse155o"]: Creating...
module.base.module.base_backend.libvirt_volume.volumes["opensuse155o"]: Still creating... [10s elapsed]
module.base.module.base_backend.libvirt_volume.volumes["opensuse155o"]: Still creating... [20s elapsed]
module.base.module.base_backend.libvirt_volume.volumes["opensuse155o"]: Still creating... [30s elapsed]
...

and then received this error message:

Error: Error while uploading source https://download.opensuse.org/distribution/leap/15.5/appliances/openSUSE-Leap-15.5-Minimal-VM.x86_64-Cloud.qcow2: Error while copying source to volume stream error: stream ID 5; PROTOCOL_ERROR; received from peer
     with module.base.module.base_backend.libvirt_volume.volumes["opensuse155o"],
     on backend_modules/libvirt/base/main.tf line 57, in resource "libvirt_volume" "volumes":
     57: resource "libvirt_volume" "volumes" {

I appreciate your help.

[cbosdo]

But now I got another error, after running terraform apply again, I kept getting these logs:

module.base.module.base_backend.libvirt_volume.volumes["opensuse155o"]: Creating...
module.base.module.base_backend.libvirt_volume.volumes["opensuse155o"]: Still creating... [10s elapsed]
module.base.module.base_backend.libvirt_volume.volumes["opensuse155o"]: Still creating... [20s elapsed]
module.base.module.base_backend.libvirt_volume.volumes["opensuse155o"]: Still creating... [30s elapsed]
...

These are indicating the image is being downloaded, so they are good.

and then received this error message:

Error: Error while uploading source https://download.opensuse.org/distribution/leap/15.5/appliances/openSUSE-Leap-15.5-Minimal-VM.x86_64-Cloud.qcow2: Error while copying source to volume stream error: stream ID 5; PROTOCOL_ERROR; received from peer
     with module.base.module.base_backend.libvirt_volume.volumes["opensuse155o"],
     on backend_modules/libvirt/base/main.tf line 57, in resource "libvirt_volume" "volumes":
     57: resource "libvirt_volume" "volumes" {

Could that be a permission issue on the pool folder? Here are the permissions I have on my pool:

  <target>
    <path>/public/vms</path>
    <permissions>
      <mode>0755</mode>
      <owner>1000</owner>
      <group>100</group>
    </permissions>
  </target>

Debugging this would probably be easier on gitter.

[johnathankahn]

Hey @agraul @cbosdo
what will be the selection criteria ? what weightage will contributions hold ?

[cbosdo]

Hey @agraul @cbosdo what will be the selection criteria ? what weightage will contributions hold ?

This is more a gut feeling or rule of thumb than anything quantifiable. A solid and researched application is always the key. But the quality of the contributions and the communications surrounding them will be modifiers for us. No application is a no-go since it breaks rules, and we clearly prefer one nice contribution with good team work rather than a dozen of poor quality ones.