vBMC container missing mount after initial cluster deployment

[abays]

Describe the bug
While I am able to use dev-scripts to initially deploy a cluster, an issue appears after deployment if I want to add more nodes (VMs) to the virtual cluster. I try to add a new vBMC endpoint for a new VM, and I get this error within the vBMC container:

Error: Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': No such file or directory

If I stop and remove the container, and then recreate it with an additional -v "/var/run/libvirt:/var/run/libvirt:Z" mount, the problem disappears.

To Reproduce

1. Install RHEL 8.1
2. Deploy a virtual cluster with dev-scripts
3. Once the cluster is up, create an additional libvirt VM
4. Try to add a vBMC endpoint for the new VM and see the aforementioned error

Expected/observed behavior
The action in step #4 above should successfully add a vBMC endpoint for the new VM.

[asalkeld]

/assign

[asalkeld]

Hi @abays
I am rather new to the team can you be more explicit with the steps 3 and 4 that you performed?
I can't find any docs in the repo that explain how to do that.

[hardys]

Note that a potentially simpler way to achieve this is to use NUM_EXTRA_WORKERS - I added some docs about that yesterday in #1114

It would be good to understand this issue though, as it should be possible to manually create additional workers (it's just not well documented/tested at this point).

[abays]

@hardys Ah, thanks for sharing the info!

I've just recently started using dev-scripts, having previously used other deployment tools. But anyhow, a little more detail on steps 3 and 4 for @asalkeld:

1. I create an additional libvirt VM like so:

virt-install --ram NNNNN --vcpus N --os-variant rhel8.0 --cpu host-passthrough \
        --disk size=NNN,pool=default,device=disk,bus=virtio,format=qcow2 --import --noautoconsole \
        --vnc --network=bridge:ostestpr,mac="52:54:00:82:68:60" \
        --network=bridge:ostestbm,mac="52:54:00:82:69:60" --name "ostest-storage-0" \
        --os-type=linux --events on_reboot=restart --boot hd,network --noreboot

virsh net-update ostestbm add ip-dhcp-host "<host mac='52:54:00:82:69:60' name='storage-0' ip='192.168.111.30' />" --live --config
virsh net-update ostestbm add dns-host "<host ip='192.168.111.30'><hostname>storage-0</hostname></host>" --live --config

2. I then add the vBMC like so:

vbmc add "ostest-storage-0" --address 192.168.111.1 --port "6260" --username XXXXX --password XXXXX
vbmc start "ostest-storage-0"

It's the vbmc commands that hit the error I reported in this issue. Hope this helps!