:: Re: [DNG] Running libvirt fails und…
Góra strony
Delete this message
Reply to this message
Autor: wirelessduck
Data:  
Dla: dng
Temat: Re: [DNG] Running libvirt fails under runit-init
Hi,

On Thu, 21 Sept 2023 at 03:56, Bob Proulx via Dng <dng@???> wrote:
>
> wirelessduck--- wrote:
> > I'm having trouble running virsh under runit-init.
> >
> > It was working previously under sysvinit-core, but after I install
> > runit-init and reboot I can no longer access the hypervisor.
>
> I don't know much detail but I know how I would start to debug this.
> On a sysvinit booted system I would look for running processes related
> to libvirt.
>
>     root@calamity:~# ps -efH | grep libvirt
>     root      1657     1  0 Sep09 ?        00:05:18   /usr/sbin/libvirtd -d
>     libvirt+  3651     1 38 Sep09 ?        4-01:18:14   qemu-system-x86_64 -enable-kvm -name guest=...

>
> On a working system "libvirtd -d" is running in deamon mode.


On my working system with sysvinit-core installed, I get slightly
different with no qemu process but it still works and I can create/run
VM in virt-manager and run virsh without error.
I also get the same output when I have runit-init installed.

# ps -efH | grep libvirt
root      1823     1  0 Sep25 ?        00:00:01   /usr/sbin/libvirtd -d
nobody    2049     1  0 Sep25 ?        00:00:00   /usr/sbin/dnsmasq
--conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro
--dhcp-script=/usr/lib/libvirt/libvirt_leaseshelper
root      2050  2049  0 Sep25 ?        00:00:00     /usr/sbin/dnsmasq
--conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro
--dhcp-script=/usr/lib/libvirt/libvirt_leaseshelper



>     root@calamity:~# lsof -p 1657 | grep -i -e sock -e stream
>     libvirtd 1657 root  mem       REG               8,18    97464  135073016 /usr/lib/x86_64-linux-gnu/libboost_iostreams.so.1.62.0
>     libvirtd 1657 root   11u     unix 0xffff922018c85400      0t0      18513 /var/run/libvirt/libvirt-sock type=STREAM
>     libvirtd 1657 root   12u     unix 0xffff922017582400      0t0      18515 /var/run/libvirt/libvirt-sock-ro type=STREAM
>     libvirtd 1657 root   13u     unix 0xffff922015ee6400      0t0      18517 /var/run/libvirt/libvirt-admin-sock type=STREAM
>     libvirtd 1657 root   15u     unix 0xffff922018e0f000      0t0      18550 type=STREAM
>     libvirtd 1657 root   16u     sock                0,8      0t0      16620 protocol: TCP
>     libvirtd 1657 root   22u     unix 0xffff921fcfd63c00      0t0      29737 /var/run/libvirt/libvirt-sock type=STREAM
>     libvirtd 1657 root   26u     unix 0xffff922017582800      0t0      30981 type=STREAM


Once again, slightly different under sysvinit-core for me:

# lsof -p 1823 | grep -i -e sock -e stream
lsof: WARNING: can't stat() fuse.portal file system /run/user/1001/doc
      Output information may be incomplete.
libvirtd 1823 root    9u     sock                0,8      0t0
23356 protocol: NETLINK
libvirtd 1823 root   10u     unix 0x00000000430f3988      0t0
23357 /run/libvirt/libvirt-sock type=STREAM (LISTEN)
libvirtd 1823 root   11u     unix 0x0000000028d785f9      0t0
23358 /run/libvirt/libvirt-sock-ro type=STREAM (LISTEN)
libvirtd 1823 root   12u     unix 0x000000000d0f7fb5      0t0
23359 /run/libvirt/libvirt-admin-sock type=STREAM (LISTEN)
libvirtd 1823 root   15u     unix 0x000000008f5b86d7      0t0
23368 type=STREAM (CONNECTED)


When I check again under runit-init, I am missing the last output line:

# lsof -p 1856 | grep -i -e sock -e stream
lsof: WARNING: can't stat() fuse.portal file system /run/user/1001/doc
      Output information may be incomplete.
libvirtd 1856 root    9u     sock                0,8      0t0
14760 protocol: NETLINK
libvirtd 1856 root   10u     unix 0x0000000021f3f3ac      0t0
14761 /run/libvirt/libvirt-sock type=STREAM (LISTEN)
libvirtd 1856 root   11u     unix 0x00000000460ca424      0t0
14762 /run/libvirt/libvirt-sock-ro type=STREAM (LISTEN)
libvirtd 1856 root   12u     unix 0x000000004055b5e6      0t0
14763 /run/libvirt/libvirt-admin-sock type=STREAM (LISTEN)


Perhaps this is telling me that something socket-related didn't
connect/activate properly under runit-init? But I have no problems
running `virsh list` as root under runit-init. The hypervisor connect
error is only appearing when running as my local user. Hmmm....

>
>     root@calamity:~# strace -v virsh list 2>&1 | grep sock
>     socket(AF_UNIX, SOCK_STREAM, 0)         = 5
>     connect(5, {sa_family=AF_UNIX, sun_path="/var/run/libvirt/libvirt-sock"}, 110) = 0
>     getsockname(5, {sa_family=AF_UNIX}, [128->2]) = 0


Running as root under runit-init:

# strace -v virsh list 2>&1 | grep sock
access("/var/run/libvirt/libvirt-sock", F_OK) = 0
socket(AF_UNIX, SOCK_STREAM, 0)         = 5
connect(5, {sa_family=AF_UNIX,
sun_path="/var/run/libvirt/libvirt-sock"}, 110) = 0
getsockname(5, {sa_family=AF_UNIX}, [128 => 2]) = 0


Running as my local user, it seems to be accessing a different socket
that doesn't exist:

$ strace -v virsh list 2>&1 | grep sock
access("/var/run/libvirt/virtqemud-sock", F_OK) = -1 ENOENT (No such
file or directory)
access("/var/run/libvirt/libvirt-sock", F_OK) = 0
socket(AF_UNIX, SOCK_STREAM, 0)         = 5
connect(5, {sa_family=AF_UNIX,
sun_path="/var/run/libvirt/libvirt-sock"}, 110) = 0
getsockname(5, {sa_family=AF_UNIX}, [128 => 2]) = 0



>
>     root@calamity:~# ll /var/run/libvirt/libvirt-sock
>     srwxrwxrwx 1 root root 0 Sep  9 19:48 /var/run/libvirt/libvirt-sock


# ls -l /var/run/libvirt/libvirt*
srwx------ 1 root root 0 Sep 27 10:16 /var/run/libvirt/libvirt-admin-sock
srwxrwxrwx 1 root root 0 Sep 27 10:16 /var/run/libvirt/libvirt-sock
srwxrwxrwx 1 root root 0 Sep 27 10:16 /var/run/libvirt/libvirt-sock-ro

>
> It uses /var/run/libvirt/libvirt-sock for communications. That's the
> area where things are failing for you. The client virsh can't connect
> to the running libvirtd daemon.


>
>     root@calamity:~# dpkg -S /etc/init.d/libvirtd
>     libvirt-daemon-system: /etc/init.d/libvirtd

>
>     root@calamity:~# dpkg -L libvirt-daemon-system | grep /etc/ | grep /etc/init.d/
>     /etc/init.d/libvirt-guests
>     /etc/init.d/libvirtd
>     /etc/init.d/virtlogd

>
>     root@calamity:~# dpkg -L libvirt-daemon-system | grep /etc/ | less

>
> In the case of libvirtd AFAIK there is only sysvinit scripts. Which
> means runit will be running them in legacy compatibility mode. That
> seems to be the likely part where things are not working.


On daedalus, the init script was split into
libvirt-daemon-system-sysv. I also have libvirt-daemon-driver-qemu
installed.
Given that it works under root but not a regular user account, I'm
guessing there is some sort of permission problem somewhere.
The /etc/libvirt/*.conf files mention polkit so I'm not sure if that
is related here.

> I would boot runit and then look to see how things are different. Is
> the libvirtd running? Are there are any error messages at boot time
> around the running of those scripts. Then to further debug run those
> scripts manually. Trace them.
>
>     root@calamity:~# ll /etc/init.d/libvirtd
>     -rwxr-xr-x 1 root root 5600 Oct  1  2020 /etc/init.d/libvirtd

>
>     root@calamity:~# less /etc/init.d/libvirtd

>
>     root@calamity:~# service libvirtd status
>     Checking status of libvirt management daemon: libvirtd not running.

>
>     root@calamity:~# service libvirtd start

>
> One should normally use "service" to start and stop init scripts
> because it cleans the environment, changes directory, and so forth.
> All needed things. Of course runit also has a native interface. For
> more debugging I would run the script directly tracing it.
>
>     root@calamity:~# sh -x /etc/init.d/libvirtd

>
> That will produce a lot of shell tracing output. I can't really
> speculate more without more detail. Hopefully by looking at what it
> is doing the reason for it not doing it under runit will become
> apparent. It might be a problem with the required cgroups setup. It
> might be a different problem. Check the syslog to see what errors are
> logged there.


Thanks for that suggestion to run initscript with `sh -x`. It appears
to show an error mounting cgroups in the init script, but perhaps that
is ok because I already have cgroupfs-mount package installed for
docker and that is already mounting the cgroups.

Sorry for the jumbled responses, but I'm testing as I write the reply here...

Hmmmm.. I noticed the output line of:

+ check_mount_cgroup_options
+ [ ! = yes ]
+ return 1

which looked a bit suspicious. So I checked the init script and it is
reading $mount_cgroups from /etc/default/libvirtd. The value is
commented out by default so I uncommented and set:

mount_cgroups=no

because I am already mounting cgroups via the cgroupfs-mount package.

I then stopped/restarted the libvirtd service and got some better output.

+ check_mount_cgroup_options
+ [ ! no = yes ]
+ return 1

> Hope that helps!


Yes! It appears that last shell debugging of init script definitely helped.
After setting mount_cgroups=no in /etc/default/libvirtd and restarting
libvirtd service, I can now successfully run `virsh list` and
`virt-manager` from my regular user account!

>
> Bob


Thanks for your assistance on this weird problem!

Tom