DNS native host resolving no longer works with vz VMs since v1.0.0

[dhealio]

Description

When using the vz VM and not using user-v2 networking, the DNS behavior mentioned in the documentation is no longer working as advertised when hostResolver.enabled is true (or unassigned).

This prevents being able to use the native OS resolver, which is needed for VPN configurations, split-DNS setups, as well as mDNS, local /etc/hosts etc.

The breaking change seems to be here, which prevents args.UDPDNSLocalPort etc. from getting assigned.

[dhealio]

To be more specific, DNS resolving of VPN hostnames on macOS was working on v0.23.2 when starting lima with the default configuration/parameters, but no longer works with v1.0.0 or v1.0.1.

FWIW, I have also tried starting lima with user-v2 networking via limactl start --network=lima:user-v2, but I am still unable to resolve VPN hostnames within lima. This seems to contradict the behavior mentioned here, but perhaps I am doing something wrong.

If this is by design, then how would I go about configuring lima to allow DNS resolving of VPN hostnames? The documentation should be updated to reflect any changes of behavior in this regard.

Otherwise, this is a breaking change that I'm hoping can be addressed soon.

[nirs]

Based on the comment this a regression.

[balajiv113]

@dhealio
The reason for that condition is, vz uses gvisor implementation for network. For that dns resolution is taken care by gvisor-tap-vsock. Which will take care of reading /etc/hosts to populate nameservers

The change in behaviour is due to gvisor-tap-vsock breaking change. This is fixed later. So after upgrade gvisor-tap-vsock it should work fine. Will check with gvisor-tap-vsock for a release
containers/gvisor-tap-vsock#398

[dhealio]

Thank you for the quick response!

It's low priority, but the network documentation would ideally be updated to reflect the changes in behavior mentioned by @balajiv113. This may save others from wasting time troubleshooting configuration issues etc.

[dhealio]

I have updated to v1.0.2, and unfortunately I am still seeing the issue (with and without user-v2 networking).

In case it helps, here are some details on my host system:

• macOS 15.1.1
• OpenVPN connect client v3.4.9
• The /etc/resolv.conf file on my host system does not contain any nameserver entries
• I see a VPN specific resolver when running scutil --dns
• I am testing with ping, e.g. lima ping host.vpn-domain.com; this returns "Name or service not known"

[subpop]

Using v1.0.2 as well. When I start up my default guest, this gets printed:

INFO[0010] [hostagent] Not forwarding TCP 127.0.0.53:53

That's the address that ends up in /etc/resolv.conf in the guest. Is that relevant?

[jandubois]

127.0.0.53:53 is the address for systemd-resolved, which will be running inside the VM.

You can run resolvectl status inside the VM to see what it is using to resolve names.

[nirs]

Should we reopen this issue or open a new one?

[subpop]

I've been experimenting with a guest and my hostagent resolver. When the hostagent starts, it successfully starts up the DNS resolver and listens on a random port (in this example, 50563). From my Mac, querying the resolver directly works for both VPN and non-VPN hosts:

dig @127.0.0.1 -p 50563 google.com

I can see the handleQuery received DNS query log entry when this happens. From within a guest, I can also succeed in querying the resolver if I dig at that port.

I modified the cloud-init generated systemd.network file (/run/systemd/network/10-netplan-eth0.network) and set the DNS= entry to 192.168.5.2:50563 (instead of just the IP address) and restarted systemd-networkd. After that, resolvectl query will successfully resolve VPN hosts and non-VPN hosts, and I can see handleQuery log entries.

So it appears that the hostagent resolver is working, but something is misconfigured with the ports so that the resolver is not listening on a port that the guest is querying.

[dhealio]

@subpop From my understanding of the comment made by @balajiv113, the functionality that you're referring to is intentionally no longer wired up, with the gvisor-tap-vsock DNS implementation taking over instead.

It seems as though that the gvisor-tap-vsock DNS implementation is still lacking functionality as compared to the hostagent-based resolver. It doesn't appear to be taking into account all the resolvers that are available on the host system. It's also possible that it's not respecting domain-specific resolver rules either.

If an upstream fix is required, it may take some time. If this is the case, I'm hoping that lima offers an alternative solution, such as adding a config option that wires up the original hostagent-resolver, or at least rolls back the logic that prevents using it.

[subpop]

I see. That makes sense in context now. Thank you.

+1 to a template option to use the hostagent resolver. That would be very useful.

[subpop]

I don't know if it's a true workaround, but I was able to get my VPN hostnames resolving in a guest by creating a guest using:

networks:
- vzNAT: true
vmType: vz

This seems functional enough for my needs.

[dhealio]

This is another place where the documentation could be improved, i.e. the vzNAT section could be expanded to elaborate on what it does differently and when to use it. etc.

I can confirm that vzNAT does allow queries on the VM itself to work with common utilities such as ping and dig.

Unfortunately, this workaround is not viable for me, as the resolve behavior is somehow not being picked up by select things that are running on the VM (like k3s). This may require having to add custom configuration there to get things to work.

[subpop]

In a Fedora guest using systemd-networkd and systems-resolved, I have not had any issues. On guests running NetworkManager (like CentOS), I had to manually set the ipv4.dns-priority value of the lima0 interface to a number higher than 0 so that it would be listed earlier in the generated /etc/resolv.conf. I did this by adding a connection override in /etc/NetworkManager/conf.d/dnf.conf:

[connection-cloud-init-lima0]
match-device=interface-name:lima0
ipv4.dns-priority=50

[nirs]

On guests running NetworkManager (like CentOS)

Does this apply to Ubuntu 24.04?

[subpop]

On guests running NetworkManager (like CentOS)

Does this apply to Ubuntu 24.04?

I just created a 24.04 guest and it appears to be using systemd-networkd/systemd-resolved like my Fedora guest. As long as it's created with --network vzNAT and has both an eth0 and a lima0 Ethernet interface, both VPN and non-VPN domains resolve correctly. No additional configuration changes are required. I didn't even need to restart any services. I started the guest and then after it was up, connected to VPN. After connecting to VPN, the VPN domains started resolving within the guest after a second or two of uptime.

[dhealio]

Has there been any progress on this?

The previous behavior provided by the host resolver shouldn't be replaced by gvisor-tap-vsock until gvisor-tap-vsock can provide the same level of service.

It's unfortunate to lose functionality. Reverting to v0.23.2 is currently the only option to gain it back.

Every workaround mentioned here introduces new issues and limitations in some form or another. Just to name a few: VPN traffic breaks altogether, k8s implementations running on lima cannot access resolvers on lima's loopback adapter, the priority is wrong when using multiple resolvers, etc.

Could the behavior be reverted (with an option to opt-in to gvisor-tap-vsock)? If not, could an option be added to explicitly use the previous behavior?

[dhealio]

[...] vz uses gvisor implementation for network. For that dns resolution is taken care by gvisor-tap-vsock. Which will take care of reading /etc/hosts to populate nameservers

I think it's worth noting that reading /etc/hosts to populate nameservers is not sufficient in terms of picking up all the nameservers in use by macOS. In fact, my /etc/hosts file contains this notice:

# macOS Notice
#
# This file is not consulted for DNS hostname resolution, address
# resolution, or the DNS query routing mechanism used by most
# processes on this system.
#
# To view the DNS configuration used by this system, use:
#   scutil --dns
#
# SEE ALSO
#   dns-sd(1), scutil(8)
#
# This file is automatically generated.

For me, /etc/hosts only contains nameserver entries for active adapters that are listed in System Settings -> Network. It does not contain nameserver entries that are in use by my VPN connection.