Weird DNS issue over VPN

[Kilou]

Hi,

my laptop runs on Manjaro and I'm using Remmina RDP to connect to my office desktop over my corporate's VPN. This VPN is launched from a terminal session using

sudo openconnect --juniper --no-dtls <vpn-server> -u <username>

This works fine on a freshly booted system and Remmina is able to connect to my desktop computer remotely. Once I'm done with my RDP session, I close Remmina and shut down the VPN connection. The problem is that if I want to start a new RDP session, I can start the VPN successfully with the above command but Remmina cannot connect to my desktop anymore and says "Unable to find the address of RDP server". The RDP server is given as an address instead of an IP in Remmina. It seems that after the first VPN connection, the DNS resolver in my VPN no longer works. In fact when using freerdp instead of Remmina, I get the following error:

freerdp_set_last_error ERRCONNECT_DNS_NAME_NOT_FOUND

So it seems that while DNS can be successfully resolved on the first VPN connection, they no longer work after that and the name of my RDP server cannot be found. Only a reboot allows the RDP connection to work again. Note that no matter if I'm connected to the VPN or not, I always get the IP of my router in /etc/resolv.conf. But this is also so when the DNS resolution works properly right after a reboot (which is kind of weird, no?). The content of this file is only altered if I connect to the VPN using network-manager GUI. But here again, the DNS resolution only works on the first time after booting, any subsequent attempts fail as well.

I have tried to delete the file known_hosts found in ~/.config/freerdp folder as this has been recommanded by some people. However this doesn't help with the problem. Only a reboot allows to connect to my RDP server again and rebooting is necessary each time I want to connect.

Any idea on what could be the cause of this? It seems that after the first VPN connection, something breaks in the DNS resolution and only a reboot can fix it. Thanks for any suggestion!

EDIT: I'm running kernel 4.19.24-1 but I also tried with 4.14.101-1 and the same problem occurs.

EDIT2: well I just found out that rebooting not always works Currently, Remmina is unable to connect to my RDP server although I just rebooted the laptop. VPN is up and running and I can browse internet. But probably the DNS server is not the one of my VPN, which prevents Remmina from finding the RDP server...

[Kilou]

Some more information:

I though that rebooting was a temporary fix but it is not. I've rebooted several times and still cannot connect to my RDP server over VPN.

The file /etc/resolv.conf is not a symlink of /run/systemd/resolve/resolv.conf. The two files have different contents when I'm connected to the VPN:

Content of /etc/resolv.conf:

# Generated by NetworkManager
nameserver 192.168.X.X

Content of /run/systemd/resolve/resolv.conf:

# This file is managed by man:systemd-resolved(8). Do not edit.
#
# This is a dynamic resolv.conf file for connecting local clients directly to
# all known uplink DNS servers. This file lists all configured search domains.
#
# Third party programs must not access this file directly, but only through the
# symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a different way,
# replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of

# operation for /etc/resolv.conf.
nameserver 192.168.X.X
nameserver 155.105.XXX.XX
nameserver 155.105.XXX.XXX
search intranet.mycompany vpn.intranet.mycompany

With this setup, Remmina does not find my RDP server.

Two questions here:

1. The file /run/systemd/resolve/resolv.conf correctly mentions the DNS server suitable for my VPN (those beginning by 155) and the search domains are also mentioned in this file. Nevertheless, the RDP server cannot be found. Is that because /etc/resolv.conf does not contain those lines? Basically, should I create a symlink for /etc/resolv.conf pointing to /run/systemd/resolve/resolv.conf?

2. my router's IP (beginning by 192) is in first position in the above file. Is that a problem? Does it mean that some traffic is not going through the VPN?

[jonathon]

@Kilou

sudo openconnect

Is there a reason you're not using Network Manager for this?

It sounds like your manual connection command and NM's DNS server handling are conflicting.

[Downloading]

My 2 cents here:
You can skip the DNS resolution by doing the following:

echo "xxx.xxx.xx.xx my.domain.com" >> /etc/hosts

Remmina uses libfreerdp.
From browsing the source at:
tcp.c

You can see it uses getaddrinfo() which checks /etc/hosts and /etc/resolv.conf for resolution, it doesn't check systemd's resolv.conf.

So to overwrite /etc/resolv.conf I would try to use the built in network manager to connect to the VPN. You can specify the DNS in your connection's settings!

I haven't tried but I think it's worth a shot!

[Kilou]

I use sudo openconnect because I need the --no-dtls option to maintain the VPN connection over long periods. Without this option, the VPN connection drops randomly after 20-30 minutes and I need to restart the VPN. It is not possible yet to pass that --no-dtls option to openconnect using network manager so I use the command line as a workaround. However even if I use network manager to start the VPN connection, I'm also unable to connect to my RDP server.

[Kilou]

You can see it uses getaddrinfo() which checks /etc/hosts and /etc/resolv.conf for resolution, it doesn’t check systemd’s resolv.conf.

So to overwrite /etc/resolv.conf I would try to use the built in network manager to connect to the VPN.

Doesn't this speak in favor of creating a symlink for /etc/resolv.conf pointing at /run/systemd/resolve/resolv.conf instead? Systemd's resolv.conf seems to be correct when I connect to the VPN from command line as well...(except that the first IP adress is the one of my router whereas Network Manager puts the same addresses but in the other way around (i.e. DNS server for the VPN first). Not sure if there is any drawback to creating that symlink but I don't know why it was not created in the first hand. Anyway, this symlink is mentioned in systemd's resolv.conf file:

# Third party programs must not access this file directly, but only through the
# symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a different way,
# replace this symlink by a static file or a different symlink

Using Network Manager to connect to VPN is not a viable option currently because I need to pass the --no-dtls option to openconnect to maintain the connection and Network manager is unable to do that. Without that option, the VPN connection drops randomly every 20-30 minutes or so...

EDIT: by the way, after the last stable update (23.02.2019), systemd's resolved.service was disabled and I wasn't able to connect to the VPN at all. I restarted it manually (see [Stable Update] 2019-02-23 - Kernels, Systemd, Browsers, Plasma5, Python, Haskell). Could my DNS problems be caused by doing this? Was the last update intentionally disabling systemd's resolved.service?

[Kilou]

I tested the symlink in a VM by issuing:

sudo ln -sf /run/systemd/resolve/resolv.conf /etc/resolv.conf

It does indeed seem to solve the problem: both systemd's resolv.conf and /etc/resolv.conf are now identical and the RDP appears able to connect several times in a row. It seems now that resolv.conf has the correct layout and puts the IPs of my VPN first. It's content while connected to the VPN using sudo openconnect is now:

nameserver 155.105.XXX.XX
nameserver 155.105.XXX.XXX
nameserver 192.168.X.X
search intranet.mycompany vpn.intranet.mycompany

Note that it works also when connecting to the VPN with network manager but now resolv.conf does not contain the row with my router's IP... This is the only difference in resolv.conf between connecting with sudo openconnect or using network manager.

So creating the symlink seems to solve the problem although I'm not sure whether it creates other issues elsewhere and whether the fact that my router's IP appears in resolv.conf when using sudo openconnect is a problem or not.

Do you guys have that symlink or are systemd's resolv.conf and /etc/resolv.conf two separate files?

[Downloading]

I'm not sure this is a viable solution since NetworkManager will overwrite /etc/resolv.conf every time. And thus your systemd resolv.conf since it's linked.

[Kilou]

That's right but is it a problem in practice ? (sorry I'm a newbie with Linux...). The symlink definitely solves my issue so far so I can mark this thread as solved (please let me know if this is not appropriate). There is some information on such symlink on the Arch wiki (see https://wiki.archlinux.org/index.php/systemd-networkd#Basic_DHCP_network) and it basically says that creating such symlink and enabling systemd-resolved.service means that DNS is set up by DHCP (whatever this means...). So I guess there is no real drawback if this is something possible according to Arch wiki.

EDIT: This page https://wiki.archlinux.org/index.php/Systemd-resolved#DNS also mentions to create a symlink at /etc/resolv.conf but pointing at systemd's stub-resolv.conf file instead of systemd's resolv.conf file. I did also test this and it works when connecting the VPN using sudo openconnect. However it doesn't work when the VPN is launched with network manager. Not sure what all this is doing but for my problem, letting /etc/resolv.conf be a symlink pointing at /run/systemd/resolve/resolv.conf works fine both with sudo openconnect and network manager (but I need sudo openconnect with --no-dtls option to maintain the VPN connection over long periods).

Thanks for your help and suggestions anyway!

[Downloading]

As a matter of fact your solution is indeed a viable solution!

https://wiki.archlinux.org/index.php/NetworkManager#DNS_caching_and_split_DNS

Under systemd-resolved:

systemd-resolved
NetworkManager can use systemd-resolved as a DNS resolver and cache. Make sure that systemd-resolved is properly configured and that systemd-resolved.service is started before using it.

systemd-resolved will be used automatically if /etc/resolv.conf is a symlink to /run/systemd/resolve/stub-resolv.conf, /run/systemd/resolve/resolv.conf or /usr/lib/systemd/resolv.conf.

So NetworkManager will simply use systemd-resolved when it detects a symlink, so all is well!

But now for the learning experience:
Have you turned on systemd-resolved by yourself or has openconnect somehow turned it on?
Because by default I think it's turned off (well it is atleast on my system).

[Kilou]

I did turn it on myself but only after the last stable update on 23.09.2019. So I don't know whether it was turned on or off prior to that update and whether this update did change the behaviour of systemd-resolved.service (it was an update about systemd so it could be...). The reason I turned it on is that after that update, openconnect complained that the resolve service was not found (see [Stable Update] 2019-02-23 - Kernels, Systemd, Browsers, Plasma5, Python, Haskell).

Now while this symlink did the trick for me inside a VM, I still seem to have issues with it on my actual system. It works when the VPN is launched with sudo openconnect but when it is launched using network manager, the RDP server is still not found... Can't understand that but I need to test it a little further.

[Kilou]

Some more information: while the symlink technique did work, I found out I had some DNS leaks when doing a test using dnsleaktest.com (I could see my ISP IP while connected to the VPN). Overall I was not very happy with this "solution". I was a bit concerned that I had enabled systemd-resolved manually to allow sudo openconnect to work while @Downloading mentioned this service is turned off on his system. After searching a bit more on this topic, I found this:

http://www.ubuntugeek.com/how-to-fix-dns-problems-after-upgrading-ubuntu-17-10-from-ubuntu-17-04-16-10-16-04.html

It's about Ubuntu but the solutions are working for Arch/Manjaro as well. I chose solution 3 because it allowed to disable systemd-resolved and so revert my system back to its original state. So basically what I did:

1. run sudo systemctl stop systemd-resolved
2. run sudo systemctl disable systemd-resolved
3. Removed symlink between /etc/resolv.conf and /run/systemd/resolve/resolv.conf
4. Commented out the line with hosts in /etc/nsswitch.conf
5. Reboot

Now everything works fine, both with sudo openconnect and network manager There is one minor glitch though: the RDP server is not found when the VPN is launched using Network Manager IF sudo openconnect was used before. That's the only remaining tiny issue but I can live with that. However only using network manager several times in a row works, and sudo openconnect always works whether or not network manager was used before or not.

My understanding is that commenting out the line with hosts in /etc/nsswitch.conf allows sudo openconnect to work without systemd-resolved. Indeed, openconnect relies on /etc/vpnc/vpnc-script to detect which type of DNS resolver is used and if it finds a row containing the word "resolve" in /etc/nsswitch.conf, it thinks systemd-resolved is being used even if it is disabled. This is why I got an error when using sudo openconnect with systemd-resolved disabled. And this is also why I did enable this service afterwards. However it seems this caused some errors and even if the symlink technique did work, I was still having some issues with DNS leaks. Well, simply commenting this line in /etc/nsswitch.conf is what is recommended in the above link and it indeed works. Now, although systemd-resolved is being disabled (the default in Manjaro apparently), sudo openconnect doesn't complain anymore about it and works properly.

I don't know whether vpnc-script is at fault here: it should not detect systemd-resolved as the DNS resolver being used based on that row in nsswitch.conf. It should check if systemd-reolved is enabled IMHO.

So I'm changing the solution to that post instead of the one on the symlink.

Note that I also enabled DNS caching by creating /etc/NetworkManager/conf.d/dns.conf and inserting
[main]
dns=dnsmasq

This works when using the VPN with Network Manager (I can see nameserver 127.0.0.1 in /etc/resolv.conf) but it doesn't work with sudo openconnect. But I don't mind that because obviously this dnsmasq option is inserted in a config file for Network Manager so sudo openconnect can't access it.

I think this solves all my issues with DNS. Thanks to anyone for having contributed!! Great forum!