Re: Packets not passing from domU to dom0

[Chuck Zmudzinski <brchuckz@xxxxxxx>]

On 1/24/2023 6:50 AM, Klaus Darilion wrote:
>
> Hi all!
>
>  
>
> I have spent hours debugging network issues between domU and dom0 - without 
> success. I hope you have some ideas what could be the cause of the problem, 
> or you can give me some hints how to debug this issue.
>
>  
>
> Situation: dom0 and PV domU are Ubuntu22.04 with XEN
>
> 4.16.0-1~ubuntu2.1(I had the same issues with Ubuntu 20.04 and XEN
>
> 4.11.3+24-g14b62ab3e5-1ubuntu2.3). The dom0 was setup by my provider
>
> where I rent this baremetal server. We have 20+ other servers with other 
> providers
>
> with more or less identical setup and without issues.
>
>  
>
>  
>
> dom0  vif1.0 <---------------> eth0 domU
>
>   192.168.0.1/24            192.168.0.2/24
>
>  
>
> No nftables, iptables, ebtables.
>
>  
>
> Packets pass from dom0 to domU, but almost not from domU to dom0.
>
> "Almost not" means, that once in a while a packet pass through from the
>
> domU to the dom0.
>
>  
>
> For example, dom0 pings 102.168.0.2. I use tcpdump in dom0 with "-i
>
> vif1.0" and in domU with "-i eth0".
>
>  
>
> In the dom0 I see the ARP request for 192.168.0.2.
>
> In the domU I see the ARP request for 192.168.0.2 arriving and I see the
>
> ARP response back.
>
> But the ARP response does not arrive in the dom0 (I do not see it in
>
> tcpdump).
>
>  
>
> Once in a while the ARP response passes through to the domU. Then the
>
> dom0 send ICMP ping request, which arrives in the domU, and the domU
>
> responds with an ICMP response which does not arrive in the domU. Again,
>
> once in a while a ping response arrives in the domU.
>
>  
>
> RX/TX statistics in the domU show plenty of packets coming in and sent
>
> out.
>
> RX/TX statistics in the dom0 show plenty of packets sent out, but only a
>
> few packets coming in. For example TX=5000 packets, RX=20 packets.
>
> No "dropped" or "error" indications.
>
>  
>
> As said before I have no idea where or why the packets do not pass
>
> through to the dom0. I have plenty of identical setups on other servers.
>
> I have enabled log_martians, checked verious sysctl setttings.
>
> Everything looks fine. Also no indications in the kernel logs.
>
>  
>
>  
>
> Questions: First, I would like to find out where the packets get 
> dropped/filtered -
>
> in the vif eth0 of the domU, in the vif vif1.0 of the dom0, or somewhere
>
> inbetween?
>

IIUC it is a direct connection between the vif in the dom0 and the
eth0 in the domU. But networking scripts that run in dom0 shipped
by Xen can introduce some iptables rules that can filter things. See,
for example:

http://xenbits.xen.org/gitweb/?p=xen.git;a=tree;f=tools/hotplug/Linux;h=1e5e4f9af283c68626fc518559f593eee810b629;hb=HEAD

In there is a bunch of hotplug scripts for Linux that run in dom0,
including the networking scripts (vif-common, vif-bridge, vif-route,
etc). Some of them, such as vif-common.sh, do setup some iptables
rules, among other things. These scripts don't always work in every
networking setup and sometimes they need to be tweaked on a
particular server in a particular networking setup. Your provider must
not have setup those vif-* scripts correctly for the network setup of
your provider. You should probably report the problem to the provider
and simply tell them if they can't fix it you will just use another provider.

>  
>
> Who forwards packets between the two vifs - the hypervisor or the dom0?
>
>  
>
> Can I somehow debug if a packet enters the vif but get silenty dropped?
>
>  
>
> Thanks
>
> Klaus
>
>  
>
>  
>
> -- 
>
> Klaus Darilion, Head of Operations
>
> nic.at GmbH, Jakob-Haringer-Straße 8/V
>
> 5020 Salzburg, Austria
>
>  
>