Re: Xen 4.18rc/ARM64 on Raspberry Pi 4B: Traffic in DomU crashing Dom0 when using VLANs

[Paul Leiber <paul@xxxxxxxxxxxxxxxx>]

Am 13.10.2023 um 20:56 schrieb Paul Leiber:
> Hi Xen developers list,
>
> TL;DR:
> ------
>
> Causing certain web server traffic on a secondary VLAN on Raspberry Pi 
> under vanilla Debian/UEFI in combination with Xen leads to complete 
> system reboot (watchdog triggering for Dom0). Other strange things are 
> happening.
> Description:
> ----------
>
> I recently set up Xen (self compiled, Version 4.18-rc) on a Raspberry Pi 
> 4B (on vanilla Debian Bookworm, UEFI boot mode). Until some time ago, 
> everything worked well with Dom0, one DomU and one bridge.
> Then I wanted to actually make use of the virtualization and started to 
> set up a second Debian Bookworm DomU (using xen-create-image) for 
> monitoring my systems with zabbix (a webserver based system monitoring 
> solution). The bridge used for this setup was the device bridging the 
> hardware NIC. I installed zabbix, set it up, and everything went well, I 
> could access the web interface without any problem.
> Then I set up VLANs (initally using VLAN numbers 1 and 2) to separate 
> network traffic between the DomUs. I made the existing device bridge 
> VLAN 1 (bridge 1) and created a secondary device for bridging VLAN 2 
> (bridge 2). Using only bridge 1 / VLAN 1 everything works well, I can 
> access the zabbix web interface without any noticeable issue. After 
> switching the zabbix DomU to VLAN 2 / bridge 2, everything seemingly 
> keeps on working well, I can ping different devices in my network from 
> the zabbix DomU and vice versa, I can ssh into the machine.
> However, as soon as I remotely access the zabbix web interface, the 
> complete system (DomUs and Dom0) becomes unresponsive and reboots after 
> some time (usually seconds, sometimes 1-2 minutes). The reboot is 
> reliably reproducable.
> I didn't see any error message in any log (zabbix, DomU syslog, Dom0 
> syslog) except for the following lines immediately before the system 
> reboots on the Xen serial console:
> (XEN) Watchdog timer fired for domain 0
> (XEN) Hardware Dom0 shutdown: watchdog rebooting machine
>
> As soon as I change the bridge to bridge 1 (with or without VLAN setup), 
> the web interface is accessible again after booting the zabbix DomU, no 
> reboots.
> So I assume that causing specific traffic on the virtual NIC when using 
> a VLAN setup with more than one VLAN under Xen makes the Dom0 system 
> hard crash. Of course, there might be other causes that I'm not aware 
> of, but to me, this seems to be the most likely explanation right now.
> What I tried:
> -------------
>
> 1. I changed the VLAN numbers. First to 101, 102, 103 etc. This was when 
> I noticed another strange thing: VLANs with numbers >99 simply don't 
> work on my Raspberry Pi under Debian, with or without Xen. VLAN 99 
> works, VLAN 100 (or everything else >99 that I tried) doesn't work. If I 
> choose a number >99, the VLAN is not configured, "ip a" doesn't list it. 
> Other Debian systems on x64 architecture don't show this behavior, 
> there, it was no problem to set up VLANs > 99. Therefore, I've changed 
> the VLANs to 10, 20, 30 etc., which worked. But it didn't solve the 
> initial problem of the crashing Dom0 and DomUs.
> 2. Different bridge options, without noticable effect:
> bridge_stp off      # dont use STP (spanning tree proto)
> bridge_waitport 0   # dont wait for port to be available
> bridge_fd 0         # no forward delay
>
> 3. Removing IPv6: No noticable effect.
>
> 4. Network traffic analysis: Now, here it becomes _really_ strange. I 
> started tcpdumps on Dom0, and depending on on which interface/bridge 
> traffic was logged, the problem went away, meaning, the DomU was running 
> smoothly for hours, even when accessing the zabbix web interface. 
> Stopping the log makes the system crash (as above, after seconds up to 
> 1-2 minutes) reproducably if I access the zabbix web interface.
> Logging enabcm6e4ei0 (NIC): no crashes
> Logging enabcm6e4ei0.10 (VLAN 10): instant crash
> Logging enabcm6e4ei0.20 (VLAN 20): no crashes
> Logging xenbr0 (on VLAN 10): instant crash
> Logging xenbr1 (on VLAN 20): no crashes
>
> I am clinging to the thought that there must be a rational explanation 
> for why logging the traffic on certain interfaces/bridges should avoid 
> the crash of the complete system, while logging other interfaces/bridges 
> doesn't. I myself can't think of one.
> I checked the dumps of enabcm6e4ei0.10 and xenbr0 (where the system 
> crashes) with wireshark, nothing sticks out to me (but I am really no 
> expert in analyzing network traffic). Dumps can be provided.
> 5. Watchdog: I tried to dig deeper into the cause for the watchdog 
> triggering. However, I didn't find any useful documentation on the web 
> on how the watchdog works or how to enable logging.
> 6. Eliminating Xen as cause: I booted the Debian system (which in Xen 
> setup would be Dom0) without Xen and set it up to use the VLAN 20 bridge 
> (the same that leads to a reboot when using it in the DomU) as primary 
> network interface. Everything seemed to be working, I could download 
> large files from the internet without any problem. Setting up Zabbix on 
> the base Debian system showed that the same setup (VLANs 10 and 20, 
> bridges 1 and 2, using bridge 2 as interface for Zabbix) without Xen is 
> working reliably, no reboots. This points to some Xen related component 
> being the root cause, I think.
> 7. Eliminating Apache as root cause: Reloading the Apache starting page 
> hosted on DomU several times per second didn't lead to a reboot.
> 8. Recompiling Xen: Independent of which Xen master branch version I was 
> using (all 4.18), the behavior was the same. I didn't get Xen working on 
> ARM64/UEFI in version 4.17.
> Current situation:
> ------------------
>
> I am out of ideas what to do next. Everything that was recommended to me 
> on xen-users didn't lead to significant insight or solve the problem.
> I'd appreciate any hints how to troubleshoot this and/or how to proceed 
> otherwise.
O.k., let's try to break that issue down.

Firstly, how can I get more information on why the Xen watchdog 
triggers? Is there documentation? Are there any logs? I couldn't find 
anything useful with my search skills.
Thanks,

Paul