Xen 4.11 - Unable to start X server: Nvidia out of memory error

[Mathieu Tarral <mathieu.tarral@xxxxxxxxxxxxxx>]

Hi,

I have Ubuntu 20.04 installed, and I'm using the provided Xen package: 
xen-hypervisor-4.11-amd64

Xen boots as expected, but I can't get my Gnome desktop, since the X server 
fails to start.

Looking at the journalctl logs, I can see the errors related the Gnome:
/usr/lib/gdm3/gdm-x-session[1962]: Unable to run X server

And a few lines before, an error related to Nvidia:
/usr/lib/gdm3/gdm-x-session[1966]: (WW) NVIDIA: Failed to bind sideband socket 
to
/usr/lib/gdm3/gdm-x-session[1966]: (WW) NVIDIA:     
'/var/run/nvidia-xdriver-66ed5ce3' Permission denied
/usr/lib/gdm3/gdm-x-session[1966]: (II) NVIDIA: Using 24576.00 MB of virtual 
memory for indirect memory
/usr/lib/gdm3/gdm-x-session[1966]: (II) NVIDIA:     access.
/usr/lib/gdm3/gdm-x-session[1966]: (EE) NVIDIA(0): Failed to allocate software 
rendering cache surface: out of
/usr/lib/gdm3/gdm-x-session[1966]: (EE) NVIDIA(0):     memory.
/usr/lib/gdm3/gdm-x-session[1966]: (EE) NVIDIA(0):  *** Aborting ***
/usr/lib/gdm3/gdm-x-session[1966]: (EE)
/usr/lib/gdm3/gdm-x-session[1966]: Fatal server error:
/usr/lib/gdm3/gdm-x-session[1966]: (EE) NVIDIA: A GPU exception occurred during 
X server initialization

I can also see a GPU MMU fault in dmesg:
kernel: NVRM: Xid (PCI:0000:01:00): 31, pid=285, Ch 00000008, intr 00000000. 
MMU Fault: ENGINE GRAPHICS HUBCLIENT_FE faulted @ 0x0_00000000. Fault is of 
type FAULT_PDE ACCESS_TYPE_VIRT_WRITE

In a normal boot I don't have these errors (with nouveau or nvidia driver),
so it's safe to assume that Xen is doing something that the drivers were not 
prepared for.

I'm using the latest drivers available: nvidia-440

Have you already seen this error before ?
Any ideas how to fix this issue ?
Do you have contacts at Nvidia who could help ?

Best regards,
Mathieu Tarral