Failed to start instances on our Ganeti Xen cluster nodes

[CamZie <camzie@xxxxxxxxxxxxxx>]

Hello,

We are having problems starting instances on our two Ganeti (Xen) cluster nodes.

Below is the error we get when starting one of the instances that are failing. This only happens to the instances that are primarily running on our first node. The instances on the second node seems to be working without any problems.

node1

• OS: Debian 11.5
• Ganeti version: 3.0.2-1~deb11u1
• Xen Hypervisor: 4.14.5+24-g87d90d511c-1
• Storage: DRBD

node2

• OS: Debian 11.5
• Ganeti version: 3.0.2-1~deb11u1
• Xen Hypervisor: 4.14.5+24-g87d90d511c-1
• Storage: DRBD

ERROR MESSAGE: it somehow detects that the domain already exists

2022-10-13 10:15:10,818: ganeti-noded pid=165950 ERROR Hypervisor error: Failed to start instance ***REMOVED***: exited with exit code 3 (Parsing config from /etc/xen/***REMOVED***

libxl: error: libxl_domain.c:81:libxl__domain_rename: Domain 1116:Domain with name "***REMOVED***" already exists.

libxl: error: libxl_create.c:1233:initiate_domain_create: Domain 1116:cannot make domain: -6

libxl: error: libxl_domain.c:1182:libxl__destroy_domid: Domain 1116:Non-existant domain

libxl: error: libxl_domain.c:1136:domain_destroy_callback: Domain 1116:Unable to destroy guest

libxl: error: libxl_domain.c:1063:domain_destroy_cb: Domain 1116:Destruction of domain failed

). Moved config file to /var/log/ganeti/xen/***REMOVED***-2022-10-13_10_15_10

Traceback (most recent call last):

  File "/usr/share/ganeti/3.0/ganeti/backend.py", line 2921, in StartInstance

    hyper.StartInstance(instance, block_devices, startup_paused)

  File "/usr/share/ganeti/3.0/ganeti/hypervisor/hv_xen.py", line 959, in StartInstance

    raise errors.HypervisorError("Failed to start instance %s: %s (%s). Moved"

ganeti.errors.HypervisorError: Failed to start instance ***REMOVED***: exited with exit code 3 (Parsing config from /etc/xen/***REMOVED***

libxl: error: libxl_domain.c:81:libxl__domain_rename: Domain 1116:Domain with name "***REMOVED***" already exists.

libxl: error: libxl_create.c:1233:initiate_domain_create: Domain 1116:cannot make domain: -6

libxl: error: libxl_domain.c:1182:libxl__destroy_domid: Domain 1116:Non-existant domain

libxl: error: libxl_domain.c:1136:domain_destroy_callback: Domain 1116:Unable to destroy guest

libxl: error: libxl_domain.c:1063:domain_destroy_cb: Domain 1116:Destruction of domain failed

Thanks in advance.