Re: Unable to start more than 14 domUs - libxl__xs_transaction_commit: could not commit xenstore transaction

[Jürgen Groß <jgross@xxxxxxxx>]

On 03.06.20 15:01, J. Roeleveld wrote:
> On Wednesday, June 3, 2020 2:56:06 PM CEST Jürgen Groß wrote:
> > On 03.06.20 14:15, J. Roeleveld wrote:
> > > Hi all,
> > >
> > > I upgraded to Xen 4.12.2 recently and since the upgrade, I am unable to
> > > start more than 14 domUs. >
> > > The error, when trying to start the 15th, is:
> > > Parsing config from /etc/xen/boot/stage7/dbtest
> > > libxl: error: libxl_xshelp.c:265:libxl__xs_transaction_commit: could not
> > > commit xenstore transaction: No space left on device
> > > libxl: error: libxl_xshelp.c:265:libxl__xs_transaction_commit: could not
> > > commit xenstore transaction: No space left on device
> > > libxl: error: libxl_create.c:1318:domcreate_launch_dm: Domain 20:unable to
> > > add disk devices
> > > libxl: error: libxl_domain.c:1038:libxl__destroy_domid: Domain
> > > 20:Non-existant domain
> > > libxl: error: libxl_domain.c:993:domain_destroy_callback: Domain 20:Unable
> > > to destroy guest
> > > libxl: error: libxl_domain.c:920:domain_destroy_cb: Domain 20:Destruction
> > > of domain failed
> > >
> > >
> > >
> > > The domUs showing the issue get the disks from a driver-domain.
> > Ah, this might be related to hitting some Xenstore quota.
> >
> > As you are seeing the problem in libxl__xs_transaction_commit I guess
> > you should increase the number of Xenstore entries per domain.
> >
> > The default is 1000, you can set the value via the -E start parameter of
> > xenstored (e.g. "-E 5000" to set it to 5000), or if you are using
> > oxenstored by editing your /etc/xen/oxenstored.conf file (entry
> > "quota-maxentity").
> Has this default been changed/lowered?
> Or has there been an increase in amount of values in xenstore?
Depends. Which Xen version did you use before, and which Xenstore
variant (xenstored, oxenstored, or xenstore-stubdom) are you using?
Did you change anything in the driver domain?

> As this will require a reboot of the server, I will not be able to test this
> before the weekend. Is there any way to confirm this is actually the case
> without a restart?
There are strong hints in the data you presented:

libxl__xs_transaction_commit() is only calling xs_transaction_end()
and the error you are seeing is ENOSPC. In transaction end handling
the only reason for ENOSPC is hitting the quota for the number of
Xenstore nodes per domain.

You can easily check this by calling

xenstore-ls -p

in dom0 and count the number of entries which are owned by the driver
domain, an entry looks like:

      console = "" . . . . . . . . . . . . . . . . . . . . .  (n0,n1)

where "(n0,n1)" are the permissions of the entry. The first permission
entry "n0" is the owner (drop the letter, and you get the domid of the
owner, in my example "0").


Juergen