[Solved] Re: Unable to start more than 14 domUs - libxl__xs_transaction_commit: could not commit xenstore transaction

["J. Roeleveld" <joost@xxxxxxxxxxxx>]

TLDR version: Setting "-E 10000" for xenstored and rebooting solved the issue.


On 3 June 2020 15:26:19 CEST, "Jürgen Groß" <jgross@xxxxxxxx> wrote:
>On 03.06.20 15:01, J. Roeleveld wrote:
>> On Wednesday, June 3, 2020 2:56:06 PM CEST Jürgen Groß wrote:
>>> On 03.06.20 14:15, J. Roeleveld wrote:
>>>> Hi all,
>>>>
>>>> I upgraded to Xen 4.12.2 recently and since the upgrade, I am
>unable to
>>>> start more than 14 domUs. >
>>>> The error, when trying to start the 15th, is:
>>>> Parsing config from /etc/xen/boot/stage7/dbtest
>>>> libxl: error: libxl_xshelp.c:265:libxl__xs_transaction_commit:
>could not
>>>> commit xenstore transaction: No space left on device
>>>> libxl: error: libxl_xshelp.c:265:libxl__xs_transaction_commit:
>could not
>>>> commit xenstore transaction: No space left on device
>>>> libxl: error: libxl_create.c:1318:domcreate_launch_dm: Domain
>20:unable to
>>>> add disk devices
>>>> libxl: error: libxl_domain.c:1038:libxl__destroy_domid: Domain
>>>> 20:Non-existant domain
>>>> libxl: error: libxl_domain.c:993:domain_destroy_callback: Domain
>20:Unable
>>>> to destroy guest
>>>> libxl: error: libxl_domain.c:920:domain_destroy_cb: Domain
>20:Destruction
>>>> of domain failed
>>>>
>>>>
>>>>
>>>> The domUs showing the issue get the disks from a driver-domain.
>>>
>>> Ah, this might be related to hitting some Xenstore quota.
>>>
>>> As you are seeing the problem in libxl__xs_transaction_commit I
>guess
>>> you should increase the number of Xenstore entries per domain.
>>>
>>> The default is 1000, you can set the value via the -E start
>parameter of
>>> xenstored (e.g. "-E 5000" to set it to 5000), or if you are using
>>> oxenstored by editing your /etc/xen/oxenstored.conf file (entry
>>> "quota-maxentity").
>> 
>> Has this default been changed/lowered?
>> Or has there been an increase in amount of values in xenstore?
>
>Depends. Which Xen version did you use before, and which Xenstore
>variant (xenstored, oxenstored, or xenstore-stubdom) are you using?
>Did you change anything in the driver domain?

I need to check the logs to see which version it was. I wasn't that far behind 
in version numbers.
I am running Gentoo as my OS and afaik, I only have 1 option: "xenstored"  not 
heard of the other options yet, will do some research. (If you have some 
bookmarks, I would appreciate it. Otherwise it'll be whatever I can find)

I already noticed the issue before updating the driver domain. Initially I 
suspected a version mismatch and waited for a chance to update the driver 
domain. As that also didn't solve it and I still couldn't find anything usefull 
online, I asked for help here.

>> As this will require a reboot of the server, I will not be able to
>test this
>> before the weekend. Is there any way to confirm this is actually the
>case
>> without a restart?
>
>There are strong hints in the data you presented:
>
>libxl__xs_transaction_commit() is only calling xs_transaction_end()
>and the error you are seeing is ENOSPC. In transaction end handling
>the only reason for ENOSPC is hitting the quota for the number of
>Xenstore nodes per domain.
>
>You can easily check this by calling
>
>xenstore-ls -p
>
>in dom0 and count the number of entries which are owned by the driver
>domain, an entry looks like:
>
>       console = "" . . . . . . . . . . . . . . . . . . . . .  (n0,n1)
>
>where "(n0,n1)" are the permissions of the entry. The first permission
>entry "n0" is the owner (drop the letter, and you get the domid of the
>owner, in my example "0").

I will investigate the output later this week. Did see a strong hint at the 
"1000" value. The "ldb" file for xenstored was 1000kb in size.
I set the option to "-E 10000" to be certain I won't have this issue again for 
a long time and managed to organize a slot to reboot the system.
And, thank you so very much, the issue has now been solved.

Many thanks,

Joost


-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.