Re: storagedriver domain limited to 61 "shares" - Xen 4.16.6 pre2

["J. Roeleveld" <joost@xxxxxxxxxxxx>]

On Monday, February 12, 2024 7:40:39 AM CET Juergen Gross wrote:
> On 11.02.24 17:21, J. Roeleveld wrote:
> 
> > Hi all,
> > 
> > I am running into an issue with my storagedomain.
> > I have the HBAs assigned to the domain,
> > Added "driver_domain=1" to the config.
> > 
> > And am accessing filesystems on this domain from other domains.
> > 
> > This works as expected.
> > 
> > However, I am only able to assign 61 filesystems to other domains. As soon
> > as I
 attempt to assign a 62nd filesystem, it fails.
> > 
> > I am completely at a loss where this limit of 61 is coming from and am
> > unable
 to find anything with over 2 days of google searches.
> > 
> > I also noticed that "xl devd" isn't actually running on my storagedomain
> > anymore. It used to run in a previous version, but as the whole
> > environment
 was booting succesfully I never bothered to keep checking.
> > 
> > I do remember that it was running in the past. Currently, I think the
> > init- script is stopping because "/proc/xen/capabilities" is empty.
> > 
> > Please let me know which config-files are needed to troubleshoot this.
> 
> 
> My first guess would be that the driver domain is limited by the max number
> of Xenstore nodes it is allowed to own.
> 
> You can raise the default of 1000 nodes per domain (how to do that depends
> on the Xenstore type you are running, xenstored or oxenstored).

Hi Juergen,

I just tested with -E set to 15000 (15k) but am still seeing the same issue.

On the storagedomain, I see the following in dmesg:
===
[58855.383841] vbd vbd-16-51729: 7 adding watch on /local/domain/16/device/
vbd/51729/state
[58855.384545] vbd vbd-16-51729: xenbus: watch_otherend on backend/vbd/
16/51729 failed.
[58855.384548] vbd: probe of vbd-16-51729 failed with error -7
===

followed by several like:
===
[58855.407134] vbd vbd-16-51730: 7 adding watch on backend/vbd/16/51730/
physical-device
[58855.408205] xen-blkback: xen_blkbk_probe failed
[58855.408242] vbd vbd-16-51730: 7 xenbus_dev_probe on backend/vbd/16/51730
[58855.408405] vbd: probe of vbd-16-51730 failed with error -7
===

The part "adding watch" makes me wonder about the option:
-W, --watch-nb <nb>     limit the number of watches per domain,

>From the code, I find the default is 128. Is there any way to find out how 
>many 
are currently set on my system and what a good amount would be?

I am considering trying 256 for this.
Is there anything I need to be aware off before making this change?

--
Joost