Re: storagedriver domain limited to 61 "shares" - Xen 4.16.6 pre2

[Juergen Gross <jgross@xxxxxxxx>]

On 12.02.24 10:22, J. Roeleveld wrote:
> On Monday, February 12, 2024 7:40:39 AM CET Juergen Gross wrote:
> > On 11.02.24 17:21, J. Roeleveld wrote:
> >
> > > Hi all,
> > >
> > > I am running into an issue with my storagedomain.
> > > I have the HBAs assigned to the domain,
> > > Added "driver_domain=1" to the config.
> > >
> > > And am accessing filesystems on this domain from other domains.
> > >
> > > This works as expected.
> > >
> > > However, I am only able to assign 61 filesystems to other domains. As soon
> > > as I
>   attempt to assign a 62nd filesystem, it fails.
> > > I am completely at a loss where this limit of 61 is coming from and am
> > > unable
>   to find anything with over 2 days of google searches.
> > > I also noticed that "xl devd" isn't actually running on my storagedomain
> > > anymore. It used to run in a previous version, but as the whole
> > > environment
>   was booting succesfully I never bothered to keep checking.
> > > I do remember that it was running in the past. Currently, I think the
> > > init- script is stopping because "/proc/xen/capabilities" is empty.
> > >
> > > Please let me know which config-files are needed to troubleshoot this.
> >
> > My first guess would be that the driver domain is limited by the max number
> > of Xenstore nodes it is allowed to own.
> >
> > You can raise the default of 1000 nodes per domain (how to do that depends
> > on the Xenstore type you are running, xenstored or oxenstored).
> Hi Juergen,
>
> I just tested with -E set to 15000 (15k) but am still seeing the same issue.
>
> On the storagedomain, I see the following in dmesg:
> ===
> [58855.383841] vbd vbd-16-51729: 7 adding watch on /local/domain/16/device/
> vbd/51729/state
> [58855.384545] vbd vbd-16-51729: xenbus: watch_otherend on backend/vbd/
> 16/51729 failed.
> [58855.384548] vbd: probe of vbd-16-51729 failed with error -7
> ===
>
> followed by several like:
> ===
> [58855.407134] vbd vbd-16-51730: 7 adding watch on backend/vbd/16/51730/
> physical-device
> [58855.408205] xen-blkback: xen_blkbk_probe failed
> [58855.408242] vbd vbd-16-51730: 7 xenbus_dev_probe on backend/vbd/16/51730
> [58855.408405] vbd: probe of vbd-16-51730 failed with error -7
> ===
>
> The part "adding watch" makes me wonder about the option:
> -W, --watch-nb <nb>     limit the number of watches per domain,
>
> > From the code, I find the default is 128. Is there any way to find out how many 
> are currently set on my system and what a good amount would be?
Oh, the relevant pieces have been backported due to several XSAs for
Xenstore.

You should be able to issue:

xenstore-control quota

For showing the current settings and

xenstore-control quota set watches 256

for changing the current quota value for watches. Same should work for
other quota.

xenstore-control quota <domid>

shows the current number of used up resources of a specific domain.

> I am considering trying 256 for this.
> Is there anything I need to be aware off before making this change?
Using above commands: no.


Juergen

Attachment: OpenPGP_0xB0DE9DD628BF132F.asc
Description: OpenPGP public key

Attachment: OpenPGP_signature.asc
Description: OpenPGP digital signature