Re: [Xen-devel] Assigning contiguous memory to a driver domain

["Jan Beulich" <JBeulich@xxxxxxxxxx>]

>>> On 15.09.10 at 13:07, Joanna Rutkowska <joanna@xxxxxxxxxxxxxxxxxxxxxx> 
>>> wrote:
> On 09/15/10 12:59, Jan Beulich wrote:
>>>>> On 15.09.10 at 12:42, Joanna Rutkowska <joanna@xxxxxxxxxxxxxxxxxxxxxx> 
>>>>> wrote:
>>> Wait! Are you saying there is no *way* to guarantee proper operation of
>>> a driver domain in Xen?
>> 
>> No, that's to strict a statement. The admin can guarantee this. Xen
>> on its own can't.
>> 
> Can they? How? Your answer below suggest it cannot be *guaranteed*...

Perhaps we have a different understanding of what "guaranteeing"
here means.

>>> Sure, we can tune our memory balancer to always keep some 100MB (or
>>> 200MB, or maybe 500MB?) of xen free memory, and *hope* that it will
>>> contain enough continues pages, in case some driver in some driver
>>> domain calls dma_alloc_coherent(), so the call will succeed.
>> 
>> It likely doesn't need to be that much, because allocation happens
>> top-down. That is, Xen will try to avoid the (no longer explicitly
>> separate) DMA pool unless explicitly requested (or unless otherwise
>> not being able to fulfill a request). Therefore, if you always keep
>> just enough memory in Xen so that it'll never touch a reasonably
>> small set of pages you want to be available for DMA in guests,
>> these regions (the combined set of what Xen has and what was
>> given out as contiguous memory to the driver domain) will remain
>> contiguous. It'll get more problematic if you have more than one
>> driver domain.
>> 
> 
> Well, we do have 2 driver domains in Qubes: one is the netvm (NIC
> drivers), and the other one is Dom0 (storage, video, etc). BTW, how is
> it that the drivers in Dom0 never has this problem with allocating
> continues memory (even if we keep xen free memory = 0)?

Merely because post-boot there would not usually be allocations.

>> Further, most drivers will prefer to allocate DMA memory once
>> (at startup or when enabling use of the device) rather than
>> repeatedly in the I/O path. In that case, you don't even need
>> to continuously make guarantees on available memory in Xen.
>> 
> 
> But it is not feasible to modify/ensure e.g. all the Linux networking
> drivers behave like this. That would require review/patching of hundreds
> of drivers, unless there is some generic way to ensure that?

The fact that in Dom0 things work flawlessly with most drivers
(and yes, I know of occasional exceptions, though usually it's a
shortage of swiotlb space that we get reported, not one of Xen
memory) speaks for itself I think. But yes, there is a level of
uncertainty which, afaict, can be overcome only by auditing
individual drivers (or groups of them if they share a common
framework).

And you certainly realize that there are other things some
drivers do that don't work under Xen without modification -
these cases also only get handled as people run into them.
When a pattern results from the necessary changes, one
can certainly go through the rest of the tree to find other
instances needing fixing. But that only helps until the next
such issue gets found (or introduced anew).

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel