Re: [Xen-devel] Assigning contiguous memory to a driver domain

["Jan Beulich" <JBeulich@xxxxxxxxxx>]

>>> On 15.09.10 at 12:42, Joanna Rutkowska <joanna@xxxxxxxxxxxxxxxxxxxxxx> 
>>> wrote:
> On 09/15/10 11:50, Jan Beulich wrote:
>>>>> On 15.09.10 at 11:39, Rafal Wojtczuk <rafal@xxxxxxxxxxxxxxxxxxxxxx> wrote:
>>> On Wed, Sep 15, 2010 at 11:08:42AM +0200, Rafal Wojtczuk wrote:
>>>> On Tue, Sep 14, 2010 at 10:36:56AM +0100, Jan Beulich wrote:
>>>>>>>> On 14.09.10 at 11:24, Rafal Wojtczuk <rafal@xxxxxxxxxxxxxxxxxxxxxx> 
>>>>>>>> wrote:
>>>>>> Hello,
>>>>>>
>>>>>> Could someone guide me in the right direction with the topic of assigning
>>>>>> contiguous memory to a domain.
>>>>>>
>>>>>> I have an issue with a PV domain that is assigned a PCI device. 
>>>>>> Sometimes, 
>>>
>>>>>> the 
>>>>>> driver fails to load
>>>>>> Sep 13 10:36:43 localhost kernel: [  103.651858] iwlagn 0000:00:01.0:
>>>>>> firmware: requesting iwlwifi-4965-2.ucode
>>>>>> Sep 13 10:36:43 localhost kernel: [  103.669105] iwlagn 0000:00:01.0: 
>>> loaded
>>>>>> firmware version 228.61.2.24
>>>>>> Sep 13 10:36:43 localhost kernel: [  103.669263] iwlagn 0000:00:01.0: 
>>> failed
>>>>>> to allocate pci memory
>>>>>>
>>>>>> The reason seems to be that the domain does not have enough contiguous
>>>>>> memory, in mfn terms.
>>>>>
>>>>> No, how (dis)contiguous the memory of a domain is doesn't matter
>>>>> here. What matters is whether the domain can *make* the requested
>>>>> memory contiguous, and that depends on how much contiguous
>>>>> memory Xen has at the point of the allocation.
>>>>
>>>> Ah, so you are saying that regardless of whether a domain has some
>>>> contiguous memory, the driver will call xen_create_contiguous_region when 
>>> allocating
>>>> memory (via dma_alloc_coherent ?).
>>>>
>>>> Slightly out-of-the-list-scope: is there a convention when a driver should
>>>> allocate DMA-able memory ? Is it safe to assume that as soon as the driver
>>>> has loaded, it will no longer need to call xen_create_contiguous_region
>>>> anymore and we can use up all free Xen memory ? 
>>>
>>> Hmm, at least in case of tg3 driver, if xen free memory = 0, after I have
>>> done "ifconfig eth0 down" in the driver domain, the subsequent "ifconfig 
>>> eth0 up" 
>>> failed with "SIOCSIFFLAGS: Cannot allocate memory".
>> 
>> Sure - why would the driver waste resources when the device may
>> not be used.
>> 
>>> So, it looks like in order to make a PV driver domain work, there must be 
>>> some Xen free memory 
>>> all the time ?
>> 
>> Potentially yes, but this really depends on how the respective
>> driver is written.
>> 
>>> Moreover, this free Xen memory must be contiguous to some
>>> extent; is there any way to assure this ?
>> 
>> No.
>> 
> 
> Wait! Are you saying there is no *way* to guarantee proper operation of
> a driver domain in Xen?

No, that's to strict a statement. The admin can guarantee this. Xen
on its own can't.

> Sure, we can tune our memory balancer to always keep some 100MB (or
> 200MB, or maybe 500MB?) of xen free memory, and *hope* that it will
> contain enough continues pages, in case some driver in some driver
> domain calls dma_alloc_coherent(), so the call will succeed.

It likely doesn't need to be that much, because allocation happens
top-down. That is, Xen will try to avoid the (no longer explicitly
separate) DMA pool unless explicitly requested (or unless otherwise
not being able to fulfill a request). Therefore, if you always keep
just enough memory in Xen so that it'll never touch a reasonably
small set of pages you want to be available for DMA in guests,
these regions (the combined set of what Xen has and what was
given out as contiguous memory to the driver domain) will remain
contiguous. It'll get more problematic if you have more than one
driver domain.

Further, most drivers will prefer to allocate DMA memory once
(at startup or when enabling use of the device) rather than
repeatedly in the I/O path. In that case, you don't even need
to continuously make guarantees on available memory in Xen.

> But this is not a good solution: not only because it's a waste of memory
> (I'd rather use this memory for Dom0/storage domain page cache instead)
> but also, and most importantly, because I don't want to build a system
> based on *hope*!
> 
> Can we do something about it?

I can't easily imagine anything beyond how it's working today.

Jan

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel