RE: [Xen-devel] iommu=0 leading to panic when system defaults to using x2apic

["Kay, Allen M" <allen.m.kay@xxxxxxxxx>]

OK, forget what I said about cluster mode in previous message.  I investigated 
into this in more detail today and here is what I found:

Keir's interpretation in previous message is basically correct.  Currently, 
both MSI and IOAPIC address register consists of 32 bits.  Bits 31:20 are 
occupied by 0xFEE so there aren't enough bits to accommodate full 32-bit APIC 
ID's.  As a result new format is define for x2apic to accommodate full 32-bit 
APIC ID's.

The problem is legacy MSI and IOAPIC devices still have 8-bit APIC ID's so 
interrupt remapping is used to be the intermediary between the legacy MSI 
devices and ioapics and CPUs in x2APIC mode.

Although it is possible to do turn on x2APIC on systems with less than 256 
CPU's without using interrupt remapping, this configuration is not supported by 
the architecture in order to simplify HW validation.

Instead, BIOS is required to start the system in xAPIC mode for systems with 
less than 256 CPUs.  It will turn on x2APIC mode for systems with more than 256 
CPU's.  On these systems, Xen will need to enable VT-d/interrupt remapping for 
x2APIC to work properly.

Allen 

-----Original Message-----
From: Keir Fraser [mailto:keir.xen@xxxxxxxxx] On Behalf Of Keir Fraser
Sent: Monday, December 13, 2010 11:45 PM
To: Kay, Allen M; Jan Beulich; Zhang, Yang Z
Cc: Han, Weidong; xen-devel@xxxxxxxxxxxxxxxxxxx
Subject: Re: [Xen-devel] iommu=0 leading to panic when system defaults to using 
x2apic

Well, if it is a restriction imposed by cluster mode, you know the next
question is obvious: Why do we bother with cluster mode at all? I don't see
that it yields us any advantage over physical mode, and we could use
physical mode without interrupt remapping, that would seem to be a big bonus
and simplification? Could we just kill our x2apic cluster mode logic?

 -- Keir

On 14/12/2010 02:25, "Kay, Allen M" <allen.m.kay@xxxxxxxxx> wrote:

> Keir/Jan,
> 
> My understanding is that cluster mode requires it.  I will get back to you
> guys after I dig out the details on this - did not get a chance to do this
> today.
> 
> Allen
> 
> -----Original Message-----
> From: Keir Fraser [mailto:keir.xen@xxxxxxxxx] On Behalf Of Keir Fraser
> Sent: Monday, December 13, 2010 1:03 AM
> To: Jan Beulich; Kay, Allen M; Zhang, Yang Z
> Cc: Han, Weidong; xen-devel@xxxxxxxxxxxxxxxxxxx
> Subject: Re: [Xen-devel] iommu=0 leading to panic when system defaults to
> using x2apic
> 
> On 13/12/2010 08:15, "Jan Beulich" <JBeulich@xxxxxxxxxx> wrote:
> 
>>>>> On 11.12.10 at 01:07, "Kay, Allen M" <allen.m.kay@xxxxxxxxx> wrote:
>>> Yes, interrupt remapping is needed to be the intermediary between legacy
>>> IOxAPIC and MSI devices and the new x2APIC in the CPU.
>> 
>> But isn't this only when there are APIC IDs beyond 255?
> 
> Apparently not, since even Linux requires irq remapping even when none of
> the APIC IDs are greater than 255. Unless running on kvm or xen. I don't
> fully understand this particular restriction, mind you.
> 
> Actually, my guess is that x2apic mode requires a different format of APIC
> message with a 32-bit APICID field, legacy IOxAPIC and MSI devices do not
> support the new message format, and so irq remapping hardware is required to
> bridge the two formats, even if no actual irq remapping is occurring.
> 
> Is that a canny guess, Allen?
> 
>  -- Keir
> 
>> Jan
>> 
>>> -----Original Message-----
>>> From: Keir Fraser [mailto:keir.xen@xxxxxxxxx] On Behalf Of Keir Fraser
>>> Sent: Friday, December 10, 2010 10:50 AM
>>> To: Kay, Allen M; Jan Beulich; Zhang, Yang Z
>>> Cc: xen-devel@xxxxxxxxxxxxxxxxxxx; Han, Weidong
>>> Subject: Re: [Xen-devel] iommu=0 leading to panic when system defaults to
>>> using x2apic
>>> 
>>> Ah, and the interrupt remapping dependency is because PCI(e) devices cannot
>>> address 32-bit APIC IDs?
>>> 
>>>  -- Keir
>>> 
>>> On 10/12/2010 18:26, "Kay, Allen M" <allen.m.kay@xxxxxxxxx> wrote:
>>> 
>>>> The architectural requirement is actually between interrupt remapping and
>>>> x2apic.  Since interrupt remapping is part of the VT-d feature so current
>>>> software requires all VT-d features enabled in order for x2apic to be
>>> enabled.
>>>> 
>>>> Strictly speaking DMA remapping is not required for x2apic.  However,
>>>> queued
>>>> invalidation is required since interrupt remapping requires queued
>>>> invalidation.  So x2apic dependency is as follows:
>>>> 
>>>>     x2apic->interrupt remapping->queued invalidation
>>>> 
>>>> Due to historical reasons, the new VT-d features were built on top of the
>>>> old
>>>> ones as they become available.  Is there a requirement to separate this
>>>> out?
>>>> If so, we will need to re-design iommu boot parameter which took a while to
>>>> get it right so most systems can now boot successfully.
>>>> 
>>>> Allen
>> 
>> 
> 
> 



_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel