[Xen-devel] HVM CPU enumeration, mapping to VCPU ID (Was: Re: [Xen-users] FreeBSD PVHVM call for testing)

[Ian Campbell <Ian.Campbell@xxxxxxxxxx>]

On Thu, 2013-05-30 at 10:16 -0700, Matt Wilson wrote:
> On Thu, May 30, 2013 at 10:31:43AM +0100, Ian Campbell wrote:
> > On Wed, 2013-05-29 at 15:10 -0700, Matt Wilson wrote:
> > > On Wed, May 29, 2013 at 06:25:56PM +0100, Ian Campbell wrote:
> > > > On Wed, 2013-05-29 at 19:03 +0200, Roger Pau Monnà wrote:
> > > > > 
> > > > > Since I'm not able to reproduce the cpuid != acpi_id case, could you
> > > > > give it a try and report the results?
> > > > 
> > > > Konrad,
> > > > 
> > > > Might this same problem be related to the issue you saw doing PVHVM VCPU
> > > > hotplug which you mentioned the other day?
> > > >
> > > > In general for HVM I suppose there isn't a strict relationship between
> > > > the CPU number the kernel chooses for a CPU (which is somewhat
> > > > arbitrarily up to the kernel) and the hypervisors VCPU number (which is
> > > > exposed via the virtual APIC ID).
> > > 
> > > The CPU enumeration should follow the ACPI ID order in MADT (or
> > > MP-table), right?
> 
> > You mean the guest's enumeration?
> 
> Right.
> 
> > I'd have thought that was entirely up to the guest. Using the APIC
> > ID might be a reasonable implementation but there are surely others
> > (e.g. collapsing sparse LAPIC IDs into contiguous cpu ids, using the
> > top bits for NUMA node, etc etc).
> 
> On bare metal x86 Linux, the kernel enumerates CPUs based on an order
> defined by the BIOS.
>  Typically this means that all the cores are
> enumerated first, followed by logical processors (HT/SMT). For Linux,
> maxcpus=N/2 should disable HT on systems that enumerate processors in
> the recommended order. Some history:
>   https://bugzilla.kernel.org/show_bug.cgi?id=2317

How the guest chooses to enumerate the CPUs is not terribly relevant so
long as the Xen specific code for that OS knows how to invert that
mapping to get at the underlying ABI which determines Xen's VCPUID for a
CPU.

I think I was wrong to focus on the guest enumeration scheme before,
what actually matters is where in our ABI we expose the VCPUID, which
isn't at all clear to me.

> > >   The local APIC ID (returned by cpuid
> > > EAX=0000_0001h => EBX bits 31:24) shouldn't be used for
> > > enumeration. Under Xen, the LAPIC ID is hardcoded to vCPU idx * 2:
> > > 
> > > void hvm_cpuid(unsigned int input, unsigned int *eax, unsigned int *ebx,
> > > [...]
> > >      case 0x1:
> > >          /* Fix up VLAPIC details. */
> > >          *ebx &= 0x00FFFFFFu;
> > >          *ebx |= (v->vcpu_id * 2) << 24;
> > > [...]
> > >      case 0xb:
> > >          /* Fix the x2APIC identifier. */
> > >          *edx = v->vcpu_id * 2;
> > >          break;
> > > 
> > > This isn't the way it works on some of our EC2 instances.  On our high
> > > performance instance types, we provide x2APIC IDs that distinguish
> > > threads, cores and sockets to provide the guest OS with CPU topology
> > > information. The OS and applications can use CPUID EAX=0000_000Bh,
> > > ECX=level (HT=0, Core=1, Socket=2) => EAX bits 4:0 to determine the
> > > topology.
> > 
> > Hrm, that does beg the question of how an HVM guest is supposed to
> > determine what Xen's idea of the VCPU number is for a given CPU.
> 
> The virtual BIOS provides both ACPI tables and a legacy MP-table that
> gives the LAPIC id mapping. The guest could infer the Xen vCPU ID from
> a processor's position in these tables.

Do we consider the ordering given in any of those tables to be an HVM
guest ABI? What about the lapic_id == 2*vcpuid -- is that multiplication
factor part of the ABI (i.e. is the guest expected to pass lapic_id/2 to
vcpuop)?

>  Or we could add a VCPUOP that an enlightened guest could use to get
> the information more directly.

I'm hoping that there is some existing interface which I simply don't
know about, but yes this could be the answer if such a thing doesn't
exist.

> One question: why does a hypercall take a parameter that only has one
> valid value? That value can be determined by looking at the current
> running vCPU.

The generic prototype is:
        vcpu_op(int cmd, int vcpuid, void *extra_args)
Some cmds can act on any vcpuid and others can only act on the current
vcpu. In an ideal world we would have had VCPUID_SELF or something but
its a bit late for that.

> > I can't see anything which reverses that * 2 on the vcpu_op path and I
> > don't suppose you implemented the inverse of your scheme.
> 
> The *2 is just for assigning the LAPIC ID, and I'm pretty sure that
> Linux is assigning processor IDs sequentially at ACPI parse time.

That probably doesn't matter, what matters is the Xen specific parts of
the kernel's ability to reverse that assignment to get at the underlying
APIC ID, assuming that is actually an ABI from which we can infer the
VCPU ID...

Ian


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel