Re: [Xen-devel] [PATCH v2] x86/apicv: fix RTC periodic timer and apicv issue

["Tian, Kevin" <kevin.tian@xxxxxxxxx>]

Sorry I haven't got on this thread yet, but want you know it's on my list 
(hopefully will respond next week).

> -----Original Message-----
> From: Xuquan (Quan Xu) [mailto:xuquan8@xxxxxxxxxx]
> Sent: Monday, October 17, 2016 5:28 PM
> To: Xuquan (Quan Xu); Tian, Kevin; Jan Beulich
> Cc: Andrew Cooper; George.Dunlap@xxxxxxxxxxxxx; yang.zhang.wz@xxxxxxxxx;
> Hanweidong (Randy); Jiangyifei; Nakajima, Jun; xen-devel@xxxxxxxxxxxxx;
> konrad.wilk@xxxxxxxxxx; Tim Deegan
> Subject: RE: [PATCH v2] x86/apicv: fix RTC periodic timer and apicv issue
> 
> On October 11, 2016 7:11 PM, Xuquan < xuquan8@xxxxxxxxxx > wrote:
> >On October 11, 2016 3:49 PM, Tian, Kevin <Kevin.tian@xxxxxxxxx>
> >>> From: Xuquan (Quan Xu) [mailto:xuquan8@xxxxxxxxxx]
> >>> Sent: Monday, October 10, 2016 6:49 PM
> >>>
> >>> On October 10, 2016 5:40 PM, Jan Beulich < JBeulich@xxxxxxxx > wrote:
> >>> >>>>>> >>> On 20.09.16 at 15:30, <xuquan8@xxxxxxxxxx> wrote:
> >>> >>>>>> > --- a/xen/arch/x86/hvm/vlapic.c
> >>> >>>>>> > +++ b/xen/arch/x86/hvm/vlapic.c
> >>> >>>>>> > @@ -433,6 +433,12 @@ void vlapic_EOI_set(struct vlapic
> >>> >>>>>> > *vlapic) void vlapic_handle_EOI(struct vlapic *vlapic, u8 
> >>> >>>>>> > vector)  {
> >>> >>>>>> >      struct domain *d = vlapic_domain(vlapic);
> >>> >>>>>> > +    struct vcpu *v = vlapic_vcpu(vlapic);
> >>> >>>>>> > +    struct hvm_intack pt_intack;
> >>> >>>>>> > +
> >>> >>>>>> > +    pt_intack.vector = vector;
> >>> >>>>>> > +    pt_intack.source = hvm_intsrc_lapic;
> >>> >>>>>> > +    pt_intr_post(v, pt_intack);
> >>> >>>>>>
> >>> >>>>>> This also sits on the EOI LAPIC register write path, i.e. the
> >>> >>>>>> change then also affects non-apicv environments.
> >>> >>>>>
> >>> >>>>>The new logic should be entered only when EOI-induced exit happens.
> >>> >>>>>
> >>> >>>>
> >>> >>>> Yes, more that the EOI-induced exit is conditional,
> >>> >>>> specifically, the bitmap is set by vmx_set_eoi_exit_bitmap().
> >>> >>>> Jan, what do you think? While I recall from v1 discussion, you
> >>> >>>> have the same comment. We can dig it deep..
> >>> >>>
> >>> >>>See my reply to Kevin sent a minute ago. As I'm not sure what
> >>> >>>Kevin means to state with several of his responses, I can't
> >>> >>>properly respond for now. And then what you say doesn't really
> >>> >>>address my concern - things being conditional elsewhere doesn't
> >>> >>>mean we won't get here too in the non-apicv case, at least not in
> >>> >>>a way that I can follow
> >>right away.
> >>> >>
> >>> >> Jan, any idea now?
> >>> >
> >>> >I don't think there was anything left open on the other sub-thread;
> >>> >if there is, please point out specific aspects which are still unclear.
> >>> >
> >>>
> >>> Sorry, I overlooked the other sub-thread after holiday(10.1-10.7)..
> >>> I will read it again..
> >>>
> >>> Quan
> >>
> >>Is there any discussion after 10.1? I didn't see it.
> >>
> >>Back to the main open before holiday - multiple EOIs may come to clear
> >>irq_issued before guest actually handles the very vpt injection
> >>(possible if vpt vector is shared with other sources). I don't see a
> >>good solution on that open... :/
> >>
> >>We've discussed various options which all fail in one or another place
> >>- either miss an injection, or incur undesired injections.
> >>Possibly we should consider another direction - fall back to non-apicv
> >>path when we see vpt vector pending but it's not the highest one.
> >>
> >>Original condition to enter virtual intr delivery:
> >>        else if ( cpu_has_vmx_virtual_intr_delivery &&
> >>              intack.source != hvm_intsrc_pic &&
> >>              intack.source != hvm_intsrc_vector )
> >>
> >>now new condition:
> >>        else if ( cpu_has_vmx_virtual_intr_delivery &&
> >>              intack.source != hvm_intsrc_pic &&
> >>              intack.source != hvm_intsrc_vector &&
> >>                    (pt_vector == -1 || intack.vector == pt_vector) )
> >>
> >>Thoughts?
> >>
> >Kevin,
> >When I try to fix it as your suggestion, I cannot boot the guest, with below
> >message(from xl dmesg):
> 
> with Kevin's patch, the hypervisor always calls ' vmx_inject_extint() ->
> __vmx_inject_exception()' to inject exception, then vm-entry on loop..
> the interrupt (PT or IPI, or others) can't deliver to guest..
> 
> and so far, we suppress MSR-based APIC suggestion when having APIC-V by
> http://xenbits.xen.org/gitweb/?p=xen.git;a=commitdiff;h=7f2e992b824ec62a2818e643
> 90ac2ccfbd74e6b7
> so I think we couldn't fallback to non-apicv dynamically here..
> 
> Quan
> 
> 
> 
> >(d1) HVM Loader
> >(d1) Detected Xen v4.8-unstable
> >(d1) Xenbus rings @0xfeffc000, event channel 1
> >(d1) System requested SeaBIOS
> >(d1) CPU speed is 2394 MHz
> >(d1) Relocating guest memory for lowmem MMIO space disabled
> >(XEN) irq.c:275: Dom1 PCI link 0 changed 0 -> 5
> >(d1) PCI-ISA link 0 routed to IRQ5
> >(XEN) irq.c:275: Dom1 PCI link 1 changed 0 -> 10
> >(d1) PCI-ISA link 1 routed to IRQ10
> >(XEN) irq.c:275: Dom1 PCI link 2 changed 0 -> 11
> >(d1) PCI-ISA link 2 routed to IRQ11
> >(XEN) irq.c:275: Dom1 PCI link 3 changed 0 -> 5
> >(d1) PCI-ISA link 3 routed to IRQ5
> >(d1) pci dev 01:3 INTA->IRQ10
> >(d1) pci dev 02:0 INTA->IRQ11
> >(d1) RAM in high memory; setting high_mem resource base to 20f800000
> >(d1) pci dev 03:0 bar 10 size 002000000: 0f0000008
> >(d1) pci dev 02:0 bar 14 size 001000000: 0f2000008
> >(d1) pci dev 03:0 bar 30 size 000010000: 0f3000000
> >(d1) pci dev 03:0 bar 14 size 000001000: 0f3010000
> >(d1) pci dev 02:0 bar 10 size 000000100: 00000c001
> >(d1) pci dev 01:1 bar 20 size 000000010: 00000c101
> >(d1) Multiprocessor initialisation:
> >(d1)  - CPU0 ... 46-bit phys ... fixed MTRRs ... var MTRRs [1/8] ... done.
> >(d1)  - CPU1 ... 46-bit phys ... fixed MTRRs ... var MTRRs [1/8] ... done.
> >(d1) Testing HVM environment:
> >(d1)  - REP INSB across page boundaries ... passed
> >(d1)  - GS base MSRs and SWAPGS ... passed
> >(d1) Passed 2 of 2 tests
> >(d1) Writing SMBIOS tables ...
> >(d1) Loading SeaBIOS ...
> >(d1) Creating MP tables ...
> >(d1) Loading ACPI ...
> >(d1) vm86 TSS at fc00a300
> >(d1) BIOS map:
> >(d1)  10000-100e3: Scratch space
> >(d1)  c0000-fffff: Main BIOS
> >(d1) E820 table:
> >(d1)  [00]: 00000000:00000000 - 00000000:000a0000: RAM
> >(d1)  HOLE: 00000000:000a0000 - 00000000:000c0000
> >(d1)  [01]: 00000000:000c0000 - 00000000:00100000: RESERVED
> >(d1)  [02]: 00000000:00100000 - 00000000:f0000000: RAM
> >(d1)  HOLE: 00000000:f0000000 - 00000000:fc000000
> >(d1)  [03]: 00000000:fc000000 - 00000001:00000000: RESERVED
> >(d1)  [04]: 00000001:00000000 - 00000002:0f800000: RAM
> >(d1) Invoking SeaBIOS ...
> >(d1) SeaBIOS (version rel-1.9.3-0-ge2fc41e)
> >(d1) BUILD: gcc: (SUSE Linux) 4.3.4 [gcc-4_3-branch revision 152973] 
> >binutils:
> >(GNU
> >(d1) Binutils; SUSE Linux Enterprise 11) 2.23.1
> >(d1)
> >(d1) Found Xen hypervisor signature at 40000000
> >(d1) Running on QEMU (i440fx)
> >(d1) xen: copy e820...
> >(d1) Relocating init from 0x000d8fa0 to 0xeffabc40 (size 82736)
> >(d1) Found 6 PCI devices (max PCI bus is 00)
> >(d1) Allocated Xen hypercall page at effff000
> >(d1) Detected Xen v4.8-unstable
> >(d1) xen: copy BIOS tables...
> >(d1) Copying SMBIOS entry point from 0x00010020 to 0x000f5b60
> >(d1) Copying MPTABLE from 0xfc001170/fc001180 to 0x000f5a60
> >(d1) Copying PIR from 0x00010040 to 0x000f59e0
> >(d1) Copying ACPI RSDP from 0x000100c0 to 0x000f59b0
> >(d1) Using pmtimer, ioport 0xb008
> >(d1) Scan for VGA option rom
> >(d1) Running option rom at c000:0003
> >(XEN) stdvga.c:174:d1v0 entering stdvga mode
> >(d1) pmm call arg1=0
> >(d1) Turning on vga text mode console
> >(d1) SeaBIOS (version rel-1.9.3-0-ge2fc41e)
> >(d1) Machine UUID 59e20ef4-565a-49cb-9559-cde6d391cdf4
> >(d1) All threads complete.
> >(d1) Found 0 lpt ports
> >(d1) Found 0 serial ports
> >(d1) ATA controller 1 at 1f0/3f4/0 (irq 14 dev 9)
> >(d1) ATA controller 2 at 170/374/0 (irq 15 dev 9)
> >(d1) PS2 keyboard initialized
> >(d1) ata0-0: QEMU HARDDISK ATA-7 Hard-Disk (30720 MiBytes)
> >(d1) Searching bootorder for: /pci@i0cf8/*@1,1/drive@0/disk@0
> >(d1) All threads complete.
> >(d1) Scan for option roms
> >(d1)
> >(d1) Press ESC for boot menu.
> >(d1)
> >(d1) Searching bootorder for: HALT
> >(d1) drive 0x000f5940: PCHS=16383/16/63 translation=lba LCHS=1024/255/63
> >s=62914561
> >(d1) Space available for UMB: c9800-ec800, f5380-f5940
> >(d1) Returned 258048 bytes of ZoneHigh
> >(d1) e820 map has 7 items:
> >(d1)   0: 0000000000000000 - 000000000009fc00 = 1 RAM
> >(d1)   1: 000000000009fc00 - 00000000000a0000 = 2 RESERVED
> >(d1)   2: 00000000000f0000 - 0000000000100000 = 2 RESERVED
> >(d1)   3: 0000000000100000 - 00000000effff000 = 1 RAM
> >(d1)   4: 00000000effff000 - 00000000f0000000 = 2 RESERVED
> >(d1)   5: 00000000fc000000 - 0000000100000000 = 2 RESERVED
> >(d1)   6: 0000000100000000 - 000000020f800000 = 1 RAM
> >(d1) enter handle_19:
> >(d1)   NULL
> >(d1) Booting from Hard Disk...
> >(d1) Booting from 0000:7c00
> >(XEN) stdvga.c:179:d1v0 leaving stdvga mode
> >(XEN) Failed vm entry (exit reason 0x80000021) caused by invalid guest state 
> >(0).
> >(XEN) ************* VMCS Area **************
> >(XEN) *** Guest State ***
> >(XEN) CR0: actual=0x0000000080010031, shadow=0x0000000080010031,
> >gh_mask=ffffffffffffffff
> >(XEN) CR4: actual=0x00000000000426f8, shadow=0x00000000000406b8,
> >gh_mask=ffffffffffffffff
> >(XEN) CR3 = 0x0000000000185000
> >(XEN) PDPTE0 = 0x0000000000186001  PDPTE1 = 0x0000000000187001
> >(XEN) PDPTE2 = 0x0000000000188001  PDPTE3 = 0x0000000000189001
> >(XEN) RSP = 0x000000008ce53a6c (0x000000008ce53a6c)  RIP =
> >0x000000008161dd21 (0x000000008161dd21)
> >(XEN) RFLAGS=0x00200046 (0x00200046)  DR7 = 0x0000000000000400
> >(XEN) Sysenter RSP=000000008078b000 CS:RIP=0008:000000008168c0c0
> >(XEN)        sel  attr  limit   base
> >(XEN)   CS: 0008 0c09b ffffffff 0000000000000000
> >(XEN)   DS: 0023 0c0f3 ffffffff 0000000000000000
> >(XEN)   SS: 0010 0c093 ffffffff 0000000000000000
> >(XEN)   ES: 0023 0c0f3 ffffffff 0000000000000000
> >(XEN)   FS: 0030 04093 00003748 0000000081779c00
> >(XEN)   GS: 0000 1c000 ffffffff 0000000000000000
> >(XEN) GDTR:            000003ff 0000000081553000
> >(XEN) LDTR: 0000 1c000 ffffffff 0000000000000000
> >(XEN) IDTR:            000007ff 0000000081553400
> >(XEN)   TR: 0028 0008b 000020ab 00000000801ad000
> >(XEN) EFER = 0x0000000000000000  PAT = 0x0007010600070106
> >(XEN) PreemptionTimer = 0x00000000  SM Base = 0x00000000
> >(XEN) DebugCtl = 0x0000000000000000  DebugExceptions =
> >0x0000000000000000
> >(XEN) Interruptibility = 00000000  ActivityState = 00000000
> >(XEN) InterruptStatus = d100
> >(XEN) *** Host State ***
> >(XEN) RIP = 0xffff82d0801ff560 (vmx_asm_vmexit_handler)  RSP =
> >0xffff83187e20ff90
> >(XEN) CS=e008 SS=0000 DS=0000 ES=0000 FS=0000 GS=0000 TR=e040
> >(XEN) FSBase=0000000000000000 GSBase=0000000000000000
> >TRBase=ffff830839dfec00
> >(XEN) GDTBase=ffff83187e36f000 IDTBase=ffff83187e37b000
> >(XEN) CR0=0000000080050033 CR3=0000000821f6e000
> >CR4=00000000001526e0
> >(XEN) Sysenter RSP=ffff83187e20ffc0 CS:RIP=e008:ffff82d080244a00
> >(XEN) EFER = 0x0000000000000000  PAT = 0x0000050100070406
> >(XEN) *** Control State ***
> >(XEN) PinBased=000000bf CPUBased=b6a065fa SecondaryExec=0000576b
> >(XEN) EntryControls=000051ff ExitControls=000fefff
> >(XEN) ExceptionBitmap=00060002 PFECmask=00000000 PFECmatch=00000000
> >(XEN) VMEntry: intr_info=800000e1 errcode=00000000 ilen=00000000
> >(XEN) VMExit: intr_info=00000000 errcode=00000000 ilen=00000006
> >(XEN)         reason=80000021 qualification=0000000000000000
> >(XEN) IDTVectoring: info=00000000 errcode=00000000
> >(XEN) TSC Offset = 0xfffd7fc7c2ca5a1c  TSC Multiplier = 0x0000000000000000
> >(XEN) TPR Threshold = 0x00  PostedIntrVec = 0xf4
> >(XEN) EPT pointer = 0x0000000821f8501e  EPTP index = 0x0000
> >(XEN) PLE Gap=00000080 Window=00001000
> >(XEN) Virtual processor ID = 0x5f7a VMfunc controls = 0000000000000000
> >(XEN) **************************************
> >(XEN) domain_crash called from vmx.c:3111
> >(XEN) Domain 1 (vcpu#0) crashed on cpu#12:
> >(XEN) ----[ Xen-4.8-unstable  x86_64  debug=y   Not tainted ]----
> >(XEN) CPU:    12
> >(XEN) RIP:    0008:[<000000008161dd21>]
> >(XEN) RFLAGS: 0000000000200046   CONTEXT: hvm guest (d1v0)
> >(XEN) rax: 00000000000c0000   rbx: 000000000000000a   rcx:
> >00000000000c00d1
> >(XEN) rdx: 0000000000000000   rsi: 0000000000000000   rdi:
> >000000008ce53acc
> >(XEN) rbp: 000000008ce53a6c   rsp: 000000008ce53a6c   r8:
> >0000000000000000
> >(XEN) r9:  0000000000000000   r10: 0000000000000000   r11:
> >0000000000000000
> >(XEN) r12: 0000000000000000   r13: 0000000000000000   r14:
> >0000000000000000
> >(XEN) r15: 0000000000000000   cr0: 0000000080010031   cr4:
> >00000000000406b8
> >(XEN) cr3: 0000000000185000   cr2: 000000008cc09000
> >(XEN) ds: 0023   es: 0023   fs: 0030   gs: 0000   ss: 0010   cs: 0008
> >(XEN) HVM2 save: CPU
> >(XEN) HVM2 save: PIC
> >(XEN) HVM2 save: IOAPIC
> >(XEN) HVM2 save: LAPIC
> >(XEN) HVM2 save: LAPIC_REGS
> >(XEN) HVM2 save: PCI_IRQ
> >(XEN) HVM2 save: ISA_IRQ
> >(XEN) HVM2 save: PCI_LINK
> >(XEN) HVM2 save: PIT
> >(XEN) HVM2 save: RTC
> >(XEN) HVM2 save: HPET
> >(XEN) HVM2 save: PMTIMER
> >(XEN) HVM2 save: MTRR
> >(XEN) HVM2 save: VIRIDIAN_DOMAIN
> >(XEN) HVM2 save: CPU_XSAVE
> >(XEN) HVM2 save: VIRIDIAN_VCPU
> >(XEN) HVM2 save: VMCE_VCPU
> >(XEN) HVM2 save: TSC_ADJUST
> >(XEN) HVM2 restore: CPU 0
> >
> >
> >Quan

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel