Re: [Xen-devel] [PATCH v2] x86/apicv: fix RTC periodic timer and apicv issue

["Xuquan (Quan Xu)" <xuquan8@xxxxxxxxxx>]

On October 11, 2016 7:11 PM, Xuquan < xuquan8@xxxxxxxxxx > wrote:
>On October 11, 2016 3:49 PM, Tian, Kevin <Kevin.tian@xxxxxxxxx>
>>> From: Xuquan (Quan Xu) [mailto:xuquan8@xxxxxxxxxx]
>>> Sent: Monday, October 10, 2016 6:49 PM
>>>
>>> On October 10, 2016 5:40 PM, Jan Beulich < JBeulich@xxxxxxxx > wrote:
>>> >>>>>> >>> On 20.09.16 at 15:30, <xuquan8@xxxxxxxxxx> wrote:
>>> >>>>>> > --- a/xen/arch/x86/hvm/vlapic.c
>>> >>>>>> > +++ b/xen/arch/x86/hvm/vlapic.c
>>> >>>>>> > @@ -433,6 +433,12 @@ void vlapic_EOI_set(struct vlapic
>>> >>>>>> > *vlapic) void vlapic_handle_EOI(struct vlapic *vlapic, u8 vector)  
>>> >>>>>> > {
>>> >>>>>> >      struct domain *d = vlapic_domain(vlapic);
>>> >>>>>> > +    struct vcpu *v = vlapic_vcpu(vlapic);
>>> >>>>>> > +    struct hvm_intack pt_intack;
>>> >>>>>> > +
>>> >>>>>> > +    pt_intack.vector = vector;
>>> >>>>>> > +    pt_intack.source = hvm_intsrc_lapic;
>>> >>>>>> > +    pt_intr_post(v, pt_intack);
>>> >>>>>>
>>> >>>>>> This also sits on the EOI LAPIC register write path, i.e. the
>>> >>>>>> change then also affects non-apicv environments.
>>> >>>>>
>>> >>>>>The new logic should be entered only when EOI-induced exit happens.
>>> >>>>>
>>> >>>>
>>> >>>> Yes, more that the EOI-induced exit is conditional,
>>> >>>> specifically, the bitmap is set by vmx_set_eoi_exit_bitmap().
>>> >>>> Jan, what do you think? While I recall from v1 discussion, you
>>> >>>> have the same comment. We can dig it deep..
>>> >>>
>>> >>>See my reply to Kevin sent a minute ago. As I'm not sure what
>>> >>>Kevin means to state with several of his responses, I can't
>>> >>>properly respond for now. And then what you say doesn't really
>>> >>>address my concern - things being conditional elsewhere doesn't
>>> >>>mean we won't get here too in the non-apicv case, at least not in
>>> >>>a way that I can follow
>>right away.
>>> >>
>>> >> Jan, any idea now?
>>> >
>>> >I don't think there was anything left open on the other sub-thread;
>>> >if there is, please point out specific aspects which are still unclear.
>>> >
>>>
>>> Sorry, I overlooked the other sub-thread after holiday(10.1-10.7)..
>>> I will read it again..
>>>
>>> Quan
>>
>>Is there any discussion after 10.1? I didn't see it.
>>
>>Back to the main open before holiday - multiple EOIs may come to clear
>>irq_issued before guest actually handles the very vpt injection
>>(possible if vpt vector is shared with other sources). I don't see a
>>good solution on that open... :/
>>
>>We've discussed various options which all fail in one or another place
>>- either miss an injection, or incur undesired injections.
>>Possibly we should consider another direction - fall back to non-apicv
>>path when we see vpt vector pending but it's not the highest one.
>>
>>Original condition to enter virtual intr delivery:
>>          else if ( cpu_has_vmx_virtual_intr_delivery &&
>>              intack.source != hvm_intsrc_pic &&
>>              intack.source != hvm_intsrc_vector )
>>
>>now new condition:
>>          else if ( cpu_has_vmx_virtual_intr_delivery &&
>>              intack.source != hvm_intsrc_pic &&
>>              intack.source != hvm_intsrc_vector &&
>>                      (pt_vector == -1 || intack.vector == pt_vector) )
>>
>>Thoughts?
>>
>Kevin,
>When I try to fix it as your suggestion, I cannot boot the guest, with below
>message(from xl dmesg):

with Kevin's patch, the hypervisor always calls ' vmx_inject_extint() -> 
__vmx_inject_exception()' to inject exception, then vm-entry on loop..
the interrupt (PT or IPI, or others) can't deliver to guest..

and so far, we suppress MSR-based APIC suggestion when having APIC-V by 
http://xenbits.xen.org/gitweb/?p=xen.git;a=commitdiff;h=7f2e992b824ec62a2818e64390ac2ccfbd74e6b7
so I think we couldn't fallback to non-apicv dynamically here..

Quan



>(d1) HVM Loader
>(d1) Detected Xen v4.8-unstable
>(d1) Xenbus rings @0xfeffc000, event channel 1
>(d1) System requested SeaBIOS
>(d1) CPU speed is 2394 MHz
>(d1) Relocating guest memory for lowmem MMIO space disabled
>(XEN) irq.c:275: Dom1 PCI link 0 changed 0 -> 5
>(d1) PCI-ISA link 0 routed to IRQ5
>(XEN) irq.c:275: Dom1 PCI link 1 changed 0 -> 10
>(d1) PCI-ISA link 1 routed to IRQ10
>(XEN) irq.c:275: Dom1 PCI link 2 changed 0 -> 11
>(d1) PCI-ISA link 2 routed to IRQ11
>(XEN) irq.c:275: Dom1 PCI link 3 changed 0 -> 5
>(d1) PCI-ISA link 3 routed to IRQ5
>(d1) pci dev 01:3 INTA->IRQ10
>(d1) pci dev 02:0 INTA->IRQ11
>(d1) RAM in high memory; setting high_mem resource base to 20f800000
>(d1) pci dev 03:0 bar 10 size 002000000: 0f0000008
>(d1) pci dev 02:0 bar 14 size 001000000: 0f2000008
>(d1) pci dev 03:0 bar 30 size 000010000: 0f3000000
>(d1) pci dev 03:0 bar 14 size 000001000: 0f3010000
>(d1) pci dev 02:0 bar 10 size 000000100: 00000c001
>(d1) pci dev 01:1 bar 20 size 000000010: 00000c101
>(d1) Multiprocessor initialisation:
>(d1)  - CPU0 ... 46-bit phys ... fixed MTRRs ... var MTRRs [1/8] ... done.
>(d1)  - CPU1 ... 46-bit phys ... fixed MTRRs ... var MTRRs [1/8] ... done.
>(d1) Testing HVM environment:
>(d1)  - REP INSB across page boundaries ... passed
>(d1)  - GS base MSRs and SWAPGS ... passed
>(d1) Passed 2 of 2 tests
>(d1) Writing SMBIOS tables ...
>(d1) Loading SeaBIOS ...
>(d1) Creating MP tables ...
>(d1) Loading ACPI ...
>(d1) vm86 TSS at fc00a300
>(d1) BIOS map:
>(d1)  10000-100e3: Scratch space
>(d1)  c0000-fffff: Main BIOS
>(d1) E820 table:
>(d1)  [00]: 00000000:00000000 - 00000000:000a0000: RAM
>(d1)  HOLE: 00000000:000a0000 - 00000000:000c0000
>(d1)  [01]: 00000000:000c0000 - 00000000:00100000: RESERVED
>(d1)  [02]: 00000000:00100000 - 00000000:f0000000: RAM
>(d1)  HOLE: 00000000:f0000000 - 00000000:fc000000
>(d1)  [03]: 00000000:fc000000 - 00000001:00000000: RESERVED
>(d1)  [04]: 00000001:00000000 - 00000002:0f800000: RAM
>(d1) Invoking SeaBIOS ...
>(d1) SeaBIOS (version rel-1.9.3-0-ge2fc41e)
>(d1) BUILD: gcc: (SUSE Linux) 4.3.4 [gcc-4_3-branch revision 152973] binutils:
>(GNU
>(d1) Binutils; SUSE Linux Enterprise 11) 2.23.1
>(d1)
>(d1) Found Xen hypervisor signature at 40000000
>(d1) Running on QEMU (i440fx)
>(d1) xen: copy e820...
>(d1) Relocating init from 0x000d8fa0 to 0xeffabc40 (size 82736)
>(d1) Found 6 PCI devices (max PCI bus is 00)
>(d1) Allocated Xen hypercall page at effff000
>(d1) Detected Xen v4.8-unstable
>(d1) xen: copy BIOS tables...
>(d1) Copying SMBIOS entry point from 0x00010020 to 0x000f5b60
>(d1) Copying MPTABLE from 0xfc001170/fc001180 to 0x000f5a60
>(d1) Copying PIR from 0x00010040 to 0x000f59e0
>(d1) Copying ACPI RSDP from 0x000100c0 to 0x000f59b0
>(d1) Using pmtimer, ioport 0xb008
>(d1) Scan for VGA option rom
>(d1) Running option rom at c000:0003
>(XEN) stdvga.c:174:d1v0 entering stdvga mode
>(d1) pmm call arg1=0
>(d1) Turning on vga text mode console
>(d1) SeaBIOS (version rel-1.9.3-0-ge2fc41e)
>(d1) Machine UUID 59e20ef4-565a-49cb-9559-cde6d391cdf4
>(d1) All threads complete.
>(d1) Found 0 lpt ports
>(d1) Found 0 serial ports
>(d1) ATA controller 1 at 1f0/3f4/0 (irq 14 dev 9)
>(d1) ATA controller 2 at 170/374/0 (irq 15 dev 9)
>(d1) PS2 keyboard initialized
>(d1) ata0-0: QEMU HARDDISK ATA-7 Hard-Disk (30720 MiBytes)
>(d1) Searching bootorder for: /pci@i0cf8/*@1,1/drive@0/disk@0
>(d1) All threads complete.
>(d1) Scan for option roms
>(d1)
>(d1) Press ESC for boot menu.
>(d1)
>(d1) Searching bootorder for: HALT
>(d1) drive 0x000f5940: PCHS=16383/16/63 translation=lba LCHS=1024/255/63
>s=62914561
>(d1) Space available for UMB: c9800-ec800, f5380-f5940
>(d1) Returned 258048 bytes of ZoneHigh
>(d1) e820 map has 7 items:
>(d1)   0: 0000000000000000 - 000000000009fc00 = 1 RAM
>(d1)   1: 000000000009fc00 - 00000000000a0000 = 2 RESERVED
>(d1)   2: 00000000000f0000 - 0000000000100000 = 2 RESERVED
>(d1)   3: 0000000000100000 - 00000000effff000 = 1 RAM
>(d1)   4: 00000000effff000 - 00000000f0000000 = 2 RESERVED
>(d1)   5: 00000000fc000000 - 0000000100000000 = 2 RESERVED
>(d1)   6: 0000000100000000 - 000000020f800000 = 1 RAM
>(d1) enter handle_19:
>(d1)   NULL
>(d1) Booting from Hard Disk...
>(d1) Booting from 0000:7c00
>(XEN) stdvga.c:179:d1v0 leaving stdvga mode
>(XEN) Failed vm entry (exit reason 0x80000021) caused by invalid guest state 
>(0).
>(XEN) ************* VMCS Area **************
>(XEN) *** Guest State ***
>(XEN) CR0: actual=0x0000000080010031, shadow=0x0000000080010031,
>gh_mask=ffffffffffffffff
>(XEN) CR4: actual=0x00000000000426f8, shadow=0x00000000000406b8,
>gh_mask=ffffffffffffffff
>(XEN) CR3 = 0x0000000000185000
>(XEN) PDPTE0 = 0x0000000000186001  PDPTE1 = 0x0000000000187001
>(XEN) PDPTE2 = 0x0000000000188001  PDPTE3 = 0x0000000000189001
>(XEN) RSP = 0x000000008ce53a6c (0x000000008ce53a6c)  RIP =
>0x000000008161dd21 (0x000000008161dd21)
>(XEN) RFLAGS=0x00200046 (0x00200046)  DR7 = 0x0000000000000400
>(XEN) Sysenter RSP=000000008078b000 CS:RIP=0008:000000008168c0c0
>(XEN)        sel  attr  limit   base
>(XEN)   CS: 0008 0c09b ffffffff 0000000000000000
>(XEN)   DS: 0023 0c0f3 ffffffff 0000000000000000
>(XEN)   SS: 0010 0c093 ffffffff 0000000000000000
>(XEN)   ES: 0023 0c0f3 ffffffff 0000000000000000
>(XEN)   FS: 0030 04093 00003748 0000000081779c00
>(XEN)   GS: 0000 1c000 ffffffff 0000000000000000
>(XEN) GDTR:            000003ff 0000000081553000
>(XEN) LDTR: 0000 1c000 ffffffff 0000000000000000
>(XEN) IDTR:            000007ff 0000000081553400
>(XEN)   TR: 0028 0008b 000020ab 00000000801ad000
>(XEN) EFER = 0x0000000000000000  PAT = 0x0007010600070106
>(XEN) PreemptionTimer = 0x00000000  SM Base = 0x00000000
>(XEN) DebugCtl = 0x0000000000000000  DebugExceptions =
>0x0000000000000000
>(XEN) Interruptibility = 00000000  ActivityState = 00000000
>(XEN) InterruptStatus = d100
>(XEN) *** Host State ***
>(XEN) RIP = 0xffff82d0801ff560 (vmx_asm_vmexit_handler)  RSP =
>0xffff83187e20ff90
>(XEN) CS=e008 SS=0000 DS=0000 ES=0000 FS=0000 GS=0000 TR=e040
>(XEN) FSBase=0000000000000000 GSBase=0000000000000000
>TRBase=ffff830839dfec00
>(XEN) GDTBase=ffff83187e36f000 IDTBase=ffff83187e37b000
>(XEN) CR0=0000000080050033 CR3=0000000821f6e000
>CR4=00000000001526e0
>(XEN) Sysenter RSP=ffff83187e20ffc0 CS:RIP=e008:ffff82d080244a00
>(XEN) EFER = 0x0000000000000000  PAT = 0x0000050100070406
>(XEN) *** Control State ***
>(XEN) PinBased=000000bf CPUBased=b6a065fa SecondaryExec=0000576b
>(XEN) EntryControls=000051ff ExitControls=000fefff
>(XEN) ExceptionBitmap=00060002 PFECmask=00000000 PFECmatch=00000000
>(XEN) VMEntry: intr_info=800000e1 errcode=00000000 ilen=00000000
>(XEN) VMExit: intr_info=00000000 errcode=00000000 ilen=00000006
>(XEN)         reason=80000021 qualification=0000000000000000
>(XEN) IDTVectoring: info=00000000 errcode=00000000
>(XEN) TSC Offset = 0xfffd7fc7c2ca5a1c  TSC Multiplier = 0x0000000000000000
>(XEN) TPR Threshold = 0x00  PostedIntrVec = 0xf4
>(XEN) EPT pointer = 0x0000000821f8501e  EPTP index = 0x0000
>(XEN) PLE Gap=00000080 Window=00001000
>(XEN) Virtual processor ID = 0x5f7a VMfunc controls = 0000000000000000
>(XEN) **************************************
>(XEN) domain_crash called from vmx.c:3111
>(XEN) Domain 1 (vcpu#0) crashed on cpu#12:
>(XEN) ----[ Xen-4.8-unstable  x86_64  debug=y   Not tainted ]----
>(XEN) CPU:    12
>(XEN) RIP:    0008:[<000000008161dd21>]
>(XEN) RFLAGS: 0000000000200046   CONTEXT: hvm guest (d1v0)
>(XEN) rax: 00000000000c0000   rbx: 000000000000000a   rcx:
>00000000000c00d1
>(XEN) rdx: 0000000000000000   rsi: 0000000000000000   rdi:
>000000008ce53acc
>(XEN) rbp: 000000008ce53a6c   rsp: 000000008ce53a6c   r8:
>0000000000000000
>(XEN) r9:  0000000000000000   r10: 0000000000000000   r11:
>0000000000000000
>(XEN) r12: 0000000000000000   r13: 0000000000000000   r14:
>0000000000000000
>(XEN) r15: 0000000000000000   cr0: 0000000080010031   cr4:
>00000000000406b8
>(XEN) cr3: 0000000000185000   cr2: 000000008cc09000
>(XEN) ds: 0023   es: 0023   fs: 0030   gs: 0000   ss: 0010   cs: 0008
>(XEN) HVM2 save: CPU
>(XEN) HVM2 save: PIC
>(XEN) HVM2 save: IOAPIC
>(XEN) HVM2 save: LAPIC
>(XEN) HVM2 save: LAPIC_REGS
>(XEN) HVM2 save: PCI_IRQ
>(XEN) HVM2 save: ISA_IRQ
>(XEN) HVM2 save: PCI_LINK
>(XEN) HVM2 save: PIT
>(XEN) HVM2 save: RTC
>(XEN) HVM2 save: HPET
>(XEN) HVM2 save: PMTIMER
>(XEN) HVM2 save: MTRR
>(XEN) HVM2 save: VIRIDIAN_DOMAIN
>(XEN) HVM2 save: CPU_XSAVE
>(XEN) HVM2 save: VIRIDIAN_VCPU
>(XEN) HVM2 save: VMCE_VCPU
>(XEN) HVM2 save: TSC_ADJUST
>(XEN) HVM2 restore: CPU 0
>
>
>Quan

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel