RE: [Xen-devel] [PATCH] [HVM] Fix virtual apic irq distribution

["Dong, Eddie" <eddie.dong@xxxxxxxxx>]

Keir:
        This has a long history behind this:-) It is not caused by TSC hardware 
acceleration. But how we handle periodic time IRQ.
        In X86 VMM, the guest platform time like PIT, RTC has an periodic IRQ 
(same for guest local APIC time) whose pending interrupt is accumulated 
(pending_intr_nr) if the host time goes more.  In this periodic time model, the 
guest time seen can be explained like:

        Pit_time = total_pit_int_nr * T0 + CNT0 + Off0
        T0: period
        CNT0: The time elapsed since last IRQ fires. It can be known from the 
“Counter” register
                CNT0 = (LATCH-1- “Counter”)* T0 / LATCH
        Off0: A constant offset standing for the time when 1st IRQ happens.


        Here CNT0 is always less than T0. Say guest see guest_pit0 at last PIT 
IRQ injection time, now before we inject the next PIT IRQ, the guest time seen 
from guest PIT is actually limited within (guest_pit0,  guest_pit0 + T0) no 
matter how many physical time elapsed. If we sync guest TSC with host TSC using 
a fixed OFFSET, the guest time seen from TSC may be much ahead than guest PIT 
time if the host time elapsed a lot. Due to this, we freeze guest time (TSC 
time) at domain switch time to solve the problem. It works fine for UP.
        In SMP guest, each VP has its TSC time and local APIC time which we 
need to sync them together like above UP approach. Ideally this per VP time 
should sync with platform time too like PIT (and thus all VP guest time is 
synced). But the problem is that some VPs may be deactive and thus no way to 
inject perioidic time IRQ no matter for LOCAL APIC time or PIT time. The 
solution will be very complicated to this and it may need to change scheduler 
to do switch in/out with affinity for VPs in same VM.
        After some investigation, we think this is too complicate for now, and 
we want to go with simple solution first, i.e. Each guest time is synced, but 
the guest time among different VP is out of sync if the pending_intr_nr is not 
0. In this way the guest application (may migrate among processors) is OK 
because pending_intr_nr is always 0 before a guest application get executed on 
a VP, and thus the guest application see persistently guest time going ahead 
even it is migrated to anothre VP. We assume guest kernel critical path will 
not migrate between processors.
        With this approach, the platform time (PIT here) must be pinned to one 
processor (VP0 here), and guest TSC, PIT & local APIC time are synced in VP0 
(other VPs only sync TSC time and local APIC time). If it is routed to 
different VP dynamically say VP1, then the guest time seen within same VP (here 
VP1) will be not synced. That will cause the "well-known" guest lost too many 
ticks issue (refer CSET: 7478, 9324).


thx,eddie


-----Original Message-----
From: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx 
[mailto:xen-devel-bounces@xxxxxxxxxxxxxxxxxxx] On Behalf Of Keir Fraser
Sent: 2006年6月27日 20:53
To: Li, Xin B
Cc: xen-devel@xxxxxxxxxxxxxxxxxxx
Subject: Re: [Xen-devel] [PATCH] [HVM] Fix virtual apic irq distribution


On 27 Jun 2006, at 12:39, Li, Xin B wrote:

> But on our VM, we will have to synchronize TSC from time to time, so 
> PIT
> irq handler on different vcpu may see big TSC diff and complain about
> the unreliable TSC, then maybe it will try to do TSC sync, which make
> guest time keeping complex and unreliable.

How out-of-sync do TSCs of different VCPUs get?

I'd like to see a non-optimised TSC mode in which RDTSC always vmexits 
and we implement constant-rate always-sync'ed TSC in Xen. It'd be good 
to see if we could at least get that time mode working properly, and 
we'll need it anyway for save/restore/migration between machines with 
(even only slightly) different clock speeds or the guest will get very 
confused.

  -- Keir


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel