Re: [Xen-devel] [xen-unstable test] 6947: regressions - trouble: broken/fail/pass

["Jan Beulich" <JBeulich@xxxxxxxxxx>]

>>> On 02.05.11 at 13:22, Keir Fraser <keir.xen@xxxxxxxxx> wrote:
> On 02/05/2011 10:01, "Jan Beulich" <JBeulich@xxxxxxxxxx> wrote:
>> I have to admit that I find it bogus to not be allowed to call xmalloc()
>> with interrupts disabled. There's no equivalent restriction on kmalloc()
>> in Linux.
> 
> Well, the reason for the restriction on IRQ-disabled status on spinlock
> acquisition (IRQs disabled *only*, or IRQs disabled *never*) is because of
> the TSC synchronising rendezvous in x86/time.c:time_calibration().
> 
> A few options:
> 
> (1) Revert that rendezvous to using softirq or similar. The reason it was
> turned into hardirq rendezvous is that Dan Magenheimer measured that it
> reduced TSC skew by an order of magnitude or more. Perhaps it matters less
> on modern CPUs, or perhaps we could come up with some other smart workaround
> that would once again let us acquire IRQ-unsafe spinlocks with IRQs
> disabled. See (2) for why alloc_heap_pages() may still be IRQs-disabled
> unsafe however.

I admit I would want to avoid touching this (fragile) code.

> (2) Change the xmalloc lock to spin_lock_irqsave(). This would also have to
> be transitively applied to at least the heap_lock in page_alloc.c. One issue
> with this (and indeed with calling alloc_heap_pages at all with IRQs
> disabled) is that alloc_heap_pages does actually assume IRQs are enabled
> (for example, it calls flush_tlb_mask()) -- actually I think this limitation
> probably predates the tsc rendezvous changes, and could be a source of
> latent bugs in earlier Xen releases.

(2b) Make only the xmalloc() lock disable IRQs, and don't allow it to
go into the page allocator when IRQs were disabled on entry. Have
a reserve page available on each pCPU (requires that in a single
hypercall there can't be allocations adding up to more than PAGE_SIZE),
and when consumed, re-fill this page e.g. from a softirq or tasklet.

> (3) Restructure the interrupt code to do less work in IRQ context. For
> example tasklet-per-irq, and schedule on the local cpu. Protect a bunch of
> the PIRQ structures with a non-IRQ lock. Would increase interrupt latency if
> the local CPU is interrupted in hypervisor context. I'm not sure about this
> one -- I'm not that happy about the amount of work now done in hardirq
> context, but I'm not sure on the performance impact of deferring the work.

I'm not inclined to make changes in this area for the purpose at hand
either (again, Linux gets away without this - would have to check how
e.g. KVM gets the TLB flushing done, or whether they don't defer
flushes like we do).

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel