Re: [Xen-devel] Load increase after memory upgrade (part2)

[Carsten Schiers <carsten@xxxxxxxxxx>]

Here with two cards enabled and creating a bit "work" by watching TV with one 
oft hem:

[   23.842720] Starting SWIOTLB debug thread.
[   23.842750] swiotlb_start_thread: Go!
[   23.842838] xen_swiotlb_start_thread: Go!
[   28.841451] 0 [budget_av 0000:00:01.0] bounce: from:435596(slow:0)to:0 
map:658 unmap:0 sync:435596
[   28.841592] SWIOTLB is 4% full
[   33.840147] 0 [budget_av 0000:00:01.0] bounce: from:127652(slow:0)to:0 map:0 
unmap:0 sync:127652
[   33.840283] SWIOTLB is 4% full
[   33.844222] 0 budget_av 0000:00:01.0 alloc coherent: 8, free: 0
[   38.840227] 0 [budget_av 0000:00:01.0] bounce: from:128310(slow:0)to:0 map:0 
unmap:0 sync:128310
[   38.840361] SWIOTLB is 4% full
[   43.840182] 0 [budget_av 0000:00:01.0] bounce: from:128310(slow:0)to:0 map:0 
unmap:0 sync:128310
[   43.840323] SWIOTLB is 4% full
[   48.840094] 0 [budget_av 0000:00:01.0] bounce: from:127652(slow:0)to:0 map:0 
unmap:0 sync:127652
[   48.840154] SWIOTLB is 4% full
[   53.840160] 0 [budget_av 0000:00:01.0] bounce: from:119756(slow:0)to:0 map:0 
unmap:0 sync:119756
[   53.840301] SWIOTLB is 4% full
[   58.840202] 0 [budget_av 0000:00:01.0] bounce: from:128310(slow:0)to:0 map:0 
unmap:0 sync:128310
[   58.840339] SWIOTLB is 4% full
[   63.840626] 0 [budget_av 0000:00:01.0] bounce: from:128310(slow:0)to:0 map:0 
unmap:0 sync:128310
[   63.840686] SWIOTLB is 4% full
[   68.840122] 0 [budget_av 0000:00:01.0] bounce: from:127323(slow:0)to:0 map:0 
unmap:0 sync:127323
[   68.840180] SWIOTLB is 4% full
[   73.840647] 0 [budget_av 0000:00:01.0] bounce: from:211547(slow:0)to:0 map:0 
unmap:0 sync:211547
[   73.840784] SWIOTLB is 4% full
[   78.840204] 0 [budget_av 0000:00:01.0] bounce: from:255962(slow:0)to:0 map:0 
unmap:0 sync:255962
[   78.840344] SWIOTLB is 4% full
[   83.840114] 0 [budget_av 0000:00:01.0] bounce: from:255304(slow:0)to:0 map:0 
unmap:0 sync:255304
[   83.840178] SWIOTLB is 4% full
[   88.840158] 0 [budget_av 0000:00:01.0] bounce: from:256620(slow:0)to:0 map:0 
unmap:0 sync:256620
[   88.840302] SWIOTLB is 4% full
[   93.840185] 0 [budget_av 0000:00:00.0] bounce: from:250040(slow:0)to:0 map:0 
unmap:0 sync:250040
[   93.840319] SWIOTLB is 4% full
[   98.840181] 0 [budget_av 0000:00:00.0] bounce: from:255962(slow:0)to:0 map:0 
unmap:0 sync:255962
[   98.841563] SWIOTLB is 4% full
[  103.841221] 0 [budget_av 0000:00:00.0] bounce: from:255962(slow:0)to:0 map:0 
unmap:0 sync:255962
[  103.841361] SWIOTLB is 4% full
[  108.840247] 0 [budget_av 0000:00:00.0] bounce: from:255962(slow:0)to:0 map:0 
unmap:0 sync:255962
[  108.840389] SWIOTLB is 4% full
[  113.840157] 0 [budget_av 0000:00:00.0] bounce: from:261555(slow:0)to:0 map:0 
unmap:0 sync:261555
[  113.840298] SWIOTLB is 4% full
[  118.840119] 0 [budget_av 0000:00:00.0] bounce: from:295442(slow:0)to:0 map:0 
unmap:0 sync:295442
[  118.840259] SWIOTLB is 4% full
[  123.841025] 0 [budget_av 0000:00:00.0] bounce: from:295113(slow:0)to:0 map:0 
unmap:0 sync:295113
[  123.841164] SWIOTLB is 4% full
[  128.840175] 0 [budget_av 0000:00:00.0] bounce: from:294784(slow:0)to:0 map:0 
unmap:0 sync:294784
[  128.840310] SWIOTLB is 4% full
[  133.840194] 0 [budget_av 0000:00:00.0] bounce: from:293797(slow:0)to:0 map:0 
unmap:0 sync:293797
[  133.840330] SWIOTLB is 4% full
[  138.840498] 0 [budget_av 0000:00:00.0] bounce: from:341502(slow:0)to:0 map:0 
unmap:0 sync:341502
[  138.840637] SWIOTLB is 4% full
[  143.840173] 0 [budget_av 0000:00:00.0] bounce: from:341502(slow:0)to:0 map:0 
unmap:0 sync:341502
[  143.840313] SWIOTLB is 4% full
[  148.840215] 0 [budget_av 0000:00:00.0] bounce: from:341831(slow:0)to:0 map:0 
unmap:0 sync:341831
[  148.840355] SWIOTLB is 4% full
[  153.840205] 0 [budget_av 0000:00:01.0] bounce: from:329658(slow:0)to:0 map:0 
unmap:0 sync:329658
[  153.840341] SWIOTLB is 4% full
[  158.840137] 0 [budget_av 0000:00:00.0] bounce: from:342160(slow:0)to:0 map:0 
unmap:0 sync:342160
[  158.840277] SWIOTLB is 4% full
[  163.841288] 0 [budget_av 0000:00:00.0] bounce: from:341502(slow:0)to:0 map:0 
unmap:0 sync:341502
[  163.841424] SWIOTLB is 4% full
[  168.840198] 0 [budget_av 0000:00:00.0] bounce: from:341502(slow:0)to:0 map:0 
unmap:0 sync:341502
[  168.840339] SWIOTLB is 4% full
[  173.840167] 0 [budget_av 0000:00:00.0] bounce: from:341502(slow:0)to:0 map:0 
unmap:0 sync:341502
[  173.840304] SWIOTLB is 4% full
[  178.840184] 0 [budget_av 0000:00:00.0] bounce: from:328013(slow:0)to:0 map:0 
unmap:0 sync:328013
[  178.840324] SWIOTLB is 4% full
[  183.840129] 0 [budget_av 0000:00:00.0] bounce: from:341831(slow:0)to:0 map:0 
unmap:0 sync:341831
[  183.840269] SWIOTLB is 4% full
[  188.840123] 0 [budget_av 0000:00:01.0] bounce: from:340515(slow:0)to:0 map:0 
unmap:0 sync:340515
[  188.841647] SWIOTLB is 4% full
[  193.840192] 0 [budget_av 0000:00:00.0] bounce: from:338541(slow:0)to:0 map:0 
unmap:0 sync:338541
[  193.840329] SWIOTLB is 4% full
[  198.840148] 0 [budget_av 0000:00:01.0] bounce: from:330316(slow:0)to:0 map:0 
unmap:0 sync:330316
[  198.840230] SWIOTLB is 4% full
[  203.840860] 0 [budget_av 0000:00:00.0] bounce: from:341831(slow:0)to:0 map:0 
unmap:0 sync:341831
[  203.841000] SWIOTLB is 4% full
[  208.840562] 0 [budget_av 0000:00:01.0] bounce: from:337883(slow:0)to:0 map:0 
unmap:0 sync:337883
[  208.840698] SWIOTLB is 4% full
[  213.840171] 0 [budget_av 0000:00:00.0] bounce: from:341502(slow:0)to:0 map:0 
unmap:0 sync:341502
[  213.840311] SWIOTLB is 4% full
[  218.840214] 0 [budget_av 0000:00:01.0] bounce: from:320117(slow:0)to:0 map:0 
unmap:0 sync:320117
[  218.840354] SWIOTLB is 4% full
[  223.840238] 0 [budget_av 0000:00:01.0] bounce: from:299390(slow:0)to:0 map:0 
unmap:0 sync:299390
[  223.840373] SWIOTLB is 4% full
[  228.841415] 0 [budget_av 0000:00:01.0] bounce: from:298732(slow:0)to:0 map:0 
unmap:0 sync:298732
[  228.841560] SWIOTLB is 4% full
[  233.840705] 0 [budget_av 0000:00:00.0] bounce: from:299061(slow:0)to:0 map:0 
unmap:0 sync:299061
[  233.840844] SWIOTLB is 4% full
[  238.840145] 0 [budget_av 0000:00:01.0] bounce: from:293468(slow:0)to:0 map:0 
unmap:0 sync:293468
[  238.840280] SWIOTLB is 4% full

-----Ursprüngliche Nachricht-----
Von: Konrad Rzeszutek Wilk [mailto:konrad@xxxxxxxxxx] 
Gesendet: Freitag, 2. Dezember 2011 16:24
An: Konrad Rzeszutek Wilk
Cc: Ian Campbell; xen-devel; Carsten Schiers; zhenzhong.duan@xxxxxxxxxx; 
lersek@xxxxxxxxxx
Betreff: Re: [Xen-devel] Load increase after memory upgrade (part2)

> > > > > That is a puzzle. It should not. The code is very much the same - both
> > > > > use the generic SWIOTLB which has not changed for years.
> > > > 
> > > > The swiotlb-xen used by classic-xen kernels (which I assume is what
> > > > Carsten means by "Xenified") isn't exactly the same as the stuff in
> > > > mainline Linux, it's been heavily refactored for one thing. It's not
> > > > impossible that mainline is bouncing something it doesn't really need
> > > > to.
> > > 
> > > The usage, at least with 'pci_alloc_coherent' is that there is no bouncing
> > > being done. The alloc_coherent will allocate a nice page, underneath the 
> > > 4GB
> > > mark and give it to the driver. The driver can use it as it wishes and 
> > > there
> > > is no need to bounce buffer.
> > 
> > Oh, I didn't realise dma_alloc_coherent was part of swiotlb now. Only a
> > subset of swiotlb is in use then, all the bouncing stuff _should_ be
> > idle/unused -- but has that been confirmed?
> 
> Nope. I hope that the diagnostic patch I have in mind will prove/disprove 
> that.
> Now I just need to find a moment to write it :-)

Done!

Carsten, can you please patch your kernel with this hacky patch and
when you have booted the new kernel, just do

modprobe dump_swiotlb

it should give an idea of how many bounces are happening, coherent
allocations, syncs, and so on.. along with the last driver that
did those operations.



_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel