Re: [Xen-users] add_random must be set to 1 for me - Archlinux HVM x64 - XenServer 7 Latest Patched

[WebDawg <webdawg@xxxxxxxxx>]

On Tue, Oct 25, 2016 at 6:29 AM, Austin S. Hemmelgarn
<ahferroin7@xxxxxxxxx> wrote:
> On 2016-10-24 14:53, WebDawg wrote:
>>
>> On Wed, Oct 19, 2016 at 2:23 PM, Austin S. Hemmelgarn
>> <ahferroin7@xxxxxxxxx> wrote:
>> So adding 3 more vCPU for a total of 4x on the domU, this just by
>> itself speeds up the dd write to xvda to 20MB a second.  But, the IO
>> load also adds a sy aka system cpu time load to almost all the CPU's.
>> (the CPU load has been sy load the entire time) All in all it sticks
>> to about 200-300% CPU use at this point.
>
>
>> The only thing that changes anything at this point is to add the
>> oflag=direct to dd.  When I add that CPU use dramatically lowers and
>> write speed goes much higher.  Still the CPU use compared to debian is
>> no change.
>
> OK, this suggests the issue is somewhere in the caching in the guest OS.  My
> first thoughts knowing that are:
> 1. How much RAM does the VM have?
> 2. What value does `cat /proc/sys/vm/vfs_cache_pressure` show?
> 3. Are you doing anything with memory ballooning?

 /proc/sys/vm/vfs_cache_pressure shows a value of 100 on both domU's.

THE BALLOONING IS THE ANSWER.

Okay,

So I do not know what the deal is but when you mentioned ballooning I
was looking at the memory settings of the domU.  These are the
settings that I had:

Static:  128 MiB/ 2 GiB
Dynamic:  2 GiB/ 4 GiB

I cannot even set this at command line.  I wanted to replicate the
problem after I fixed it and tried this:

xe vm-memory-limits-set dynamic-max=400000000 dynamic-min=200000000
static-max=200000000 static-min=16777216 name-label=domU-name

Error code: MEMORY_CONSTRAINT_VIOLATION

Error parameters: Memory limits must satisfy: static_min <=
dynamic_min <= dynamic_max <= static_max

The dynamic MAX was bigger then the static MAX, which is impossible to
set, but somehow happened.

I do not know if it happened from import from XenServer 6.5 to
XenServer 7, or the multiple software products I was using to manage
it or just something got corrupted.

So I have been checking it all out after setting everything to this:

Static:  128 MiB/ 2 GiB
Dynamic:  2 GiB/ 2 GiB

It is all working as expected now!

Like I said...I cannot understand how the dynamic MAX was bigger then
the static MAX when XenServer does not allow you to set this.  Does
anyone have any experience in setting these bad settings and can
explain why I was having such bad CPU use issues?

>>
>> Debian is:  3.16.0-4-amd64
>>
>> archlinux is:  4.8.4-1-ARCH #1 SMP PREEMPT Sat Oct 22 18:26:57 CEST
>> 2016 x86_64 GNU/Linux
>>
>>
>> scsi_mod.use_blk_mq=0 and it looks like it did nothing.  My
>> /proc/cmdline shows that it is there and it should be doing
>> something....but my scheduler setting in the queue dir still says
>> none.
>>
>> I think looking into this that the none is a result of the xen PVHVM
>> block front driver?
>
> Probably.  I've not been able to find any way to turn it off for the Xen PV
> block device driver (which doesn't really surprise me, xen-blkfront has
> multiple (virtual) 'hardware' queues, and stuff like that is exactly what
> blk-mq was designed to address (although it kind of sucks for anything like
> that except NVMe devices right now)).
>

If I remember too, the kernel line stuff to tune this, is not in the
docs yet :/  At least the ones I was looking at.

>>
>>>>
>>>>
>>>> If someone could shed some insight why enabling IO generation/linking
>>>> of timing/entropy data to /dev/random makes the 'system work' this
>>>> would be great.  Like I said, I am just getting into this and I will
>>>> be doing more tuning if I can.
>>>
>>>
>> ONCE AGAIN, I am wrong here.  add_random does nothing to help me
>> anymore.  In fact I cannot find any setting under queue that does
>> anything to help, at least in what I am trying to fix.
>>
>> I am sorry for this false information.
>>
>>> I'm kind of surprised at this though, since I've
>>> got half a dozen domains running fine with blk-mq getting within 1% of
>>> the
>>> disk access speed the host sees (and the host is using blk-mq too, both
>>> in
>>> the device-mapper layer, and the lower block layer).  Some info about the
>>> rest of the storage stack might be helpful (ie, what type of backing
>>> storage
>>> are you using for the VM disks (on LVM, MD RAID, flat partitions, flat
>>> files, etc), what Xen driver (raw disk, blktap, something else?), and
>>> what
>>> are you accessing in the VM (raw disk, partition, LVM volume, etc))?
>>
>>
>> This is a RAID 6 SAS Array.
>>
>> The kernel that I am using (archlinux: linux), is all vanilla except
>> for, it looks like, one patch:
>>
>>
>> https://git.archlinux.org/svntogit/packages.git/tree/trunk?h=packages/linux
>>
>> That patch changes from 7 to
>> CONSOLE_LOGLEVEL_DEFAULT 4
>>
>> These are the results from some tests:
>>
>>  dd if=/dev/zero of=/root/test2 bs=1M oflag=direct
>>  19.7 MB/s
>>  CPU:  20%
>>
>>  dd if=/dev/zero of=/root/test2 bs=1M
>>  2.5 MB/s
>>  CPU:  100%
>
> This brings one other thing to mind: What filesystem are you using in the
> domU?  My guess is that this is some kind of interaction between the
> filesystem and the blk-mq code.  One way to check that would be to try
> writing directly to a disk from the domU instead of through the filesystem.
>

I am using ext4.

>>
>> This is sy CPU / system CPU use; so something in the kernel?
>>
>> One the debian domU almost no CPU is hit.
>>
>> I am also thinking that 20MB/s is bad in general for my RAID6 as
>> almost nothing is reading and writing to it.  But one thing at a time
>> and the only reason I mention it, is that it might help to figure out
>> this issue.
>
>

So, now that it looks like I have figured out what is wrong, but not
figured out how it got that wrong:  Does anyone have any pointers for
increasing the speed of the Local Storage array?  I know I can add the
backup battery, but even without that...a SAS RAID6 running at 20mb a
second in domU seems so slow....

In dom0 I get about 20-21 mb a second with dd oflag=direct

I think XenServer or Xen has some type of disk io load balancing stuff
or am I wrong?

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxx
https://lists.xen.org/xen-users