[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Revisiting Slow Apt (actually disk write)

Sorry for the long message, but wanted to give a lack-of-progress report

Did a lot of troubleshooting but it hasn’t gotten me very far:

System:  SuperMicro
              Xeon E3-1230 V3 @ 3.3 Ghz
             12 GB RAM
             Physical Disk:  4 x 2 TB ST2000NM0033 - Enterprise

Dom0 OS:  Debian Buster 10.10, kernel 4.19.0-16-amd64
                 Xen 4.11

DomUs:  All run debian buster as well. they are using Disk.Img for their swap and root drives.  All have at least 1 GB of RAM allocated.  My test VM has 2GB.

When I ran debian wheezy, I was not having this problem.

Dom0 is fast in disk read and writes.

Problem:   DomU's Seem to be very slow disk writes.  For example, if I just to an "apt update" this is my output:

Get:1 http://security.debian.org buster/updates InRelease [65.4 kB]
Get:2 http://ftp.us.debian.org/debian buster InRelease [122 kB]
Get:3 http://security.debian.org buster/updates/main Sources [195 kB]
Get:4 http://security.debian.org buster/updates/main amd64 Packages [297 kB]
Get:5 http://ftp.us.debian.org/debian buster/non-free Sources [85.7 kB]
Get:6 http://ftp.us.debian.org/debian buster/main Sources [7836 kB]
Get:7 http://ftp.us.debian.org/debian buster/contrib Sources [42.5 kB]
Get:8 http://ftp.us.debian.org/debian buster/main amd64 Packages [7907 kB]
Get:9 http://ftp.us.debian.org/debian buster/main Translation-en [5968 kB]
Get:10 http://ftp.us.debian.org/debian buster/contrib amd64 Packages [50.1 kB] Get:11 http://ftp.us.debian.org/debian buster/contrib Translation-en [44.2 kB] Get:12 http://ftp.us.debian.org/debian buster/non-free amd64 Packages [87.7 kB] Get:13 http://ftp.us.debian.org/debian buster/non-free Translation-en [88.9 kB]
Fetched 22.7 MB in 52min 3s (7277 B/s)
Reading package lists... Done

Notice that it says "52min 3s", this is something that Dom0 takes about 3 seconds to do to the same servers.  Meaning, it is not a network issue.

What I've tried:
 - I first went through the I/O tweaks from the xen wiki pages. Zero effect

 - Moved all the DomUs off to another server except one (tester).

 - Formatted and reinstalled entire OS from scratch on Dom0.

 - Tried using LVM disk instead of disk images.   Slight improvement, but still extremely slow.

 - Tried using physical disk, dedicating one drive entirely to the test DomU, mounting with "phys" in the config.   This was also a slight improvement, but still very slow disk writes.

"top" on the DomU shows WA very high while trying to get the update.  Drops to <1 when idle.  All other "top" values are <1.

I thought I solved it by going back to an earlier kernel, 4.19.0-14-amd64, and I tried doing that again and it had zero effect this time.

Is it possible that there is a kernel flag that is turned off that would effect VM disk writes?   Reading seems to be normal speed.   Writing to a network drive from the DomU is very fast as well.

Flag Info from /proc/cpu on Dom0

flags        : fpu de tsc msr pae mce cx8 apic sep mca cmov pat clflush acpi mmx fxsr sse sse2 ss ht syscall nx

lm constant_tsc rep_good nopl nonstop_tsc cpuid pni pclmulqdq monitor est ssse3 cx16 sse4_1 sse4_2

popcnt xsave avx f16c rdrand hypervisor lahf_lm cpuid_fault ssbd ibrs ibpb stibp fsgsbase erms xsaveopt md_clear



Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.