[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-devel] [BUG] After upgrade to Xen 4.12.0 iommu=no-igfx


  • To: Roman Shaposhnik <roman@xxxxxxxxxx>, <xen-devel@xxxxxxxxxxxxxxxxxxxx>
  • From: Andrew Cooper <andrew.cooper3@xxxxxxxxxx>
  • Date: Sat, 20 Jul 2019 17:39:29 +0100
  • Authentication-results: esa6.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none; spf=None smtp.pra=andrew.cooper3@xxxxxxxxxx; spf=Pass smtp.mailfrom=Andrew.Cooper3@xxxxxxxxxx; spf=None smtp.helo=postmaster@xxxxxxxxxxxxxxx
  • Autocrypt: addr=andrew.cooper3@xxxxxxxxxx; prefer-encrypt=mutual; keydata= mQINBFLhNn8BEADVhE+Hb8i0GV6mihnnr/uiQQdPF8kUoFzCOPXkf7jQ5sLYeJa0cQi6Penp VtiFYznTairnVsN5J+ujSTIb+OlMSJUWV4opS7WVNnxHbFTPYZVQ3erv7NKc2iVizCRZ2Kxn srM1oPXWRic8BIAdYOKOloF2300SL/bIpeD+x7h3w9B/qez7nOin5NzkxgFoaUeIal12pXSR Q354FKFoy6Vh96gc4VRqte3jw8mPuJQpfws+Pb+swvSf/i1q1+1I4jsRQQh2m6OTADHIqg2E ofTYAEh7R5HfPx0EXoEDMdRjOeKn8+vvkAwhviWXTHlG3R1QkbE5M/oywnZ83udJmi+lxjJ5 YhQ5IzomvJ16H0Bq+TLyVLO/VRksp1VR9HxCzItLNCS8PdpYYz5TC204ViycobYU65WMpzWe LFAGn8jSS25XIpqv0Y9k87dLbctKKA14Ifw2kq5OIVu2FuX+3i446JOa2vpCI9GcjCzi3oHV e00bzYiHMIl0FICrNJU0Kjho8pdo0m2uxkn6SYEpogAy9pnatUlO+erL4LqFUO7GXSdBRbw5 gNt25XTLdSFuZtMxkY3tq8MFss5QnjhehCVPEpE6y9ZjI4XB8ad1G4oBHVGK5LMsvg22PfMJ ISWFSHoF/B5+lHkCKWkFxZ0gZn33ju5n6/FOdEx4B8cMJt+cWwARAQABtClBbmRyZXcgQ29v cGVyIDxhbmRyZXcuY29vcGVyM0BjaXRyaXguY29tPokCOgQTAQgAJAIbAwULCQgHAwUVCgkI CwUWAgMBAAIeAQIXgAUCWKD95wIZAQAKCRBlw/kGpdefoHbdD/9AIoR3k6fKl+RFiFpyAhvO 59ttDFI7nIAnlYngev2XUR3acFElJATHSDO0ju+hqWqAb8kVijXLops0gOfqt3VPZq9cuHlh IMDquatGLzAadfFx2eQYIYT+FYuMoPZy/aTUazmJIDVxP7L383grjIkn+7tAv+qeDfE+txL4 SAm1UHNvmdfgL2/lcmL3xRh7sub3nJilM93RWX1Pe5LBSDXO45uzCGEdst6uSlzYR/MEr+5Z JQQ32JV64zwvf/aKaagSQSQMYNX9JFgfZ3TKWC1KJQbX5ssoX/5hNLqxMcZV3TN7kU8I3kjK mPec9+1nECOjjJSO/h4P0sBZyIUGfguwzhEeGf4sMCuSEM4xjCnwiBwftR17sr0spYcOpqET ZGcAmyYcNjy6CYadNCnfR40vhhWuCfNCBzWnUW0lFoo12wb0YnzoOLjvfD6OL3JjIUJNOmJy RCsJ5IA/Iz33RhSVRmROu+TztwuThClw63g7+hoyewv7BemKyuU6FTVhjjW+XUWmS/FzknSi dAG+insr0746cTPpSkGl3KAXeWDGJzve7/SBBfyznWCMGaf8E2P1oOdIZRxHgWj0zNr1+ooF /PzgLPiCI4OMUttTlEKChgbUTQ+5o0P080JojqfXwbPAyumbaYcQNiH1/xYbJdOFSiBv9rpt TQTBLzDKXok86LkCDQRS4TZ/ARAAkgqudHsp+hd82UVkvgnlqZjzz2vyrYfz7bkPtXaGb9H4 Rfo7mQsEQavEBdWWjbga6eMnDqtu+FC+qeTGYebToxEyp2lKDSoAsvt8w82tIlP/EbmRbDVn 7bhjBlfRcFjVYw8uVDPptT0TV47vpoCVkTwcyb6OltJrvg/QzV9f07DJswuda1JH3/qvYu0p vjPnYvCq4NsqY2XSdAJ02HrdYPFtNyPEntu1n1KK+gJrstjtw7KsZ4ygXYrsm/oCBiVW/OgU g/XIlGErkrxe4vQvJyVwg6YH653YTX5hLLUEL1NS4TCo47RP+wi6y+TnuAL36UtK/uFyEuPy wwrDVcC4cIFhYSfsO0BumEI65yu7a8aHbGfq2lW251UcoU48Z27ZUUZd2Dr6O/n8poQHbaTd 6bJJSjzGGHZVbRP9UQ3lkmkmc0+XCHmj5WhwNNYjgbbmML7y0fsJT5RgvefAIFfHBg7fTY/i kBEimoUsTEQz+N4hbKwo1hULfVxDJStE4sbPhjbsPCrlXf6W9CxSyQ0qmZ2bXsLQYRj2xqd1 bpA+1o1j2N4/au1R/uSiUFjewJdT/LX1EklKDcQwpk06Af/N7VZtSfEJeRV04unbsKVXWZAk uAJyDDKN99ziC0Wz5kcPyVD1HNf8bgaqGDzrv3TfYjwqayRFcMf7xJaL9xXedMcAEQEAAYkC HwQYAQgACQUCUuE2fwIbDAAKCRBlw/kGpdefoG4XEACD1Qf/er8EA7g23HMxYWd3FXHThrVQ HgiGdk5Yh632vjOm9L4sd/GCEACVQKjsu98e8o3ysitFlznEns5EAAXEbITrgKWXDDUWGYxd pnjj2u+GkVdsOAGk0kxczX6s+VRBhpbBI2PWnOsRJgU2n10PZ3mZD4Xu9kU2IXYmuW+e5KCA vTArRUdCrAtIa1k01sPipPPw6dfxx2e5asy21YOytzxuWFfJTGnVxZZSCyLUO83sh6OZhJkk b9rxL9wPmpN/t2IPaEKoAc0FTQZS36wAMOXkBh24PQ9gaLJvfPKpNzGD8XWR5HHF0NLIJhgg 4ZlEXQ2fVp3XrtocHqhu4UZR4koCijgB8sB7Tb0GCpwK+C4UePdFLfhKyRdSXuvY3AHJd4CP 4JzW0Bzq/WXY3XMOzUTYApGQpnUpdOmuQSfpV9MQO+/jo7r6yPbxT7CwRS5dcQPzUiuHLK9i nvjREdh84qycnx0/6dDroYhp0DFv4udxuAvt1h4wGwTPRQZerSm4xaYegEFusyhbZrI0U9tJ B8WrhBLXDiYlyJT6zOV2yZFuW47VrLsjYnHwn27hmxTC/7tvG3euCklmkn9Sl9IAKFu29RSo d5bD8kMSCYsTqtTfT6W4A3qHGvIDta3ptLYpIAOD2sY3GYq2nf3Bbzx81wZK14JdDDHUX2Rs 6+ahAA==
  • Cc: Paul Durrant <Paul.Durrant@xxxxxxxxxx>, Wei Liu <wl@xxxxxxx>, Jan Beulich <JBeulich@xxxxxxxx>, Roger Pau Monne <roger.pau@xxxxxxxxxx>
  • Delivery-date: Sat, 20 Jul 2019 16:40:11 +0000
  • Ironport-sdr: GP7T8pwSlhcrPlrzfH61BLkQtekBDmFqk552quVmfJ542erLUspKCfHBmxsYTlXkCZ1BUyYewO oskvZEG0rg9m1XDMr9X0Oam1qW2FogV9qZdUU6PS+laVl9HyxjyYNQJ2198yFrwuBWfHVmTUTx F7XSSbHNL3+dKXcEqUPefb4nW9ccS04acry+kRsR3XsKu8pydRclbtv7hUsgtuEjXmvIqzxCV8 xTaXtdz/H9QUpij+/iQZu77Lkfg91EpFmcf75QiL2KJfaO38BbM4WuRRa2U9nOicNXpHqGjW5c 9p0=
  • List-id: Xen developer discussion <xen-devel.lists.xenproject.org>
  • Openpgp: preference=signencrypt

On 19/07/2019 20:31, Roman Shaposhnik wrote:
Hi!

we're using Xen on Advantech ARK-2250 Embedded Box PC:
    https://www.elmark.com.pl/web/uploaded/karty_produktow/advantech/ark-2250l/ark-2250l_instrukcja-uzytkownika.pdf

After upgrading to Xen 4.12.0 from 4.11.0 we now have to utilize  iommu=no-igfx
workaround as per:
     https://xenbits.xen.org/docs/unstable/misc/xen-command-line.html#iommu

Without the workaround the screen appears to be garbled with colored
static noise and the following message keeps showing up:
(XEN) printk: 26235 messages suppressed.
(XEN) [VT-D]DMAR:[DMA Read] Request device [0000:00:02.0] fault addr
8e43c000, iommu reg = ffff82c00021d000
(XEN) printk: 26303 messages suppressed.
(XEN) [VT-D]DMAR:[DMA Read] Request device [0000:00:02.0] fault addr
8e2c6000, iommu reg = ffff82c00021d000

Once iommu=no-igfx is applied the box can boot fine.

At the end of this email, you can see a full log of the box booting
all the way into Dom0 with iommu=no-igfx applied. I am also attaching
similar log without no-igfx

Please let me know if you need any more information to help us diagnose this.

This will be a consequence of trying to remove various pieces of stupidity with the preexisting IOMMU logic, in an attempt to unify the PV and PVH paths.

As for the symptoms you're seeing, that is because the GPU is not being given access to the RAM stolen for graphics purposes.

Picking the log apart:

(XEN) EFI RAM map:
(XEN)  0000000000000000 - 0000000000058000 (usable)
(XEN)  0000000000058000 - 0000000000059000 (reserved)
(XEN)  0000000000059000 - 000000000009f000 (usable)
(XEN)  000000000009f000 - 00000000000a0000 (reserved)
(XEN)  0000000000100000 - 000000008648a000 (usable)
(XEN)  000000008648a000 - 000000008648b000 (ACPI NVS)
(XEN)  000000008648b000 - 00000000864b5000 (reserved)
(XEN)  00000000864b5000 - 000000008c224000 (usable)
(XEN)  000000008c224000 - 000000008c528000 (reserved)
(XEN)  000000008c528000 - 000000008c736000 (usable)
(XEN)  000000008c736000 - 000000008cea7000 (ACPI NVS)
(XEN)  000000008cea7000 - 000000008d2ff000 (reserved)
(XEN)  000000008d2ff000 - 000000008d300000 (usable)
(XEN)  000000008d300000 - 000000008d400000 (reserved)
(XEN)  00000000e0000000 - 00000000f0000000 (reserved)
(XEN)  00000000fe000000 - 00000000fe011000 (reserved)
(XEN)  00000000fec00000 - 00000000fec01000 (reserved)
(XEN)  00000000fee00000 - 00000000fee01000 (reserved)
(XEN)  00000000ff000000 - 0000000100000000 (reserved)
(XEN)  0000000100000000 - 000000016e000000 (usable)
...
(XEN) Enabling APIC mode:  Flat.  Using 1 I/O APICs
(XEN) [VT-D]  RMRR address range 8d800000..8fffffff not in reserved memory; need "iommu_inclusive_mapping=1"?
(XEN) Switched to APIC driver x2apic_cluster.
...
(XEN) [VT-D]DMAR:[DMA Read] Request device [0000:00:02.0] fault addr 8e480000, iommu reg = ffff82c00021d000
(XEN) [VT-D]DMAR: reason 06 - PTE Read access is not set
(XEN) [VT-D]INTR-REMAP: Request device [0000:f0:1f.0] fault index 0, iommu reg = ffff82c00021f000
(XEN) [VT-D]INTR-REMAP: reason 22 - Present field in the IRTE entry is clear

The RMRR identified is a hole in the e820, and the range which is causing IOMMU faults.  Clearly it isn't being set up correctly.

First of all, can you check what effect booting with iommu_inclusive_mapping=1 has please?  While at it, iommu=debug would also be helpful.

Back to the log.  Strictly speaking, this is a violation of the VT-d spec.  Section 8.4 Reserved Memory Region Reporting Structure says:

"BIOS must report the RMRR reported memory addresses as reserved (or as EFI runtime) in the system memory map returned through methods such as INT15, EFI GetMemoryMap etc."

However, Xen's logic here is very broken, and in need of fixing.

For that message, it only checks the first and last address for being reserved, not the entire region, which will give it plenty of false negatives.

For RMRRs themselves, system firmware is well known for abiding by the spec [citation needed], but an RMRR must be honoured, because the entire purpose of them is to state "this device won't function without access to this area".

An RMRR in a hole, while a violation of the spec, is obviously fine to use in practice, so we should just accept it and stop complaining.

OTOH, RMRRs which hit other memory (particularly RAM) need more care, and probably want to force override the e820 to reserved.  Nothing good will come from trusting the e820 over the DMAR table here, seeing as there is clearly an error somewhere in the firmware-provided information.

However - I'm struggling to locate anywhere which actually walks dom0's RMRR list and inserts them into the IOMMU.  Anyone got any hints?

~Andrew
_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel

 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.