[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [PATCH 0/9] xen/x86: PVH Dom0 fixes and fallout adjustments

  • To: Jan Beulich <jbeulich@xxxxxxxx>
  • From: Roger Pau Monné <roger.pau@xxxxxxxxxx>
  • Date: Tue, 14 Sep 2021 14:41:06 +0200
  • Arc-authentication-results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=citrix.com; dmarc=pass action=none header.from=citrix.com; dkim=pass header.d=citrix.com; arc=none
  • Arc-message-signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=nmoya7Wy3/dlPb46qHqs9Pc/h7Q4JGS/lDNSIMdu+4o=; b=EYtE/PyPNi4PWDX7UgwMePtzJyealh66xG21V/8teMH6x8h7xzIGSfgvbbe9S9nwHckGvKbV70do5J0iaJxgzQLpIegQbVveAx1dpwKP/LWoe/30PHdMB646Lg+H8Va+gNjTr5Qc0wwSFZOscDJ9ml7fYStgks9z7K51YoNl4hMA5gWwtEN4uzH5jiCCAiWgL8on+s69+AdvsEtiWcAO9sWLuTTTmkXjsoXU+TZ+qxyIRfaMta84BADMxF243skAx8XRw5aLRIZaTiwKfvelwUl0mlmdjbwJk8PcYpPqd5rnQH8gjXvQNBHrbasDBxPkHxQhqGdcNIDjbxvjjrkrKw==
  • Arc-seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=TuztZ2yH4zUyAlzu1dNi2duolQlh4BUaatOU9bN6YYBcVUFhuguKN/quekcL+ASke0GN8LxLG3qYdOVmARMstGfAERTAxC5RtasTK+uz8A/6lYLAKats6fVCzO8VmAC62zeqbMWA3xoRsCd6U/xf4vHp0AHGUqLUUyB1L6nTHEyhAIcBifQnwtddAY08MvYcnETXQSj2/qPIv5lZH3jY8piG1jyVHRtPAjupLBrJVf1RvMcciPTQEn6TWpJe0L5lv2niJCmeYwYwPt+wtyRvSKDF5SC6T0wRbX6WIVgiRFZ9S/9IoXB7AdbZ1nA/e0qBf6c+MxYUBNxyp6pdvODeFQ==
  • Authentication-results: esa2.hc3370-68.iphmx.com; dkim=pass (signature verified) header.i=@citrix.onmicrosoft.com
  • Cc: Juergen Gross <jgross@xxxxxxxx>, Boris Ostrovsky <boris.ostrovsky@xxxxxxxxxx>, Stefano Stabellini <sstabellini@xxxxxxxxxx>, lkml <linux-kernel@xxxxxxxxxxxxxxx>, "xen-devel@xxxxxxxxxxxxxxxxxxxx" <xen-devel@xxxxxxxxxxxxxxxxxxxx>
  • Delivery-date: Tue, 14 Sep 2021 12:41:22 +0000
  • Ironport-data: A9a23:GZ/FLqoWHKaT7yZjuo5a4P5nLOleBmKPYhIvgKrLsJaIsI4StFCzt garIBmPM6mCZjDxfthxPo7n/RgG7ZTcnYMwTQBkrSs0EigS8ZuZCYyVIHmrMnLJJKUvbq7GA +byyDXkBJppJpMJjk71atANlZT4vE2xbuKU5NTsY0idfic5Dnd84f5fs7Rh2Ncw0IHlW1rlV e7a+KUzBnf0g1aYDUpMg06zgEsHUCPa4W5wUvQWPJinjXeG/5UnJMt3yZKZdhMUdrJ8DO+iL 9sv+Znilo/vE7XBPfv++lrzWhVirrc/pmFigFIOM0SpqkAqSiDfTs/XnRfTAKtao2zhojx/9 DlCnbKLVxlqOvCUobQAcytlS31BJ6sBqaCSdBBTseTLp6HHW37lwvEoB0AqJ4wIvO1wBAmi9 9RBdmpLNErawbvrnvTrEYGAhex6RCXvFJkYtXx6iynQEN4tQIzZQrWM7thdtNs1rp0UQ6iHO JdHAdZpRE7RaiBsAkg4M79g2/2ulHavWiBjh03A8MLb5ECMlVcsgdABKuH9YceWTM9YmkKZo GPu/GnjBBwectuFxlKt9nOqm/+Kni7hXo8WPKO3++Qsg1CJwGEXThoMWjOTsfS/z0KzRd9bA 0gV4TY167g/8lSxSdvwVAH+p2SL1jYeUddNF+wx6CmW17HZpQ2eAwAsTDFbb8c9nNQrXjFs3 ViM9/vjAiZuq/uSUm6H8amPriKaPjIcJmsPIyQDSGM4D8LL+d9pyEiVF5A6TfDz3oad9SzML y6ighMgmfYX0JYyh7ibz22f3w22oN+YUVtgjunIZV5J/j+Vdab8OdfxtAmEsqgZRGqKZgLe5 ylfwqBy+MhLVMvUxXLXGI3hCZn0v67tDdHKvbJ483DNHRyW8ni/dMh75DhkLS+F2e5VJGe0P Cc/Ve5XjaK/3UdGj4csOOpd6OxwlMAM8OgJsNiOMLKihbArKGe6ENlGPxL44owUuBFEfVsD1 XKnnSCEVydy5UNPl2Heegvg+eVzmnBWKZ37HMimp/hY7VZuTCHMEupUWLd/Rss48LmFsG3oH yV3bpDRoyizpNbWO3GNmaZKdAhiBSFiWfje9pwGHsbec1EOMDxwVJfsLUYJJtUNc1J9zbyTo BlQmyZwlTLCuJEwAV7WMysyNOy2Bs8XQLBSFXVEAGtEEkMLOO6HxKwea4E2bf8g8ulixuRzV P4LZ4OLBfEnd9gN0211gUDVoNMweRK1qxiJOib5MjEzc4Q5H17C+8P+fxup/y4LV3Llucw7q rym9wXaXZtcGFgyUJeIMKqinwGroHwQuONuRE+UcNNdT1rhrdpxICvrg/5pf8xVcUffxiGX3 hq9CAsDobWfuJc89dTE3PjWr4qgH+ZkMFBdGm3XseS/OSXApzLxyo5cSueYOzvaUTqsqqmlY OxUydD6MeEGwwkW49YtTe4zwPtntdX1prJcwgB1J1nxbgymWuF6P32L/chTrakRlLVXjhS7B xCU8d5ANLTXZM68SAwNJBAoZ/io3O0PnmWA9uw8JUj36XMl/LeDVkkObRCAhDYEcelwOYIhh +wgpNQX+0q0jR9zaoSKiSVd9mKtKH0cUvp46sFGUdGz0gd7mEtfZZH8CzPt5MDdYtpBBUAmP zuIifeQnL9b3EfDLyI+GHWlMTCxXnjSVMSmFGM/Gmk=
  • Ironport-hdrordr: A9a23:5Hwc6aB271b39C/lHejesseALOsnbusQ8zAXPh9KJyC9I/b2qy nxppgmPEfP+UossHFJo6HlBEDyewKiyXcV2/heAV7GZmOGhILGFvAb0WKP+UyDJ8S6zJ8h6U 4CSdk+NDSTNykAsS+S2mDReLtBsbq6GeKT9J3jJhxWPGZXgtRbnn5E43GgYytLrWd9dP8EPa vZwvACiyureHwRYMj+LGICRfL/q9rCk4+jSQIaBjY8gTP+zg+A2frfKVy1zx0eWzRAzfMJ6m 7eiTH04a2lrrWS1gLc7WnO9J5b8eGRiOerRfb8z/T9GA+czTpAV74RHYFqewpF+d1H3Wxa1O UkZS1QZ/ibpUmhJV1d6iGdpTUImAxemkMKj2XowUfLkIjBXzQ9BNNGhYVFNjXky2dIhqAg7I t7m1uDsZxZFBXBmzm4wePpeVVFqmqYyEBSy9L6qRRkINQjgXtq3NAiFQpuYec9NTO/54Y9HO Z0CsbAoP5QbFOBdnjc+nJi2dq2Qx0Ib1q7q2U5y4eoOgJt7TpEJoojtbsit2ZF8Ih4R4hP5u zCPKgtnLZSTtUOZaY4AOsaW8O4BmHEXBqJaQupUBvaPbBCP2iIp4/84b0z6u3vcJsUzIEqkJ CEVF9Dr2Y9d0/nFMXL1pxW9RLGRnm7QF3Wu41jzok8vqe5SKvgMCWFRlxrm8y8o+8HCsmeQP q3MII+OY6qEYIvI/cB4+TaYegnFZAzarxmhj8LYSP5niuQEPyYigXySoenGIbQ
  • Ironport-sdr: QTZxJnGufd4l0xrPjj0NRz41Wgfh6cBGcMdq2xC+ZU25WCKXrJNeW1Pxyh6IJPtkl4RBtDwU9F DT5bFgbBRvECF2RKOX989Ptm2pGfLQ9Fzewhp2Ox3x53Fp1COM8LepRFk5hd7EqwEgPNMbK/TN psgmWz/PCUwT4ME6u/afh4tBpZ/oJzNYQP7XAESOuR6iwx/umt46IGuv+zM1QhVTzX7Hj1A0PW giyRU6/S/W1K0OsaqaRkRMtKfFfiYR1KY9xX0nRkR6b88Y6ERPz/lnHH6vMqk2cYEFYmPlT7bg 3klFld9A9om1RDN/WLho5+48
  • List-id: Xen developer discussion <xen-devel.lists.xenproject.org>

On Tue, Sep 14, 2021 at 01:58:29PM +0200, Jan Beulich wrote:
> On 14.09.2021 13:15, Roger Pau Monné wrote:
> > On Tue, Sep 14, 2021 at 11:03:23AM +0200, Jan Beulich wrote:
> >> On 14.09.2021 10:32, Roger Pau Monné wrote:
> >>> On Tue, Sep 07, 2021 at 12:04:34PM +0200, Jan Beulich wrote:
> >>>> In order to try to debug hypervisor side breakage from XSA-378 I found
> >>>> myself urged to finally give PVH Dom0 a try. Sadly things didn't work
> >>>> quite as expected. In the course of investigating these issues I actually
> >>>> spotted one piece of PV Dom0 breakage as well, a fix for which is also
> >>>> included here.
> >>>>
> >>>> There are two immediate remaining issues (also mentioned in affected
> >>>> patches):
> >>>>
> >>>> 1) It is not clear to me how PCI device reporting is to work. PV Dom0
> >>>>    reports devices as they're discovered, including ones the hypervisor
> >>>>    may not have been able to discover itself (ones on segments other
> >>>>    than 0 or hotplugged ones). The respective hypercall, however, is
> >>>>    inaccessible to PVH Dom0. Depending on the answer to this, either
> >>>>    the hypervisor will need changing (to permit the call) or patch 2
> >>>>    here will need further refinement.
> >>>
> >>> I would rather prefer if we could limit the hypercall usage to only
> >>> report hotplugged segments to Xen. Then Xen would have to scan the
> >>> segment when reported and add any devices found.
> >>>
> >>> Such hypercall must be used before dom0 tries to access any device, as
> >>> otherwise the BARs won't be mapped in the second stage translation and
> >>> the traps for the MCFG area won't be setup either.
> >>
> >> This might work if hotplugging would only ever be of segments, and not
> >> of individual devices. Yet the latter is, I think, a common case (as
> >> far as hotplugging itself is "common").
> > 
> > Right, I agree to use hypercalls to report either hotplugged segments
> > or devices. However I would like to avoid mandating usage of the
> > hypercall for non-hotplug stuff, as then OSes not having hotplug
> > support don't really need to care about making use of those
> > hypercalls.
> > 
> >> Also don't forget about SR-IOV VFs - they would typically not be there
> >> when booting. They would materialize when the PF driver initializes
> >> the device. This is, I think, something that can be dealt with by
> >> intercepting writes to the SR-IOV capability.
> > 
> > My plan was to indeed trap SR-IOV capability accesses, see:
> > 
> > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore.kernel.org%2Fxen-devel%2F20180717094830.54806-1-roger.pau%40citrix.com%2F&amp;data=04%7C01%7Croger.pau%40citrix.com%7C35d2502d0128484e229e08d97777087f%7C335836de42ef43a2b145348c2ee9ca5b%7C0%7C0%7C637672175399546062%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=sSeE%2F4wEo5%2Fplkj2yH%2B1kpHi5c15lxJxeUxx6Cbyr4s%3D&amp;reserved=0
> > 
> > I just don't have time ATM to continue this work.
> > 
> >> But I wonder whether
> >> there might be other cases where devices become "visible" only while
> >> the Dom0 kernel is already running.
> > 
> > I would consider those kind of hotplug devices, and hence would
> > require the use of the hypercall in order to notify Xen about them.
> So what does this mean for the one patch? Should drivers/xen/pci.c
> then be built for PVH (and then have logic added to filter boot
> time device discovery), or should I restrict this to be PV-only (and
> PVH would get some completely different logic added later)?

I think we can reuse the same hypercalls for PVH, and maybe the same
code in Linux. For PVH we just need to be careful to make the
hypercalls before attempting to access the BARs (or the PCI
configuration space for the device) since there won't be any traps
setup, and BARs won't be mapped on the p2m.

It might be easier for Linux to just report every device it finds to
Xen, like it's currently done for PV dom0, instead of filtering on
whether the device has been hotplugged.

> >>>> 2) Dom0, unlike in the PV case, cannot access the screen (to use as a
> >>>>    console) when in a non-default mode (i.e. not 80x25 text), as the
> >>>>    necessary information (in particular about VESA-bases LFB modes) is
> >>>>    not communicated. On the hypervisor side this looks like deliberate
> >>>>    behavior, but it is unclear to me what the intentions were towards
> >>>>    an alternative model. (X may be able to access the screen depending
> >>>>    on whether it has a suitable driver besides the presently unusable
> >>>>    /dev/fb<N> based one.)
> >>>
> >>> I had to admit most of my boxes are headless servers, albeit I have
> >>> one NUC I can use to test gfx stuff, so I don't really use gfx output
> >>> with Xen.
> >>>
> >>> As I understand such information is fetched from the BIOS and passed
> >>> into Xen, which should then hand it over to the dom0 kernel?
> >>
> >> That's how PV Dom0 learns of the information, yes. See
> >> fill_console_start_info(). (I'm in the process of eliminating the
> >> need for some of the "fetch from BIOS" in Xen right now, but that's
> >> not going to get us as far as being able to delete that code, no
> >> matter how much in particular Andrew would like that to happen.)
> >>
> >>> I guess the only way for Linux dom0 kernel to fetch that information
> >>> would be to emulate the BIOS or drop into realmode and issue the BIOS
> >>> calls?
> >>
> >> Native Linux gets this information passed from the boot loader, I think
> >> (except in the EFI case, as per below).
> >>
> >>> Is that an issue on UEFI also, or there dom0 can fetch the framebuffer
> >>> info using the PV EFI interface?
> >>
> >> There it's EFI boot services functions which can be invoked before
> >> leaving boot services (in the native case). Aiui the PVH entry point
> >> lives logically past any EFI boot services interaction, and hence
> >> using them is not an option (if there was EFI firmware present in Dom0
> >> in the first place, which I consider difficult all by itself - this
> >> can't be the physical system's firmware, but I also don't see where
> >> virtual firmware would be taken from).
> >>
> >> There is no PV EFI interface to obtain video information. With the
> >> needed information getting passed via start_info, PV has no need for
> >> such, and I would be hesitant to add a fundamentally redundant
> >> interface for PVH. The more that the information needed isn't EFI-
> >> specific at all.
> > 
> > I think our only option is to expand the HVM start info information to
> > convey that data from Xen into dom0.
> PHV doesn't use the ordinary start_info, does it?

No, it's HVM start info as described in:


We have already extended it once to add a memory map, we could extend
it another time to add the video information.




Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.