[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [PATCH] xen/console: do not drop serial output from the hardware domain


  • To: Jan Beulich <jbeulich@xxxxxxxx>
  • From: Roger Pau Monné <roger.pau@xxxxxxxxxx>
  • Date: Tue, 14 Jun 2022 10:41:20 +0200
  • Arc-authentication-results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=citrix.com; dmarc=pass action=none header.from=citrix.com; dkim=pass header.d=citrix.com; arc=none
  • Arc-message-signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=xGPGaFqBVOi5BK+qIiLCapLMrmlUB2cPOrnh/DVTg+g=; b=jsW/rkhS1lzDIaZtK3P8jF0KVRU9T/7Hhn0Qg5xZzBxcoFXyjw9bRZaIbKmXTooaZe2oV41DNjNmy7klE0ZUwPYGAhgp5UOEA60YazRJPECqiEI/WDwbyPOCgE7+3U7aW3NErxdM3gtyhcquTm+xfEkCYPLK1GTNWQAQ28fnUCxcINdN/4Q4rpuZfCoHQfdFA75WkWQaobLNEZMLfC8wxrZEoEGCSgmm9Ru70dMxxx/bXeC9EMiQJlu5aqpeet9w03f/qnCz9lv3t0uJgSPEhlCgvt+WbvRaRy6rfKQgM7OUcSElSTMesoLaKsWu0KDutDNQR1kkFwh1UuGjJVSyNA==
  • Arc-seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Scd/xg+AVDsTC2+JHQadlWQnDW41Lmc7DJfqqkhlZRJyPNgjG75zqi8v2SvztsDg/LShBghMTkPzV8Lsp5/CtnMXnCUPX7DRji6KW831usVNR+T6loqBHURuST8wmuJaSntgTWhvet0aMa1XkUxNnNBZl8rCY50LWqEQv9ljJRM7yEFThoYh8lJMp9O5egpNQOKrfMkdIHIjkFqfAH2/0jRjJAC6vpWQ5lIR7+EyMzl/OrYMIcJGkl2tBUUyLONobhmzUaKm30Md/077HV3B1Npg+u3jDKSlUizbSPY9BbPACLGgjUJm8E9rRlx7Adx8sfF8No8SixeD4whsTgGQOg==
  • Authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=citrix.com;
  • Cc: Andrew Cooper <andrew.cooper3@xxxxxxxxxx>, George Dunlap <george.dunlap@xxxxxxxxxx>, Julien Grall <julien@xxxxxxx>, Stefano Stabellini <sstabellini@xxxxxxxxxx>, Wei Liu <wl@xxxxxxx>, xen-devel@xxxxxxxxxxxxxxxxxxxx
  • Delivery-date: Tue, 14 Jun 2022 08:41:58 +0000
  • Ironport-data: A9a23:PeI9m6P+4NihD2nvrR26lsFynXyQoLVcMsEvi/4bfWQNrUpzgzJWz jcfDW2FMqrcajbxctl+Ooyw/UsEsZCGndVrSQto+SlhQUwRpJueD7x1DKtR0wB+jCHnZBg6h ynLQoCYdKjYdleF+lH1dOKJQUBUjclkfJKlYAL/En03FFYMpBsJ00o5wbZn29Iw2LBVPivW0 T/Mi5yHULOa82Yc3lI8s8pvfzs24ZweEBtB1rAPTagjUG32zhH5P7pGTU2FFFPqQ5E8IwKPb 72rIIdVXI/u10xF5tuNyt4Xe6CRK1LYFVDmZnF+A8BOjvXez8CbP2lS2Pc0MC9qZzu1c99Zx osRhKGMRhgQb4bulf1aVh59OA1SIvgTkFPHCSDXXc276WTjKiOp79AwSUY8MMsf5/p9BnxI+ boAMjcRYxufhuWwhrWmVu1rgcdlJ87uVG8dkig4kXeFUrB7EdaaG/WiCdxwhV/cguhUGvnTf YwBYCdHZxXceRxffFwQDfrSmc/33SKuL2MJ9jp5o4Icvnr+1zwowYL1OYfVWPy4R+lQhgGh8 zeuE2PRR0ty2Mak4TiP/2+oh+TPtTjmQ49UH7q9ntZonVmSy2o7GBAQE1yhrpGRkVWiUthSL 0gV/CsGrqUo8kGvCN7nUHWQv3qsrhMaHd1KHIUS+AyLj6bZ/QudLmwFVSJaLswrstcsQj4n3 UPPmMnmbQGDq5WQQHOZs7uR8zW7PHFNKXdYPHdUCwwY/9PkvYc/yArVScpuG7K0iduzHizsx zeNr241gLB7YdM36phXNGvv21qEzqUlhCZvum07gkrNAttFWbOY
  • Ironport-hdrordr: A9a23:SqKO8apPdhC+1ga1PCryv8waV5u5L9V00zEX/kB9WHVpm5Oj+v xGzc5w6farsl0ssREb9uxo9pPwJE800aQFmbX5Wo3SJzUO2VHYVb2KiLGP/9SOIU3DH4JmpM Rdmu1FeafN5DtB/LnHCWuDYrEdKbC8mcjH5Ns2jU0dKz2CA5sQkzuRYTzrdnGeKjM2Z6bQQ/ Gnl7d6TnebCD0qR/X+IkNAc/nIptXNmp6jSRkaByQ/4A3LqT+z8rb1HzWRwx9bClp0sPwf2F mAtza8yrSosvm9xBOZ/2jP765OkN+k7tdYHsSDhuUcNz2poAe1Y4ZKXaGEoVkO0amSwWdvtO OJjwYrPsx15X+UVmapoSH10w2l6zoq42+K8y7tvVLT5ejCAB4qActIgoxUNjHD7VA7gd162K VXm0qEqpt+F3r77WvAzumNcysvulu/oHIkn+JWpWdYS5EiZLhYqpFa1F9JEa0HADnx5OkcYa VT5fnnlbdrmG6hHjDkVjEF+q3uYp1zJGbKfqE6gL3a79AM90oJjXfxx6Qk7wI9HdwGOtx5Dt //Q9VVfYF1P7ErhJ1GdZc8qOuMexvwqEH3QRSvyWqOLtB1B1v977jK3Z4S2MaGPLQ18bpaou WybLofjx95R37T
  • List-id: Xen developer discussion <xen-devel.lists.xenproject.org>

On Tue, Jun 14, 2022 at 10:32:53AM +0200, Roger Pau Monné wrote:
> On Tue, Jun 14, 2022 at 10:10:03AM +0200, Jan Beulich wrote:
> > On 14.06.2022 08:52, Roger Pau Monné wrote:
> > > On Mon, Jun 13, 2022 at 03:56:54PM +0200, Jan Beulich wrote:
> > >> On 13.06.2022 14:32, Roger Pau Monné wrote:
> > >>> On Mon, Jun 13, 2022 at 11:18:49AM +0200, Jan Beulich wrote:
> > >>>> On 13.06.2022 11:04, Roger Pau Monné wrote:
> > >>>>> On Mon, Jun 13, 2022 at 10:29:43AM +0200, Jan Beulich wrote:
> > >>>>>> On 13.06.2022 10:21, Roger Pau Monné wrote:
> > >>>>>>> On Mon, Jun 13, 2022 at 09:30:06AM +0200, Jan Beulich wrote:
> > >>>>>>>> On 10.06.2022 17:06, Roger Pau Monne wrote:
> > >>>>>>>>> Prevent dropping console output from the hardware domain, since 
> > >>>>>>>>> it's
> > >>>>>>>>> likely important to have all the output if the boot fails without
> > >>>>>>>>> having to resort to sync_console (which also affects the output 
> > >>>>>>>>> from
> > >>>>>>>>> other guests).
> > >>>>>>>>>
> > >>>>>>>>> Do so by pairing the console_serial_puts() with
> > >>>>>>>>> serial_{start,end}_log_everything(), so that no output is dropped.
> > >>>>>>>>
> > >>>>>>>> While I can see the goal, why would Dom0 output be (effectively) 
> > >>>>>>>> more
> > >>>>>>>> important than Xen's own one (which isn't "forced")? And with this
> > >>>>>>>> aiming at boot output only, wouldn't you want to stop the 
> > >>>>>>>> overriding
> > >>>>>>>> once boot has completed (of which, if I'm not mistaken, we don't
> > >>>>>>>> really have any signal coming from Dom0)? And even during boot I'm
> > >>>>>>>> not convinced we'd want to let through everything, but perhaps just
> > >>>>>>>> Dom0's kernel messages?
> > >>>>>>>
> > >>>>>>> I normally use sync_console on all the boxes I'm doing dev work, so
> > >>>>>>> this request is something that come up internally.
> > >>>>>>>
> > >>>>>>> Didn't realize Xen output wasn't forced, since we already have rate
> > >>>>>>> limiting based on log levels I was assuming that non-ratelimited
> > >>>>>>> messages wouldn't be dropped.  But yes, I agree that Xen (non-guest
> > >>>>>>> triggered) output shouldn't be rate limited either.
> > >>>>>>
> > >>>>>> Which would raise the question of why we have log levels for 
> > >>>>>> non-guest
> > >>>>>> messages.
> > >>>>>
> > >>>>> Hm, maybe I'm confused, but I don't see a direct relation between log
> > >>>>> levels and rate limiting.  If I set log level to WARNING I would
> > >>>>> expect to not loose _any_ non-guest log messages with level WARNING or
> > >>>>> above.  It's still useful to have log levels for non-guest messages,
> > >>>>> since user might want to filter out DEBUG non-guest messages for
> > >>>>> example.
> > >>>>
> > >>>> It was me who was confused, because of the two log-everything variants
> > >>>> we have (console and serial). You're right that your change is 
> > >>>> unrelated
> > >>>> to log levels. However, when there are e.g. many warnings or when an
> > >>>> admin has lowered the log level, what you (would) do is effectively
> > >>>> force sync_console mode transiently (for a subset of messages, but
> > >>>> that's secondary, especially because the "forced" output would still
> > >>>> be waiting for earlier output to make it out).
> > >>>
> > >>> Right, it would have to wait for any previous output on the buffer to
> > >>> go out first.  In any case we can guarantee that no more output will
> > >>> be added to the buffer while Xen waits for it to be flushed.
> > >>>
> > >>> So for the hardware domain it might make sense to wait for the TX
> > >>> buffers to be half empty (the current tx_quench logic) by preempting
> > >>> the hypercall.  That however could cause issues if guests manage to
> > >>> keep filling the buffer while the hardware domain is being preempted.
> > >>>
> > >>> Alternatively we could always reserve half of the buffer for the
> > >>> hardware domain, and allow it to be preempted while waiting for space
> > >>> (since it's guaranteed non hardware domains won't be able to steal the
> > >>> allocation from the hardware domain).
> > >>
> > >> Getting complicated it seems. I have to admit that I wonder whether we
> > >> wouldn't be better off leaving the current logic as is.
> > > 
> > > Another possible solution (more like a band aid) is to increase the
> > > buffer size from 4 pages to 8 or 16.  That would likely allow to cope
> > > fine with the high throughput of boot messages.
> > 
> > You mean the buffer whose size is controlled by serial_tx_buffer?
> 
> Yes.
> 
> > On
> > large systems one may want to simply make use of the command line
> > option then; I don't think the built-in default needs changing. Or
> > if so, then perhaps not statically at build time, but taking into
> > account system properties (like CPU count).
> 
> So how about we use:
> 
> min(16384, ROUNDUP(1024 * num_possible_cpus(), 4096))

Er, sorry, that should be max(...) instead.

Thanks, Roger.



 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.