Re: [Xen-devel] XSA-180 follow-up: repurpose xenconsoled for logging

[George Dunlap <george.dunlap@xxxxxxxxxx>]

On 03/06/16 18:38, Andrew Cooper wrote:
> On 01/06/16 15:00, Wei Liu wrote:
>> Hi all
>>
>> During the discussion of XSA-180 Ian came up with the idea that we
>> repurpose xenconsoled to handle logging. I've done some research (and
>> some coding as well!).
>>
>> In my reply to George the other day:
>>
>>> I just read the code of virtlogd and xenconsoled.
>>>
>>> I think xenconsoled is missing at least things.
>>>
>>> From a higher level:
>>>
>>> 1. Abstraction of rotating file.
>>> 2. Abstraction of client.
>>> 3. IPC interface to libxl -- presumably we need to create a socket.
>>>
>> I've done #1 and port existing code to use that -- would be useful in
>> general.
>>
>> #2 is not too hard because xenconsoled already has concept of a domain.
>> I suspect some refactoring will be fine.
>>
>> #3 requires a bit more thinking. Virtlogd has an RPC interface -- I'm
>> pretty sure we *don't* want that for xenconsoled. So I spent some time
>> this morning and write up a draft for a xenstore based protocol. See
>> below.
>>
>> Also there is an implication here: we put xenconsoled in a really
>> critical path. If for some reason it doesn't work all guests are
>> blocked. Do we really want to do this?
> 
> Sorry to be late to this thread, but I think all my Xen 4.7 tasks are
> now done.
> 
> Architecturally, this is a bad idea, and -1 from me.
> 
> First of all, you are making the assumption that xenconsoled and all
> qemus are in the same domain.  This is not necessarily the case,
> although I suppose that `xl console` does depend on xl and xenconsoled
> being in the same domain.
> 
> At the moment, if xenconsoled crashes, the worst that happens is that
> you cant interact with guest consoles, and the guests will notice that
> their console rings aren't being drained.  This is a safe, albeit
> suboptimal, situation which allows guests to continue running unimpeeded.
> 
> If qemu is connected via xenconsoled, and xenconsoled crashes, qemu will
> block.  Once qemu blocks, the VM is for all intents and purposes, dead. 
> As you identify, there is a security implication here, where a guest
> which can crash xenconsoled can also DoS all other HVM domains in the
> system.
> 
> From the scalability side, XenServer supports running 1000 windows VMs
> on a single host (subject to sufficient RAM), and xenstored is a big
> monlithic single point of failure.  We would like to avoid adding a second.

qemu stubdoms is a red herring; that's a separate codepath which will be
modified appropriately for its use case.   The potential for qemu to
block if the daemon dies, as well as scalability concerns, are good
points which should be considered; and we certainly need to allow other
ways of dealing with logging.

But...

> If someone builds Xen from source and finds later that their disks are
> full up with logs, then tough.

...I cannot express how wrong-headed I think this statement is; and...

> If someone installs Xen from a distro,
> they should reasonably expect logging, and log management, to be
> consistent with other packages from the same distro.

...the degree to which this is true depends on the degree to which
distributions are actually equipped to deal with these sorts of things.

I agree that if logging using existing logging systems were workable,
that would be a better solution.  Wei did actually spend a decent chunk
of time looking at other options, including logrotate and journald
before settling on using xenconsoled.

A lot of this brainstorming and discussion happened off-list because
XSA-180 was still embargoed, so I can understand why it looks like this
came out of nowhere.  It would probably be good for Wei to report here
what he found and why he decided to propose this solution instead.

FWIW, the libvirt project has exactly the same problem, and they did the
analog of what Wei is proposing -- they added a new daemon, virtlogd, to
handle all the console and debug log rotation in a fashion resistant to
DoSing.  Without reading their discussion, it's reasonable to assume
that using system logging was at least considered using system-level
logging before deciding to write their own code.

We already have a daemon to do logging of consoles; it just doesn't have
any of the logrotate features that are needed to make it robust against
DoS.  There's no sense in having log rotation code in two places, so
upgrading xenconsoled to do what virtlogd is doing makes more sense than
say, either writing our own, or stealing virtlogd.

> I think the best solution here is to manage expectations, not to follow
> the kneejerk reaction and hack up a solution.

Hopefully you can see now that this is not a knee-jerk solution. :-)
Thank you for making sure other options is properly discussed in public.

Tying off a few random technical threads:

What does xapi do at the moment?

Re the interface, it seems like having the option to log to {nothing,
file, pipe, xenconsoled} should cover the reasonble usecases: libvirt
for instance could take the pipe and pipe it to virtlogd (and xapi could
do something similar if it wanted).

Regarding qemu grinding to a halt if xenconsoled dies: That may be
something worth considering; but at least on Linux you can set pipes as
O_NONBLOCK and you'll get EAGAIN when writing to a full pipe instead of
blocking.  We'd have to make sure that the error didn't cause any other
failures (ignoring it and letting the write drop is probably the best
thing), but that shouldn't be terribly difficult to do.

 -George

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel