Re: [Xen-devel] Is:livepatch-build-tools.git declare it supported? Was:Re: [PATCH for-4.9] livepatch: Declare live patching as a supported feature

[George Dunlap <george.dunlap@xxxxxxxxxx>]

On 08/06/2017 01:07 AM, Konrad Rzeszutek Wilk wrote:
> On Thu, Aug 03, 2017 at 06:21:30PM +0100, George Dunlap wrote:
>> On 08/03/2017 06:20 PM, George Dunlap wrote:
>>> On 07/03/2017 03:53 PM, Ross Lagerwall wrote:
>>>> On 06/30/2017 02:42 PM, George Dunlap wrote:
>>>>> On 06/28/2017 05:18 PM, Ross Lagerwall wrote:
>>>>>> On 06/27/2017 10:17 AM, George Dunlap wrote:
>>>>>>> On 26/06/17 18:30, Andrew Cooper wrote:
>>>>>>>> On 26/06/17 18:00, George Dunlap wrote:
>>>>>>>>> On 26/06/17 16:36, Ross Lagerwall wrote:
>>>>>> ...
>>>>>>> You seem to be simply refusing to use your imagination.  Step back.
>>>>>>> Imagine yourself in one year.  You come to the office and find an
>>>>>>> e-mail
>>>>>>> on security@ which says, "Livepatch tools open a security hole when
>>>>>>> compiling with gcc x.yy".  You realize that XenVerson ${LATEST-2} uses
>>>>>>> gcc x.yy, so you take a closer look at that livepatch, only to discover
>>>>>>> that the livepatches generated actually do contain the bug, but you
>>>>>>> missed it because ${LATEST-[0,1]} were perfectly fine (since they used
>>>>>>> newer versions of gcc), the difference was subtle, and it passed all
>>>>>>> the
>>>>>>> functional tests.
>>>>>>>
>>>>>>> Now all of the customers that have applied those patches are
>>>>>>> vulnerable.
>>>>>>>
>>>>>>> Do you:
>>>>>>>
>>>>>>> 1. Tell the reporter to post it publicly to xen-devel immediately,
>>>>>>> since
>>>>>>> livepatch tools are not security supported -- thus "zero-day"-ing all
>>>>>>> your customers (as well as anyone else who happens to have used x.yy to
>>>>>>> build a hypervisor)?
>>>>>>>
>>>>>>> 2. Secretly take advantage of Citrix' privileged position on the
>>>>>>> security list, and try to get an update out to your customers before it
>>>>>>> gets announced (but allowing everyone *else* using gcc x.yy to
>>>>>>> experience a zero-day)?
>>>>>>>
>>>>>>> 3. Issue an XSA so that everyone has the opportunity to fix things up
>>>>>>> before making a public announcement, and so that anyone not on the
>>>>>>> embargo list gets an alert, so they know to either update their own
>>>>>>> livepatches, or look for updates from their software provider?
>>>>>>>
>>>>>>> I think #3 is the only possible choice.
>>>>>>>
>>>>>>>    -George
>>>>>>>
>>>>>>
>>>>>> The issue here is that any bug in livepatch-build-tools which still
>>>>>> results in output being generated would be a security issue, because
>>>>>> someone might have used it to patch a security issue.
>>>>>> livepatch-build-tools is certainly not stable enough yet (ever?) to be
>>>>>> treated in this fashion.
>>>>>
>>>>> You didn't answer my question.  If the situation described happens, what
>>>>> position do you want Andrew to be put in?  (If I missed a potential
>>>>> action, let me know.)
>>>>>
>>>>
>>>> I would choose #3 as it is the obvious choice. But I still don't think
>>>> it is a sensible idea to have security support for the build tools, at
>>>> least at this point. The same scenario could be posed for a nasty bug
>>>> that affects Xen 4.4 only, but it is now just out of security support.
>>>> IMO something being not supported doesn't preclude it from having an XSA
>>>> released if there is a particularly nasty vulnerability found.
>>>
>>> Well basically I think we agree, but we're using different terms.  You
>>> want to say, "This isn't security supported, but if important bug is
>>> actually found then we'll issue an XSA".  I want to say, "This is
>>> security supported, because if an important bug is actually found we'll
>>> issue an XSA."
>>>
>>> So it seems to me there are likely two things that make you resistant to
>>> calling it "security supported":
>>>
>>> 1. The fear that we'll be issuing XSAs over trivial things that don't matter
>>>
>>> 2. The fear that people will not do due diligence when creating patches
>>> with the tools.
>>>
>>> I think #1 is just a misconception.  *Every* bug reported to us about
>>> any part of the code we go through the process of trying to determine
>>> its impact and whether we need to issue an XSA or not.  All of the
>>> examples put forward of things we don't want to issue an XSA for are
>>> things that I'm sure we would not issue an XSA for.
>>>
>>> For #2, that is a reasonable fear, but we can deal with that in a
>>> different way than calling the tools "unsupported".  We can, for
>>> instance, mention that in the documents.  We can add a warning message
>>> that the build tools output saying that the result should be manually
>>> inspected for correctness.
>>
>> We need to get a resolution on this.  Anyone else (particarly
>> committers) want to give their opinion?
> 
> Changing title as this is all about now livepatch-build-tools.
> 
> The livepatch-build-tools get a lot of usage around XSA times.
> And that is when the corner cases are being found. The three of them:
> 0c10457 Remove section alignment requirement
> b30d34c Ignore .discard sections
> 6327ab9 create-diff-object: Update fixup offsets in .rela.ex_table
> 
> where thanks to generating XSAs. Now the folks who use these tools
> are also the ones that do pre-disclosures. And the folks who
> work on these tools also are the ones who have to get the livepatches out.
> 
> It is a stressful time and in the past the issues were off:
> 'oh, livepatch-build-tools won't generate the livepatch' 
> 
> which I don't even know how to classify - is it an XSA that it could not
> create an livepatch?
> 
> And if the livepatch-build-tools does generate something mighty wrong
> then the folks on the XSA pre-disclosure list should be let know
> (and that has been happening).
> 
> But I am not really a fan of 'Oh, and one more XSA'

Thanks for weighing in, Konrad.

So it seems that people are still not quite clear about what I'm proposing.

In general, a bug is a security issue if a human told the computer to do
safe thing X, and the computer appeared to do safe thing X, but in fact
did unsafe thing Y.

Consider the question: "Is it an XSA that domain creation will fail for
a kernel with feature A enabled?"  No, that's not a security issue: The
human told the computer to do safe thing X ("boot with kernel A"), and
it did safe thing Z ("not boot").

Consider this question: "Is it an XSA that if you detach a block device
from one domain, and then attach it to another block device, that the
second user can read potentially sensitive data from the first domain?"
No, that's not a security issue: The human told the computer to do
unsafe thing Y ("expose block device from domain B to domain C"), and
the computer did unsafe thing Y.

So.  Suppose a developer / package maintainer / sysadmin tries to build
a livepatch, and the livepatch throws an error.  Is this a security
issue?  No -- the human told it to do safe thing X ("build me a
livepatch"), and it did safe thing Z ("print an error").

Suppose someone builds a livepatch that, when loaded, immediately
crashes the hypervisor in all cases.  Is this a security issue?  No --
the human told it to do safe thing X ("patch this vulnerability") and it
did safe thing Z ("crash the hypervisor immediately").  (This is safe
because we assume you do a reasonable amount of testing before deploying
a livepatch.)

According to my understanding, the above three fixes that happened as
part of devloping XSAs were like one of the above two examples: The tool
either just didn't make a livepatch at all, or it made one that clearly
didn't work.  As such, they would not be considered XSAs.

Suppose someone builds the livepatch with two different versions of the
compiler; or against stale versions of the binaries; and the livepatch
tool doesn't notice, and generates a livepatch which opens up a security
hole.  Is this a security issue?  No -- the human told it to do unsafe
thing Y, and it did unsafe thing Y.

Suppose someone builds a livepatch that with buggy fix-up code that
inadvertently gives PV guests access to hypervisor memory.  Is this a
security issue?  No -- the human told it to do unsafe thing Y ("give PV
guest access to hypervisor memory") and it did unsafe thing Y.

Suppose someone builds a livepatch with the correct compiler, with a
correct patch (that would fix the bug if rebooted into a new
hypervisor), with correct fix-up code.  Suppose that the bug passes all
reasonable testing; but that, *due to a bug in the tools*, the patch
also gives PV guests access to hypervisor memory.  Is this a security
issue?  Yes -- the human told it to do safe thing X ("build a livepatch
based on correct inputs to fix this bug") and it did unsafe thing Y
("build a livepatch that opens up a new security hole").

We could even place more restrictions on the scope if we wanted to.  We
could say that we only support the livepatch tools generating patches
for XSAs.

> This is very similar to what XSA-155 was - the GCC compiler optimizations
> added a nice jump table that was accessed twice. And the offset was
> retrieved from the shared ring.
>
> But we didn't do an XSA-155 for the GCC compiler. That is we didn't
> file a ticket with GCC saying 'Hey, your compiler can create an race
> on shared memory. Could you make your compiler be smarter in these cases'
> We instead wrote code with this optimization in mind with more
> barriers.

Right -- so the gcc compiler guys are using a specification that allows
that behavior.  So from their perspective, we told the compiler to do
unsafe thing Y (or at least, said that we were OK with it doing unsafe
thing Y), and it did unsafe thing Y -- a security issue for Xen, but not
for gcc.  If gcc had *violated* the spec when causing the security
issue, then we certainly would have called that a security issue in gcc.

> I think livepatch-build-tools is in the same category as GCC or linkers.

Indeed; gcc is "security-supported" in the way I am proposing
livepatching be security supported.

I don't think this is a hill worth dying on; if everyone agrees we
should call them "security un-supported", we can go with that.  I'm only
still talking because people seem to be confused about what I'm proposing.

 -George

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel