Re: [Xen-devel] [PATCH 2/2] altp2m: Implement p2m_get_mem_access for altp2m views

["Lengyel, Tamas" <tlengyel@xxxxxxxxxxx>]

On Thu, Jan 28, 2016 at 10:04 AM, Razvan Cojocaru <rcojocaru@xxxxxxxxxxxxxxx> wrote:

On 01/28/2016 06:40 PM, Lengyel, Tamas wrote:
>
>
> On Thu, Jan 28, 2016 at 9:32 AM, Razvan Cojocaru
> <rcojocaru@xxxxxxxxxxxxxxx <mailto:rcojocaru@xxxxxxxxxxxxxxx>> wrote:
>
>Â Â ÂOn 01/28/2016 05:58 PM, Lengyel, Tamas wrote:
>Â Â Â>
>Â Â Â>
>Â Â Â> On Thu, Jan 28, 2016 at 8:20 AM, Razvan Cojocaru
>Â Â Â> <rcojocaru@xxxxxxxxxxxxxxx <mailto:rcojocaru@xxxxxxxxxxxxxxx>

>Â Â Â<mailto:rcojocaru@xxxxxxxxxxxxxxx
>Â Â Â<mailto:rcojocaru@xxxxxxxxxxxxxxx>>> wrote:
>Â Â Â>
>Â Â Â>Â Â ÂOn 01/28/2016 05:12 PM, Lengyel, Tamas wrote:
>Â Â Â>Â Â Â>
>Â Â Â>Â Â Â> On Jan 28, 2016 8:02 AM, "Razvan Cojocaru" <rcojocaru@xxxxxxxxxxxxxxx <mailto:rcojocaru@xxxxxxxxxxxxxxx>
>Â Â Â<mailto:rcojocaru@xxxxxxxxxxxxxxx <mailto:rcojocaru@xxxxxxxxxxxxxxx>>
>Â Â Â>Â Â Â> <mailto:rcojocaru@xxxxxxxxxxxxxxx
>Â Â Â<mailto:rcojocaru@xxxxxxxxxxxxxxx> <mailto:rcojocaru@xxxxxxxxxxxxxxx
>Â Â Â<mailto:rcojocaru@xxxxxxxxxxxxxxx>>>> wrote:
>Â Â Â>Â Â Â>>
>Â Â Â>Â Â Â>> On 01/28/2016 04:42 PM, Lengyel, Tamas wrote:
>Â Â Â>Â Â Â>> >
>Â Â Â>Â Â Â>> > On Jan 28, 2016 6:38 AM, "Jan Beulich" <JBeulich@xxxxxxxx
>Â Â Â<mailto:JBeulich@xxxxxxxx> <mailto:JBeulich@xxxxxxxx
>Â Â Â<mailto:JBeulich@xxxxxxxx>>
>Â Â Â>Â Â Â> <mailto:JBeulich@xxxxxxxx <mailto:JBeulich@xxxxxxxx>
>Â Â Â<mailto:JBeulich@xxxxxxxx <mailto:JBeulich@xxxxxxxx>>>
>Â Â Â>Â Â Â>> > <mailto:JBeulich@xxxxxxxx <mailto:JBeulich@xxxxxxxx>
>Â Â Â<mailto:JBeulich@xxxxxxxx <mailto:JBeulich@xxxxxxxx>>
>Â Â Â>Â Â Â<mailto:JBeulich@xxxxxxxx <mailto:JBeulich@xxxxxxxx>
>Â Â Â<mailto:JBeulich@xxxxxxxx <mailto:JBeulich@xxxxxxxx>>>>> wrote:
>Â Â Â>Â Â Â>> >>
>Â Â Â>Â Â Â>> >> >>> On 27.01.16 at 21:06, <tlengyel@xxxxxxxxxxx
>Â Â Â<mailto:tlengyel@xxxxxxxxxxx> <mailto:tlengyel@xxxxxxxxxxx
>Â Â Â<mailto:tlengyel@xxxxxxxxxxx>>
>Â Â Â>Â Â Â> <mailto:tlengyel@xxxxxxxxxxx <mailto:tlengyel@xxxxxxxxxxx>
>Â Â Â<mailto:tlengyel@xxxxxxxxxxx <mailto:tlengyel@xxxxxxxxxxx>>>
>Â Â Â>Â Â Â>> > <mailto:tlengyel@xxxxxxxxxxx
>Â Â Â<mailto:tlengyel@xxxxxxxxxxx> <mailto:tlengyel@xxxxxxxxxxx
>Â Â Â<mailto:tlengyel@xxxxxxxxxxx>>
>Â Â Â>Â Â Â<mailto:tlengyel@xxxxxxxxxxx <mailto:tlengyel@xxxxxxxxxxx>
>Â Â Â<mailto:tlengyel@xxxxxxxxxxx <mailto:tlengyel@xxxxxxxxxxx>>>>> wrote:
>Â Â Â>Â Â Â>> >> > --- a/xen/arch/x86/mm/p2m.c
>Â Â Â>Â Â Â>> >> > +++ b/xen/arch/x86/mm/p2m.c
>Â Â Â>Â Â Â>> >> > @@ -1572,7 +1572,9 @@ void
>Â Â Âp2m_mem_access_emulate_check(struct
>Â Â Â>Â Â Â> vcpu *v,
>Â Â Â>Â Â Â>> >> >Â Â Â Â Â bool_t violation = 1;
>Â Â Â>Â Â Â>> >> >Â Â Â Â Â const struct vm_event_mem_access *data =""> >Â Â Â>Â Â Â&rsp->u.mem_access;
>Â Â Â>Â Â Â>> >> >
>Â Â Â>Â Â Â>> >> > -Â Â Â Â if ( p2m_get_mem_access(v->domain,
>Â Â Â_gfn(data->gfn),
>Â Â Â>Â Â Â>> > &access) == 0 )
>Â Â Â>Â Â Â>> >> > +Â Â Â Â if ( p2m_get_mem_access(v->domain,
>Â Â Â_gfn(data->gfn),
>Â Â Â>Â Â Â>> >> > +
>Â Â Âaltp2m_active(v->domain) ?
>Â Â Â>Â Â Â>> > vcpu_altp2m(v).p2midx : 0,
>Â Â Â>Â Â Â>> >> > +Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â &access) == 0 )
>Â Â Â>Â Â Â>> >>
>Â Â Â>Â Â Â>> >> This looks to be a behavioral change beyond what title and
>Â Â Â>Â Â Â>> >> description say, and it's not clear whether that's
>Â Â Âactually the
>Â Â Â>Â Â Â>> >> behavior everyone wants.
>Â Â Â>Â Â Â>> >
>Â Â Â>Â Â Â>> > I'm fairly comfident its exactly the expected behavior
>Â Â Âwhen one
>Â Â Â>Â Â Âuses
>Â Â Â>Â Â Â>> > mem_access in altp2m tables and emulation. Right now because
>Â Â Â>Â Â Âthe lack of
>Â Â Â>Â Â Â>> > this AFAIK emulation would not work correctly with
>Â Â Âaltp2m. But
>Â Â Â>Â Â ÂRazvan
>Â Â Â>Â Â Â>> > probably can chime in as he uses this path actively.
>Â Â Â>Â Â Â>>
>Â Â Â>Â Â Â>> I've done an experiment to see how much slower using altp2m
>Â Â Âwould
>Â Â Â>Â Â Âbe as
>Â Â Â>Â Â Â>> compared to emulation - so I'm not a big user of the
>Â Â Âfeature, but
>Â Â Â>Â Â ÂI did
>Â Â Â>Â Â Â>> find it cumbersome to have to work with two sets of APIs
>Â Â Â(one for
>Â Â Â>Â Â Âwhat
>Â Â Â>Â Â Â>> could arguably be called the default altp2m view, i.e. the
>Â Â Âregular
>Â Â Â>Â Â Â>> xc_set_mem_access(), and one for altp2m, i.e.
>Â Â Â>Â Â Â>> xc_altp2m_set_mem_access()). Furthermore, the APIs do not
>Â Â Âcurrently
>Â Â Â>Â Â Â>> offer the same features (most notably,
>Â Â Âxc_altp2m_get_mem_access() is
>Â Â Â>Â Â Â>> completely missing). I've mentioned this to Tamas while
>Â Â Âinitially
>Â Â Â>Â Â Âtrying
>Â Â Â>Â Â Â>> to get it to work.
>Â Â Â>Â Â Â>>
>Â Â Â>Â Â Â>> Now, whether the behaviour I expect is what everyone
>Â Â Âexpects is, of
>Â Â Â>Â Â Â>> course, wide open to debate. But I think we can all agree
>Â Â Âthat the
>Â Â Â>Â Â Â>> altp2m interface can, and probably should, be improved.
>Â Â Â>Â Â Â>>
>Â Â Â>Â Â Â>
>Â Â Â>Â Â Â> There is that, but also, what is the exact logic behind
>Â Â Âdoing this
>Â Â Â>Â Â Âcheck
>Â Â Â>Â Â Â> before emulation? AFAIU emulation happens in response to a
>Â Â Âvm_event so
>Â Â Â>Â Â Â> we should be fairly certain that this check succeeds as it just
>Â Â Â>Â Â Âverifies
>Â Â Â>Â Â Â> that indeed the permissions are restricted by mem_access in the
>Â Â Â>Â Â Âp2m (and
>Â Â Â>Â Â Â> with altp2m this should be the active one). But when is this
>Â Â Âcheck
>Â Â Â>Â Â Â> normally expected to fail?
>Â Â Â>
>Â Â Â>Â Â ÂThat check is important, please do not remove it. A vm_event
>Â Â Âis sent
>Â Â Â>Â Â Âinto userspace to our monitoring application, but the monitoring
>Â Â Â>Â Â Âapplication can actually remove the page restrictions before
>Â Â Âreplying,
>Â Â Â>Â Â Âso in that case emulation is pointless - there will be no more
>Â Â Âpage
>Â Â Â>Â Â Âfaults for that instruction.
>Â Â Â>
>Â Â Â>
>Â Â Â> I see, but then why would you reply with VM_EVENT_FLAG_EMULATE?
>Â Â ÂYou know
>Â Â Â> you removed the permission before sending the reply, so this
>Â Â Âsounds like
>Â Â Â> something specific to your application.
>
>Â Â ÂIt's cheap insurance that things go right. If there's some issue with
>Â Â Âpage rights, or some external tool somehow does an xc_set_mem_access(),
>Â Â Âthings won't go wrong.
>
>
> I can see this working for your application if you don't cache the
> mem_access permissions locally and you don't want to query for it before
> deciding to send the emulate flag in the response or not. Although, I
> think that would be the best way to go here.

Querying is out of the question, for obvious performance reasons. That's
why we've cached the registers in the vm_event request - we could have
not done that and instead just query them via libxc. But one small
decision like that and the monitored guest is running twice as slow.
This way, you can just set the emulate flag and have the hypervisor do
the right thing anyway, with no extra userspace <-> hypervisor roundtrips.

Caching might work, but then again that's extra work, memory used in the
application (in _each_ application, not just ours). So on one hand, we
have the current scenario where things can't go wrong and the solution
is in one place, vs. the other scenario, where each application needs to
solve the problem by doing tracking / caching / querying that the HV
does anyway in p2m, and pay with a possible guest crash or freeze for
failure.

>Â Â ÂAnd they will go wrong if Xen thinks it should
>Â Â Âemulate the next instruction and the next instruction is not the one
>Â Â Âthat has caused the original fault.
>
>
> How could that happen? When the vCPU is resumed after the fault, isn't
> the same instruction guaranteed to be retried?

The instruction is the same, but if the page restrictions have been
lifted (somehow) and the EMULATE flag is still set, the original
instruction will run normally (because it won't trigger another page
fault). But the HV will still think that it needs to emulate the next
page fault, and so it will emulate whatever instruction causes the next
page fault (if it matches the emulate conditions).

>Â Â ÂI would think that benefits any
>Â Â Âapplication.
>
>
> It's just a bit of an obscure exception. From an API perspective I would
> rather have Xen do what I tell it to do - in this case emulate - rather
> then it doing something else silently behind the scenes that you really
> only find out about if you read the code.

But the way the emulation code works now, it _can't_ emulate (see above
explanation). Emulation currently only happens as a result of a page
fault, and there will be no page fault if the page restriction are
lifted. I am thinking about a better way to achieve this, but until then
I think it's a good idea to keep the check in.

I hope I've been able to shed more light on this.

Sure, make sense. Since AFAIK you guys are the only one really using this path I'm cool with keeping it as it is, was really just wondering for the logic behind it. Without a reference implementation using this path it's not exactly trivial trying to figure out why things are the way they are.

Jan,
with the explanation above by Razvan, when using emulation with altp2m the correct check here is to see if the altp2m permissions are still restricted, otherwise no need to emulate. So this patch actually makes the two systems correctly work together. Without this patch only the hostp2m permissions are checked which may not have the restrictions that actually caused the fault and lead to infinite faults and hanging the VM.

Tamas

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel