Re: [Xen-devel] Branch Trace Storage for guests andVPMUinitialization

[<Kevin.Mayer@xxxxxxxx>]

> -----UrsprÃngliche Nachricht-----
> Von: Boris Ostrovsky [mailto:boris.ostrovsky@xxxxxxxxxx]
> Gesendet: Donnerstag, 26. Februar 2015 17:35
> An: Dietmar Hahn; xen-devel@xxxxxxxxxxxxx
> Cc: Mayer, Kevin
> Betreff: Re: [Xen-devel] Branch Trace Storage for guests and
> VPMUinitialization
> 
> On 02/26/2015 03:56 AM, Dietmar Hahn wrote:
> > Am Mittwoch 25 Februar 2015, 11:31:31 schrieb Boris Ostrovsky:
> >> On 02/25/2015 10:12 AM, Kevin.Mayer@xxxxxxxx wrote:
> >>>> -----UrsprÃngliche Nachricht-----
> >>>> Von: Boris Ostrovsky [mailto:boris.ostrovsky@xxxxxxxxxx]
> >>>> Gesendet: Dienstag, 24. Februar 2015 18:13
> >>>> An: Mayer, Kevin; xen-devel@xxxxxxxxxxxxx
> >>>> Betreff: Re: [Xen-devel] Branch Trace Storage for guests and VPMU
> >>>> initialization
> >>>>
> >>>> On 02/24/2015 10:27 AM, Kevin.Mayer@xxxxxxxx wrote:
> >>>>> Hi guys
> >>>>>
> >>>>> I`m trying to set up the BTS so that I can log the branches taken
> >>>>> in the guest using Xen 4.4.1 with a WinXP SP3 guest on a Core i7
> >>>>> Sandy Bridge.
> >>>>>
> >>>>> I added the vpmu=bts boot parameter to my grub2 configuration and
> >>>>> extended the libxl,libxc,domctl,â with an own command so that I
> >>>>> can trigger the activation of the BTS whenever I want.
> >>>>>
> >>>> I am not sure why you are doing all these changes to Xen code. BTS
> >>>> is supposed to be managed from the guest. For example, a Fedora
> HVM
> >>>> guest will produce this:
> >>>>
> >>>> [root@dhcp-burlington7-2nd-B-east-10-152-55-140 ~]# perf record -e
> >>>> branches:u -c 1 -d sleep 1 [ perf record: Woken up 3838 times to
> >>>> write data ] [ perf record: Captured and wrote 0.704 MB perf.data
> >>>> (~30756 samples) ]
> >>>> [root@dhcp-burlington7-2nd-B-east-10-152-55-140 ~]# perf script -f
> >>>> ip,addr,sym,dso,symoff --show-kernel-path
> >>>>     ffffffff8167c347 native_irq_return_iret+0x0 (/proc/kcore) =>
> >>>> 328c001590 [unknown] (/proc/kcore)
> >>>>     ffffffff8167c347 native_irq_return_iret+0x0 (/proc/kcore) =>
> >>>> 328c001590 [unknown] ([unknown])
> >>>>           328c001593 [unknown] ([unknown]) =>       328c004b70 [unknown]
> >>>> ([unknown])
> >>>> ...
> >>>>
> >>> I want to be able to log the taken branches (of the guest) without the
> need to modify the guest at all.
> >>> This means I have to do all the logic in the hypervisor, or am I wrong?
> >> In that case, yes. But then you have to make sure that at least
> >>    * you don't load guest's VPMU (or, at least, BTS-related
> >> registers) on context switch
> > But you need to modify PMU registers when switching to/from the guest
> > context to get PMU running.
> 
> 
> 
> I was thinking that all BTS stuff can be controlled from dom0 and so we can
> use dom0's version of these registers. I didn't realize that DS_AREA would
> have to be accessed in guest's address space (and that DEBUGCTL is loaded
> from VMCS).
> 
> Which is what I think I said in response to this message (which didn't show up
> on the list because Kevin accidentally dropped xen-devel).
> 
> -boris
 
Terribly sorry about that...

So the VPMU doesnât get loaded when there is a VMENTER?
I thought I could set the domU->vcpu->vpmu to enable BTS while in dom0 (with 
modified versions of msr_write_intercept, vpmu_do_wrmsr and core2_vpmu_do_wrmsr 
of course since the build in ones use the current-vcpu which would be the 
dom0-vcpu)
and as soon as there is a context switch to domU the vpmu gets loaded and the 
guest starts logging.
If the described behavior is correct the only problem I can see is with 
allocating memory in dom0 in a way that the guest can access it.
But if I got it wrong please explain how the vpmu really works.

Cheers

Kevin


> 
> 
> > I didn't think of using the VPMU stuff with modifying the context from
> > outside the guest.
> >
> >>    * You don't send the interrupt to the guest (meaning that you will
> >> need to somehow inform dom0 of the BTS interrupt)
> >>
> >> and probably more.
> >>
> >> Essentially, you want dom0 to profile the guest. I have been working
> >> on patches that would allow that but they are still under review.
> >>
> >>
> >>>>> In this command I do the following:
> >>>>>
> >>>>> I set up the memory region for the BTS Buffer and the DS Buffer
> >>>>> Management Area using xzalloc_bytes
> >>>>>
> >>>> I don't think you should be allocating BTS buffers in the
> >>>> hypervisor, they are in guest's memory.
> >>> I agree. As I said I think this is where my main problem is at the moment.
> >>> Is there any way I can allocate memory in the hypervisor in a way the
> guest can access it?
> >> I am not sure this is what you want since you seem to *not* want the
> >> guest to process the samples, right?
> >>
> >> But yes, you can. E.g. something like what map_vcpu_info() does. (I
> >> have no idea how you'd do this from Windows.)
> > The DS buffer has to be mapped within the guests address space so the
> > CPU running in guest context can access this area. Otherwise you get
> > this triple fault.
> > So I would think you need a mixture of writing some stuff in Windows
> > and patching the hypervisor.
> >
> > Dietmar.
> >
> >>
> >>> Of course the guest must not be able to use this memory in its normal
> operations but just for BTS.
> >>> Is this even possible? I am rather confused at the moment. :-D
> >>>
> >>>>> Then I write the pointer to the BTS Buffer into the DS Buffer
> >>>>> Management Area at +0x0 and +0x8 (BTS Buffer Base and BTS Index)
> >>>>>
> >>>>> When I use vmx_msr_write_intercept to store the value in
> >>>>> MSR_IA32_DS_AREA the host reboots (my idea is he tries to access a
> >>>>> vpmu-struct that isnÂt there in the current vcpu and panics).
> >>
> >> Who is trying to write to MSR_IA32_DS_AREA? The guest or dom0? I
> >> thought you said that you want dom0 to do sampling. Or are you trying
> >> to setup DS area from your guest and control it from dom0? I am
> somewhat confused.
> >>
> >>>> Can you post hypervisor log? (hard to say how helpful it will be
> >>>> without seeing your code changes though)
> >>>>
> >>> Right after enabling the BTS I get a triple fault.
> >>> hvm.c:1357:d2 Triple fault on VCPU0 - invoking HVM shutdown action 1.
> >>
> >> That's not host reboot, this is your guest dying.
> >>
> >>
> >>>>> When I use a modified version of vmx_msr_write_intercept I donât
> >>>>> get any crashes as long as I donât enable BTS and TR in the
> >>>>> GUEST_IA32_DEBUGCTL (BTR works). When I enable the BTS (and TR)
> >>>>> the guest crashes. I suppose he gets killed by the hypervisor for
> >>>>> accessing forbidden memory.
> >>>>>
> >>>> Possibly because DS area point to hypervisor memory.
> >>>>
> >>>>
> >>>> Having said all this, I am not sure how well BTS works. You did
> >>>> notice this in the hypervisor log:
> >>>>
> >>>> (XEN)
> ******************************************************
> >>>> (XEN) ** WARNING: Emulation of BTS Feature is switched on **
> >>>> (XEN) ** Using this processor feature in a virtualized **
> >>>> (XEN) ** environment is not 100% safe. **
> >>>> (XEN) ** Setting the DS buffer address with wrong values **
> >>>> (XEN) ** may lead to hypervisor hangs or crashes. **
> >>>> (XEN) ** It is NOT recommended for production use! **
> >>>> (XEN)
> ******************************************************
> >>>>
> >>> Yes, I saw that. It doesnât state that BTS is not working at all, just 
> >>> that it is
> not that safe to use.
> >>> As I understand it as long as I set the DS buffer address correctly I 
> >>> should
> be fine, right?
> >> Right. Except that I am not convinced you did set this buffer
> >> correctly, which is possibly why your hypervisor crashed (I am not
> >> sure I understood under what circumstances though).
> >>
> >> -boris
> >>
> >>> Since I donât want to use for production that is fine with me. At least 
> >>> for
> now.
> >>>
> >>>
> >>> Kevin
> >>>> -boris
> >>>>
> >>>>
> >>>>> The modified version of vmx_msr_write_intercept takes a
> >>>>> vcpu-struct as a parameter and uses this instead of the current vcpu.
> >>>>>
> >>>>> Instead of
> >>>>>
> >>>>> staticint vmx_msr_write_intercept(unsigned int msr, uint64_t
> >>>> msr_content)
> >>>>> {
> >>>>>
> >>>>>       struct vcpu *v = current;
> >>>>>
> >>>>> I just have
> >>>>>
> >>>>> staticint own_vmx_msr_write_intercept(unsigned int msr, uint64_t
> >>>>> msr_content, struct vcpu *v)
> >>>>>
> >>>>> I get this vcpu by d->vcpu[0] as I have limited my guest domain to
> >>>>> one vcpu atm.
> >>>>>
> >>>>> Of course I also use similarly modified version of the called
> >>>>> functions(vpmu_do_wrmsr,â).
> >>>>>
> >>>>> IÂm pretty sure that my problem is with a wrong scope/usage of the
> >>>>> vcpus/memory, but I have no idea how to fix this.
> >>>>>
> >>>>> I can see a potential problem with the memory allocation (in the
> >>>>> host) into which the cpu in guest-mode is supposed to write.
> >>>>>
> >>>>> Or maybe I got the principle of a vcpu/vpmu all wrong.
> >>>>>
> >>>>> Since I couldnât find any project that uses the BTS for the guest,
> >>>>> I am wondering if anyone has ever done this and if it is possible at 
> >>>>> all.
> >>>>>
> >>>>> Any input is welcome as I am pretty much stuck atmâ
> >>>>>
> >>>>> Cheers
> >>>>>
> >>>>> Kevin
> >

____________
Virus checked by G Data MailSecurity
Version: AVA 25.435 dated 26.02.2015
Virus news: www.antiviruslab.com
_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel