Re: [Xen-devel] Branch Trace Storage for guests and VPMUinitialization

[Boris Ostrovsky <boris.ostrovsky@xxxxxxxxxx>]

On 02/26/2015 03:56 AM, Dietmar Hahn wrote:
> Am Mittwoch 25 Februar 2015, 11:31:31 schrieb Boris Ostrovsky:
> > On 02/25/2015 10:12 AM, Kevin.Mayer@xxxxxxxx wrote:
> > > > -----UrsprÃngliche Nachricht-----
> > > > Von: Boris Ostrovsky [mailto:boris.ostrovsky@xxxxxxxxxx]
> > > > Gesendet: Dienstag, 24. Februar 2015 18:13
> > > > An: Mayer, Kevin; xen-devel@xxxxxxxxxxxxx
> > > > Betreff: Re: [Xen-devel] Branch Trace Storage for guests and VPMU
> > > > initialization
> > > >
> > > > On 02/24/2015 10:27 AM, Kevin.Mayer@xxxxxxxx wrote:
> > > > > Hi guys
> > > > >
> > > > > I`m trying to set up the BTS so that I can log the branches taken in
> > > > > the guest using Xen 4.4.1 with a WinXP SP3 guest on a Core i7 Sandy
> > > > > Bridge.
> > > > >
> > > > > I added the vpmu=bts boot parameter to my grub2 configuration and
> > > > > extended the libxl,libxc,domctl,â with an own command so that I can
> > > > > trigger the activation of the BTS whenever I want.
> > > > I am not sure why you are doing all these changes to Xen code. BTS is
> > > > supposed to be managed from the guest. For example, a Fedora HVM guest
> > > > will produce this:
> > > >
> > > > [root@dhcp-burlington7-2nd-B-east-10-152-55-140 ~]# perf record -e
> > > > branches:u -c 1 -d sleep 1 [ perf record: Woken up 3838 times to write data ] [
> > > > perf record: Captured and wrote 0.704 MB perf.data (~30756 samples) ]
> > > > [root@dhcp-burlington7-2nd-B-east-10-152-55-140 ~]# perf script -f
> > > > ip,addr,sym,dso,symoff --show-kernel-path
> > > >     ffffffff8167c347 native_irq_return_iret+0x0 (/proc/kcore) =>
> > > > 328c001590 [unknown] (/proc/kcore)
> > > >     ffffffff8167c347 native_irq_return_iret+0x0 (/proc/kcore) =>
> > > > 328c001590 [unknown] ([unknown])
> > > >           328c001593 [unknown] ([unknown]) =>       328c004b70 [unknown]
> > > > ([unknown])
> > > > ...
> > > I want to be able to log the taken branches (of the guest) without the need to 
> > > modify the guest at all.
> > > This means I have to do all the logic in the hypervisor, or am I wrong?
> > In that case, yes. But then you have to make sure that at least
> >    * you don't load guest's VPMU (or, at least, BTS-related registers) on
> > context switch
> But you need to modify PMU registers when switching to/from the guest context
> to get PMU running.


I was thinking that all BTS stuff can be controlled from dom0 and so we 
can use dom0's version of these registers. I didn't realize that DS_AREA 
would have to be accessed in guest's address space (and that DEBUGCTL is 
loaded from VMCS).
Which is what I think I said in response to this message (which didn't 
show up on the list because Kevin accidentally dropped xen-devel).
-boris



> I didn't think of using the VPMU stuff with modifying the context from outside
> the guest.
>
> >    * You don't send the interrupt to the guest (meaning that you will
> > need to somehow inform dom0 of the BTS interrupt)
> >
> > and probably more.
> >
> > Essentially, you want dom0 to profile the guest. I have been working on
> > patches that would allow that but they are still under review.
> >
> >
> > > > > In this command I do the following:
> > > > >
> > > > > I set up the memory region for the BTS Buffer and the DS Buffer
> > > > > Management Area using xzalloc_bytes
> > > > I don't think you should be allocating BTS buffers in the hypervisor, they are
> > > > in guest's memory.
> > > I agree. As I said I think this is where my main problem is at the moment.
> > > Is there any way I can allocate memory in the hypervisor in a way the guest can 
> > > access it?
> > I am not sure this is what you want since you seem to *not* want the
> > guest to process the samples, right?
> >
> > But yes, you can. E.g. something like what map_vcpu_info() does. (I have
> > no idea how you'd do this from Windows.)
> The DS buffer has to be mapped within the guests address space so the CPU
> running in guest context can access this area. Otherwise you get this
> triple fault.
> So I would think you need a mixture of writing some stuff in Windows and
> patching the hypervisor.
>
> Dietmar.
>
> > > Of course the guest must not be able to use this memory in its normal 
> > > operations but just for BTS.
> > > Is this even possible? I am rather confused at the moment. :-D
> > >
> > > > > Then I write the pointer to the BTS Buffer into the DS Buffer
> > > > > Management Area at +0x0 and +0x8 (BTS Buffer Base and BTS Index)
> > > > >
> > > > > When I use vmx_msr_write_intercept to store the value in
> > > > > MSR_IA32_DS_AREA the host reboots (my idea is he tries to access a
> > > > > vpmu-struct that isnÂt there in the current vcpu and panics).
> > Who is trying to write to MSR_IA32_DS_AREA? The guest or dom0? I thought
> > you said that you want dom0 to do sampling. Or are you trying to setup
> > DS area from your guest and control it from dom0? I am somewhat confused.
> >
> > > > Can you post hypervisor log? (hard to say how helpful it will be without
> > > > seeing your code changes though)
> > > Right after enabling the BTS I get a triple fault.
> > > hvm.c:1357:d2 Triple fault on VCPU0 - invoking HVM shutdown action 1.
> > That's not host reboot, this is your guest dying.
> >
> >
> > > > > When I use a modified version of vmx_msr_write_intercept I donât get
> > > > > any crashes as long as I donât enable BTS and TR in the
> > > > > GUEST_IA32_DEBUGCTL (BTR works). When I enable the BTS (and TR) the
> > > > > guest crashes. I suppose he gets killed by the hypervisor for
> > > > > accessing forbidden memory.
> > > > Possibly because DS area point to hypervisor memory.
> > > >
> > > >
> > > > Having said all this, I am not sure how well BTS works. You did notice
> > > > this in the hypervisor log:
> > > >
> > > > (XEN) ******************************************************
> > > > (XEN) ** WARNING: Emulation of BTS Feature is switched on **
> > > > (XEN) ** Using this processor feature in a virtualized **
> > > > (XEN) ** environment is not 100% safe. **
> > > > (XEN) ** Setting the DS buffer address with wrong values **
> > > > (XEN) ** may lead to hypervisor hangs or crashes. **
> > > > (XEN) ** It is NOT recommended for production use! **
> > > > (XEN) ******************************************************
> > > Yes, I saw that. It doesnât state that BTS is not working at all, just that it 
> > > is not that safe to use.
> > > As I understand it as long as I set the DS buffer address correctly I should be 
> > > fine, right?
> > Right. Except that I am not convinced you did set this buffer correctly,
> > which is possibly why your hypervisor crashed (I am not sure I
> > understood under what circumstances though).
> >
> > -boris
> >
> > > Since I donât want to use for production that is fine with me. At least for now.
> > >
> > >
> > > Kevin
> > > > -boris
> > > >
> > > >
> > > > > The modified version of vmx_msr_write_intercept takes a vcpu-struct as
> > > > > a parameter and uses this instead of the current vcpu.
> > > > >
> > > > > Instead of
> > > > >
> > > > > staticint vmx_msr_write_intercept(unsigned int msr, uint64_t
> > > > msr_content)
> > > > > {
> > > > >
> > > > >       struct vcpu *v = current;
> > > > >
> > > > > I just have
> > > > >
> > > > > staticint own_vmx_msr_write_intercept(unsigned int msr, uint64_t
> > > > > msr_content, struct vcpu *v)
> > > > >
> > > > > I get this vcpu by d->vcpu[0] as I have limited my guest domain to one
> > > > > vcpu atm.
> > > > >
> > > > > Of course I also use similarly modified version of the called
> > > > > functions(vpmu_do_wrmsr,â).
> > > > >
> > > > > IÂm pretty sure that my problem is with a wrong scope/usage of the
> > > > > vcpus/memory, but I have no idea how to fix this.
> > > > >
> > > > > I can see a potential problem with the memory allocation (in the host)
> > > > > into which the cpu in guest-mode is supposed to write.
> > > > >
> > > > > Or maybe I got the principle of a vcpu/vpmu all wrong.
> > > > >
> > > > > Since I couldnât find any project that uses the BTS for the guest, I
> > > > > am wondering if anyone has ever done this and if it is possible at all.
> > > > >
> > > > > Any input is welcome as I am pretty much stuck atmâ
> > > > >
> > > > > Cheers
> > > > >
> > > > > Kevin

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel