RE: [Xen-devel] [PATCH][VT] Patch to allow VMX domainsto be destroyed or shut down cleanly

["Jiang, Yunhong" <yunhong.jiang@xxxxxxxxx>]

I think the Xin Xiaohui's patch resolved most problem for the page left
issue.

Also another small issue found is the on vmx_set_cr0. On Windows, the
guest will enter/leave protected mode many times.  Following code cause
problem.
    if ((value & X86_CR0_PE) && (value & X86_CR0_PG) && !paging_enabled)
{
        /*
         * The guest CR3 must be pointing to the guest physical.
         */
        if ( !VALID_MFN(mfn = get_mfn_from_pfn(
            d->arch.arch_vmx.cpu_cr3 >> PAGE_SHIFT)) ||
             !get_page(pfn_to_page(mfn), d->domain) )
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        {
            printk("Invalid CR3 value = %lx", d->arch.arch_vmx.cpu_cr3);
            domain_crash_synchronous(); /* need to take a clean path */
        }

We should place the get_page to when guest set cr3 when not paging, i.e
:
   case 3:
    {
        unsigned long old_base_mfn, mfn;

        /*
         * If paging is not enabled yet, simply copy the value to CR3.
         */
        if (!vmx_paging_enabled(d)) {
                ..... get page here....

After above two change, windows destroied succesfully after
create/login/open IE/destroy.

Thanks
Yunhong Jiang

Khoa Huynh wrote:
> Keir Fraser wrote:
>> I mean forcibly decrement them to zero and free them right there and
>> then. Of course, as you point out, the problem is that some of the
>> pages are mapped in domain0. I'm not sure how we can distinguish
>> tainted refcnts from genuine external references. Perhaps there's a
>> proper way we should be destructing the full shadow pagetables such
>> that the refcnts end up at zero.
> 
> Thanks for your comment.  I have done extensive tracing through
> the domain destruction code in the hypervisor in the last few
> days.
> 
> The bottom line:  after domain destruction code in the hypervisor
> is done, all shadow pages were indeed freed up - even though
> the shadow_tainted_refcnts flag was set.  I now believe the
> remaining pages are genuinely externally referenced (possibly
> by the qemu device model still running in domain0).
> 
> Here are more details on what I have found:
> 
> Ideally, when we destroy or shut down a VMX domain, the general
> page reference counts ended up at 0 in shadow mode, so that the
> pages can be released properly from the domain.
> 
> I have traced quite a bit of code for different scenarios
> involving Windows XP running in a VMX domain.  I only
> did simple operations in Windows XP, but I tried to destroy
> the VMX domain at different times (e.g. during Windows XP boot,
> during simple operations, after Windows XP has been shutdown, etc.)
> 
> For non-VMX (Linux) domains, after we relinquish memory in
> domain_relinquish_resources(), all pages in the domain's page
> list indeed had reference count of 0 and were properly freed from
> the xen heap - just like we expected.
> 
> For VMX (e.g., Windows XP) domains, after we relinquish memory in
> domain_relinquish_resources(), depending on how many activities
> were done in Windows XP, there were anywhere from 2 to 100 pages
> remaining just before the domain's structures were freed up
> by the hypervisor.  Most of these pages still have page
> reference counts of 1, and therefore, could not be freed
> from the heap by the hypervisor.  This prevents the rest
> of the domain's resources from being released, and therefore,
> 'xm list' still shows the VMX domains after they were destroyed.
> 
> In shadow mode, the following things could be reflected
> in the page (general) reference counts:
> 
> (a) General stuff:
>     - page is allocated (PGC_allocated)
>     - page is pinned
>     - page is pointed by CR3's
> (b) Shadow page tables (l1, l2, hl2, etc.)
> (c) Out-of-sync entries
> (d) Grant table mappings
> (e) External references (not through grant table)
> (f) Monitor page table references (external shadow mode)
> (g) Writable PTE predictions
> (h) GDTs/LDTs
> 
> So I put in a lot of instrumentation and tracing code,
> and made sure that the above things were taken into
> account and removed from the page reference counts
> during the domain destruction code sequence in the
> hypervisor.  During this code sequence, we disable
> shadow mode (shadow_mode_disable()) and the
> shadow_tainted_refcnts flag was set.  However,
> much to my surprise, the page reference counts
> were properly taken care of in shadow mode, and
> all shadow pages (including those in l1, l2, hl2
> tables and snapshots) were all freed up.
> 
> In particular, here's where each of the things
> in the above list was taken into account during
> the domain destruction code sequence in the
> hypervisor:
> 
> (a) General stuff:
>     - None of remaining pages have PGC_allocated
>       flag set
>     - None of remaining pages are still pinned
>     - The monitor shadow ref was 0, and all
>       pages pointed to by CR3's were taken care
>       of in free_shadow_pages()
> (b) All shadow pages (including those pages in
>     l1, l2, hl2, snapshots) were freed properly.
>     I implemented counters to track all shadow
>     page promotions/allocations and demotions/
>     deallocations throughout the hypervisor code,
>     and at the end after we relinquished all domain
>     memory pages, these counters did indeed
>     return to 0 - as we expected.
> (c) out-of-sync entries -> in free_out_of_sync_state()
>     called by free_shadow_pages().
> (d) grant table mappings -> the count of active
>     grant table mappings is 0 after the domain
>     destruction sequence in the hypervisor is
>     executed.
> (e) external references not mapped via grant table
>     -> I believe that these include the qemu-dm
>     pages which still remain after we relinquish
>     all domain memory pages - as the qemu-dm may
>     still be active after a VMX domain has been
>     destroyed.
> (f) external monitor page references -> all references
>     from monitor page table are dropped in
>     vmx_relinquish_resources(), and monitor table
>     itself is freed in domain_destruct().  In fact,
>     in my code traces, the monitor shadow reference
>     count was 0 after the domain destruction code in
>     the hypervisor.
> (g) writable PTE predictions -> I didn't see any pages in
>     this category in my code traces, but if there
>     are, they would be freed up in free_shadow_pages().
> (h) GDTs/LDTs -> these were destroyed in destroy_gdt() and
>     invalidate_shadow_ldt() called from domain_relinquish_
>     resources().
> 
> Based on the code instrumentation and tracing above, I am
> pretty confident that the shadow page reference counts
> were handled properly during the domain destruction code
> sequence in the hypervisor.  There is a problem in keeping
> track of shadow page counts (domain->arch.shadow_page_count),
> and I will submit a patch to fix this shortly.  However, this
> does not really impact how shadow pages are handled.
> 
> Consequently, the pages that still remain after the domain
> destruction code sequence in the hypervisor are externally
> referenced and may belong to the qemu device model running
> in domain0.  The fact that qemu-dm is still active for some
> time after a VMX domain has been torn down in the hypervisor
> is evident by examining the tools code (python).  In fact,
> if I forcibly free these remaining pages from the xen heap,
> the system/dom0 crashed.
> 
> Am I missing anything ? Your comments, suggestions, etc.,
> are welcome!  Thanks for reading this rather long email :-)
> 
> Khoa H.
> 
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@xxxxxxxxxxxxxxxxxxx
> http://lists.xensource.com/xen-devel


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel