Re: [BUG] Possible FATAL PAGE FAULT in domain_crash function on Xen 4.14.6

[Andrew Cooper <andrew.cooper3@xxxxxxxxxx>]

On 06/02/2024 12:14 pm, Giuseppe De Rosa wrote:
> Bug detailed description:
> 
> ----------------
> 
> While booting a Linux Debian 7 "Wheezy" VM, Xen crashes with a FATAL
> PAGE FAULT. 
> 
>  
> 
> Environment :
> 
> ----------------
> 
> HW: Intel(R) Xeon(R) CPU E5-1620 v2 @ 3.70GHz (2 CPU, Xen in nested
> virtualization upon QEMU/KVM), 4GB RAM
> 
> Xen: Xen 4.14.6 (xen-hypervisor-4.14-amd64 package)
> 
> Dom0: Linux 5.10.0-26-amd64 (Debian 11 "Bullseye"), 1 CPU, 1024GB RAM
> 
>  
> 
> Reproduce steps:
> 
> ----------------
> 
> 1.      Install debian 11 and xen package
> 
> 2.      Boot a clean debian 7 image in hardware-assisted virtualization
> 
> 3.      Single bit-flip bit number 2 of VMCS field
> "VM_EXIT_QUALIFICATION" (field number 6400). In my case, value changed
> from [100049] -> [10004d]
> 
> 4.      Leave the debian7 guest executing after the bit flip.
> 
>  
> 
> Current result:
> 
> ----------------
> 
> Xen crash
> 
>  
> 
> Console error log:
> 
> ----------------
> 
> (XEN) d1v0 Unexpected PIO status 1, port 0x10 read 0x00000000ffff
> 
> (XEN) domain_crash called from io.c:166
> (XEN) Domain 1 (vcpu#0) crashed on cpu#1:
> (XEN) ----[ Xen-4.14.6  x86_64  debug=n   Not tainted ]----
> (XEN) CPU:    1
> (XEN) RIP:    0010:[<ffffffff8100712e>]
> (XEN) RFLAGS: 0000000000000046   CONTEXT: hvm guest (d1v0)
> (XEN) rax: 0000000000000000   rbx: 0000000040000000   rcx: 0000000000000001
> (XEN) rdx: 0000000000000000   rsi: ffffffff81666a80   rdi: ffffffff81617830
> (XEN) rbp: 0000000000000020   rsp: ffffffff81601e78   r8:  0000000000000200
> (XEN) r9:  ffffffff8168f2a0   r10: 0000000000000007   r11: 0000000000000007
> (XEN) r12: ffffffff81601f58   r13: ffffffffffffffff   r14: 000000000008c800
> (XEN) r15: 0000000000001000   cr0: 0000000080050033   cr4: 00000000001000a0
> (XEN) cr3: 0000000001605000   cr2: 0000000000000000
> (XEN) fsb: 0000000000000000   gsb: ffffffff81696000   gss: 0000000000000000
> (XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: 0000   cs: 0010
> (XEN) d1v0 Bad rIP ffffffff8100712e for mode 0
> (XEN) domain_crash called from vmx.c:4413
> (XEN) ----[ Xen-4.14.6  x86_64  debug=n   Not tainted ]----
> (XEN) CPU:    1
> (XEN) RIP:    e008:[<ffff82d040206fa9>] __domain_crash+0x9/0x80
> (XEN) RFLAGS: 0000000000010296   CONTEXT: hypervisor (d1v0)
> (XEN) rax: ffff830139c0506c   rbx: ffff8301308a0000   rcx: 0000000000000000
> (XEN) rdx: ffff830136ddffff   rsi: 000000000000000a   rdi: 0000000000000000
> (XEN) rbp: 0000000000000000   rsp: ffff830136ddfee0   r8:  0000000000000001
> (XEN) r9:  0000000000004000   r10: 0000000000000001   r11: ffff82d040372d40
> (XEN) r12: 0000000000000000   r13: 0000000000000000   r14: 0000000000000000
> (XEN) r15: 0000000000000000   cr0: 0000000080050033   cr4: 00000000001526e0
> (XEN) cr3: 0000000136da6000   cr2: 0000000000000208
> (XEN) fsb: 0000000000000000   gsb: 0000000000000000   gss: 0000000000000000
> (XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: 0000   cs: e008
> (XEN) Xen code around <ffff82d040206fa9> (__domain_crash+0x9/0x80):
> (XEN)  0f 1e fa 55 48 89 fd 53 <80> bf 08 02 00 00 00 75 2d 48 89 e3 0f
> b7 37 48
> (XEN) Xen stack trace from rsp=ffff830136ddfee0:
> (XEN)    ffff8301308a0000 0000000000000000 ffff82d0402a1798 0000000000001000
> (XEN)    000000000008c800 ffffffffffffffff ffffffff81601f58 0000000000000020
> (XEN)    0000000040000000 0000000000000007 0000000000000007 ffffffff8168f2a0
> (XEN)    0000000000000200 0000000000000000 0000000000000001 0000000000000000
> (XEN)    ffffffff81666a80 ffffffff81617830 000000fa00000000 ffffffff8100712e
> (XEN)    0000000000000000 0000000000000046 ffffffff81601e78 0000000000000000
> (XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
> (XEN)    0000e01000000001 ffff8301308a0000 00000030f9686000 00000000001526e0
> (XEN)    0000000000000000 0000000000000000 0006020200000000 0000000000000000
> (XEN) Xen call trace:
> (XEN)    [<ffff82d040206fa9>] R __domain_crash+0x9/0x80
> (XEN)    [<ffff82d0402a1798>] S vmx_asm_vmexit_handler+0xf8/0x210
> (XEN)
> (XEN) Pagetable walk from 0000000000000208:
> (XEN)  L4[0x000] = 0000000000000000 ffffffffffffffff
> (XEN)
> (XEN) ****************************************
> (XEN) Panic on CPU 1:
> (XEN) FATAL PAGE FAULT
> (XEN) [error_code=0000]
> (XEN) Faulting linear address: 0000000000000208
> (XEN) ****************************************
> (XEN)
> (XEN) Reboot in five seconds...
> 
>  
> 
> HVM guest config file:
> 
> --------------------------------
> 
> name = "debian7"
> 
> builder= "hvm"
> memory = 1024
> vcpus = 1
> cpus= ["1"]
> disk=["qcow2:/home/test/debian7.qcow2,xvda,w"]
> vnc = 1
> vnclisten = '127.0.0.1'
> vncdisplay = 1
> boot = "d"
> serial = "pty"
> 
> Discussion
> --------------------------------
> 
> Hello, i am conducting robustness tests on Xen 4.14. I bit flipped the
> field VM_EXIT_QUALIFICATION during the exit handling of an
> IO_INSTRUCTION. After a VM Read on that field, Xen crashes with this
> error log. This is my (possible) explanation:
> 
> Xen correctly detects an error in the field, resulting in a domain crash
> (Unexpected PIO status 1) called from this point in the code:
> 
> xen/arch/x86/hvm/io.c
> ```
> gprintk(XENLOG_ERR, "Unexpected PIO status %d, port %#x %s 0x%0*x\n",
>                 rc, port, dir == IOREQ_WRITE ? "write" : "read",
>                 size * 2, data & ((1u << (size * 8)) - 1));
> domain_crash(curr->domain);
> return false;
> ```
> 
> In the `handle_pio` function, in the handling of the `IO_INSTRUCTION`
> exit reason. However, the code continues and ends here for an issue in
> the processor mode:
> 
> /xen/arch/x86/hvm/vmx/vmx.c
> ```
> mode = vmx_guest_x86_mode(v);
>     if ( mode == 8 ? !is_canonical_address(regs->rip)
>                    : regs->rip != regs->eip )
>     {
>         gprintk(XENLOG_WARNING, "Bad rIP %lx for mode %u\n", regs->rip,
> mode);
> 
>         if ( vmx_get_cpl() )
>         {
>             __vmread(VM_ENTRY_INTR_INFO, &intr_info);
>             if ( !(intr_info & INTR_INFO_VALID_MASK) )
>                 hvm_inject_hw_exception(TRAP_gp_fault, 0);
>             /* Need to fix rIP nevertheless. */
>             if ( mode == 8 )
>                 regs->rip = (long)(regs->rip << (64 - VADDR_BITS)) >>
>                             (64 - VADDR_BITS);
>             else
>                 regs->rip = regs->eip;
>         }
>         else
>             domain_crash(v->domain);
>     }
> ```
> 
> However, the domain pointer has already been deallocated due to the
> previous domain crash, resulting in a page fault that leads to Xen crashing.
> 
> I would like to report this crash and ask for your opinion. Based on
> previous research, I do not believe it has been reported before. I am
> unsure if it could potentially be a security issue, hence I am posting
> it here. Let me know if I should provide other results. Thank you for
> your kind response in advance.


Answering somewhat out of order.

You've posted this publicly so the cat is out of the bag regardless.  If
you have concerns about security, please email security@xxxxxxx as your
first point of contact.

That said, nested virt is not security supported.  It's still an
experimental feature, so bugs like this are fine to come straight to the
public mailing list.

The "Bad rIP" logic is buggy and has since been deleted.  Xen 4.14 is a
very old version of Xen, and is outside of general bugfix support.

It is quite likely that this bug still exists, but please use an
up-to-date version of Xen.  Fixes need developing against master and are
unlikely to be backported in this case, given its experimental status.

Also, you should be using a debug build of Xen generally for work like
this.  (I have no idea if it would alter your observations.)


I'm not sure what I think about bitflipping the exit qualification.  In
other places, that will definitely cause more severe crashes, and Xen
won't be getting in the game of auditing the VMX implementation against
the VMX spec.


For this crash you've got, there should be no way for the domain pointer
to have been freed in the sequence you've described.  While the vCPU is
still scheduled, the structures will remain.

Looking at the second backtrace, It looks suspiciously like a NULL
pointer was passed into __domain_crash() from vmx_asm_vmexit_handler()
but there's no such call call, nor a tailcall out of the handler, even
taking a peek at the 4.14 code.

If you have local changes, I'd look at those first.

~Andrew