[Xen-devel] questions of vm save/restore on arm64

[Chenxiao Zhao <chenxiao.zhao@xxxxxxxxx>]

Hi, 

My board is Hikey on which have octa-core of arm cortex-a53. I have applied patches [1] to try vm save/restore on arm.

These patches originally do not working on arm64. I have made some changes based on patch set [2].

What I have got so far is

1. if I run 'xl save -p guest memState' to leave guest in suspend state, then run 'xl unpause guest'.

    the guest can resume successfully. so I suppose the guest works find on suspend/resume. 

2. if I run 'xl restore -p memState' to restore guest and use xenctx to dump all vcpu's registers.

    all the registers are identical to the state on save. After I run 'xl unpause guest', I got no error but can not connect to console.

After restore the guest's PC is at a function called user_disable_single_step(), which is called by single_step_handler(). 

My question is

1. How could I debug guest on restore progress? are there any tools available?

2. From my understanding, the restore not working is because some status is missing when saving.

 e.g. on cpu_save, it know the domain is 64bit, but on cpu_load, it always think it is a 32bit domain. so I have hard coded the domain type to DOMAIN_64BIT.

Am I correct?

3. How could I dump all VM's status? I only found xenctx can dump vcpu's registers.

I have attached my patch and log below.

Looking forward for your feedback.

Thanks

xl list

Name                                        ID   Mem VCPUs      State   Time(s)

Domain-0                                     0  1024     8     r-----      11.7

root@linaro-alip:~# xl create guest.cfg

Parsing config from guest.cfg

[   39.238723] xen-blkback: ring-ref 8, event-channel 3, protocol 1 (arm-abi) persistent grants

root@linaro-alip:~# xl save -p guest memState

Saving to memState new xl format (info 0x3/0x0/931)

xc: info: Saving domain 1, type ARM

(XEN) HVM1 save: VCPU

(XEN) HVM1 save: A15_TIMER

(XEN) HVM1 save: GICV2_GICD

(XEN) HVM1 save: GICV2_GICC

(XEN) HVM1 save: GICV3

root@linaro-alip:~# /usr/lib/xen/bin/xenctx -a 1

PC:       ffffffc0000ab028

LR:       ffffffc00050458c

ELR_EL1:  ffffffc000086b34

CPSR:     200001c5

SPSR_EL1: 60000145

SP_EL0:   0000007ff6f2a850

SP_EL1:   ffffffc0140a7ca0

 x0: 0000000000000001    x1: 00000000deadbeef    x2: 0000000000000002

 x3: 0000000000000002    x4: 0000000000000004    x5: 0000000000000000

 x6: 000000000000001b    x7: 0000000000000001    x8: 000000618e589e00

 x9: 0000000000000000   x10: 0000000000000000   x11: 0000000000000000

x12: 00000000000001a3   x13: 000000001911a7d9   x14: 0000000000002ee0

x15: 0000000000000005   x16: 00000000deadbeef   x17: 0000000000000001

x18: 0000000000000007   x19: 0000000000000000   x20: ffffffc014163d58

x21: ffffffc014163cd8   x22: 0000000000000001   x23: 0000000000000140

x24: ffffffc000d5bb18   x25: ffffffc014163cd8   x26: 0000000000000000

x27: 0000000000000000   x28: 0000000000000000   x29: ffffffc0140a7ca0

SCTLR: 34d5d91d

TTBCR: 00000032b5193519

TTBR0: 002d000054876000

TTBR1: 0000000040dcf000

root@linaro-alip:~# xl destroy guest

(XEN) mm.c:1265:d0v1 gnttab_mark_dirty not implemented yet

root@linaro-alip:~# xl restore -p memState

Loading new save file memState (new xl fmt info 0x3/0x0/931)

 Savefile contains xl domain config in JSON format

Parsing config from <saved>

xc: info: (XEN) HVM2 restore: VCPU 0

Found ARM domain from Xen 4.7

xc: info: Restoring domain

(XEN) HVM2 restore: A15_TIMER 0

(XEN) HVM2 restore: A15_TIMER 0

(XEN) HVM2 restore: GICV2_GICD 0

(XEN) HVM2 restore: GICV2_GICC 0

(XEN) GICH_LRs (vcpu 0) mask=0

(XEN)    VCPU_LR[0]=0

(XEN)    VCPU_LR[1]=0

(XEN)    VCPU_LR[2]=0

(XEN)    VCPU_LR[3]=0

xc: info: Restore successful

xc: info: XenStore: mfn 0x39001, dom 0, evt 1

xc: info: Console: mfn 0x39000, dom 0, evt 2

root@linaro-alip:~# /usr/lib/xen/bin/xenctx -a 2

PC:       ffffffc0000ab028

LR:       ffffffc00050458c

ELR_EL1:  ffffffc000086b34

CPSR:     200001c5

SPSR_EL1: 60000145

SP_EL0:   0000007ff6f2a850

SP_EL1:   ffffffc0140a7ca0

 x0: 0000000000000000    x1: 00000000deadbeef    x2: 0000000000000002

 x3: 0000000000000002    x4: 0000000000000004    x5: 0000000000000000

 x6: 000000000000001b    x7: 0000000000000001    x8: 000000618e589e00

 x9: 0000000000000000   x10: 0000000000000000   x11: 0000000000000000

x12: 00000000000001a3   x13: 000000001911a7d9   x14: 0000000000002ee0

x15: 0000000000000005   x16: 00000000deadbeef   x17: 0000000000000001

x18: 0000000000000007   x19: 0000000000000000   x20: ffffffc014163d58

x21: ffffffc014163cd8   x22: 0000000000000001   x23: 0000000000000140

x24: ffffffc000d5bb18   x25: ffffffc014163cd8   x26: 0000000000000000

x27: 0000000000000000   x28: 0000000000000000   x29: ffffffc0140a7ca0

SCTLR: 34d5d91d

TTBCR: 00000000b5193519

TTBR0: 002d000054876000

TTBR1: 0000000040dcf000

root@linaro-alip:~# xl unpause guest

root@linaro-alip:~# xl list

Name                                        ID   Mem VCPUs      State   Time(s)

Domain-0                                     0  1024     8     r-----      22.2

guest                                        2     0     1     r-----       4.8

root@linaro-alip:~# /usr/lib/xen/bin/xenctx -a 2

PC:       ffffffc000084a00

LR:       ffffffc00050458c

ELR_EL1:  ffffffc000084a00

CPSR:     000003c5

SPSR_EL1: 000003c5

SP_EL0:   0000007ff6f2a850

SP_EL1:   ffffffc0140a7ca0

 x0: 0000000000000000    x1: 00000000deadbeef    x2: 0000000000000002

 x3: 0000000000000002    x4: 0000000000000004    x5: 0000000000000000

 x6: 000000000000001b    x7: 0000000000000001    x8: 000000618e589e00

 x9: 0000000000000000   x10: 0000000000000000   x11: 0000000000000000

x12: 00000000000001a3   x13: 000000001911a7d9   x14: 0000000000002ee0

x15: 0000000000000005   x16: 00000000deadbeef   x17: 0000000000000001

x18: 0000000000000007   x19: 0000000000000000   x20: ffffffc014163d58

x21: ffffffc014163cd8   x22: 0000000000000001   x23: 0000000000000140

x24: ffffffc000d5bb18   x25: ffffffc014163cd8   x26: 0000000000000000

x27: 0000000000000000   x28: 0000000000000000   x29: ffffffc0140a7ca0

SCTLR: 34d5d91d

TTBCR: 00000000b5193519

TTBR0: 002d000054876000

TTBR1: 0000000040dcf000

root@linaro-alip:~# xl console guest

xenconsole: Could not read tty from store: Success

root@linaro-alip:~#

diff --git a/xen/arch/arm/hvm.c b/xen/arch/arm/hvm.c

index aee3353..411bab4 100644

--- a/xen/arch/arm/hvm.c

+++ b/xen/arch/arm/hvm.c

@@ -120,7 +120,8 @@ static int cpu_save(struct domain *d, hvm_domain_context_t *h)

         ctxt.dfar = v->arch.dfar;

         ctxt.dfsr = v->arch.dfsr;

 #else

-        /* XXX 64-bit */

+       ctxt.far = v->arch.far;

+       ctxt.esr = v->arch.esr;

 #endif

 #ifdef CONFIG_ARM_32

@@ -187,6 +188,9 @@ static int cpu_load(struct domain *d, hvm_domain_context_t *h)

     if ( hvm_load_entry(VCPU, h, &ctxt) != 0 )

         return -EINVAL;

+#ifdef CONFIG_ARM64

+    v->arch.type = DOMAIN_64BIT;

+#endif

     v->arch.sctlr = ctxt.sctlr;

     v->arch.ttbr0 = ctxt.ttbr0;

     v->arch.ttbr1 = ctxt.ttbr1;

@@ -199,7 +203,8 @@ static int cpu_load(struct domain *d, hvm_domain_context_t *h)

     v->arch.dfar = ctxt.dfar;

     v->arch.dfsr = ctxt.dfsr;

 #else

-    /* XXX 64-bit */

+    v->arch.far = ctxt.far;

+    v->arch.esr = ctxt.esr;

 #endif

 #ifdef CONFIG_ARM_32

diff --git a/xen/include/public/arch-arm/hvm/save.h b/xen/include/public/arch-arm/hvm/save.h

index db916b1..89e6e89 100644

--- a/xen/include/public/arch-arm/hvm/save.h

+++ b/xen/include/public/arch-arm/hvm/save.h

@@ -46,8 +46,12 @@ DECLARE_HVM_SAVE_TYPE(HEADER, 1, struct hvm_save_header);

 struct hvm_hw_cpu

 {

+#ifdef CONFIG_ARM_32

     uint64_t vfp[34]; /* Vector floating pointer */

     /* VFP v3 state is 34x64 bit, VFP v4 is not yet supported */

+#else

+    uint64_t vfp[66];

+#endif

     /* Guest core registers */

     struct vcpu_guest_core_regs core_regs;

@@ -60,6 +64,9 @@ struct hvm_hw_cpu

     uint32_t dacr;

     uint64_t par;

+    uint64_t far;

+    uint64_t esr;

+

     uint64_t mair0, mair1;

     uint64_t tpidr_el0;

     uint64_t tpidr_el1;

[1] http://lists.xen.org/archives/html/xen-devel/2015-12/msg01053.html

[2] http://lists.xen.org/archives/html/xen-devel/2014-04/msg01544.html

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel