Xen project Mailing List

This is late reply but, thank you for your kind reply, Dario Faggioli.

I made the new Xen4.5 binary with 'debug=y' option that I modified and install it.

Then, there was a kernel panic caused by the debugging code triggered by 'debug=y' during booting process(of dom0):

(XEN) ----[ Xen-4.5.0 x86_64 debug=y Not tainted ]----

(XEN) CPU: 7

(XEN) RIP: e008:[<ffff82d08012ba6c>] vcpu_migrate+0x1bd/0x374

(XEN) RFLAGS: 0000000000010096 CONTEXT: hypervisor

(XEN) rax: ffff82d080413320 rbx: ffff830461c3e068 rcx: ffff82d080428e60

(XEN) rdx: 0000000000201da0 rsi: ffff830086d2f000 rdi: ffff82d080280bc0

(XEN) rbp: ffff83045e77fe48 rsp: ffff83045e77fdd8 r8: 0000000000000007

(XEN) r9: 00000000deadbeef r10: ffff82d08024d870 r11: 0000000000000246

(XEN) r12: ffff830461c3e068 r13: 0000000000000007 r14: ffff82d080428e60

(XEN) r15: ffff830461c3e068 cr0: 0000000080050033 cr4: 00000000003526f0

(XEN) cr3: 0000000459c0c000 cr2: ffff82d081422020

(XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e010 cs: e008

(XEN) Xen stack trace from rsp=ffff83045e77fdd8:

(XEN) ffff82d080428e60 ffff82d080428e60 ffff83045e77fdf8 ffff82d080428e60

(XEN) ffff83045e77fe00 ffff830086d2f000 00201da000000086 0000000000000246

(XEN) ffff82d08012e0c6 ffff830086d2f000 ffff830461c3e068 ffff82d080428e60

(XEN) ffff82d080413320 ffff83045e78a000 ffff83045e77fe78 ffff82d08012be23

(XEN) 0000000000000007 ffff830086d2f000 0000000000000000 0000000000000000

(XEN) ffff83045e77fef8 ffff82d080107052 ffffffff00000000 0000000000000008

(XEN) 0000000000a0fb00 0000000000000000 ffffffffffffffff 0000000001000c02

(XEN) ffff83045e77fe32 ffff82d080196da9 0f00000000000001 ffff830086d2f000

(XEN) ffff880456bcc4c0 0000000000000007 0000000000000000 0000000000000000

(XEN) 00007cfba18800c7 ffff82d080234d9b ffffffff8100130a 0000000000000018

(XEN) 0000000000000000 0000000000000002 0000000001000c02 ffff880450b43e3c

(XEN) ffff880450b43e88 ffff880456bcc4c0 0000000000000246 0000000000000004

(XEN) 0000000000000000 0000000000000004 0000000000000018 ffffffff8100130a

(XEN) 0000000000000000 0000000000000007 0000000000000007 0001010000000000

(XEN) ffffffff8100130a 000000000000e033 0000000000000246 ffff880450b43e70

(XEN) 000000000000e02b 0000000000000000 0000000000000000 0000000000000000

(XEN) 0000000000000000 0000000000000007 ffff830086d2f000 00000033e1815200

(XEN) 0000000000000000

(XEN) Xen call trace:

(XEN) [<ffff82d08012ba6c>] vcpu_migrate+0x1bd/0x374

(XEN) [<ffff82d08012be23>] vcpu_force_reschedule+0x9e/0xa7

(XEN) [<ffff82d080107052>] do_vcpu_op+0x2e7/0x69d

(XEN) [<ffff82d080234d9b>] syscall_enter+0xeb/0x145

(XEN)

(XEN) Pagetable walk from ffff82d081422020:

(XEN) L4[0x105] = 0000000086092063 ffffffffffffffff

(XEN) L3[0x142] = 000000008608f063 ffffffffffffffff

(XEN) L2[0x00a] = 0000000000000000 ffffffffffffffff

(XEN)

(XEN) ****************************************

(XEN) Panic on CPU 7:

(XEN) FATAL PAGE FAULT

(XEN) [error_code=0000]

(XEN) Faulting linear address: ffff82d081422020

(XEN) ****************************************

Because I received a solution from my professor, I think it is a hard work to change Xen version.

Anyway, even if I turned on the 'debug=y' option, I could not get accurate information like with 'debug=n'; I get only linear address(ffff82d081422020).

So, I want to use a dis-assembly utility like 'addr2line' or 'objdump', which binaries can I use as input to the utility?

I'm using Ubuntu and previously I used '/boot/xen-syms-4.5.0' as input to the utilities.

But I could get wrong information, which told me a code line that is never related this problem.

I know that a beginner in the Xen developer community like me might be annoying you, but I ask you one more help.

Thanks for your help again.

On Fri, Sep 8, 2017 at 4:40 PM, Dario Faggioli <dario.faggioli@xxxxxxxxxx> wrote:

On Fri, 2017-09-08 at 14:33 +0900, Minjun Hong wrote:
> 1) I worked on the scheduler(credit scheduler) and I had a kernel
> panic by my modification.
>
To do what?

Can you show here what you are changing, e.g., by putting together a
quick patch?

It does not have to be properly formatted and follow all the rules of a
proper patch submission... it can just be a diff against original code,
to understand what you are changing.

> 2) I tried to get any information for debugging so that, I used
> serial console and could gain the serial logs like following:
>
> (XEN) ----[ Xen-4.5.0 x86_64 debug=n Not tainted ]----
>
First of all, when debugging, you should use a debug hypervisor, i.e.,
a build of Xen, done with 'debug=y', or in general, with debug enabled.

Also, Xen-4.5.0. Can you move to a more recent version?

> (XEN) CPU: 2
> (XEN) RIP: e008:[<ffff82d080120973>] csched_schedule+0x373/0x1180
> (XEN) RFLAGS: 0000000000010086 CONTEXT: hypervisor
> [..]
> (XEN) Xen call trace:
> (XEN) [<ffff82d080120973>]
> csched_schedule+0x373/0x1180
> (XEN) [<ffff82d080128cb3>] schedule+0xf3/0x590
> (XEN) [<ffff82d08015f295>] reprogram_timer+0x75/0xe0
> (XEN) [<ffff82d08012f64e>] timer_softirq_action+0x13e/0x210
> (XEN) [<ffff82d08012c03c>] __do_softirq+0x7c/0xd0
> (XEN) [<ffff82d080162e3a>] idle_loop+0x3a/0x70
>
Again, use a debug hypervisor, compiled with frame pointers.

> (XEN) Pagetable walk from ffff830088002c98:
> (XEN) L4[0x106] = 0000000086075063 ffffffffffffffff
> (XEN) L3[0x002] = 0000000086071063 ffffffffffffffff
> (XEN) L2[0x040] = 0000000000000000 ffffffffffffffff
> (XEN)
> (XEN) ****************************************
> (XEN) Panic on CPU 2:
> (XEN) FATAL PAGE FAULT
> (XEN) [error_code=0000]
> (XEN) Faulting linear address: ffff830088002c98
> (XEN) ****************************************

> I want to know where I should start debugging from.
> However, although I'm using serial console, I could get not enough
> clues only from the kernel log:
> 1) I could figure out what line and file caused the panic by its call
> trace, but it is too rough so it does not help me.
>
That's exactly from where you usually start: looking at what's at the
instruction that cause the system to explode, in your case, at address
0xffff82d080120973.

You can figure that out by disassembling the Xen hypervisor binary,
with `objdump', and looking up that address. Or you can use addr2line,
to have an indication of the same thing, but in the C sources.

I'm not sure what you mean with "it is too rough". At least the address
of the instruction that caused the system to fail in the Xen binary, is
usually pretty accurate (then, of course, you have to look at
surrounding instructions, check how you got there, etc.).

Again, make sure you use a debug hypervisor.

> 2) What linear address brings about this situation; 'Faulting linear
> address', but it is just an address and not recognizable something
> that human cannot read.
>
> I think, literally, the 'Faulting linear address' is key point
> because I heard that it represents bad address that I should never
> access.
>
It is, but the only way of understanding why you hit such an access
violation, is understand what the code is doing when it happens.

Regards,
Dario
--
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

Re: [Xen-devel] Faulting linear address??