Re: [Xen-devel] Running Xen on Nvidia Jetson-TK1

[Meng Xu <xumengpanda@xxxxxxxxx>]

On Sat, Mar 12, 2016 at 3:23 PM, Dushyant Behl
<myselfdushyantbehl@xxxxxxxxx> wrote:
> Hi Meng,
>
> On Sat, Mar 12, 2016 at 8:57 PM, Meng Xu <mengxu@xxxxxxxxxxxxx> wrote:
>>
>> On Sat, Mar 12, 2016 at 9:20 AM, Dushyant Behl
>> <myselfdushyantbehl@xxxxxxxxx> wrote:
>> > Hi Julien,
>> >
>> > Thanks for the quick reply.
>> >
>> > On Thu, Mar 10, 2016 at 10:38 AM, Julien Grall <julien.grall@xxxxxxx>
>> > wrote:
>> >>
>> >> Hi Dushyant,
>> >>
>> >> On 09/03/2016 20:37, Wei Liu wrote:
>> >>>
>> >>> On Tue, Mar 08, 2016 at 08:23:29AM +0000, Dushyant K Behl wrote:
>> >>>>
>> >>>> I'm working on a research project with IBM, and I want to run Xen on
>> >>>> Nvidia
>> >>>> Tegra Jetson-tk1 board.
>> >>>> I looked at a post on this mailing list
>> >>>> (http://lists.xenproject.org/archives/
>> >>>> html/xen-devel/2015-03/msg01122.html),
>> >>>> and I am using this git tree -
>> >>>>
>> >>>> git://xenbits.xen.org/people/ianc/xen.git
>> >>>> and branch  - tegra-tk1-jetson-v1
>> >>>>
>> >>>> But when I try to boot Xen on the board I am not able to see any
>> >>>> output
>> >>>> (even
>> >>>> with earlyprintk enabled).
>> >>>> After jumping to Xen the board just resets without showing any
>> >>>> output.
>> >>>>
>> >>>> I am using upstream u-boot with non secure mode enabled. I have also
>> >>>> tested
>> >>>> booting the Linux kernel on the same setup
>> >>>> and Linux 4.0 is able to boot with all 4 cores in HYP mode and kvm
>> >>>> enabled.
>> >>>>
>> >>>> Can anyone help me as to what I might have done wrong while using
>> >>>> Xen?
>> >>
>> >>
>> >> I never tried to boot Xen on Jetson-TK1 myself.
>> >>
>> >> Could you provide the command line you use to compile Xen (with
>> >> earlyprintk enabled) and the U-boot runes?
>> >
>> >
>> > I am running a ubuntu trusty schroot and I am using the Ubuntu/Linaro
>> > arm-linux-gnueabihf-gcc 4.7.3
>> > as the cross compiler to compile everything. The tree which I pointed
>> > towards (in my previous mail)
>> > contains the earlyprintk configurations for Jetson-TK1, so I am using
>> > this
>> > command to compile Xen with earlyprintk -
>> >
>> > make dist-xen debug=y CONFIG_EARLY_PRINTK=jetson XEN_TARGET_ARCH=arm32
>> >
>> > I want to update my current stage with Xen, after using this toolchain I
>> > am
>> > able to get Xen to print output
>> > on the Jetson-TK1's serial console and Xen is able to boot correctly
>> > with
>> > all 4 cores in HYP mode.
>> >
>> > But the problem is when Xen tries to boot the dom0 kernel and transfer
>> > control, I receive no output on the serial port.
>> > I am using Linux 4.1.0 as the dom0 kernel and the kernel is compiled
>> > with
>> > Xen specific options enabled.
>> > Also the same linux kernel is able to boot on the board without Xen.
>> >
>> > I am passing the dom0 kernel argument to Xen in the /chosen node in the
>> > device tree using these commands  -
>> >
>> > fdt mknod /chosen module
>> > fdt set /chosen/module compatible "xen,linux-zimage"
>> > "xen,multiboot-module"
>> > fdt set /chosen/module reg <0x0 $kernel_addr_r 0x0 0x$filesize>
>> > fdt set /chosen/module bootargs "$bootargs"
>>
>> I haven't run Xen on ARM yet. But just a random thought:
>> Is it possible that you need to provide the console=hvc0 for the boot
>> cmdline of dom0?
>>
>> dom0 needs share the serial port with Xen.
>
>
> I have added the serial port in the bootargs passed to the kernel. The
> bootargs
> passed to the dom0 when executing with Xen are the same bootargs which I
> pass
> when I run the kernel standalone without Xen and at that time I am able to
> see the
> kernel bootlog and a tty console on the serial port.

Ahha, that's the problem!

when Linux runs in native env., it will use serial port, say com1, you
can use console=com1 or console ttyS0 for the boot cmd.
However, when you run linux in dom0, linux won't see the hardware
ttyS0 directly. It must go through Xen to see it. So the linux's
cmdline in dom0 must be console=hvc0, instead of ttyS0 or com1.

Can you try this and see if it fix the problem?

>
>>
>> >
>> > I am loading Xen at top of the ram and linux and device tree at their
>> > default locations (near starting of ram).
>> >
>> > This is the Xen BootLog which I receive through the earlyprintk serial
>> > port
>> > -
>> >
>> > - UART enabled -
>> > - CPU 00000000 booting -
>> > - Xen starting in Hyp mode -
>> > - Zero BSS -
>> > - Setting up control registers -
>> > - CPU Init Done -- Turning on paging -
>> > - Ready -
>> > (XEN) Checking for initrd in /chosen
>> > (XEN) RAM: 0000000080000000 - 00000000ffefffff
>> > (XEN)
>> > (XEN) MODULE[0]: 0000000082000000 - 0000000082010000 Device Tree
>> > (XEN) MODULE[1]: 0000000081000000 - 0000000081544448 Kernel
>> > (XEN)  RESVD[0]: 0000000082000000 - 0000000082010000
>> > (XEN)
>> > (XEN) Command line: <NULL>
>> > (XEN) Placing Xen at 0x00000000ffc00000-0x00000000ffe00000
>> > (XEN) Update BOOTMOD_XEN from 00000000fd000000-00000000fd0f9701 =>
>> > 00000000ffc00000-00000000ffcf9701
>> > (XEN) Xen heap: 00000000fa000000-00000000fe000000 (16384 pages)
>> > (XEN) Dom heap: 507648 pages
>> > (XEN) Domain heap initialised
>> > (XEN) Platform: TEGRA124
>> > (XEN) No dtuart path configured
>> > (XEN) Bad console= option 'dtuart'
>> >  Xen 4.6-unstable
>> > (XEN) Xen version 4.6-unstable (root@) (arm-linux-gnueabihf-gcc
>> > (Ubuntu/Linaro 4.7.3-11ubuntu1) 4.7.3) debug=y Fri Mar 11 10:09:07 UTC
>> > 2016
>> > (XEN) Latest ChangeSet:
>> > (XEN) Processor: 413fc0f3: "ARM Limited", variant: 0x3, part 0xc0f, rev
>> > 0x3
>> > (XEN) 32-bit Execution:
>> > (XEN)   Processor Features: 00001131:00011011
>> > (XEN)     Instruction Sets: AArch32 A32 Thumb Thumb-2 ThumbEE Jazelle
>> > (XEN)     Extensions: GenericTimer Security
>> > (XEN)   Debug Features: 02010555
>> > (XEN)   Auxiliary Features: 00000000
>> > (XEN)   Memory Model Features: 10201105 40000000 01240000 02102211
>> > (XEN)  ISA Features: 02101110 13112111 21232041 11112131 10011142
>> > 00000000
>> > (XEN) Using PSCI-0.1 for SMP bringup
>> > (XEN) Generic Timer IRQ: phys=30 hyp=26 virt=27 Freq: 12000 KHz
>> > (XEN) GICv2 initialization:
>> > (XEN)         gic_dist_addr=0000000050041000
>> > (XEN)         gic_cpu_addr=0000000050042000
>> > (XEN)         gic_hyp_addr=0000000050044000
>> > (XEN)         gic_vcpu_addr=0000000050046000
>> > (XEN)         gic_maintenance_irq=25
>> > (XEN) GICv2: 192 lines, 4 cpus, secure (IID 0000043b).
>> > (XEN) Using scheduler: SMP Credit Scheduler (credit)
>> > (XEN) I/O virtualisation disabled
>> > (XEN) Allocated console ring of 32 KiB.
>> > (XEN) VFP implementer 0x41 architecture 4 part 0x30 variant 0xf rev 0x0
>> > (XEN) Bringing up CPU1
>> > - CPU 00000001 booting -
>> > - Xen starting in Hyp mode -
>> > - Setting up control registers -
>> > - CPU Init Done -- Turning on paging -
>> > - Ready -
>> > (XEN) CPU 1 booted.
>> > (XEN) Bringing up CPU2
>> > - CPU 00000002 booting -
>> > - Xen starting in Hyp mode -
>> > - Setting up control registers -
>> > - CPU Init Done -- Turning on paging -
>> > - Ready -
>> > (XEN) CPU 2 booted.
>> > (XEN) Bringing up CPU3
>> > - CPU 00000003 booting -
>> > - Xen starting in Hyp mode -
>> > - Setting up control registers -
>> > - CPU Init Done -- Turning on paging -
>> > - Ready -
>> > (XEN) CPU 3 booted.
>> > (XEN) Brought up 4 CPUs
>> > (XEN) P2M: 40-bit IPA
>> > (XEN) P2M: 3 levels with order-1 root, VTCR 0x80003558
>> > (XEN) *** LOADING DOMAIN 0 ***
>> > (XEN) Loading kernel from boot module @ 0000000081000000
>> > (XEN) Allocating 1:1 mappings totalling 128MB for dom0:
>> > (XEN) BANK[0] 0x00000088000000-0x00000090000000 (128MB)
>> > (XEN) Additional MMIO 40000-40040 (IRAM)
>> > (XEN) Additional MMIO 54200-54240 (Display A)
>> > (XEN) Additional MMIO 54240-54280 (Display B)
>> > (XEN) Additional MMIO 6000f-60010 (EXCEPTION VECTORS)
>> > (XEN) Additional MMIO 6000c-6000d (SYSREG)
>> > (XEN) Additional MMIO 1000-1001 (PCI CFG0)
>> > (XEN) Additional MMIO 1001-1002 (PCI CFG1)
>> > (XEN) Additional MMIO 12000-12010 (PCI IO)
>> > (XEN) Additional MMIO 13000-20000 (PCI MEM)
>> > (XEN) Additional MMIO 20000-40000 (PCI MEM (PREFETCH))
>> > (XEN) Additional IRQ 105 (DISPLAY)
>> > (XEN) TEGRA: Routing IRQ105 to dom0, ICTLR2, mask 0x000200
>> > (XEN) Additional IRQ 106 (DISPLAY B)
>> > (XEN) TEGRA: Routing IRQ106 to dom0, ICTLR2, mask 0x000400
>> > (XEN) Loading zImage from 0000000081000000 to
>> > 000000008fa00000-000000008ff44448
>> > (XEN) Allocating PPI 16 for event channel interrupt
>> > (XEN) Loading dom0 DTB to 0x000000008f800000-0x000000008f80f4a4
>> > (XEN) Scrubbing Free RAM on 1 nodes using 4 CPUs
>> > (XEN) ....done.
>> > (XEN) Initial low memory virq threshold set at 0x4000 pages.
>> > (XEN) Std. Loglevel: All
>> > (XEN) Guest Loglevel: All
>> > (XEN) *** Serial input -> DOM0 (type 'CTRL-a' three times to switch
>> > input to
>> > Xen)
>> > (XEN) Freed 272kB init memory.
>> >
>> > After this last line the system just hangs with no output on the serial
>> > console.
>>
>> But is dom0 booted up?
>> If you have network, can you connect to it?
>> If dom0 did boot up, then at least we know it's some configuration issue.
>> Again, I haven't tried on ARM yet, but I'd love to... :-)
>
>
> Yes, I have network on the board and I have configured the board to get
> fixed ip from my local
> network but even after waiting sometime I am not able to see any network
> response from the board.
> So I guessed maybe the dom0 is not booting correctly.

Hmm, probably... If Linux in dom0 have network on bare metal, it
should have network in dom0 after you boot Xen 4.5+.
Probably you can try the above suggestion and see if you can see some
interesting log?

Meng

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel