Re: [Minios-devel] [UNIKRAFT PATCHv4 21/43] plat/kvm: Add Arm64 basic entry code

[Julien Grall <julien.grall@xxxxxxx>]

Hi Wei,

On 16/07/18 07:42, Wei Chen wrote:
> > -----Original Message-----
> > From: Julien Grall <julien.grall@xxxxxxx>
> > Sent: 2018年7月13日 18:42
> > To: Wei Chen <Wei.Chen@xxxxxxx>; minios-devel@xxxxxxxxxxxxxxxxxxxx;
> > simon.kuenzer@xxxxxxxxx
> > Cc: Kaly Xin <Kaly.Xin@xxxxxxx>; nd <nd@xxxxxxx>
> > Subject: Re: [Minios-devel] [UNIKRAFT PATCHv4 21/43] plat/kvm: Add Arm64 basic
> > entry code
> >
> >
> >
> > On 13/07/18 11:11, Wei Chen wrote:
> > > Hi Julien,
> > Hi Wei,
> >
> > > > -----Original Message-----
> > > > From: Julien Grall <julien.grall@xxxxxxx>
> > > > Sent: 2018年7月12日 18:06
> > > > To: Wei Chen <Wei.Chen@xxxxxxx>; minios-devel@xxxxxxxxxxxxxxxxxxxx;
> > > > simon.kuenzer@xxxxxxxxx
> > > > Cc: Kaly Xin <Kaly.Xin@xxxxxxx>; nd <nd@xxxxxxx>
> > > > Subject: Re: [Minios-devel] [UNIKRAFT PATCHv4 21/43] plat/kvm: Add Arm64
> > basic
> > > > entry code
> > > >
> > > >
> > > >
> > > > On 12/07/18 10:43, Wei Chen wrote:
> > > > > Hi Julien,
> > > > Hi Wei,
> > > >
> > > > > > -----Original Message-----
> > > > > > From: Julien Grall <julien.grall@xxxxxxx>
> > > > > > Sent: 2018年7月11日 23:52
> > > > > > To: Wei Chen <Wei.Chen@xxxxxxx>; minios-devel@xxxxxxxxxxxxxxxxxxxx;
> > > > > > simon.kuenzer@xxxxxxxxx
> > > > > > Cc: Kaly Xin <Kaly.Xin@xxxxxxx>; nd <nd@xxxxxxx>
> > > > > > Subject: Re: [Minios-devel] [UNIKRAFT PATCHv4 21/43] plat/kvm: Add Arm64
> > > > basic
> > > > > > entry code
> > > > > >
> > > > > >
> > > > > >
> > > > > > On 11/07/18 10:50, Wei Chen wrote:
> > > > > > > Hi Julien,
> > > > > > Hi Wei,
> > > > > >
> > > > > > > > -----Original Message-----
> > > > > > > > From: Julien Grall <julien.grall@xxxxxxx>
> > > > > > > > Sent: 2018年7月8日 6:24
> > > > > > > > To: Wei Chen <Wei.Chen@xxxxxxx>; minios-devel@xxxxxxxxxxxxxxxxxxxx;
> > > > > > > > simon.kuenzer@xxxxxxxxx
> > > > > > > > Cc: Kaly Xin <Kaly.Xin@xxxxxxx>; nd <nd@xxxxxxx>
> > > > > > > > Subject: Re: [Minios-devel] [UNIKRAFT PATCHv4 21/43] plat/kvm: Add
> > Arm64
> > > > > > basic
> > > > > > > > entry code
> > > > > > > >
> > > > > > > > Hi,
> > > > > > > >
> > > > > > > > On 07/06/2018 10:03 AM, Wei Chen wrote:
> > > > > > > > > QEMU/KVM can boot an Arm64 elf image without multiboot. In this case,
> > > > > > > > > we can plage _libkvmplat_entry to entry64.S directly as the vCPU
> > > > > > > > > reset entry. In this basic entry code, we just initialize the boot
> > > > > > > > > stack and prepare jumping to _libkvmplat_start.
> > > > > > > > Can you clarify why you are using the ELF format and not Image? My main
> > > > > > > > concern is the former does not seem to have a clear description of the
> > > > > > > > state of the VM at boot.
> > > > > > > It's little hard for me to answer your question. This is why I reply
> > this
> > > > > > > Comment at the last. Actually, when I was selecting the elf image I
> > didn’t
> > > > > > > think so much. And most Unikernel projects that I have involved (ukvm,
> > > > mini-
> > > > > > os)
> > > > > > > are using the elf image, both for arm and x86.
> > > > > > Mini-OS ARM is using the zImage format, not ELF. For UKVM, IIRC, you
> > > > > Yes, arm32 is zImage, but x86_64 is using OUTPUT_FORMAT("elf64-x86-64").
> > > > > And about the ukvm, Yes, I wrote it by myself, because the ukvm requires
> > elf
> > > > > format, it only support elf loader. We want to make our Unikernel
> > > > application
> > > > > like a normal elf application that can run on virtual machine directly. We
> > > > don't
> > > > > want to be compatible with the Linux image boot protocol, it's too complex
> > > > for us.
> > > > > And we don't have some many information need to pass.
> > > > Why is it too complex? The Image format is basically a couple fields to
> > > > slap at the top of your binary and a few guidelines for the tools how to
> > > > deal with the VM state. Most of the boot code will be the same.
> > > >
> > > > Also, most likely if you are going to support kvmtools, you would need
> > > > to use the Image format. For Xen, support for ELF will require a lot of
> > > > reworks and to be honest it is not worth the effort (I tried it before).
> > > I have read the Xen code and Shijie's Arm64 mini-OS patches. Shijie was
> > > Using elf format as image.From his code, I didn't see lots of reworks
> > > That I need to do If I will enable elf format for Xen?
> > Arm64 Mini-OS is not using ELF. Xen tools does not have ELF support for
> > Arm, and you can look at the patch [1] to see the image been created
> > using objcopy.
> Ok, you're right, xen tools doesn't support. but he was using the ELF format in
> linking stage. Here is Shijie's branch on GITHUB:
> https://github.com/zyzii/mini-os/blob/arm64-v3/arch/arm/arm64/minios-arm64.lds.S
Because EFL format is the standard for building C application. Both Xen 
and Linux does exactly the same.
> > > Kvmtools doesn't support Arm elf loader. But it supports other architectures.
> > > And as your said, if we using the kernel image format? How about ukvm,
> > > it requires elf format? Relative to kvmtools, I am more like to support ukvm.
> > > Because in a way, qemu and kvmtools are overlapping. Actually, I don't mind
> > to
> > > provide different image format for different platform.
> > May I ask why did you decide to require using ELF for UKVM? What was the
> > rationale? Is that just because x86 was doing it?
> Yes, x86 is using the elf format too. About rationale? I haven't asked
> Mato.
This is usually the standard to boot OS for x86. Xen and KVM does 
support the same for x86.
However, on Arm64, I haven't seen many people requesting ELF so far. I 
actually dropped the support for Arm in Xen because it was rotten.
> > But then why using that for QEMU? You need to compile your ELF assuming
> Can I ask you why QEMU supports elf format image? If QEMU support it, why I
> can't use it? While I was implementing the Arm64 enablement, elf format is
> the simplest way for me to verify my code. I just need to place my code at
> the entry point, then QEMU would help to load it to CPU reset entry.
Mostly likely because it was support on x86 and was easy to add support 
for Arm.
But as I said earlier, I am not against using ELF. However, there need 
to be some documentation telling you how to boot. At the moment, it is 
close to zero. So can you write down the expectation?
> > VA == PA, which is rather fragile. You also yet haven't told me what is
> > the expect ABI (e.g state of the machine...) for ELF.
> I don't think is fragile. My application has only one memory address space,
> VA == PA is mandatory. Most of the application developers will not touch the
> link script. My expect state of the machine is MMU off on reset.
>   
> > > > > > wrote it yourself. So I guess you based your understanding of the state
> > > > > > of the VM from somewhere?
> > > > > Each SoC, include QEMU virtual machine, they will give most system
> > registers
> > > > > a default value. I am using cortex-a53 VCPU, the QEMU gives it a reset
> > value
> > > > > to cpu->reset_sctlr = 0x00c50838; MMU, I/D cache are disabled.
> > > > > But yes, you're right, I ignore the MMU has been disabled already, and
> > > > disable
> > > > > It again in entry code.
> > > > This looks very fragile to rely on for a specific processor. What if we
> > > > decide to use another Cortex-A*? Or even another Arm CPU?
> > > >
> > > > We need a clear definition of the VM state. For instance, you say the
> > > > cache is disabled. Has the kernel image/DT been cleaned to PoC by the
> > > > tools? You will also have to be careful when writing the page-table as
> > > > "Cache disabled" does not rely mean "disabled". It is more a by-pass of
> > > > the cache.
> > > I can read them from sctlr if I want. I am curious about how does other
> > > Bootloader handle such case? Don't think my code is an entry code for
> > > OS. Let's treat it as a bootloader.
> > I don't understand what you mean. Whether your code is a bootloader or
> > an OS, you have to know exactly what is the state of your platform when
> > the processor jumps to your entry code.
> I just know my MMU and Caches are disabled while I was writing the code.
> I admit my MMU disable code in entry is unnecessary.
>
> > Your Image has been loaded by QEMU in the memory. If you modify the
> > page-table with cache disabled and not clean the DT. Then you may end up
> > using stall information when re-enabling the cache.
> What does the DT mean here? Device tree? I haven't re-enable the cache, I
> just enable it only once.
It looks like I mangle two sentence. I meant cleaning the cache. See 
more below.
> > If you disable the MMU, then this will still have an impact on your
> > cache as well. This is because KVM has alias to that region you will
> > modify and therefore you may have stall data in your cache when turning
> > on the MMU.
> Even I think this scenario could not happen in current code, I still
> Want to know how to avoid such problem before I re-enable the MMU?
Can you explain why do you think it can't happen? KVM has all the memory 
mapped with cacheable attributes. So you have a cacheable alias on your 
guest memory.
However, when you will write page-table, you will write with Device 
nGnRnE attributes (because MMU is disabled). So the cache will be 
bypassed. The cache may still contain stall data that you will hit when 
enabling MMU and cache.
To prevent such issue, you need to clean the cache potentially before 
and after updating the page-table area. I also mention before because it 
looks like the page-table will not be part of the kernel (the region is 
not populated), and therefore the cache state is unknown.
I would have recommended you to look at the Linux code, but that's GPL 
code and you write BSD code :). Maybe someone in your team can look at 
Linux and summarize the problem and way to fix it to you?
> > Overall, I think you should treat Unikraft as an OS as usually the
> > format will give you more guaranty than the platform itself. This will
> > also allow you to re-use that code accross multiple environment rather
> > than tailoring for QEMU.
> Yes, I agree with you. This is why Simon and I decide to create a plat/common
> Folder. We want to share most code as we can among different platforms.
> But in current stage, QEMU is our first target, we're focusing on enable
> and verify features on QEMU platform, and then port to other platforms.
>
> About the image format, I understand your concern. But I prefer to use
> another patch series to change it to improve its compatibility. If we
> support ukvm later, it will have a separate elf format image.
That should be fine, as long as you write down the assumptions and get 
the boot code correct.
Cheers,

--
Julien Grall

_______________________________________________
Minios-devel mailing list
Minios-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/minios-devel