Re: Keystone Issue

[CodeWiz2280 <codewiz2280@xxxxxxxxx>]

On Fri, Jun 5, 2020 at 3:37 AM Bertrand Marquis
<Bertrand.Marquis@xxxxxxx> wrote:
>
> Hi,
>
> > On 5 Jun 2020, at 03:29, CodeWiz2280 <codewiz2280@xxxxxxxxx> wrote:
> >
> > On Thu, Jun 4, 2020 at 2:24 PM Julien Grall <julien@xxxxxxx> wrote:
> >>
> >> Hi,
> >>
> >> On 04/06/2020 13:07, CodeWiz2280 wrote:
> >>> On Thu, Jun 4, 2020 at 6:16 AM Julien Grall <julien@xxxxxxx> wrote:
> >>>>
> >>>> Hi,
> >>>>
> >>>> On 04/06/2020 10:08, Bertrand Marquis wrote:
> >>>>> I would have thought that linux would have need some memory, even small 
> >>>>> in the 32bit space in order to boot.
> >>>>
> >>>> Yes it needs some, but then they are switching to use the high memory
> >>>> alias after the MMU has been switch on.
> >>>>
> >>>>  From my understanding, the only difference is the page-tables will
> >>>> point to the high memory alias address rather than the low memory one.
> >>>> Linux will still be located at the same place but now accessed from the
> >>>> high memory alias rather than the low one.
> >>>>
> >>>> Note that AFAICT the secondary CPUs will still be brought-up using the
> >>>> low memory alias.
> >>>>
> >>>>> I could understand that some memory in the low address space needs to 
> >>>>> be reserved by Linux as DMA area for peripherals not supporting 36-bit 
> >>>>> addresses, but the whole low memory sounds like a big restriction.
> >>>> Many platforms have devices only supporting 32-bit DMA, but none of them
> >>>> require such aliasing. So this doesn't look to be the issue here.
> >>>>
> >>>> TBH, this code is only used by Keystone and switching address space is
> >>>> expensive (you have to turn off the MMU, updates page-tables, flush the
> >>>> cache...). I find hard to believe a developper would have come up with
> >>>> this complexity if it were possible to always use the low memory address
> >>>> range. It is even harder to believe Linux community would have accepted 
> >>>> it.
> >>>>
> >>>>>
> >>>>> Would it be possible to have a bit more information on the “problem 
> >>>>> with peripherals” here ?
> >>>>
> >>>> I am curious as well, so I looked in more depth :). Going through the
> >>>> Linux history, one of the commit message [1] suggests they are switching
> >>>> to a coherent address space. The datasheet [2] (page 75) also confirm
> >>>> that the low region is not IO coherent.
> >>>>
> >>>> So I think you would not be able to do DMA without flush the cache which
> >>>> can be pretty expensive. For a PoC, it might be possible to force Linux
> >>>> flushing the area before and after each DMA request. This should be
> >>>> possible by marking the devices as not coherent.
> >>>>
> >>>> Although, I am not entirely sure if there is any fallout.
> >>>>
> >>>> @Dave, do you think it is possible for you to have a try? I can provide
> >>>> the patch for Linux to disable DMA coherency if possible.
> >>> I attempted to do that, where I removed the "dma-coherent" flags from
> >>> the device tree.  There are likely other issues, but the most glaring
> >>> problem that I ran into is that the ethernet does not work.  Eth0
> >>> shows up in ifconfig but there is no activity on it after a small
> >>> handful of message exchanges, whereas booting without Xen it seems to
> >>> work fine even if left in 32-bit mode (with the dma-coherent
> >>> disabled).  I don't know what implications behind the scenes there are
> >>> trying to stay in the lower 0x8000_0000 alias range either though.
> >>
> >> Thank you for the answer. As wrote, Linux is working fine in 32-bit mode
> >> when dma-coherent is left in 32-bit mode. So this suggest a different
> >> issue on the platform.
> >>
> >> Given that you receive an handful of packet and then nothing, this would
> >> lead to maybe an interrupt problem. Can you check whether the number of
> >> interrupts increments the same way on baremetal and on Xen?
> >>
> >> Dumping /proc/interrupts should be sufficient.
> >>
> > I am able to ping the board from itself, do you think it could still
> > be an interrupt issue?  It just cannot seem to ping out to a different
> > host (or ping from
> > my pc).  Unfortunately, the interrupts for the netcp Ethernet driver
> > on this board don't show up in the cat /proc/interrupts output under
> > the non-Xen kernel or
> > Xen loaded kernel from what I can tell.  I'm not sure how I would confirm 
> > that.
>
> Could you check the content of /proc/interrupts ?
>
> I did raise an issue several years ago on the keystone 2 related to 
> interrupts and virtualization (no with Xen but the context should still be 
> right):
> https://e2e.ti.com/support/processors/f/791/t/462126?Keystone-2-no-interrupts-received-out-of-80-and-92-
>
> There might be something to check in regards to level vs front interrupts for 
> forwarded interrupts.
>
The Keystone uses the netcp driver, which has interrupts from 40-79
listed in the device tree (arch/arm/boot/keystone-k2e-netcp.dtsi).
I'm using the same device tree between my non-xen standalone kernel
and my dom0 kernel booted by xen.  In the standalone (non-xen) kernel
the ethernet works fine, but I don't see any of its interrupts in the
output of /proc/iomem.  I'm not seeing them in /proc/iomem when
running dom0 under Xen either.  When booting with Xen I get this
behavior where the ifconfig output shows 1 RX message and 1 TX
message, and then nothing else.

> Regards
> Bertrand
>
>
>
> >
> >>> I
> >>> would rather run it as intended by switching to the upper
> >>> 0x8_0000_0000 alias region.
> >>
> >> I agree this would be ideal :).
> >>
> >> Cheers,
> >>
> >> --
> >> Julien Grall
>