Re: Keystone Issue

[CodeWiz2280 <codewiz2280@xxxxxxxxx>]

On Thu, Jun 4, 2020 at 2:24 PM Julien Grall <julien@xxxxxxx> wrote:
>
> Hi,
>
> On 04/06/2020 13:07, CodeWiz2280 wrote:
> > On Thu, Jun 4, 2020 at 6:16 AM Julien Grall <julien@xxxxxxx> wrote:
> >>
> >> Hi,
> >>
> >> On 04/06/2020 10:08, Bertrand Marquis wrote:
> >>> I would have thought that linux would have need some memory, even small 
> >>> in the 32bit space in order to boot.
> >>
> >> Yes it needs some, but then they are switching to use the high memory
> >> alias after the MMU has been switch on.
> >>
> >>   From my understanding, the only difference is the page-tables will
> >> point to the high memory alias address rather than the low memory one.
> >> Linux will still be located at the same place but now accessed from the
> >> high memory alias rather than the low one.
> >>
> >> Note that AFAICT the secondary CPUs will still be brought-up using the
> >> low memory alias.
> >>
> >>> I could understand that some memory in the low address space needs to be 
> >>> reserved by Linux as DMA area for peripherals not supporting 36-bit 
> >>> addresses, but the whole low memory sounds like a big restriction.
> >> Many platforms have devices only supporting 32-bit DMA, but none of them
> >> require such aliasing. So this doesn't look to be the issue here.
> >>
> >> TBH, this code is only used by Keystone and switching address space is
> >> expensive (you have to turn off the MMU, updates page-tables, flush the
> >> cache...). I find hard to believe a developper would have come up with
> >> this complexity if it were possible to always use the low memory address
> >> range. It is even harder to believe Linux community would have accepted it.
> >>
> >>>
> >>> Would it be possible to have a bit more information on the “problem with 
> >>> peripherals” here ?
> >>
> >> I am curious as well, so I looked in more depth :). Going through the
> >> Linux history, one of the commit message [1] suggests they are switching
> >> to a coherent address space. The datasheet [2] (page 75) also confirm
> >> that the low region is not IO coherent.
> >>
> >> So I think you would not be able to do DMA without flush the cache which
> >> can be pretty expensive. For a PoC, it might be possible to force Linux
> >> flushing the area before and after each DMA request. This should be
> >> possible by marking the devices as not coherent.
> >>
> >> Although, I am not entirely sure if there is any fallout.
> >>
> >> @Dave, do you think it is possible for you to have a try? I can provide
> >> the patch for Linux to disable DMA coherency if possible.
> > I attempted to do that, where I removed the "dma-coherent" flags from
> > the device tree.  There are likely other issues, but the most glaring
> > problem that I ran into is that the ethernet does not work.  Eth0
> > shows up in ifconfig but there is no activity on it after a small
> > handful of message exchanges, whereas booting without Xen it seems to
> > work fine even if left in 32-bit mode (with the dma-coherent
> > disabled).  I don't know what implications behind the scenes there are
> > trying to stay in the lower 0x8000_0000 alias range either though.
>
> Thank you for the answer. As wrote, Linux is working fine in 32-bit mode
> when dma-coherent is left in 32-bit mode. So this suggest a different
> issue on the platform.
>
> Given that you receive an handful of packet and then nothing, this would
> lead to maybe an interrupt problem. Can you check whether the number of
> interrupts increments the same way on baremetal and on Xen?
>
> Dumping /proc/interrupts should be sufficient.
>
I am able to ping the board from itself, do you think it could still
be an interrupt issue?  It just cannot seem to ping out to a different
host (or ping from
my pc).  Unfortunately, the interrupts for the netcp Ethernet driver
on this board don't show up in the cat /proc/interrupts output under
the non-Xen kernel or
Xen loaded kernel from what I can tell.  I'm not sure how I would confirm that.

> > I
> > would rather run it as intended by switching to the upper
> > 0x8_0000_0000 alias region.
>
> I agree this would be ideal :).
>
> Cheers,
>
> --
> Julien Grall