Re: [PATCH v3 14/34] xen/riscv: introduce io.h

[Oleksii <oleksii.kurochko@xxxxxxxxx>]

On Mon, 2024-01-15 at 17:57 +0100, Jan Beulich wrote:
> On 22.12.2023 16:12, Oleksii Kurochko wrote:
> > --- /dev/null
> > +++ b/xen/arch/riscv/include/asm/io.h
> > @@ -0,0 +1,142 @@
> > +/* SPDX-License-Identifier: GPL-2.0-only */
> > +/*
> > + * {read,write}{b,w,l,q} based on arch/arm64/include/asm/io.h
> > + *   which was based on arch/arm/include/io.h
> > + *
> > + * Copyright (C) 1996-2000 Russell King
> > + * Copyright (C) 2012 ARM Ltd.
> > + * Copyright (C) 2014 Regents of the University of California
> > + */
> > +
> > +
> > +#ifndef _ASM_RISCV_IO_H
> > +#define _ASM_RISCV_IO_H
> > +
> > +#include <asm/byteorder.h>
> > +
> > +/*
> > + * The RISC-V ISA doesn't yet specify how to query or modify PMAs,
> > so we can't
> > + * change the properties of memory regions.  This should be fixed
> > by the
> > + * upcoming platform spec.
> > + */
> > +#define ioremap_nocache(addr, size) ioremap((addr), (size))
> > +#define ioremap_wc(addr, size) ioremap((addr), (size))
> > +#define ioremap_wt(addr, size) ioremap((addr), (size))
> 
> Nit: No need for the inner parentheses.
Thanks. I'll update that place.

> 
> > +/* Generic IO read/write.  These perform native-endian accesses.
> > */
> > +#define __raw_writeb __raw_writeb
> > +static inline void __raw_writeb(u8 val, volatile void __iomem
> > *addr)
> > +{
> > +   asm volatile("sb %0, 0(%1)" : : "r" (val), "r" (addr));
> > +}
> > +
> > +#define __raw_writew __raw_writew
> > +static inline void __raw_writew(u16 val, volatile void __iomem
> > *addr)
> > +{
> > +   asm volatile("sh %0, 0(%1)" : : "r" (val), "r" (addr));
> > +}
> > +
> > +#define __raw_writel __raw_writel
> > +static inline void __raw_writel(u32 val, volatile void __iomem
> > *addr)
> > +{
> > +   asm volatile("sw %0, 0(%1)" : : "r" (val), "r" (addr));
> > +}
> > +
> > +#ifdef CONFIG_64BIT
> > +#define __raw_writeq __raw_writeq
> > +static inline void __raw_writeq(u64 val, volatile void __iomem
> > *addr)
> > +{
> > +   asm volatile("sd %0, 0(%1)" : : "r" (val), "r" (addr));
> > +}
> > +#endif
> > +
> > +#define __raw_readb __raw_readb
> > +static inline u8 __raw_readb(const volatile void __iomem *addr)
> > +{
> > +   u8 val;
> > +
> > +   asm volatile("lb %0, 0(%1)" : "=r" (val) : "r" (addr));
> > +   return val;
> > +}
> > +
> > +#define __raw_readw __raw_readw
> > +static inline u16 __raw_readw(const volatile void __iomem *addr)
> > +{
> > +   u16 val;
> > +
> > +   asm volatile("lh %0, 0(%1)" : "=r" (val) : "r" (addr));
> > +   return val;
> > +}
> > +
> > +#define __raw_readl __raw_readl
> > +static inline u32 __raw_readl(const volatile void __iomem *addr)
> > +{
> > +   u32 val;
> > +
> > +   asm volatile("lw %0, 0(%1)" : "=r" (val) : "r" (addr));
> > +   return val;
> > +}
> > +
> > +#ifdef CONFIG_64BIT
> > +#define __raw_readq __raw_readq
> > +static inline u64 __raw_readq(const volatile void __iomem *addr)
> > +{
> > +   u64 val;
> > +
> > +   asm volatile("ld %0, 0(%1)" : "=r" (val) : "r" (addr));
> > +   return val;
> > +}
> > +#endif
> > +
> > +/*
> > + * Unordered I/O memory access primitives.  These are even more
> > relaxed than
> > + * the relaxed versions, as they don't even order accesses between
> > successive
> > + * operations to the I/O regions.
> > + */
> > +#define readb_cpu(c)               ({ u8  __r = __raw_readb(c); __r;
> > })
> > +#define readw_cpu(c)               ({ u16 __r = le16_to_cpu((__force
> > __le16)__raw_readw(c)); __r; })
> > +#define readl_cpu(c)               ({ u32 __r = le32_to_cpu((__force
> > __le32)__raw_readl(c)); __r; })
> > +
> > +#define
> > writeb_cpu(v,c)             ((void)__raw_writeb((v),(c)))
> > +#define
> > writew_cpu(v,c)             ((void)__raw_writew((__force 
> > u16)cpu_to_le16(v),(c)))
> > +#define
> > writel_cpu(v,c)             ((void)__raw_writel((__force 
> > u32)cpu_to_le32(v),(c)))
> > +
> > +#ifdef CONFIG_64BIT
> > +#define readq_cpu(c)               ({ u64 __r = le64_to_cpu((__force
> > __le64)__raw_readq(c)); __r; })
> > +#define
> > writeq_cpu(v,c)             ((void)__raw_writeq((__force 
> > u64)cpu_to_le64(v),(c)))
> > +#endif
> 
> How come there are endianness assumptions here on the MMIO accessed?
It is a hard story.

As you might expect it was copy from Linux Kernel where it was decided
to follow only LE way:
https://patchwork.kernel.org/project/linux-riscv/patch/20190411115623.5749-3-hch@xxxxxx/
One of the answers of the author of the commit:
    And we don't know if Linux will be around if that ever changes.
    The point is:
     a) the current RISC-V spec is LE only
     b) the current linux port is LE only except for this little bit
    There is no point in leaving just this bitrotting code around.  It
    just confuses developers, (very very slightly) slows down compiles
    and will bitrot.  It also won't be any significant help to a future
    developer down the road doing a hypothetical BE RISC-V Linux port.

>From the specs [1, p.5 ] it is mentioned that:
   The base ISA has been defined to have a little-endian memory system, 
   with big-endian or bi-endian as non-standard variants.

And in [3, p.6]:
   RISC-V base ISAs have either little-endian or big-endian memory 
   systems, with the privileged architecture further defining bi-endian 
   operation. Instructions are stored in memory as a sequence of 16-bit 
   little-endian parcels, regardless of memory system endianness. Parcels
   forming one instruction are stored at increasing halfword addresses, 
   with the lowest-addressed parcel holding the lowest-numbered bits in 
   the instruction specification.
   
    We originally chose little-endian byte ordering for the RISC-V memory
   system because little-endian systems are currently dominant 
   commercially (all x86 systems; iOS, Android, and Windows for ARM). A 
   minor point is that we have also found little-endian memory systems to
   be more natural for hardware designers. However, certain application 
   areas, such as IP networking, operate on big-endian data structures, 
   and certain legacy code bases have been built assuming big-endian 
   processors, so we have defined big-endian and bi-endian variants of 
   RISC-V.
   
    We have to fix the order in which instruction parcels are stored in 
   memory, independent of memory system endianness, to ensure that the 
   length-encoding bits always appear first in halfword address order. 
   This allows the length of a variable-length instruction to be quickly 
   determined by an instruction-fetch unit by examining only the first few
   bits of the first 16-bit instruction parcel.

[ this part is from source [2] which I can't find in [3] for some
uknown reason ]
   We further make the instruction parcels themselves little-endian to
   decouple the instruction encoding from the memory system endianness
   altogether. This design benefits both software tooling and bi-endian
   hardware. Otherwise, for instance, a RISC-V assembler or disassembler
   would always need to know the intended active endianness, despite that
   in bi-endian systems, the endianness mode might change dynamically
   during execution. In contrast, by giving instructions a fixed
   endianness, it is sometimes possible for carefully written software to
   be endianness-agnostic even in binary form, much like position-
   independent code.
   
   The choice to have instructions be only little-endian does have
   consequences, however, for RISC-V software that encodes or decodes
   machine instructions. Big-endian JIT compilers, for example, must swap
   the byte order when storing to instruction memory.
   
   Once we had decided to fix on a little-endian instruction encoding,
   this naturally led to placing the length-encoding bits in the LSB
   positions of the instruction format to avoid breaking up opcode fields.


[1] https://riscv.org/wp-content/uploads/2017/05/riscv-spec-v2.2.pdf
[2] https://five-embeddev.com/riscv-isa-manual/latest/intro.html
[3] https://riscv.org/wp-content/uploads/2017/05/riscv-spec-v2.2.pdf
> 
> As a file-wide remark: While I don't mind you using u<N> here for
> now,
> presumably to stay close to Linux, eventually - as we make progress
> with
> the conversion to uint<N>_t - this will need to diverge anyway.
Then I'll use uint<N>_t everywhere from the start. I am pretty sure
this file doesn't require sync often, so we can use  Xen style instead
of Linux.

> 
> > +/*
> > + * I/O memory access primitives. Reads are ordered relative to any
> > + * following Normal memory access. Writes are ordered relative to
> > any prior
> > + * Normal memory access.  The memory barriers here are necessary
> > as RISC-V
> > + * doesn't define any ordering between the memory space and the
> > I/O space.
> > + */
> > +#define __io_br()  do {} while (0)
> 
> Nit: Why are this and ...
> 
> > +#define __io_ar(v) __asm__ __volatile__ ("fence i,r" : : :
> > "memory");
> > +#define __io_bw()  __asm__ __volatile__ ("fence w,o" : : :
> > "memory");
> > +#define __io_aw()  do { } while (0)
> 
> ... this not expanding exactly the same?
I don't know the specific reason, it was done so in Linux kernel in
case when CONFIG_MMIOWB isn't supported:
https://elixir.bootlin.com/linux/latest/source/include/asm-generic/mmiowb.h#L61

https://elixir.bootlin.com/linux/latest/source/arch/riscv/include/asm/mmio.h#L136



~ Oleksii