Re: [Xen-devel] [PATCH v3 2/5] x86: use PDEP/PEXT for maddr/direct-map-offset conversion when available

[Andrew Cooper <andrew.cooper3@xxxxxxxxxx>]

On 10/09/18 11:00, Jan Beulich wrote:
>
>>>>> +                   ASM_OUTPUT2([ma] "=&r" (ma), [off] "+r" (va)),
>>>>> +                   [mask] "m" (ma_real_mask),
>>>>> +                   [shift] "m" (pfn_pdx_hole_shift),
>>>>> +                   [bmask] "m" (ma_va_bottom_mask),
>>>>> +                   [tmask] "m" (ma_top_mask)
>>>>> +                   : "ecx");
>>>>> +#undef SYMNAME
>>>>> +#else
>>>>> +    alternative_io("call do2ma",
>>>>> +                   /* pdep ma_real_mask(%rip), %rdi, %rax */
>>>>> +                   ".byte 0xc4, 0xe2, 0xc3, 0xf5, 0x05\n\t"
>>>>> +                   ".long ma_real_mask - 4 - .",
>>>>> +                   X86_FEATURE_BMI2,
>>>>> +                   ASM_OUTPUT2("=a" (ma), "+D" (va)), "m" (ma_real_mask)
>>>>> +                   : "rcx", "rdx", "rsi", "r8", "r9", "r10", "r11");
>>>>> +#endif
>>>> This is a massive clobber list in a function you've forced always
>>>> inline, and I can't see it doing nice things to the callsites.  TBH,
>>>> this still feels over-complicated for what it wants to be.
>>>>
>>>> Why not implement one single function in assembly that doesn't have
>>>> usual C calling conventions and can clobber %ecx and one other, and use
>>>> that?
>>>>
>>>> It avoids the need for potentially 256 almost-identical copies of the
>>>> function in the linkonce section, and avoids having the multiple
>>>> implementations in C/asm, avoids the need for any logic derived from
>>>> CONFIG_INDIRECT_THUNK, and avoids the need for massive clobber lists.
>>> Your response mixes things a bit too much for me to sort out what
>>> exactly you're concerned about: The massive clobber list exists only
>>> in the !CONFIG_INDIRECT_THUNK case. In that case though there
>>> aren't going to be up to 225 instances of the function. I'd be fine
>>> implementing the single one in assembly to reduce the clobber list,
>>> it was just to keep down assembly code size and also to have
>>> compiler generated code to compare against. I have to admit though
>>> that I'm not overly concerned about the !CONFIG_INDIRECT_THUNK
>>> case in the first place, so I also didn't see much reason to try to
>>> optimize it.
>>>
>>> For the CONFIG_INDIRECT_THUNK case, otoh, I'd really like to
>>> avoid dictating register allocation to the compiler. Hence the solution
>>> with the (possibly many) function instances. Overall code size still
>>> decreases with this approach, and on modern hardware the entire
>>> region of the image in which they live will remain cold. (Additionally
>>> not affecting register allocation here has the benefit of making it
>>> far easier to compare pre/post generated code.)
>> How many passes through the hypervisor hit two or more of these functions?
>>
>> I appreciate that for development, reducing the register perturbance can
>> be nice for diffing the resulting disassembly, but when it comes to
>> actually running the code, register renames at compile time are free,
>> whereas pulling multiple almost-identical copies of the stub into the
>> I-cache is not.
>>
>> Implementing this as a single function looks to be simpler in terms of
>> the change, will compile to a smaller result, and will run faster.  It
>> seems to be a win-win-win overall.
> Well, I continue to not really agree. First and foremost, as said before,
> the common (exclusive?) case is going to be that with "x86: use MOV
> for PFN/PDX conversion when possible" no calls will exist at runtime at
> all.

Taking this one step further, why don't we drop PDX entirely?

I seem to recall you saying that the one system it was introduced for
never shipped, at which point, why bother keeping the code around?

A separate point which has only just occurred to me is the humongous
pipeline stall which occurs when mixing legacy and VEX SSE instructions
on SandyBridge and later hardware.  I severely doubt that a single
transformation from ALU operations to PDEP/PEXT is going to make up for
the pipeline stall if the guest is using legacy SSE, although given how
common the PDX conversions are, I could easily believe that the net is
in the same ballpark.

> At that point all function instances could collectively be purged just
> like .init.text, if we cared enough. And then, for this particular case,
> leaving the compiler the widest possible choice of register allocation
> still seems pretty desirable to me. I'd agree with your "register
> renames at compile time are free" only if there weren't special uses of
> quite a few of the registers.
>
> As perhaps a prime example, consider the case where the
> transformation here gets done in the course of setting up another
> function's arguments. The compiler would have to avoid certain
> registers (or generate extra MOVs) if it had to first pass the input
> in a fixed register to the helper here (which then additionally also
> needs to be assumed to clobber several other ones).

Once again, I think you are focusing on the wrong aspect, and ending up
with something which is worse overall.

First, is there a single example here where the compiler sets up
registers before a function call? Given the sequence point, it is
distinctly non-trivial to optimise around.

Irrespective, a couple of extra movs (which are handled during register
renaming) is far less overhead than hitting a cold icache line, and
having 256 variations of this stub function is a very good way to hit a
lot of cold icache lines.

Ultimately, real performance numbers are the only way to say for sure,
but I expect you'll be surprised by the results you'd see.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel