Xen project Mailing List

Re: Help with Understanding vcpu xstate restore error during vm migration

To: Fonyuy-Asheri Caleb <fonyuy-asheri.caleb@xxxxxxxx>

From: Andrew Cooper <andrew.cooper3@xxxxxxxxxx>

Date: Sat, 13 Jul 2024 14:26:17 +0100

Autocrypt: addr=andrew.cooper3@xxxxxxxxxx; keydata= xsFNBFLhNn8BEADVhE+Hb8i0GV6mihnnr/uiQQdPF8kUoFzCOPXkf7jQ5sLYeJa0cQi6Penp VtiFYznTairnVsN5J+ujSTIb+OlMSJUWV4opS7WVNnxHbFTPYZVQ3erv7NKc2iVizCRZ2Kxn srM1oPXWRic8BIAdYOKOloF2300SL/bIpeD+x7h3w9B/qez7nOin5NzkxgFoaUeIal12pXSR Q354FKFoy6Vh96gc4VRqte3jw8mPuJQpfws+Pb+swvSf/i1q1+1I4jsRQQh2m6OTADHIqg2E ofTYAEh7R5HfPx0EXoEDMdRjOeKn8+vvkAwhviWXTHlG3R1QkbE5M/oywnZ83udJmi+lxjJ5 YhQ5IzomvJ16H0Bq+TLyVLO/VRksp1VR9HxCzItLNCS8PdpYYz5TC204ViycobYU65WMpzWe LFAGn8jSS25XIpqv0Y9k87dLbctKKA14Ifw2kq5OIVu2FuX+3i446JOa2vpCI9GcjCzi3oHV e00bzYiHMIl0FICrNJU0Kjho8pdo0m2uxkn6SYEpogAy9pnatUlO+erL4LqFUO7GXSdBRbw5 gNt25XTLdSFuZtMxkY3tq8MFss5QnjhehCVPEpE6y9ZjI4XB8ad1G4oBHVGK5LMsvg22PfMJ ISWFSHoF/B5+lHkCKWkFxZ0gZn33ju5n6/FOdEx4B8cMJt+cWwARAQABzSlBbmRyZXcgQ29v cGVyIDxhbmRyZXcuY29vcGVyM0BjaXRyaXguY29tPsLBegQTAQgAJAIbAwULCQgHAwUVCgkI CwUWAgMBAAIeAQIXgAUCWKD95wIZAQAKCRBlw/kGpdefoHbdD/9AIoR3k6fKl+RFiFpyAhvO 59ttDFI7nIAnlYngev2XUR3acFElJATHSDO0ju+hqWqAb8kVijXLops0gOfqt3VPZq9cuHlh IMDquatGLzAadfFx2eQYIYT+FYuMoPZy/aTUazmJIDVxP7L383grjIkn+7tAv+qeDfE+txL4 SAm1UHNvmdfgL2/lcmL3xRh7sub3nJilM93RWX1Pe5LBSDXO45uzCGEdst6uSlzYR/MEr+5Z JQQ32JV64zwvf/aKaagSQSQMYNX9JFgfZ3TKWC1KJQbX5ssoX/5hNLqxMcZV3TN7kU8I3kjK mPec9+1nECOjjJSO/h4P0sBZyIUGfguwzhEeGf4sMCuSEM4xjCnwiBwftR17sr0spYcOpqET ZGcAmyYcNjy6CYadNCnfR40vhhWuCfNCBzWnUW0lFoo12wb0YnzoOLjvfD6OL3JjIUJNOmJy RCsJ5IA/Iz33RhSVRmROu+TztwuThClw63g7+hoyewv7BemKyuU6FTVhjjW+XUWmS/FzknSi dAG+insr0746cTPpSkGl3KAXeWDGJzve7/SBBfyznWCMGaf8E2P1oOdIZRxHgWj0zNr1+ooF /PzgLPiCI4OMUttTlEKChgbUTQ+5o0P080JojqfXwbPAyumbaYcQNiH1/xYbJdOFSiBv9rpt TQTBLzDKXok86M7BTQRS4TZ/ARAAkgqudHsp+hd82UVkvgnlqZjzz2vyrYfz7bkPtXaGb9H4 Rfo7mQsEQavEBdWWjbga6eMnDqtu+FC+qeTGYebToxEyp2lKDSoAsvt8w82tIlP/EbmRbDVn 7bhjBlfRcFjVYw8uVDPptT0TV47vpoCVkTwcyb6OltJrvg/QzV9f07DJswuda1JH3/qvYu0p vjPnYvCq4NsqY2XSdAJ02HrdYPFtNyPEntu1n1KK+gJrstjtw7KsZ4ygXYrsm/oCBiVW/OgU g/XIlGErkrxe4vQvJyVwg6YH653YTX5hLLUEL1NS4TCo47RP+wi6y+TnuAL36UtK/uFyEuPy wwrDVcC4cIFhYSfsO0BumEI65yu7a8aHbGfq2lW251UcoU48Z27ZUUZd2Dr6O/n8poQHbaTd 6bJJSjzGGHZVbRP9UQ3lkmkmc0+XCHmj5WhwNNYjgbbmML7y0fsJT5RgvefAIFfHBg7fTY/i kBEimoUsTEQz+N4hbKwo1hULfVxDJStE4sbPhjbsPCrlXf6W9CxSyQ0qmZ2bXsLQYRj2xqd1 bpA+1o1j2N4/au1R/uSiUFjewJdT/LX1EklKDcQwpk06Af/N7VZtSfEJeRV04unbsKVXWZAk uAJyDDKN99ziC0Wz5kcPyVD1HNf8bgaqGDzrv3TfYjwqayRFcMf7xJaL9xXedMcAEQEAAcLB XwQYAQgACQUCUuE2fwIbDAAKCRBlw/kGpdefoG4XEACD1Qf/er8EA7g23HMxYWd3FXHThrVQ HgiGdk5Yh632vjOm9L4sd/GCEACVQKjsu98e8o3ysitFlznEns5EAAXEbITrgKWXDDUWGYxd pnjj2u+GkVdsOAGk0kxczX6s+VRBhpbBI2PWnOsRJgU2n10PZ3mZD4Xu9kU2IXYmuW+e5KCA vTArRUdCrAtIa1k01sPipPPw6dfxx2e5asy21YOytzxuWFfJTGnVxZZSCyLUO83sh6OZhJkk b9rxL9wPmpN/t2IPaEKoAc0FTQZS36wAMOXkBh24PQ9gaLJvfPKpNzGD8XWR5HHF0NLIJhgg 4ZlEXQ2fVp3XrtocHqhu4UZR4koCijgB8sB7Tb0GCpwK+C4UePdFLfhKyRdSXuvY3AHJd4CP 4JzW0Bzq/WXY3XMOzUTYApGQpnUpdOmuQSfpV9MQO+/jo7r6yPbxT7CwRS5dcQPzUiuHLK9i nvjREdh84qycnx0/6dDroYhp0DFv4udxuAvt1h4wGwTPRQZerSm4xaYegEFusyhbZrI0U9tJ B8WrhBLXDiYlyJT6zOV2yZFuW47VrLsjYnHwn27hmxTC/7tvG3euCklmkn9Sl9IAKFu29RSo d5bD8kMSCYsTqtTfT6W4A3qHGvIDta3ptLYpIAOD2sY3GYq2nf3Bbzx81wZK14JdDDHUX2Rs 6+ahAA==

Cc: xen-devel <xen-devel@xxxxxxxxxxxxxxxxxxxx>

Delivery-date: Sat, 13 Jul 2024 13:26:28 +0000

List-id: Xen developer discussion <xen-devel.lists.xenproject.org>

Ok, so trimming down to the relevant pieces, we have: On 11/07/2024 12:09 pm, Fonyuy-Asheri Caleb wrote: > ========================================================================================================= > Source server xen-cpuid -p > ========================================================================================================= > Raw policy: 32 leaves, 2 MSRs > ... > 00000001:ffffffff -> 000306f2:00100800:7ffefbff:bfebfbff > ... > 0000000d:00000000 -> 00000007:00000000:00000340:00000000 > 0000000d:00000001 -> 00000001:00000000:00000000:00000000 > 0000000d:00000002 -> 00000100:00000240:00000000:00000000 Haswell, with AVX, and ... > PV Default policy: 31 leaves, 2 MSRs > ... > 0000000d:00000000 -> 00000007:00000000:00000340:00000000 > 0000000d:00000001 -> 00000001:00000000:00000000:00000000 > 0000000d:00000002 -> 00000100:00000240:00000000:00000000 ... AVX available to PV guests. On the target, something very different. We have ... > ========================================================================================================= > Target server xen-cpuid -p > ========================================================================================================= > Raw policy: 39 leaves, 2 MSRs > ... > 00000001:ffffffff -> 00050654:00200800:7ffefbff:bfebfbff Skylake. (Which is very much not a Haswell.) > 0000000d:00000000 -> 000002ff:00000000:00000a88:00000000 > 0000000d:00000001 -> 0000000f:00000000:00000100:00000000 > 0000000d:00000002 -> 00000100:00000240:00000000:00000000 > 0000000d:00000003 -> 00000040:000003c0:00000000:00000000 > 0000000d:00000004 -> 00000040:00000400:00000000:00000000 > 0000000d:00000005 -> 00000040:00000440:00000000:00000000 > 0000000d:00000006 -> 00000200:00000480:00000000:00000000 > 0000000d:00000007 -> 00000400:00000680:00000000:00000000 > 0000000d:00000008 -> 00000080:00000000:00000001:00000000 > 0000000d:00000009 -> 00000008:00000a80:00000000:00000000 and AVX, MPX(x2), AVX512(x3), PROC_TRACE and PKRU. > Host policy: 38 leaves, 2 MSRs > CPUID: > leaf subleaf -> eax ebx ecx edx > 00000000:ffffffff -> 0000000d:756e6547:6c65746e:49656e69 > 00000001:ffffffff -> 00050654:00200800:47fee3ff:bfebfbff > 00000002:ffffffff -> 76036301:00f0b5ff:00000000:00c30000 > 00000004:00000000 -> 3c004121:01c0003f:0000003f:00000000 > 00000004:00000001 -> 3c004122:01c0003f:0000003f:00000000 > 00000004:00000002 -> 3c004143:03c0003f:000003ff:00000000 > 00000004:00000003 -> 3c07c163:0280003f:00004fff:00000004 > 00000007:00000000 -> 00000000:029cffdb:00000008:bc002400 > 0000000d:00000000 -> 0000021b:00000000:00000a88:00000000 > 0000000d:00000001 -> 0000000f:00000000:00000000:00000000 > 0000000d:00000003 -> 00000040:000003c0:00000000:00000000 > 0000000d:00000004 -> 00000040:00000400:00000000:00000000 > 0000000d:00000009 -> 00000008:00000a80:00000000:00000000 Xen has only chosen to activate MPX(x2) and PKRU, and indeed... > PV Default policy: 30 leaves, 2 MSRs > ... > 0000000d:00000000 -> 00000003:00000000:00000240:00000000 > 0000000d:00000001 -> 00000007:00000000:00000000:00000000 ... PV guests get nothing, while you're trying to migrate in a guest which... > ========================================================================================================= > VM xen-cpuid -p $domid > --------------------------------------------------------------------------------------------------------- > Domain 3 policy: 31 leaves, 2 MSRs > ... > 0000000d:00000000 -> 00000007:00000000:00000340:00000000 > 0000000d:00000001 -> 00000001:00000000:00000000:00000000 > 0000000d:00000002 -> 00000100:00000240:00000000:00000000 ... has AVX. This explains why you're hitting the "guest size > xen size" check. Migration should have failed earlier with a clearer message about the VM having features not available on the target, but I suspect I know why it didn't, and sadly its an item I've had on the todo list for a few years. Perhaps the more important question, are you booting the skylake with cpuid=no-avx on the command line by any chance? ~Andrew

©2013 Xen Project, A Linux Foundation Collaborative Project. All Rights Reserved.
Linux Foundation is a registered trademark of The Linux Foundation.
Xen Project is a trademark of The Linux Foundation.