[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [PATCH] xen/arm: smmuv1: remove iommu group when deassign a device


  • To: Mykyta Poturai <mykyta.poturai@xxxxxxxxx>
  • From: Rahul Singh <Rahul.Singh@xxxxxxx>
  • Date: Mon, 20 Jun 2022 09:38:18 +0000
  • Accept-language: en-US
  • Arc-authentication-results: i=2; mx.microsoft.com 1; spf=pass (sender ip is 63.35.35.123) smtp.rcpttodomain=lists.xenproject.org smtp.mailfrom=arm.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=arm.com; dkim=pass (signature was verified) header.d=armh.onmicrosoft.com; arc=pass (0 oda=1 ltdi=1 spf=[1,1,smtp.mailfrom=arm.com] dkim=[1,1,header.d=arm.com] dmarc=[1,1,header.from=arm.com])
  • Arc-authentication-results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=arm.com; dmarc=pass action=none header.from=arm.com; dkim=pass header.d=arm.com; arc=none
  • Arc-message-signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=6+tmg8qjPFWiP8y7jqtZEugKcyw+bDuA9E8t0SVbl7g=; b=G8i2V1WZQofq++av+3pQmoCtv2zE7D5tjW97enlNVoHX7xZj6YFWj6NmQsMQR2sKW+G4K12PhlEJOb5AG8qKE9ccEmd2Q1B483D5+HdsDCjYBb87OSbAOTwf8VC6AKzUTs/rZnrsqrA/+3BQijpGfh24sQy+568SbLE/9xesTcu1RBydPbKoa1ehWk94biuHoEe39IcT32/NTh+uXPJV+SHH3bK1cLFWRt50IOchOdyGZL3zBuRRKW123wHBeRrELyU410kiggvl98OVyircY0Q6YBDr/oe25JvERRnsEOpgjDld6cIO6rNJfkgsDRadaLZmtM5IrbfaCEiyzGPURQ==
  • Arc-message-signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=6+tmg8qjPFWiP8y7jqtZEugKcyw+bDuA9E8t0SVbl7g=; b=CsaYJDhnIbAaf1y6N9zRkkmax+I6MVZoDGUL2MXYQRHQbW33zScY6ggFVUVxALpCiICs7OYT2Rr8Hu0hK7DwZGKhBeh7IPoqbBfI9EkZne5zpx0jRRWOyEL3D7LQ5CilCLJNt5hmgWoOSA3aETiGHpx8Dv7s7XBDiu/qjLiSnQeNp6BJapqs9M5/iy0Ac6CrIEqpIJnr3Zq8S4F0AFwvNFWXgbDxYDFDR0HmOpWKB8CmxzIjeR7z118UmU/bG88mU3nhSzwoBuuyRwVsxdzHP4zKVbg4en4T3vuo3nAA4dIY++tQVmfzzY7fakk2QcMX7Acb0Xffk3wuM67Jkro5jw==
  • Arc-seal: i=2; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=pass; b=S4k4r8mY/ka34mtlXQmuGov7OMvWRqjuFUuMgNnLT+Qp03tKkrDCW/DRbteH98BEysokMVQmT1cNiT0ws/RT67EXcr5ZGhZXHG8UpxmitzZShB8jSjAAucB9MlBQTAUL9TnguYat9dxjSpZ8yRK/56Y6W0oqn1511R5wwlL6BEGTwhIVRby9n5vaSeMxj9dd5OSO8u47IDgfu1PB8U5uh0tUQYJhg1nBFjJKtUSIeBeYVccNQ96KZ9rTVXWkc35dvMTak8B9L6H1YHnOmz0pNT3eIEuE7VwdyQwa1QlOF5TMThx/71w2SvA6P82Z6HpPTqthcMY7MNuItOhsqtEdEw==
  • Arc-seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=P+xlVf/lQ5kBOHCzd7wmM6GhilqMn+55R1CKmtVuNk6UcTISBOh/i4Cs5Bc63nNCX3OPMrPPZUV93vojPsgNpWrq/Ll+HUQf6oRIiCgjEwUeP83WI+/540T04t48ZNbmMV47JF3yU5n6SSzSqGMfHO6Jt4jNQzTnsz3BtQ0vDfUlrKDfPZGUWiXjxhOsHraulc2++xIQ1WX6N0w82VrB7bVvMqQ5fL5viH3lphA91Gez2dPlCviwDqYGXd3OQkjnKm6BPb3oG06GWFOCQ9AsSN3KYcS6YxXKeSsbG0CDWaOlNroJE4wL8YMS4sGS+bXL3rX57unh86FF58OyktOk6g==
  • Authentication-results-original: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=arm.com;
  • Cc: Bertrand Marquis <Bertrand.Marquis@xxxxxxx>, "Volodymyr_Babchuk@xxxxxxxx" <Volodymyr_Babchuk@xxxxxxxx>, "julien@xxxxxxx" <julien@xxxxxxx>, "sstabellini@xxxxxxxxxx" <sstabellini@xxxxxxxxxx>, "xen-devel@xxxxxxxxxxxxxxxxxxxx" <xen-devel@xxxxxxxxxxxxxxxxxxxx>
  • Delivery-date: Mon, 20 Jun 2022 09:39:05 +0000
  • List-id: Xen developer discussion <xen-devel.lists.xenproject.org>
  • Nodisclaimer: true
  • Original-authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=arm.com;
  • Thread-index: AQHYWlIekEFYa4Z+ZkOKg6prrRxnv60EB7eAgALvxoCABl1EgIBEoW2AgAA0fACAAZfdAIAEm8YA
  • Thread-topic: [PATCH] xen/arm: smmuv1: remove iommu group when deassign a device

Hi Mykyta,

> On 17 Jun 2022, at 12:15 pm, Mykyta Poturai <mykyta.poturai@xxxxxxxxx> wrote:
> 
>> Hi Mykyta,
>> 
>>> On 16 Jun 2022, at 8:48 am, Mykyta Poturai <mykyta.poturai@xxxxxxxxx> wrote:
>>> 
>>> Hi Julien, Rahul
>>> I've encountered a similar problem with IMX8 GPU recently. It wasn't probing
>>> properly after the domain reboot.  After some digging, I came to the same
>>> solution as Rahul and found this thread. I also encountered the occasional
>>> "Unexpected global fault, this could be serious" error message when 
>>> destroying
>>> a domain with an actively-working GPU.
>>> 
>>>> Hmmmm.... Looking at the code, arm_smmu_alloc_smes() doesn't seem to use
>>>> the domain information. So why would it need to be done every time it is 
>>>> assigned?
>>> Indeed after removing the arm_smmu_master_free_smes() call, both reboot and 
>>> global
>>> fault issues are gone. If I understand correctly, device removing is not yet
>>> supported, so I can't find a proper place for the 
>>> arm_smmu_master_free_smes() call.
>>> Should we remove the function completely or just left it commented for 
>>> later or
>>> something else?
>>> 
>>> Rahul, are you still working on this or could I send my patch?
>> 
>> Yes, I have this on my to-do list but I was busy with other work and it got 
>> delayed. 
>> 
>> I created another solution for this issue, in which we don’t need to call 
>> arm_smmu_master_free_smes() 
>> in arm_smmu_detach_dev() but we can configure the s2cr value to type fault 
>> in detach function.
>> 
>> Will call new function arm_smmu_domain_remove_master() in detach function 
>> that will revert the changes done 
>> by arm_smmu_domain_add_master()  in attach function.
>> 
>> I don’t have any board to test the patch. If it is okay, Could you please 
>> test the patch and let me know the result.
>> 
>> diff --git a/xen/drivers/passthrough/arm/smmu.c 
>> b/xen/drivers/passthrough/arm/smmu.c
>> index 69511683b4..da3adf8e7f 100644
>> --- a/xen/drivers/passthrough/arm/smmu.c
>> +++ b/xen/drivers/passthrough/arm/smmu.c
>> @@ -1598,21 +1598,6 @@ out_err:
>>        return ret;
>> }
>> 
>> -static void arm_smmu_master_free_smes(struct arm_smmu_master_cfg *cfg)
>> -{
>> -    struct arm_smmu_device *smmu = cfg->smmu;
>> -       int i, idx;
>> -       struct iommu_fwspec *fwspec = arm_smmu_get_fwspec(cfg);
>> -
>> -       spin_lock(&smmu->stream_map_lock);
>> -       for_each_cfg_sme(cfg, i, idx, fwspec->num_ids) {
>> -               if (arm_smmu_free_sme(smmu, idx))
>> -                       arm_smmu_write_sme(smmu, idx);
>> -               cfg->smendx[i] = INVALID_SMENDX;
>> -       }
>> -       spin_unlock(&smmu->stream_map_lock);
>> -}
>> -
>> static int arm_smmu_domain_add_master(struct arm_smmu_domain *smmu_domain,
>>                                      struct arm_smmu_master_cfg *cfg)
>> {
>> @@ -1635,6 +1620,20 @@ static int arm_smmu_domain_add_master(struct 
>> arm_smmu_domain *smmu_domain,
>>        return 0;
>> }
>> 
>> +static void arm_smmu_domain_remove_master(struct arm_smmu_domain 
>> *smmu_domain,
>> +                                     struct arm_smmu_master_cfg *cfg)
>> +{
>> +       struct arm_smmu_device *smmu = smmu_domain->smmu;
>> +       struct arm_smmu_s2cr *s2cr = smmu->s2crs;
>> +       struct iommu_fwspec *fwspec = arm_smmu_get_fwspec(cfg);
>> +       int i, idx;
>> +
>> +       for_each_cfg_sme(cfg, i, idx, fwspec->num_ids) {
>> +               s2cr[idx] = s2cr_init_val;
>> +               arm_smmu_write_s2cr(smmu, idx);
>> +       }
>> +}
>> +
>> static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device 
>> *dev)
>> {
>>        int ret;
>> @@ -1684,10 +1683,11 @@ static int arm_smmu_attach_dev(struct iommu_domain 
>> *domain, struct device *dev)
>> 
>> static void arm_smmu_detach_dev(struct iommu_domain *domain, struct device 
>> *dev)
>> {
>> +       struct arm_smmu_domain *smmu_domain = domain->priv;
>>        struct arm_smmu_master_cfg *cfg = find_smmu_master_cfg(dev);
>> 
>>        if (cfg)
>> -               arm_smmu_master_free_smes(cfg);
>> +               return arm_smmu_domain_remove_master(smmu_domain, cfg);
>> 
>> }
>> 
>> Regards,
>> Rahul
> 
> Hello Rahul,
> 
> For me, this patch fixed the issue with the GPU not probing after domain 
> reboot.

Thanks for testing the patch.
> But not fixed the "Unexpected Global fault" that occasionally happens when 
> destroying
> the domain with an actively working GPU. Although, I am not sure if this issue
> is relevant here.

Can you please if possible share the more details and logs so that I can look 
if this issue is relevant here ?

Thanks in advance.

Regards,
Rahul

 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.