[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

grant table issues mapping a ring order 10



Hi Juergen, Boris,

I am trying to increase the size of the rings used for Xen 9pfs
connections for performance reasons and also to reduce the likehood of
the backend having to wait on the frontend to free up space from the
ring.

FYI I realized that we cannot choose order 11 or greater in Linux
because then we incur into the hard limit CONFIG_FORCE_MAX_ZONEORDER=11.
But that is not the reason why I am writing to you :-)


The reason why I am writing is that even order 10 fails for some
grant-table related reason I cannot explain. There are two rings, each
of them order 10. Mapping the first ring results into an error. (Order 9
works fine, resulting in both rings being mapped correctly.)

QEMU tries to map the refs but gets an error:

  gnttab: error: mmap failed: Invalid argument
  xen be: 9pfs-0: xen be: 9pfs-0: xengnttab_map_domain_grant_refs failed: 
Invalid argument
  xengnttab_map_domain_grant_refs failed: Invalid argument

The error comes from Xen. The hypervisor returns GNTST_bad_gntref to
Linux (drivers/xen/grant-table.c:gnttab_map_refs). Then:

        if (map->map_ops[i].status) {
                        err = -EINVAL;
                        continue;
                }

So Linux returns -EINVAL to QEMU. The ref seem to be garbage. The
following printks are in Xen in the implemenation of map_grant_ref:

(XEN) DEBUG map_grant_ref 1017 ref=998 nr=2560
(XEN) DEBUG map_grant_ref 1017 ref=999 nr=2560
(XEN) DEBUG map_grant_ref 1013 ref=2050669706 nr=2560
(XEN) grant_table.c:1015:d0v0 Bad ref 0x7a3abc8a for d1
(XEN) DEBUG map_grant_ref 1017 ref=0 nr=2560
(XEN) DEBUG map_grant_ref 1017 ref=19 nr=2560
(XEN) DEBUG map_grant_ref 1017 ref=0 nr=2560
(XEN) DEBUG map_grant_ref 1013 ref=56423797 nr=2560
(XEN) grant_table.c:1015:d0v0 Bad ref 0x35cf575 for d1
(XEN) DEBUG map_grant_ref 1013 ref=348793 nr=2560
(XEN) grant_table.c:1015:d0v0 Bad ref 0x55279 for d1
(XEN) DEBUG map_grant_ref 1013 ref=1589921828 nr=2560
(XEN) grant_table.c:1015:d0v0 Bad ref 0x5ec44824 for d1
(XEN) DEBUG map_grant_ref 1017 ref=0 nr=2560
(XEN) DEBUG map_grant_ref 1013 ref=2070386184 nr=2560
(XEN) grant_table.c:1015:d0v0 Bad ref 0x7b679608 for d1
(XEN) DEBUG map_grant_ref 1013 ref=3421871 nr=2560
(XEN) grant_table.c:1015:d0v0 Bad ref 0x3436af for d1
(XEN) DEBUG map_grant_ref 1013 ref=1589921828 nr=2560
(XEN) grant_table.c:1015:d0v0 Bad ref 0x5ec44824 for d1
(XEN) DEBUG map_grant_ref 1017 ref=0 nr=2560
(XEN) DEBUG map_grant_ref 1013 ref=875999099 nr=2560
(XEN) grant_table.c:1015:d0v0 Bad ref 0x3436af7b for d1
(XEN) DEBUG map_grant_ref 1017 ref=0 nr=2560
(XEN) DEBUG map_grant_ref 1013 ref=2705045486 nr=2560
(XEN) grant_table.c:1015:d0v0 Bad ref 0xa13bb7ee for d1
(XEN) DEBUG map_grant_ref 1013 ref=4294967295 nr=2560
(XEN) grant_table.c:1015:d0v0 Bad ref 0xffffffff for d1
(XEN) DEBUG map_grant_ref 1013 ref=213291910 nr=2560
(XEN) grant_table.c:1015:d0v0 Bad ref 0xcb69386 for d1
(XEN) DEBUG map_grant_ref 1017 ref=0 nr=2560
(XEN) DEBUG map_grant_ref 1013 ref=4912 nr=2560
(XEN) grant_table.c:1015:d0v0 Bad ref 0x1330 for d1
(XEN) DEBUG map_grant_ref 1013 ref=167788925 nr=2560
(XEN) grant_table.c:1015:d0v0 Bad ref 0xa00417d for d1
(XEN) DEBUG map_grant_ref 1017 ref=24 nr=2560
(XEN) DEBUG map_grant_ref 1013 ref=167788925 nr=2560
(XEN) grant_table.c:1015:d0v0 Bad ref 0xa00417d for d1
(XEN) DEBUG map_grant_ref 1017 ref=0 nr=2560
(XEN) DEBUG map_grant_ref 1017 ref=0 nr=2560
(XEN) DEBUG map_grant_ref 1017 ref=0 nr=2560
(XEN) DEBUG map_grant_ref 1017 ref=0 nr=2560
(XEN) DEBUG map_grant_ref 1017 ref=0 nr=2560
(XEN) DEBUG map_grant_ref 1017 ref=0 nr=2560
(XEN) DEBUG map_grant_ref 1017 ref=0 nr=2560
(XEN) DEBUG map_grant_ref 1017 ref=0 nr=2560
(XEN) DEBUG map_grant_ref 1017 ref=0 nr=2560
(XEN) DEBUG map_grant_ref 1017 ref=0 nr=2560
(XEN) DEBUG map_grant_ref 1017 ref=0 nr=2560


Full logs https://pastebin.com/QLTUaUGJ
It is worth mentioning that no limits are being reached: we are below
2500 entries per domain and below the 64 pages of grant refs per domain.

What it seems to happen is that after ref 999, the next refs are garbage.
Do you have any ideas why?


I tracked the gnttab_expand calls in Dom0 and they seemed to be done
correctly. We need 5 grant table pages:

- order 10 -> 1024 refs
- 2 rings -> 2048 refs
- 512 refs per grant table page -> 4 pages
- plus few others refs by default -> 5 pages

[    3.896558] DEBUG gnttab_expand 1287 cur=1 extra=1 max=64 rc=0
[    5.115189] DEBUG gnttab_expand 1287 cur=2 extra=1 max=64 rc=0
[    6.334027] DEBUG gnttab_expand 1287 cur=3 extra=1 max=64 rc=0
[    7.350523] DEBUG gnttab_expand 1287 cur=4 extra=1 max=64 rc=0

As expected gnttab_expand gets called 4 times to add 4 more pages to the
initial page.


Thanks,

Stefano



 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.