[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [PATCH v4 46/48] mm: shrinker: make memcg slab shrink lockless



Hi Dave,

On 2023/8/8 10:44, Dave Chinner wrote:
On Mon, Aug 07, 2023 at 07:09:34PM +0800, Qi Zheng wrote:
Like global slab shrink, this commit also uses refcount+RCU method to make
memcg slab shrink lockless.

This patch does random code cleanups amongst the actual RCU changes.
Can you please move the cleanups to a spearate patch to reduce the
noise in this one?

Sure, will do.


diff --git a/mm/shrinker.c b/mm/shrinker.c
index d318f5621862..fee6f62904fb 100644
--- a/mm/shrinker.c
+++ b/mm/shrinker.c
@@ -107,6 +107,12 @@ static struct shrinker_info 
*shrinker_info_protected(struct mem_cgroup *memcg,
                                         lockdep_is_held(&shrinker_rwsem));
  }
+static struct shrinker_info *shrinker_info_rcu(struct mem_cgroup *memcg,
+                                              int nid)
+{
+       return rcu_dereference(memcg->nodeinfo[nid]->shrinker_info);
+}

This helper doesn't add value. It doesn't tell me that
rcu_read_lock() needs to be held when it is called, for one....

How about adding a comment or an assertion here?


  static int expand_one_shrinker_info(struct mem_cgroup *memcg, int new_size,
                                    int old_size, int new_nr_max)
  {
@@ -198,7 +204,7 @@ void set_shrinker_bit(struct mem_cgroup *memcg, int nid, 
int shrinker_id)
                struct shrinker_info_unit *unit;
rcu_read_lock();
-               info = rcu_dereference(memcg->nodeinfo[nid]->shrinker_info);
+               info = shrinker_info_rcu(memcg, nid);

... whilst the original code here was obviously correct.

                unit = info->unit[shriner_id_to_index(shrinker_id)];
                if (!WARN_ON_ONCE(shrinker_id >= info->map_nr_max)) {
                        /* Pairs with smp mb in shrink_slab() */
@@ -211,7 +217,7 @@ void set_shrinker_bit(struct mem_cgroup *memcg, int nid, 
int shrinker_id)
static DEFINE_IDR(shrinker_idr); -static int prealloc_memcg_shrinker(struct shrinker *shrinker)
+static int shrinker_memcg_alloc(struct shrinker *shrinker)

Cleanups in a separate patch.

OK.


@@ -253,10 +258,15 @@ static long xchg_nr_deferred_memcg(int nid, struct 
shrinker *shrinker,
  {
        struct shrinker_info *info;
        struct shrinker_info_unit *unit;
+       long nr_deferred;
- info = shrinker_info_protected(memcg, nid);
+       rcu_read_lock();
+       info = shrinker_info_rcu(memcg, nid);
        unit = info->unit[shriner_id_to_index(shrinker->id)];
-       return 
atomic_long_xchg(&unit->nr_deferred[shriner_id_to_offset(shrinker->id)], 0);
+       nr_deferred = 
atomic_long_xchg(&unit->nr_deferred[shriner_id_to_offset(shrinker->id)], 0);
+       rcu_read_unlock();
+
+       return nr_deferred;
  }

This adds two rcu_read_lock() sections to every call to
do_shrink_slab(). It's not at all clear ifrom any of the other code
that do_shrink_slab() now has internal rcu_read_lock() sections....

The xchg_nr_deferred_memcg() will only be called in shrink_slab_memcg(),
so other code doesn't need to know that information?


@@ -464,18 +480,23 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, 
int nid,
        if (!mem_cgroup_online(memcg))
                return 0;
- if (!down_read_trylock(&shrinker_rwsem))
-               return 0;
-
-       info = shrinker_info_protected(memcg, nid);
+again:
+       rcu_read_lock();
+       info = shrinker_info_rcu(memcg, nid);
        if (unlikely(!info))
                goto unlock;
- for (; index < shriner_id_to_index(info->map_nr_max); index++) {
+       if (index < shriner_id_to_index(info->map_nr_max)) {
                struct shrinker_info_unit *unit;
unit = info->unit[index]; + /*
+                * The shrinker_info_unit will not be freed, so we can
+                * safely release the RCU lock here.
+                */
+               rcu_read_unlock();

Why - what guarantees that the shrinker_info_unit exists at this
point? We hold no reference to it, we hold no reference to any
shrinker, etc. What provides this existence guarantee?

The shrinker_info_unit is never freed unless the memcg is destroyed.
Here we hold the refcount of this memcg (mem_cgroup_iter() -->
css_tryget()), so the shrinker_info_unit will not be freed.


+
                for_each_set_bit(offset, unit->map, SHRINKER_UNIT_BITS) {
                        struct shrink_control sc = {
                                .gfp_mask = gfp_mask,
@@ -485,12 +506,14 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, 
int nid,
                        struct shrinker *shrinker;
                        int shrinker_id = calc_shrinker_id(index, offset);
+ rcu_read_lock();
                        shrinker = idr_find(&shrinker_idr, shrinker_id);
-                       if (unlikely(!shrinker || !(shrinker->flags & 
SHRINKER_REGISTERED))) {
-                               if (!shrinker)
-                                       clear_bit(offset, unit->map);
+                       if (unlikely(!shrinker || !shrinker_try_get(shrinker))) 
{
+                               clear_bit(offset, unit->map);
+                               rcu_read_unlock();
                                continue;
                        }
+                       rcu_read_unlock();
/* Call non-slab shrinkers even though kmem is disabled */
                        if (!memcg_kmem_online() &&
@@ -523,15 +546,20 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, 
int nid,
                                        set_shrinker_bit(memcg, nid, 
shrinker_id);
                        }
                        freed += ret;
-
-                       if (rwsem_is_contended(&shrinker_rwsem)) {
-                               freed = freed ? : 1;
-                               goto unlock;
-                       }
+                       shrinker_put(shrinker);

Ok, so why is this safe to call without holding the rcu read lock?
The global shrinker has to hold the rcu_read_lock() whilst calling
shrinker_put() to guarantee the validity of the list next pointer,
but we don't hold off RCU here so what guarantees a racing global
shrinker walk doesn't trip over this shrinker_put() call dropping
the refcount to zero and freeing occuring in a different context...

This will not be a problem, even if shrinker::refcount is reduced to
0 here, the racing global shrinker walk already holds the rcu lock.

        shrink_slab            shrink_slab_memcg
        ===========            =================

        rcu_read_lock()
        shrinker_put()
                               shrinker_put()

And in shrink_slab_memcg(), the shrinker is not required to traverse the
next bit in the shrinker_info_unit::map, so there is no need to hold the
rcu lock to ensure the existence of this shrinker.



+               /*
+                * We have already exited the read-side of rcu critical section
+                * before calling do_shrink_slab(), the shrinker_info may be
+                * released in expand_one_shrinker_info(), so reacquire the
+                * shrinker_info.
+                */
+               index++;
+               goto again;

With that, what makes the use of shrinker_info in
xchg_nr_deferred_memcg() in do_shrink_slab() coherent and valid?

Holding rcu lock can ensure that the old shrinker_info will not be
freed, and the shrinker_info_unit::nr_deferred can also be indexed from
the old shrinker_info::unit[x], so the updated nr_deferred will not be
lost.

Thanks,
Qi


-Dave.



 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.