From xen-users-bounces@lists.xenproject.org Tue May 01 06:42:11 2018 Return-path: Envelope-to: archives@lists.xen.org Delivery-date: Tue, 01 May 2018 06:42:11 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1fDOy0-0003fO-Mm; Tue, 01 May 2018 06:40:48 +0000 Received: from us1-rack-dfw2.inumbo.com ([104.130.134.6]) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1fDOxy-0003fJ-HJ for xen-users@lists.xenproject.org; Tue, 01 May 2018 06:40:46 +0000 X-Inumbo-ID: 62607222-4d0a-11e8-9728-bc764e045a96 Received: from mail-pg0-x243.google.com (unknown [2607:f8b0:400e:c05::243]) by us1-rack-dfw2.inumbo.com (Halon) with ESMTPS id 62607222-4d0a-11e8-9728-bc764e045a96; Tue, 01 May 2018 08:39:23 +0200 (CEST) Received: by mail-pg0-x243.google.com with SMTP id m21-v6so7843803pgv.8 for ; Mon, 30 Apr 2018 23:40:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=Ek69bEjhpoTGVBsBaolZD9UdBm4V3XwAdNU6fdxfIpc=; b=AZLcYTVEsVwOD0OELFsoKXfMJLo6Vk6j3QWK/kxnbdn47vW5tJUNC4x8dQ+zL22Y0D 5axBAPOJujwaKRwLl6k3YPlmTvFWfiqS+02kFkp3z+A3JLTkg21mlv40OQ91cPycUstY v0hqPz2vq6+uJIg1BEn/nFDz6XwBMOMMWq6s0Z4SrClRaqX+BqslcKKVhxjSeaCsGZv0 k0IzQBgWNroc+dA7Oebq7EJNuu62vdAkDJGEkk5cGY8RhyUZzEoiRdYi1J+n9RuKDXFo K4XIb9lYxYsUM+sXwnn0fzQ4bfBZIVpi1e2mVJ7FE+jSwgNthHkLvmaAIH/gGco5QK+y 2kXw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=Ek69bEjhpoTGVBsBaolZD9UdBm4V3XwAdNU6fdxfIpc=; b=gTidWk4oksuHnBEMQEJ6zspimS+odEe0bcNbxec7pn8uBGRE5fjJlcEN+Wm5t5C26H nkya5pGAmwzL3hF5l4bBXCGeENQ/Vy5hs24OkpbYu+aolQfg0xFnf6AufR9yoGPVs3kd zCndhhKLga29z+EHW/fDqbJixdutHIwRFXwpQMEVQ/Aq2pHPTeeZkx1T5cEFL4bhjuWV zaKt0CZWlN4cEgDB3wxit0pTo0YfxLS3rTZ5Az3WxpCpXLfDtU1EGRXcCT+nPkcSMtnD fA/3wqvNADrxwgErXWrmn/yahU4iDvwF9I/EpU7iwQh3BVD4y1+dUpIpbmvaaQh9TSTc Jk8Q== X-Gm-Message-State: ALQs6tC7r22lW97judEG+oswvvwRpwlTEsH2vosQThr71ArAWpjWJm7f NRxp8/Yo8mZUmw3aThA2q0RNhZTKPGqaatyNEKM= X-Google-Smtp-Source: AB8JxZrzaCcg0+qWQkP3GqVZbo59IVgT5r59UBv8WNap6e5OSGvD2KsGW87JsDIq4EwHCLyiZv+tnsDdLSfR17OhY/M= X-Received: by 2002:a65:625a:: with SMTP id q26-v6mr10026651pgv.113.1525156841638; Mon, 30 Apr 2018 23:40:41 -0700 (PDT) MIME-Version: 1.0 Received: by 10.100.179.44 with HTTP; Mon, 30 Apr 2018 23:40:41 -0700 (PDT) In-Reply-To: References: From: Tomas Mozes Date: Tue, 1 May 2018 08:40:41 +0200 Message-ID: To: Daz Day Subject: Re: [Xen-users] Xen DomU's randomly freezing X-BeenThere: xen-users@lists.xenproject.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Xen user discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: xen-users@lists.xenproject.org Content-Type: multipart/mixed; boundary="===============4504043421511245627==" Errors-To: xen-users-bounces@lists.xenproject.org Sender: "Xen-users" --===============4504043421511245627== Content-Type: multipart/alternative; boundary="000000000000c5d194056b1f3f97" --000000000000c5d194056b1f3f97 Content-Type: text/plain; charset="UTF-8" On Mon, Apr 30, 2018 at 6:56 PM, Daz Day wrote: > Hi All, > > I posted this out to the centos-virt mailing list and was asked by dunlapg > at umich.edu to post it here instead with some additional info (config > file of a VM, dmesg before a hang from inside a VM and dmesg after hang > from Dom0), so here's the original email, plus the extras at the bottom: > > Hi, > > I've tried hitting up the CentOS forums and thought I'd try here too as I > don't seem to be getting any bites. > > We've been in the process of migrating all our hypervisors over to CentOS > 7 using Xen. Once we had a few up and running we started to notice that the > DomU's would randomly freeze. They become unresponsive to any network > traffic, stop consuming CPU resources on the hypervisor and it's not > possible to log in to the console locally using: > virsh console > We can sometimes get as far as typing a username and hitting return, but > the DomU just hangs there. It doesn't seem to matter what Linux distro the > DomU is running, it affects them all. The only way we can get them back is > by destroying and recreating them (far from ideal!). > > After a bit of research and digging around, we eventually found these 2 > nuggets: > https://wiki.gentoo.org/wiki/Xen#Xen_domU_hanging_with_kernel_4.3.2B > https://www.novell.com/support/kb/doc.php?id=7018590 > > They both advise adding the command line argument: > gnttab_max_frames=256(the default is 32). > We applied this change and all hypervisors rand stable for around a week > until DomU's started freezing again (we've since tried even higher values, > to no avail). More research later led me to https://bugs.centos.org/vie > w.php?id=14258 and https://bugs.centos.org/view.php?id=14284 (which are > essentially the same report). There hasn't really been any movement on > these tickets unfortunately, but I have +1'd them. > > Have any others had issues with Xen and DomU's locking up in CentOS 7? Are > there any other fixes/workarounds? If any additional info is needed that > isn't already in the bug tickets or forum post, please let me know and I'll > be happy to provide whatever is required (these freezes are happening at > least once a day). > > Any help would be much appreciated and would mean my Ops guys could get a > decent sleep! > Cheers > Darren > > Additional info as requested: > Config file for one of the frozen VMs (there are other VMs with fewer > interfaces some with more memory, some with more vCPUs): > > t02red02 > bdbc6df1-e153-0dac-9a92-faa7b1270e97 > 2097152 > 2097152 > 1 > /usr/bin/pygrub > > linux > > > destroy > restart > restart > > > > > > > > > > > > > > >