From mirageos-devel-bounces@lists.xenproject.org Mon Jun 01 17:51:42 2020 Return-path: Envelope-to: archives@lists.xenproject.org Delivery-date: Mon, 01 Jun 2020 17:51:42 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1jfoai-0001BQ-V1; Mon, 01 Jun 2020 17:51:16 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1jfoah-0001BL-EV for mirageos-devel@lists.xenproject.org; Mon, 01 Jun 2020 17:51:15 +0000 X-Inumbo-ID: 7cbbd3ec-a430-11ea-9947-bc764e2007e4 Received: from mail-qv1-xf2d.google.com (unknown [2607:f8b0:4864:20::f2d]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id 7cbbd3ec-a430-11ea-9947-bc764e2007e4; Mon, 01 Jun 2020 17:51:14 +0000 (UTC) Received: by mail-qv1-xf2d.google.com with SMTP id dp10so422067qvb.10 for ; Mon, 01 Jun 2020 10:51:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=enmGJsXOmQplpawKCkgkQ22KYVeikdiGwCeoZ0oaSrY=; b=HB6IPC7h4qlYzG8WeKN3iFrmq1FZJ3nGBjvqB0sOeVpNQx4gla/28q+feM5M+qeEi7 1EfJiKGd6LakcGeXdnwBg/6RszZF2QuAoszf5OAAEhgzBugEvwSzI/qeCI5M3dbw/D6T TDfnYCGRoKC44bSnIYyIvb1swpVQhmqiXvRDIc2LgOBwdZByt/HgNU+6KfmdQJxJ9fyq EcasDylKWdy3xTZawEBVWGm8uj9ngbbYr88GOsFdamhdslCDorXp5N5PDiOnOLvZvMQR uSnrRzj2YAkH9LFqCBXjR54q5wOLbnWexkSSw+cZwmF0PP90EeafZ+PEV+rRAtugD2zu anHw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=enmGJsXOmQplpawKCkgkQ22KYVeikdiGwCeoZ0oaSrY=; b=FqmF0A/QxFZryzRRoPimXiSl/4/gpmMnQPLenKt5IUFK5jAE+md48XfBs7pC1Dy8eK Cn0JUxUWLb1v14Y9MyRaSpdfTfC+7WqIU2u2e5Uy6yuX3Cbfkk09duDaCIe91nU+EXOk eeij1WYbxh2mmtdZCi9X/i5w1z9ovf1ifCCbtBL5chR3wew9blijso8Q4U46ExAWt+Fo pUAF/lCHf+G5s6/49jC9OgfpNg0x2jAslIMxiuRFpd4RatFN/VfcUkLyzGEQ31fYTV2U ehco3qjar5i3uHyvxvuO/TPTi/r1JKJ+9Kh44FtSuTqP6NEF3Lm/sBsoOkiEcM6A5vbb BfXQ== X-Gm-Message-State: AOAM530eYXAl1+cKxynQ2045XJhYaVu5PQrsXr63gyWmgZ18JwjR03tR FfJSojtDYO9KZAWJuwHUg3hyhYdJEjUNc5tT8hh7HDadTYM= X-Google-Smtp-Source: ABdhPJxkoT/u85SFA9J+d8/cN2NjcL5sAoq9CEAOn2w47Mb+phsuqp1kmdfHtJsCmfaK5N+F9QpGOhSzXTkZJpHFs9A= X-Received: by 2002:a0c:8b4a:: with SMTP id d10mr10516989qvc.242.1591033874196; Mon, 01 Jun 2020 10:51:14 -0700 (PDT) MIME-Version: 1.0 References: <16034b74-790d-bbee-da24-4450e89adeae@orbitalfox.eu> In-Reply-To: <16034b74-790d-bbee-da24-4450e89adeae@orbitalfox.eu> From: Romain Calascibetta Date: Mon, 1 Jun 2020 19:51:01 +0200 Message-ID: Subject: Re: Hacl ARM compilation issue To: =?UTF-8?B?b3JiaWZ4IPCfpoo=?= Content-Type: multipart/alternative; boundary="000000000000e5ada705a709704d" X-BeenThere: mirageos-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Developer list for MirageOS List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: mirageos-devel@lists.xenproject.org Errors-To: mirageos-devel-bounces@lists.xenproject.org Sender: "MirageOS-devel" --000000000000e5ada705a709704d Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi, I suspect that the bug is into the extracted code from HACL* project where this issue appeared some days after your email: https://github.com/project-everest/hacl-star/pull/310 I did not look deeply but it seems that an extraction by hands of trunk is necessary on our side. Thanks for reporting, I will look into this next week. Le sam. 30 mai 2020 =C3=A0 16:19, orbifx =F0=9F=A6=8A a= =C3=A9crit : > Hello! > > I encountered a warning, which causes Hacl compilation failure on armv7l. > I'm guessing it can't do a 64bit shift since it's 32bit? > > Copying mirageos list as this appears to be a subproject. > > ``` > [...] > kremlib.h:498:42: warning: right shift count >=3D width of type > [-Wshift-count-overflow] > 498 | ~(FStar_UInt64_eq_mask(x >> 64, y >> 64))) | > | ^~ > kremlib.h:499:31: warning: right shift count >=3D width of type > [-Wshift-count-overflow] > 499 | (FStar_UInt64_eq_mask(x >> 64, y >> 64) & > FStar_UInt64_gte_mask(x, y)); > | ^~ > kremlib.h:499:40: warning: right shift count >=3D width of type > [-Wshift-count-overflow] > 499 | (FStar_UInt64_eq_mask(x >> 64, y >> 64) & > FStar_UInt64_gte_mask(x, y)); > | ^~ > kremlib.h:500:28: warning: left shift count >=3D width of type > [-Wshift-count-overflow] > 500 | return ((uint128_t)mask) << 64 | mask; > | ^~ > ``` > > --=20 Romain Calascibetta - http://din.osau.re/ --000000000000e5ada705a709704d Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi,

I suspect that the bug i= s into the extracted code from HACL* project where this issue appeared some= days after your email:

I did not look deeply but it seems that an= extraction by hands of trunk is necessary on our side.

Thanks for reporting, I will look into this next week.

Le=C2= =A0sam. 30 mai 2020 =C3=A0=C2=A016:19, orbifx =F0=9F=A6=8A <fox@orbitalfox.eu> a =C3= =A9crit=C2=A0:
H= ello!

I encountered a warning, which causes Hacl compilation failure on armv7l. I= 'm guessing it can't do a 64bit shift since it's 32bit?

Copying mirageos list as this appears to be a subproject.

```
[...]
kremlib.h:498:42: warning: right shift count >=3D width of type [-Wshift= -count-overflow]
=C2=A0 =C2=A0498 |=C2=A0 =C2=A0 =C2=A0 =C2=A0 ~(FStar_UInt64_eq_mask(x >= > 64, y >> 64))) |
=C2=A0 =C2=A0 =C2=A0 =C2=A0|=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 ^~
kremlib.h:499:31: warning: right shift count >=3D width of type [-Wshift= -count-overflow]
=C2=A0 =C2=A0499 |=C2=A0 =C2=A0 =C2=A0 =C2=A0(FStar_UInt64_eq_mask(x >&g= t; 64, y >> 64) & FStar_UInt64_gte_mask(x, y));
=C2=A0 =C2=A0 =C2=A0 =C2=A0|=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0^~
kremlib.h:499:40: warning: right shift count >=3D width of type [-Wshift= -count-overflow]
=C2=A0 =C2=A0499 |=C2=A0 =C2=A0 =C2=A0 =C2=A0(FStar_UInt64_eq_mask(x >&g= t; 64, y >> 64) & FStar_UInt64_gte_mask(x, y));
=C2=A0 =C2=A0 =C2=A0 =C2=A0|=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 ^~
kremlib.h:500:28: warning: left shift count >=3D width of type [-Wshift-= count-overflow]
=C2=A0 =C2=A0500 |=C2=A0 =C2=A0return ((uint128_t)mask) << 64 | mask;=
=C2=A0 =C2=A0 =C2=A0 =C2=A0|=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 ^~
```



--
Romain Cal= ascibetta - http://din.os= au.re/
--000000000000e5ada705a709704d-- From mirageos-devel-bounces@lists.xenproject.org Tue Jun 09 09:44:46 2020 Return-path: Envelope-to: archives@lists.xenproject.org Delivery-date: Tue, 09 Jun 2020 09:44:46 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1jiao4-00031a-Rk; Tue, 09 Jun 2020 09:44:32 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1jiao4-00030N-3l for mirageos-devel@lists.xenproject.org; Tue, 09 Jun 2020 09:44:32 +0000 X-Inumbo-ID: ce26b49e-aa35-11ea-b7bb-bc764e2007e4 Received: from smtp.lucina.net (unknown [62.176.169.44]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id ce26b49e-aa35-11ea-b7bb-bc764e2007e4; Tue, 09 Jun 2020 09:44:26 +0000 (UTC) Received: from nodbug.lucina.net (78-141-76-187.dynamic.orange.sk [78.141.76.187]) by smtp.lucina.net (Postfix) with ESMTPSA id 612CD122804; Tue, 9 Jun 2020 11:44:25 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=lucina.net; s=dkim-201811; t=1591695865; bh=6ZX7wnOiDM+mgpph4PTSgUowQ6Md6Yyyyb6vfm3/KaE=; h=Date:From:To:Subject:From; b=C0BeHPU9VVelKUtTN/95M9TAXB8lvMn7eL26Q9sKmjXigiQSnff7PY9PTyUJdYD+9 MEBL8YK+RbyOwA7nwDokKZk2k014DgAO/jiRjInu+4gxcM+R9xaqU+D3HwOXI5I+Gw XDRz64Fbc6cLTJTxRTdYNAHTIXq4uaL22hnnU9YYrmR5bEnqtMw1xCD43ONBq9HNa+ Lw3iPYl9ynHK0JDgHJ/fUWk4Tz2Y8PjS+kKl6kNg6W0fKzfdLWFfkiyO7a2xC5CeI0 lLtAaW+r5lPipTgnRiHy2DUz1d0D9A+ckTtTfSjzxWmsQxUTfYKioBcUyq/3kZ7gzi oThtxeePYKVqw== Received: by nodbug.lucina.net (Postfix, from userid 1000) id 4EB52265E722; Tue, 9 Jun 2020 11:44:25 +0200 (CEST) Date: Tue, 9 Jun 2020 11:44:25 +0200 From: Martin Lucina To: xen-devel@lists.xenproject.org, mirageos-devel@lists.xenproject.org Subject: XENMAPSPACE_grant_table vs. GNTTABOP_setup_table Message-ID: <20200609094425.GB9734@nodbug.lucina.net> Mail-Followup-To: Martin Lucina , xen-devel@lists.xenproject.org, mirageos-devel@lists.xenproject.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.10.1 (2018-07-13) X-BeenThere: mirageos-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Developer list for MirageOS List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: mirageos-devel-bounces@lists.xenproject.org Sender: "MirageOS-devel" Hi, I've been making progress on bootstrapping a new, PVHv2 only, Xen platform stack for MirageOS [1]. The basics are now functional and I have started to re-implement the grant table code. After studying both the Mini-OS and Linux implementations some, I don't understand the difference between using XENMAPSPACE_grant_table vs. GNTTABOP_setup_table to set up the initial grant table mapping for the guest. Are these calls just newer and older ways of accomplishing the same thing? The Linux driver seems to use both in various conditions, whereas Mini-OS uses the former on ARM and the latter on x86. If these are functionally equivalent, then for my purposes I'd rather use XENMAPSPACE_setup_table, since IIUC this lets me map the grant table at an address of my choosing rather than GNTTABOP_setup_table which at least on x86_64 appears to be returning PFNs at the top of the address space. Any advice much appreciated, -mato [1] https://github.com/mirage/mirage/issues/1159 From mirageos-devel-bounces@lists.xenproject.org Tue Jun 09 10:23:08 2020 Return-path: Envelope-to: archives@lists.xenproject.org Delivery-date: Tue, 09 Jun 2020 10:23:08 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1jibPM-0006cs-2l; Tue, 09 Jun 2020 10:23:04 +0000 Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1jibPK-0006cX-7U for mirageos-devel@lists.xenproject.org; Tue, 09 Jun 2020 10:23:02 +0000 X-Inumbo-ID: 2ef3bc18-aa3b-11ea-b301-12813bfff9fa Received: from esa4.hc3370-68.iphmx.com (unknown [216.71.155.144]) by us1-amaz-eas2.inumbo.com (Halon) with ESMTPS id 2ef3bc18-aa3b-11ea-b301-12813bfff9fa; Tue, 09 Jun 2020 10:22:56 +0000 (UTC) Authentication-Results: esa4.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none IronPort-SDR: EOsWrPTebEbtU0to4SChpf/HjOT3LEn6GUXRBVPWbATRtwthm1Jt+84v/uQlmtmaZHspttUBER x2HfHuNr9hwhzcJ6S4r8BRrVeaUvLp6wnec+vOGcaY4TBSgK3ZfHO7vUEyTILH78v1W3PpAsi8 vEgrf7EcBi0Xz73UzIXX/pEjQYmLRu55vS6Po5Z4diTeeOiaoxjMnxtbxyVHjgGLxurArb6hrG JNTjlhYxWCbIWlpZF6wZb0NSrphk0I/191MJtP65ugcQwOr6BxbAgKSk77/ZxQucDVgQ0/hy6t VQw= X-SBRS: 2.7 X-MesageID: 20318778 X-Ironport-Server: esa4.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.73,491,1583211600"; d="scan'208";a="20318778" Subject: Re: XENMAPSPACE_grant_table vs. GNTTABOP_setup_table To: Martin Lucina , , References: <20200609094425.GB9734@nodbug.lucina.net> From: Andrew Cooper Message-ID: <3c7269b9-bf3f-5359-6ce2-049f935c8e84@citrix.com> Date: Tue, 9 Jun 2020 11:22:51 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.8.0 MIME-Version: 1.0 In-Reply-To: <20200609094425.GB9734@nodbug.lucina.net> Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 8bit Content-Language: en-GB X-ClientProxiedBy: AMSPEX02CAS02.citrite.net (10.69.22.113) To AMSPEX02CL02.citrite.net (10.69.22.126) X-BeenThere: mirageos-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Developer list for MirageOS List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: mirageos-devel-bounces@lists.xenproject.org Sender: "MirageOS-devel" On 09/06/2020 10:44, Martin Lucina wrote: > Hi, > > I've been making progress on bootstrapping a new, PVHv2 only, Xen platform > stack for MirageOS [1]. The basics are now functional and I have started to > re-implement the grant table code. > > After studying both the Mini-OS and Linux implementations some, I don't > understand the difference between using XENMAPSPACE_grant_table vs. > GNTTABOP_setup_table to set up the initial grant table mapping for the > guest. > > Are these calls just newer and older ways of accomplishing the same thing? > The Linux driver seems to use both in various conditions, whereas Mini-OS > uses the former on ARM and the latter on x86. > > If these are functionally equivalent, then for my purposes I'd rather use > XENMAPSPACE_setup_table, since IIUC this lets me map the grant table at an > address of my choosing rather than GNTTABOP_setup_table which at least on > x86_64 appears to be returning PFNs at the top of the address space. > > Any advice much appreciated, There is a little bit of history here... GNTTABOP_setup_table was the original PV way of doing things (specify size as an input, get a list of frames as an output to map), and XENMAPSPACE_grant_table was the original HVM way of doing things (as mapping is the other way around - I specify a GFN which I'd like to turn into a grant table mapping). When grant v2 came along, it was only XENMAPSPACE_grant_table updated to be compatible.  i.e. you have to use XENMAPSPACE_grant_table to map the status frames even if you used GNTTABOP_setup_table previously. It is a mistake that GNTTABOP_setup_table was usable in HVM guests to being with.  Returning -1 is necessary to avoid an information leak (the physical address of the frames making up the grant table) which an HVM guest shouldn't, and has no use knowing. An with that note, ARM is extra special because the grant API is specified to use host physical addresses rather than guest physical (at least for dom0, for reasons of there generally not being an IOMMU), which is why it does use the old PV way. It is all a bit of a mess. ~Andrew From mirageos-devel-bounces@lists.xenproject.org Tue Jun 09 11:10:21 2020 Return-path: Envelope-to: archives@lists.xenproject.org Delivery-date: Tue, 09 Jun 2020 11:10:21 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1jic8z-0003L4-3J; Tue, 09 Jun 2020 11:10:13 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1jic8x-0003Ky-V8 for mirageos-devel@lists.xenproject.org; Tue, 09 Jun 2020 11:10:12 +0000 X-Inumbo-ID: c7e79196-aa41-11ea-bb8b-bc764e2007e4 Received: from honk.recoil.org (unknown [2604:1380:2001:1300::3]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id c7e79196-aa41-11ea-bb8b-bc764e2007e4; Tue, 09 Jun 2020 11:10:09 +0000 (UTC) Received: from [IPv6:2a02:390:81d4::e5a2:aa13:2826:18e8] (unknown [IPv6:2a02:390:81d4:0:e5a2:aa13:2826:18e8]) by honk.recoil.org (Postfix) with ESMTPSA id A86393C0189; Tue, 9 Jun 2020 10:50:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=recoil.org; s=selector1; t=1591699841; bh=TR8w4REWHYr5hfE0wj/02gw9m0ryu0Z1r9I3BSO9ZMs=; h=From:Subject:Date:In-Reply-To:Cc:To:References:From; b=f6lSShSP3s+IZmHoHFm+A2Z12+X+dqbZxtCVZFbjWX44UG8qdagW4B6D6RfqKiOAa kX3qcyNmMRnyvLF75X8ifsOpUpKKq04gmy2LEt01ER/Oj9lRyBVmZx4EwNRWY95uI+ cKqr/ixcJGyZZHFzU2vRHFHUtZXijXrRarx8HAbI= From: Anil Madhavapeddy Message-Id: Content-Type: multipart/alternative; boundary="Apple-Mail=_A71BAFB1-63AF-4E1D-953C-5E8D0B4A25A2" Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.11\)) Subject: Re: XENMAPSPACE_grant_table vs. GNTTABOP_setup_table Date: Tue, 9 Jun 2020 11:50:39 +0100 In-Reply-To: <3c7269b9-bf3f-5359-6ce2-049f935c8e84@citrix.com> To: Andrew Cooper References: <20200609094425.GB9734@nodbug.lucina.net> <3c7269b9-bf3f-5359-6ce2-049f935c8e84@citrix.com> X-Mailer: Apple Mail (2.3445.104.11) X-BeenThere: mirageos-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Developer list for MirageOS List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: xen-devel@lists.xenproject.org, Martin Lucina , mirageos-devel@lists.xenproject.org Errors-To: mirageos-devel-bounces@lists.xenproject.org Sender: "MirageOS-devel" --Apple-Mail=_A71BAFB1-63AF-4E1D-953C-5E8D0B4A25A2 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii On 9 Jun 2020, at 11:22, Andrew Cooper = wrote: >=20 > On 09/06/2020 10:44, Martin Lucina wrote: >> Hi, >>=20 >> I've been making progress on bootstrapping a new, PVHv2 only, Xen = platform >> stack for MirageOS [1]. The basics are now functional and I have = started to >> re-implement the grant table code. >>=20 >> After studying both the Mini-OS and Linux implementations some, I = don't >> understand the difference between using XENMAPSPACE_grant_table vs. >> GNTTABOP_setup_table to set up the initial grant table mapping for = the >> guest. >>=20 >> Are these calls just newer and older ways of accomplishing the same = thing? >> The Linux driver seems to use both in various conditions, whereas = Mini-OS >> uses the former on ARM and the latter on x86. >>=20 >> If these are functionally equivalent, then for my purposes I'd rather = use >> XENMAPSPACE_setup_table, since IIUC this lets me map the grant table = at an >> address of my choosing rather than GNTTABOP_setup_table which at = least on >> x86_64 appears to be returning PFNs at the top of the address space. >>=20 >> Any advice much appreciated, >=20 > There is a little bit of history here... >=20 > GNTTABOP_setup_table was the original PV way of doing things (specify > size as an input, get a list of frames as an output to map), and > XENMAPSPACE_grant_table was the original HVM way of doing things (as > mapping is the other way around - I specify a GFN which I'd like to = turn > into a grant table mapping). >=20 > When grant v2 came along, it was only XENMAPSPACE_grant_table updated = to > be compatible. i.e. you have to use XENMAPSPACE_grant_table to map = the > status frames even if you used GNTTABOP_setup_table previously. >=20 > It is a mistake that GNTTABOP_setup_table was usable in HVM guests to > being with. Returning -1 is necessary to avoid an information leak = (the > physical address of the frames making up the grant table) which an HVM > guest shouldn't, and has no use knowing. >=20 > An with that note, ARM is extra special because the grant API is > specified to use host physical addresses rather than guest physical = (at > least for dom0, for reasons of there generally not being an IOMMU), > which is why it does use the old PV way. Thanks, that makes sense. It doesn't seem too much of a mess from the = guest perspective to use just XENMAPSPACE_grant_table exclusively then. With Martin's work, the MirageOS Xen backend should just use the latest and = greatest APIs, with no backwards compatibility to older Xen versions required. I'm still a little confused about ARM though -- is there an equivalent = of XENMAPSPACE_grant_table there? It sounds like we can't leave the GNTTABOP macros behind entirely... regards, Anil= --Apple-Mail=_A71BAFB1-63AF-4E1D-953C-5E8D0B4A25A2 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii On = 9 Jun 2020, at 11:22, Andrew Cooper <andrew.cooper3@citrix.com> wrote:

On 09/06/2020 = 10:44, Martin Lucina wrote:
Hi,

I've = been making progress on bootstrapping a new, PVHv2 only, Xen platform
stack for MirageOS [1]. The basics are now functional and I = have started to
re-implement the grant table code.

After studying both the Mini-OS and Linux = implementations some, I don't
understand the difference = between using XENMAPSPACE_grant_table vs.
GNTTABOP_setup_table to set up the initial grant table = mapping for the
guest.

Are = these calls just newer and older ways of accomplishing the same = thing?
The Linux driver seems to use both in various = conditions, whereas Mini-OS
uses the former on ARM and the = latter on x86.

If these are functionally = equivalent, then for my purposes I'd rather use
XENMAPSPACE_setup_table, since IIUC this lets me map the = grant table at an
address of my choosing rather than = GNTTABOP_setup_table which at least on
x86_64 appears to = be returning PFNs at the top of the address space.

Any advice much appreciated,

There is a = little bit of history here...

GNTTABOP_setup_table was the original PV way of doing things = (specify
size as an = input, get a list of frames as an output to map), and
XENMAPSPACE_grant_table was the original HVM way of doing = things (as
mapping is = the other way around - I specify a GFN which I'd like to turn
into a grant = table mapping).

When grant v2 came along, it was only XENMAPSPACE_grant_table = updated to
be = compatible.  i.e. you have to use XENMAPSPACE_grant_table to map = the
status frames = even if you used GNTTABOP_setup_table previously.

It is a = mistake that GNTTABOP_setup_table was usable in HVM guests to
being = with.  Returning -1 is necessary to avoid an information leak = (the
physical = address of the frames making up the grant table) which an HVM
guest = shouldn't, and has no use knowing.

An with that note, ARM is extra special because the grant API = is
specified to = use host physical addresses rather than guest physical (at
least for = dom0, for reasons of there generally not being an IOMMU),
which is why = it does use the old PV way.

Thanks, that makes sense. It doesn't seem too much of a = mess from the guest
perspective to use just = XENMAPSPACE_grant_table exclusively then.  With
Martin's work, the MirageOS Xen backend should just use the = latest and greatest
APIs, with no backwards = compatibility to older Xen versions required.

I'm still a little confused about ARM = though -- is there an equivalent of
XENMAPSPACE_grant_table there? It sounds like we can't leave = the
GNTTABOP macros behind entirely...

regards,
Anil
= --Apple-Mail=_A71BAFB1-63AF-4E1D-953C-5E8D0B4A25A2-- From mirageos-devel-bounces@lists.xenproject.org Tue Jun 09 11:29:46 2020 Return-path: Envelope-to: archives@lists.xenproject.org Delivery-date: Tue, 09 Jun 2020 11:29:46 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1jicRp-0005UW-S7; Tue, 09 Jun 2020 11:29:41 +0000 Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1jicRo-0005UR-O0 for mirageos-devel@lists.xenproject.org; Tue, 09 Jun 2020 11:29:40 +0000 X-Inumbo-ID: 814f3254-aa44-11ea-b30e-12813bfff9fa Received: from mail.xenproject.org (unknown [104.130.215.37]) by us1-amaz-eas2.inumbo.com (Halon) with ESMTPS id 814f3254-aa44-11ea-b30e-12813bfff9fa; Tue, 09 Jun 2020 11:29:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=xen.org; s=20200302mail; h=Content-Transfer-Encoding:Content-Type:In-Reply-To: MIME-Version:Date:Message-ID:From:References:Cc:To:Subject:Sender:Reply-To: Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender: Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:List-Unsubscribe: List-Subscribe:List-Post:List-Owner:List-Archive; bh=HPqLLjE8XMoM4zusJM9774uLnLFPvCnw9hYmCxWWvdU=; b=soD3VkevoW8DLGwC97kzc6AfMv 8KQdLcLKGi1pwG3lQTkBBalS/HVDfUlyt16a34O95a11K/KbvtXAyJKXSPMdYRzWRti8Gev9/iZWE U5Gr/J9LsnGalN7ClFmKoEhRB3Jx0CUOBtzx54wugEN04IlzoMyz2AKKMHNfiTGj2z0Q=; Received: from xenbits.xenproject.org ([104.239.192.120]) by mail.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1jicRm-000587-7d; Tue, 09 Jun 2020 11:29:38 +0000 Received: from [54.239.6.185] (helo=a483e7b01a66.ant.amazon.com) by xenbits.xenproject.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.92) (envelope-from ) id 1jicRm-0006k7-0k; Tue, 09 Jun 2020 11:29:38 +0000 Subject: Re: XENMAPSPACE_grant_table vs. GNTTABOP_setup_table To: Anil Madhavapeddy , Andrew Cooper References: <20200609094425.GB9734@nodbug.lucina.net> <3c7269b9-bf3f-5359-6ce2-049f935c8e84@citrix.com> From: Julien Grall Message-ID: <314dd258-14de-2d70-eec4-8cbc0a62e75a@xen.org> Date: Tue, 9 Jun 2020 12:29:36 +0100 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:68.0) Gecko/20100101 Thunderbird/68.9.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252; format=flowed Content-Language: en-GB Content-Transfer-Encoding: 8bit X-BeenThere: mirageos-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Developer list for MirageOS List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: xen-devel@lists.xenproject.org, Martin Lucina , mirageos-devel@lists.xenproject.org Errors-To: mirageos-devel-bounces@lists.xenproject.org Sender: "MirageOS-devel" On 09/06/2020 11:50, Anil Madhavapeddy wrote: > On 9 Jun 2020, at 11:22, Andrew Cooper > wrote: >> >> On 09/06/2020 10:44, Martin Lucina wrote: >>> Hi, >>> >>> I've been making progress on bootstrapping a new, PVHv2 only, Xen >>> platform >>> stack for MirageOS [1]. The basics are now functional and I have >>> started to >>> re-implement the grant table code. >>> >>> After studying both the Mini-OS and Linux implementations some, I don't >>> understand the difference between using XENMAPSPACE_grant_table vs. >>> GNTTABOP_setup_table to set up the initial grant table mapping for the >>> guest. >>> >>> Are these calls just newer and older ways of accomplishing the same >>> thing? >>> The Linux driver seems to use both in various conditions, whereas Mini-OS >>> uses the former on ARM and the latter on x86. >>> >>> If these are functionally equivalent, then for my purposes I'd rather use >>> XENMAPSPACE_setup_table, since IIUC this lets me map the grant table >>> at an >>> address of my choosing rather than GNTTABOP_setup_table which at least on >>> x86_64 appears to be returning PFNs at the top of the address space. >>> >>> Any advice much appreciated, >> >> There is a little bit of history here... >> >> GNTTABOP_setup_table was the original PV way of doing things (specify >> size as an input, get a list of frames as an output to map), and >> XENMAPSPACE_grant_table was the original HVM way of doing things (as >> mapping is the other way around - I specify a GFN which I'd like to turn >> into a grant table mapping). >> >> When grant v2 came along, it was only XENMAPSPACE_grant_table updated to >> be compatible.  i.e. you have to use XENMAPSPACE_grant_table to map the >> status frames even if you used GNTTABOP_setup_table previously. >> >> It is a mistake that GNTTABOP_setup_table was usable in HVM guests to >> being with.  Returning -1 is necessary to avoid an information leak (the >> physical address of the frames making up the grant table) which an HVM >> guest shouldn't, and has no use knowing. >> >> An with that note, ARM is extra special because the grant API is >> specified to use host physical addresses rather than guest physical (at >> least for dom0, for reasons of there generally not being an IOMMU), >> which is why it does use the old PV way. > > Thanks, that makes sense. It doesn't seem too much of a mess from the guest > perspective to use just XENMAPSPACE_grant_table exclusively then.  With > Martin's work, the MirageOS Xen backend should just use the latest and > greatest > APIs, with no backwards compatibility to older Xen versions required. > > I'm still a little confused about ARM though -- is there an equivalent of > XENMAPSPACE_grant_table there? It sounds like we can't leave the > GNTTABOP macros behind entirely... XENMAPSPACE_grant_table exists and works perfectly fine on Arm. It is using guest physical address as it should. Cheers, -- Julien Grall From mirageos-devel-bounces@lists.xenproject.org Wed Jun 10 13:22:52 2020 Return-path: Envelope-to: archives@lists.xenproject.org Delivery-date: Wed, 10 Jun 2020 13:22:52 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1jj0gc-0008S8-2T; Wed, 10 Jun 2020 13:22:34 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1jj0ga-0008Rn-D1 for mirageos-devel@lists.xenproject.org; Wed, 10 Jun 2020 13:22:32 +0000 X-Inumbo-ID: 6cf64318-ab1d-11ea-bb8b-bc764e2007e4 Received: from smtp.lucina.net (unknown [62.176.169.44]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id 6cf64318-ab1d-11ea-bb8b-bc764e2007e4; Wed, 10 Jun 2020 13:22:26 +0000 (UTC) Received: from nodbug.lucina.net (78-141-76-187.dynamic.orange.sk [78.141.76.187]) by smtp.lucina.net (Postfix) with ESMTPSA id 7200E122804; Wed, 10 Jun 2020 15:22:25 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=lucina.net; s=dkim-201811; t=1591795345; bh=ISrEXEgbgbprbZHOuCt2lv8qBuL8OIAh7TmyZYfsp4E=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=SBYEk0tOaEovxg2jVWoFcdA2JbV9kzKhP9Yv+UHaHyrIk7BSXikshszZb2dXxL+7p qy8BGQVlQav5uenaL9yoB/QpwTCr4Z1JHf7mFu0063f4BbjI/ldZ99qpY9GL+/q65Q xrZo33tDK9W4qsnlB3uoRAilgZqShGQylxXyopJv0byelFzim2phGLDRp/K9rhZ3Fv bGiQH+oWGOQ6Fv4Ot29GTc5x7fizb3eQQAeRfBbo+n7BRwM2MXKlXiKhcJX3FpUwIK dE14rwd8+oOZCbqGhlU3HmGkXHZ+mdNK1go65DDxx5psyxz3CSS45ZqZpEIsFDPlKF la5U3NdQ83jEg== Received: by nodbug.lucina.net (Postfix, from userid 1000) id 2D052265E722; Wed, 10 Jun 2020 15:22:25 +0200 (CEST) Date: Wed, 10 Jun 2020 15:22:25 +0200 From: Martin Lucina To: Andrew Cooper Subject: Re: XENMAPSPACE_grant_table vs. GNTTABOP_setup_table Message-ID: <20200610132225.GA16839@nodbug.lucina.net> Mail-Followup-To: Martin Lucina , Andrew Cooper , xen-devel@lists.xenproject.org, mirageos-devel@lists.xenproject.org References: <20200609094425.GB9734@nodbug.lucina.net> <3c7269b9-bf3f-5359-6ce2-049f935c8e84@citrix.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <3c7269b9-bf3f-5359-6ce2-049f935c8e84@citrix.com> User-Agent: Mutt/1.10.1 (2018-07-13) X-BeenThere: mirageos-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Developer list for MirageOS List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: xen-devel@lists.xenproject.org, mirageos-devel@lists.xenproject.org Errors-To: mirageos-devel-bounces@lists.xenproject.org Sender: "MirageOS-devel" On Tuesday, 09.06.2020 at 11:22, Andrew Cooper wrote: > There is a little bit of history here... > > GNTTABOP_setup_table was the original PV way of doing things (specify > size as an input, get a list of frames as an output to map), and > XENMAPSPACE_grant_table was the original HVM way of doing things (as > mapping is the other way around - I specify a GFN which I'd like to turn > into a grant table mapping). > > When grant v2 came along, it was only XENMAPSPACE_grant_table updated to > be compatible.  i.e. you have to use XENMAPSPACE_grant_table to map the > status frames even if you used GNTTABOP_setup_table previously. > > It is a mistake that GNTTABOP_setup_table was usable in HVM guests to > being with.  Returning -1 is necessary to avoid an information leak (the > physical address of the frames making up the grant table) which an HVM > guest shouldn't, and has no use knowing. > > An with that note, ARM is extra special because the grant API is > specified to use host physical addresses rather than guest physical (at > least for dom0, for reasons of there generally not being an IOMMU), > which is why it does use the old PV way. > > It is all a bit of a mess. Thanks for explaining, this is helpful. So, going with the grant v2 ABI, is there a modern equivalent of GNTTABOP_get_status_frames? Reading memory.h I'm guessing that it might be XENMEM_add_to_physmap with space=XENMAPSPACE_grant_table and idx=(XENMAPIDX_grant_table_status + N) where N is the frame I want, but this is not explicitly mentioned anywhere and Linux uses the GNTTABOP mechanism. Further to that, what is the format of the grant status frames? grant_table.h doesn't have much to say about it. And lastly, given that I want the v2 grant ABI exclusively, I presume it's sufficient to call GNTTABOP_set_version (version=2) first thing and abort if it failed? Presumably the default is always v1 at start of day? Thanks, -mato From mirageos-devel-bounces@lists.xenproject.org Wed Jun 10 13:41:00 2020 Return-path: Envelope-to: archives@lists.xenproject.org Delivery-date: Wed, 10 Jun 2020 13:41:00 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1jj0yN-0001lV-G2; Wed, 10 Jun 2020 13:40:55 +0000 Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1jj0yL-0001lQ-SH for mirageos-devel@lists.xenproject.org; Wed, 10 Jun 2020 13:40:53 +0000 X-Inumbo-ID: 005fcfe6-ab20-11ea-b441-12813bfff9fa Received: from esa3.hc3370-68.iphmx.com (unknown [216.71.145.155]) by us1-amaz-eas2.inumbo.com (Halon) with ESMTPS id 005fcfe6-ab20-11ea-b441-12813bfff9fa; Wed, 10 Jun 2020 13:40:52 +0000 (UTC) Authentication-Results: esa3.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none IronPort-SDR: mcqXY4eDf3smpv7yilUtFR0V6rTDEP04Yid3QsdysGtdhas2vtoHzBJDiSHuQ3AlLRYLyZaW6F huATiSVfbjCIo/vvGpQ9eXOd+aobzD4S3d77Dtw2ff06wHDrxp8kZOaVhFBhh8b0Zn3ONYoxH1 LGs1LH7qrFCD46St7bdcnLhHYG4Y23vfCc8q3vJWCQp8vAqqKpFWvUErWQCNNqVl++Ra5EZdf+ SgfsanppFI0KrhqNSdFyr78za3OOBPcsEgncD03ROPay9UdlWzzU7nk6Jcyinug8uog47I7TQF EbI= X-SBRS: 2.7 X-MesageID: 19684861 X-Ironport-Server: esa3.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.73,496,1583211600"; d="scan'208";a="19684861" Subject: Re: XENMAPSPACE_grant_table vs. GNTTABOP_setup_table To: Martin Lucina , , References: <20200609094425.GB9734@nodbug.lucina.net> <3c7269b9-bf3f-5359-6ce2-049f935c8e84@citrix.com> <20200610132225.GA16839@nodbug.lucina.net> From: Andrew Cooper Message-ID: <46e87834-bf47-4003-1f32-89a47255155d@citrix.com> Date: Wed, 10 Jun 2020 14:40:48 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.8.0 MIME-Version: 1.0 In-Reply-To: <20200610132225.GA16839@nodbug.lucina.net> Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 8bit Content-Language: en-GB X-ClientProxiedBy: AMSPEX02CAS02.citrite.net (10.69.22.113) To AMSPEX02CL02.citrite.net (10.69.22.126) X-BeenThere: mirageos-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Developer list for MirageOS List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: mirageos-devel-bounces@lists.xenproject.org Sender: "MirageOS-devel" On 10/06/2020 14:22, Martin Lucina wrote: > On Tuesday, 09.06.2020 at 11:22, Andrew Cooper wrote: >> There is a little bit of history here... >> >> GNTTABOP_setup_table was the original PV way of doing things (specify >> size as an input, get a list of frames as an output to map), and >> XENMAPSPACE_grant_table was the original HVM way of doing things (as >> mapping is the other way around - I specify a GFN which I'd like to turn >> into a grant table mapping). >> >> When grant v2 came along, it was only XENMAPSPACE_grant_table updated to >> be compatible.  i.e. you have to use XENMAPSPACE_grant_table to map the >> status frames even if you used GNTTABOP_setup_table previously. >> >> It is a mistake that GNTTABOP_setup_table was usable in HVM guests to >> being with.  Returning -1 is necessary to avoid an information leak (the >> physical address of the frames making up the grant table) which an HVM >> guest shouldn't, and has no use knowing. >> >> An with that note, ARM is extra special because the grant API is >> specified to use host physical addresses rather than guest physical (at >> least for dom0, for reasons of there generally not being an IOMMU), >> which is why it does use the old PV way. >> >> It is all a bit of a mess. > Thanks for explaining, this is helpful. > > So, going with the grant v2 ABI, is there a modern equivalent of > GNTTABOP_get_status_frames? Reading memory.h I'm guessing that it might be > XENMEM_add_to_physmap with space=XENMAPSPACE_grant_table and > idx=(XENMAPIDX_grant_table_status + N) where N is the frame I want, but > this is not explicitly mentioned anywhere and Linux uses the GNTTABOP > mechanism. > > Further to that, what is the format of the grant status frames? > grant_table.h doesn't have much to say about it. > > And lastly, given that I want the v2 grant ABI exclusively, I presume it's > sufficient to call GNTTABOP_set_version (version=2) first thing and abort > if it failed? Presumably the default is always v1 at start of day? What kind of guests are you trying to target here? Since my reply, I tried to experiment, and I think you're forced to use GNTTABOP_setup_table/GNTTABOP_get_status_frames for x86 PV guests, and XENMEM_add_to_physmap for x86 HVM/PVH guests. You can't depend on version 2 being available.  Its not available for ARM at all, and may be disabled for security reasons on x86 (there was some extended fun with transitive grants in the past, and we offered "totally disable grant v2" as one mitigation). Use v1 unless you have a specific need to use v2 (transitive grants or subpage copies, or support for >16TB VMs (HVM) or hosts (PV)).  Amongst other things, its far more simple to use correctly.  (v2 devolves to infinite loops to use correctly, because there is no way to do an atomic option covering both the flags and status entries at the same time.) If you need to use v2, you must cleanly cope with it not being available, and fall back to v1. ~Andrew From mirageos-devel-bounces@lists.xenproject.org Wed Jun 10 13:56:10 2020 Return-path: Envelope-to: archives@lists.xenproject.org Delivery-date: Wed, 10 Jun 2020 13:56:10 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1jj1D3-0002nX-TW; Wed, 10 Jun 2020 13:56:05 +0000 Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1jj1D2-0002nO-QU for mirageos-devel@lists.xenproject.org; Wed, 10 Jun 2020 13:56:04 +0000 X-Inumbo-ID: 1ef3cee2-ab22-11ea-b449-12813bfff9fa Received: from smtp.lucina.net (unknown [62.176.169.44]) by us1-amaz-eas2.inumbo.com (Halon) with ESMTPS id 1ef3cee2-ab22-11ea-b449-12813bfff9fa; Wed, 10 Jun 2020 13:56:03 +0000 (UTC) Received: from nodbug.lucina.net (78-141-76-187.dynamic.orange.sk [78.141.76.187]) by smtp.lucina.net (Postfix) with ESMTPSA id 0C91D122804; Wed, 10 Jun 2020 15:56:02 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=lucina.net; s=dkim-201811; t=1591797362; bh=dQM1v/InNxn2/aJ4PouUn9hXkCZr2u+Zl52zCY32cuY=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=pOuQUOvlnw+gpAgMWeGC/cj0tznQq9o3pPrIxd3B4J7/JX2aWxbL0AZqiB1ZdSJt9 3RVDVR0TDb4w9urLfnKqXfdg+aXnRdxJtw4XGtwmDWymfBWzDcqs7udJUK3h+KTgiK tDSHbpddSAffTw6mnd6/rUc/+bWrD3JYCTSVG8xXH/n7Biuz/WAfBuH/U8Jw5mSY/s niJ1uTvx69XmjqzxM0jpAuEGSmVXDeVLZRPzTN3N6rlUSBDOJoguHNs7I1nnONmSfK X22uPtqLB+ko3j9zG9yWXixpih7FSIlOLo3hhSlA/qCzXss9CuxbBND8Wcmj1uUvcz dDlr5ajmoPeLw== Received: by nodbug.lucina.net (Postfix, from userid 1000) id E448D265E722; Wed, 10 Jun 2020 15:56:01 +0200 (CEST) Date: Wed, 10 Jun 2020 15:56:01 +0200 From: Martin Lucina To: Andrew Cooper Subject: Re: XENMAPSPACE_grant_table vs. GNTTABOP_setup_table Message-ID: <20200610135601.GB16839@nodbug.lucina.net> Mail-Followup-To: Martin Lucina , Andrew Cooper , xen-devel@lists.xenproject.org, mirageos-devel@lists.xenproject.org References: <20200609094425.GB9734@nodbug.lucina.net> <3c7269b9-bf3f-5359-6ce2-049f935c8e84@citrix.com> <20200610132225.GA16839@nodbug.lucina.net> <46e87834-bf47-4003-1f32-89a47255155d@citrix.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <46e87834-bf47-4003-1f32-89a47255155d@citrix.com> User-Agent: Mutt/1.10.1 (2018-07-13) X-BeenThere: mirageos-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Developer list for MirageOS List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: xen-devel@lists.xenproject.org, mirageos-devel@lists.xenproject.org Errors-To: mirageos-devel-bounces@lists.xenproject.org Sender: "MirageOS-devel" On Wednesday, 10.06.2020 at 14:40, Andrew Cooper wrote: > > So, going with the grant v2 ABI, is there a modern equivalent of > > GNTTABOP_get_status_frames? Reading memory.h I'm guessing that it might be > > XENMEM_add_to_physmap with space=XENMAPSPACE_grant_table and > > idx=(XENMAPIDX_grant_table_status + N) where N is the frame I want, but > > this is not explicitly mentioned anywhere and Linux uses the GNTTABOP > > mechanism. > > > > Further to that, what is the format of the grant status frames? > > grant_table.h doesn't have much to say about it. > > > > And lastly, given that I want the v2 grant ABI exclusively, I presume it's > > sufficient to call GNTTABOP_set_version (version=2) first thing and abort > > if it failed? Presumably the default is always v1 at start of day? > > What kind of guests are you trying to target here? PVHv2 only. x86_64 only for now, though the code should remain easily portable to at least ARM64, should someone decide they need that. > Since my reply, I tried to experiment, and I think you're forced to use > GNTTABOP_setup_table/GNTTABOP_get_status_frames for x86 PV guests, and > XENMEM_add_to_physmap for x86 HVM/PVH guests. > > You can't depend on version 2 being available.  Its not available for > ARM at all, and may be disabled for security reasons on x86 (there was > some extended fun with transitive grants in the past, and we offered > "totally disable grant v2" as one mitigation). I don't need v2 at all, I was just going by the comments in grant_table.h, which read: "Version 1 of the grant table entry structure is maintained purely for backwards compatibility. New guests should use version 2." Grant status frames are a v2-only thing, right? Or can I use v1 and ask for grant status frames? -mato From mirageos-devel-bounces@lists.xenproject.org Wed Jun 10 14:21:47 2020 Return-path: Envelope-to: archives@lists.xenproject.org Delivery-date: Wed, 10 Jun 2020 14:21:47 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1jj1br-0005Uk-GR; Wed, 10 Jun 2020 14:21:43 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1jj1bp-0005UN-Mx for mirageos-devel@lists.xenproject.org; Wed, 10 Jun 2020 14:21:41 +0000 X-Inumbo-ID: b08605f2-ab25-11ea-bca7-bc764e2007e4 Received: from esa4.hc3370-68.iphmx.com (unknown [216.71.155.144]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id b08605f2-ab25-11ea-bca7-bc764e2007e4; Wed, 10 Jun 2020 14:21:35 +0000 (UTC) Authentication-Results: esa4.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none IronPort-SDR: H9Vs+9QD9Q/3txKlxrcRmQxt3YsaVlpyjmrZS71lHV1YxX7WY6mOU8l2kM7OCFYhNaSq5cfzBA zF1aCXxsEt2aILvDo6BVUuNSMZv+m4qPt2vBhD6qe7Ns5Z42qy7PInzqz/89x1JvOxeW6U8ERS Y5zGM77UmNrtKovNiZjwgAlMwS9Fn3UZy8WFuyOkQaKvmtag3/BGUiNjiIA4Ce8OJo7b69eeL2 HpW3UCMRl6vmVcKwXMQv87EiN259sIGJM1AmBoYvQAKzbYhJVp+bAWdlvjihJVkOB/PceohnUi itI= X-SBRS: 2.7 X-MesageID: 20451479 X-Ironport-Server: esa4.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.73,496,1583211600"; d="scan'208";a="20451479" Subject: Re: XENMAPSPACE_grant_table vs. GNTTABOP_setup_table To: Martin Lucina , , References: <20200609094425.GB9734@nodbug.lucina.net> <3c7269b9-bf3f-5359-6ce2-049f935c8e84@citrix.com> <20200610132225.GA16839@nodbug.lucina.net> <46e87834-bf47-4003-1f32-89a47255155d@citrix.com> <20200610135601.GB16839@nodbug.lucina.net> From: Andrew Cooper Message-ID: <6b2a0091-2c56-895c-762d-719e5796985d@citrix.com> Date: Wed, 10 Jun 2020 15:21:31 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.8.0 MIME-Version: 1.0 In-Reply-To: <20200610135601.GB16839@nodbug.lucina.net> Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 8bit Content-Language: en-GB X-ClientProxiedBy: AMSPEX02CAS01.citrite.net (10.69.22.112) To AMSPEX02CL02.citrite.net (10.69.22.126) X-BeenThere: mirageos-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Developer list for MirageOS List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: mirageos-devel-bounces@lists.xenproject.org Sender: "MirageOS-devel" On 10/06/2020 14:56, Martin Lucina wrote: > On Wednesday, 10.06.2020 at 14:40, Andrew Cooper wrote: >>> So, going with the grant v2 ABI, is there a modern equivalent of >>> GNTTABOP_get_status_frames? Reading memory.h I'm guessing that it might be >>> XENMEM_add_to_physmap with space=XENMAPSPACE_grant_table and >>> idx=(XENMAPIDX_grant_table_status + N) where N is the frame I want, but >>> this is not explicitly mentioned anywhere and Linux uses the GNTTABOP >>> mechanism. >>> >>> Further to that, what is the format of the grant status frames? >>> grant_table.h doesn't have much to say about it. >>> >>> And lastly, given that I want the v2 grant ABI exclusively, I presume it's >>> sufficient to call GNTTABOP_set_version (version=2) first thing and abort >>> if it failed? Presumably the default is always v1 at start of day? >> What kind of guests are you trying to target here? > PVHv2 only. x86_64 only for now, though the code should remain easily > portable to at least ARM64, should someone decide they need that. > >> Since my reply, I tried to experiment, and I think you're forced to use >> GNTTABOP_setup_table/GNTTABOP_get_status_frames for x86 PV guests, and >> XENMEM_add_to_physmap for x86 HVM/PVH guests. >> >> You can't depend on version 2 being available.  Its not available for >> ARM at all, and may be disabled for security reasons on x86 (there was >> some extended fun with transitive grants in the past, and we offered >> "totally disable grant v2" as one mitigation). > I don't need v2 at all, I was just going by the comments in grant_table.h, > which read: "Version 1 of the grant table entry structure is maintained > purely for backwards compatibility. New guests should use version 2." Ha... That comment wasn't written with the benefit of hindsight. IMO, grant v2 is not worth the astounding quantity of XSAs its implementation actually gave us, or the complexity of the resulting code. > Grant status frames are a v2-only thing, right? Correct. The split status frames was new in v2, in an attempt to reduce cacheline ping-pong. The downside is that the guest needs an unbounded loop trying to make a change to the grant table while ensuring that the flags in the status frame don't change asynchronously. ~Andrew P.S. In some theoretical world, I was hoping to have something to live in https://xenbits.xen.org/docs/latest/guest-guide/index.html on this subject.  Do you mind if I use you as a retroactive guineapig if/when some documentation were to appear? From mirageos-devel-bounces@lists.xenproject.org Wed Jun 10 15:07:38 2020 Return-path: Envelope-to: archives@lists.xenproject.org Delivery-date: Wed, 10 Jun 2020 15:07:38 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1jj2K7-0001Cu-Il; Wed, 10 Jun 2020 15:07:27 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1jj2K6-0001Cp-JV for mirageos-devel@lists.xenproject.org; Wed, 10 Jun 2020 15:07:26 +0000 X-Inumbo-ID: 17b3199e-ab2c-11ea-8496-bc764e2007e4 Received: from smtp.lucina.net (unknown [62.176.169.44]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id 17b3199e-ab2c-11ea-8496-bc764e2007e4; Wed, 10 Jun 2020 15:07:25 +0000 (UTC) Received: from nodbug.lucina.net (78-141-76-187.dynamic.orange.sk [78.141.76.187]) by smtp.lucina.net (Postfix) with ESMTPSA id CF65B122804; Wed, 10 Jun 2020 17:07:24 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=lucina.net; s=dkim-201811; t=1591801644; bh=brXaipuMVaItSAA9lBzHtWeUBGANdyih+K9IC1ATdDs=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=aDBHv/lnU3FUIX/xeH+d8DqZSMpA8Tuo1pH7FxD7yNBYPDqAL1GtTYHSk/sccOQoO ddW0tqXnir1v93kE5+8Fcazxckas96m1g6uZ/FCXkqPJDUwWMHBMCHZQEuexFBIaZe 95eST6L+95KBJcgbLzJoJq6+OfbpGR2EnGdFQS+tb+bjd5AYQcVmQKrz0M8wzhQW3n 8c3/DWrV6Rn2S/h7Q+ORDlgXF1ja/zbnC0GI7VPC5b7eWibVw2yw++9bJ6JlVXcoWT WSg4xmCc2Lb77M0KGmW4/m1zXiNk55vT6K09AuSVwi3WNrCZu7wsDYexP/ToKNoOIV UuWjXeiSX2zjQ== Received: by nodbug.lucina.net (Postfix, from userid 1000) id B6720265E722; Wed, 10 Jun 2020 17:07:24 +0200 (CEST) Date: Wed, 10 Jun 2020 17:07:24 +0200 From: Martin Lucina To: Andrew Cooper Subject: Re: XENMAPSPACE_grant_table vs. GNTTABOP_setup_table Message-ID: <20200610150724.GC16839@nodbug.lucina.net> Mail-Followup-To: Martin Lucina , Andrew Cooper , xen-devel@lists.xenproject.org, mirageos-devel@lists.xenproject.org References: <20200609094425.GB9734@nodbug.lucina.net> <3c7269b9-bf3f-5359-6ce2-049f935c8e84@citrix.com> <20200610132225.GA16839@nodbug.lucina.net> <46e87834-bf47-4003-1f32-89a47255155d@citrix.com> <20200610135601.GB16839@nodbug.lucina.net> <6b2a0091-2c56-895c-762d-719e5796985d@citrix.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <6b2a0091-2c56-895c-762d-719e5796985d@citrix.com> User-Agent: Mutt/1.10.1 (2018-07-13) X-BeenThere: mirageos-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Developer list for MirageOS List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: xen-devel@lists.xenproject.org, mirageos-devel@lists.xenproject.org Errors-To: mirageos-devel-bounces@lists.xenproject.org Sender: "MirageOS-devel" On Wednesday, 10.06.2020 at 15:21, Andrew Cooper wrote: > > I don't need v2 at all, I was just going by the comments in grant_table.h, > > which read: "Version 1 of the grant table entry structure is maintained > > purely for backwards compatibility. New guests should use version 2." > > Ha... > > That comment wasn't written with the benefit of hindsight. > > IMO, grant v2 is not worth the astounding quantity of XSAs its > implementation actually gave us, or the complexity of the resulting code. > > > Grant status frames are a v2-only thing, right? > > Correct. > > The split status frames was new in v2, in an attempt to reduce cacheline > ping-pong. > > The downside is that the guest needs an unbounded loop trying to make a > change to the grant table while ensuring that the flags in the status > frame don't change asynchronously. > > ~Andrew > > P.S. In some theoretical world, I was hoping to have something to live > in https://xenbits.xen.org/docs/latest/guest-guide/index.html on this > subject.  Do you mind if I use you as a retroactive guineapig if/when > some documentation were to appear? No problem, sure, go for it. -mato From mirageos-devel-bounces@lists.xenproject.org Mon Jun 15 14:26:21 2020 Return-path: Envelope-to: archives@lists.xenproject.org Delivery-date: Mon, 15 Jun 2020 14:26:21 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1jkq3k-00019s-IF; Mon, 15 Jun 2020 14:26:00 +0000 Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1jkq3j-00019n-NN for mirageos-devel@lists.xenproject.org; Mon, 15 Jun 2020 14:25:59 +0000 X-Inumbo-ID: 216686ec-af14-11ea-b802-12813bfff9fa Received: from smtp.lucina.net (unknown [62.176.169.44]) by us1-amaz-eas2.inumbo.com (Halon) with ESMTPS id 216686ec-af14-11ea-b802-12813bfff9fa; Mon, 15 Jun 2020 14:25:58 +0000 (UTC) Received: from webmail.moloch.sk (w3-s.a.lucina.net [62.176.169.73]) by smtp.lucina.net (Postfix) with ESMTPSA id D151C122804; Mon, 15 Jun 2020 16:25:57 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=lucina.net; s=dkim-201811; t=1592231157; bh=ipoQyzITtXCOufopaXaEd/sFV/+23MisWaZOGj5QRF4=; h=Date:From:To:Subject:From; b=ZljQcOLci8Lxmy6VqcNFFxdz85y2zpztbaK2cudxoyfTmcSGqX3lW52gyoAKcBvo1 b/GeZYtQyTsDcj2spl7MOxlCoVoJ/4JYl7JJJ/s9g6/aW68GoOOneNFQlg+3vXerGA 8hvNpXMJzkgCJQIEGUUjX7dR+zFBGHKGvfNCqmCGuAwTXIwqL8/EChT0mGzxZdyUrr Ne5s1pJX1hbLaBsbmiVR5PcM/jeOWyhC/sFgoDLh+cCDcnxqp0/N0W7HnUe5M4GOjH nmEB8HtuNj4MY1UxuJSVD/OFV8mUUEXAdzed9atYc1Pm3JaMOAPTjBKshqg3OgAMNq VhuHHquq/WDjQ== MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Transfer-Encoding: 7bit Date: Mon, 15 Jun 2020 16:25:57 +0200 From: Martin Lucina To: xen-devel@lists.xenproject.org, mirageos-devel@lists.xenproject.org Subject: Event delivery and "domain blocking" on PVHv2 Mail-Followup-To: xen-devel@lists.xenproject.org, mirageos-devel@lists.xenproject.org, martin@lucina.net Message-ID: <62479d08f7650c22678d7a86851eafc4@lucina.net> X-Sender: martin@lucina.net User-Agent: Roundcube Webmail/1.3.3 X-BeenThere: mirageos-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Developer list for MirageOS List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: mirageos-devel-bounces@lists.xenproject.org Sender: "MirageOS-devel" Hi, puzzle time: In my continuing explorations of the PVHv2 ABIs for the new MirageOS Xen stack, I've run into some issues with what looks like missed deliveries of events on event channels. While a simple unikernel that only uses the Xen console and effectively does for (1..5) { printf("foo"); sleep(1); } works fine, once I plug in the existing OCaml Xenstore and Netfront code, the behaviour I see is that the unikernel hangs in random places, blocking as if an event that should have been delivered has been missed. Multiple runs of the unikernel have it block at different places during Netfront setup, and sometimes it will run as far as a fully setup Netfront, and then wait for network packets. However, even if it gets that far, packets are not actually being delivered: Solo5: Xen console: port 0x2, ring @0x00000000FEFFF000 | ___| __| _ \ | _ \ __ \ \__ \ ( | | ( | ) | ____/\___/ _|\___/____/ Solo5: Bindings version v0.6.5-6-gf4b47d11 Solo5: Memory map: 256 MB addressable: Solo5: reserved @ (0x0 - 0xfffff) Solo5: text @ (0x100000 - 0x28ffff) Solo5: rodata @ (0x290000 - 0x2e0fff) Solo5: data @ (0x2e1000 - 0x3fafff) Solo5: heap >= 0x3fb000 < stack < 0x10000000 gnttab_init(): pages=1 entries=256 2020-06-15 13:42:08 -00:00: INF [net-xen frontend] connect 0 >>>> Sometimes we hang here 2020-06-15 13:42:08 -00:00: INF [net-xen frontend] create: id=0 domid=0 2020-06-15 13:42:08 -00:00: INF [net-xen frontend] sg:true gso_tcpv4:true rx_copy:true rx_flip:false smart_poll:false 2020-06-15 13:42:08 -00:00: INF [net-xen frontend] MAC: 00:16:3e:30:49:52 >>>> Or here gnttab_grant_access(): ref=0x8, domid=0x0, addr=0x8f9000, readonly=0 gnttab_grant_access(): ref=0x9, domid=0x0, addr=0x8fb000, readonly=0 evtchn_alloc_unbound(remote=0x0) = 0x4 2020-06-15 13:42:08 -00:00: INF [ethernet] Connected Ethernet interface 00:16:3e:30:49:52 2020-06-15 13:42:08 -00:00: INF [ARP] Sending gratuitous ARP for 10.0.0.2 (00:16:3e:30:49:52) gnttab_grant_access(): ref=0xa, domid=0x0, addr=0x8fd000, readonly=1 2020-06-15 13:42:08 -00:00: INF [udp] UDP interface connected on 10.0.0.2 2020-06-15 13:42:08 -00:00: INF [tcpip-stack-direct] stack assembled: mac=00:16:3e:30:49:52,ip=10.0.0.2 Gntref.get(): Waiting for free grant Gntref.get(): Waiting for free grant >>>> The above are also rather odd, but not related to event channel >>>> delivery, so one problem at a time... >>>> Once we get this far, packets should be flowing but aren't (either >>>> way). However, Xenstore is obviously working, as we wouldn't get >>>> through Netfront setup without it. Given that I've essentially re-written the low-level event channel C code, I'd like to verify that the mechanisms I'm using for event delivery are indeed the right thing to do on PVHv2. For event delivery, I'm registering the upcall with Xen as follows: uint64_t val = 32ULL; val |= (uint64_t)HVM_PARAM_CALLBACK_TYPE_VECTOR << 56; int rc = hypercall_hvm_set_param(HVM_PARAM_CALLBACK_IRQ, val); assert(rc == 0); i.e. upcalls are to be delivered via IDT vector. Questions: 1. Being based on the Solo5 virtio code, the low-level setup code is doing the "usual" i8259 PIC setup, to remap the PIC IRQs to vectors 32 and above. Should I be doing this initialisation for Xen PVH at all? I'm not interested in using the PIC for anything, and all interrupts will be delivered via Xen event channels. 2. Related to the above, the IRQ handler code is ACKing the interrupt after the handler runs. Should I be doing that? Does ACKing "IRQ" 0 on the PIC have any interactions with Xen's view of event channels/pending upcalls? Next, for a PVHv2, uniprocessor only guest, is the following flow sufficient to unmask an event channel? struct shared_info *s = SHARED_INFO(); int pending = 0; atomic_sync_btc(port, &s->evtchn_mask[0]); pending = sync_bt(port, &s->evtchn_mask[0]); if (pending) { /* * Slow path: * * If pending is set here, then there was a race, and we lost the * upcall. Mask the port again and force an upcall via a call to * hyperspace. * * This should be sufficient for HVM/PVHv2 based on my understanding of * Linux drivers/xen/events/events_2l.c. */ atomic_sync_bts(port, &s->evtchn_mask[0]); hypercall_evtchn_unmask(port); } Lastly, the old PV-only Mini-OS based stack would do delays ("block the domain") by doing a HYPERVISOR_set_timer_op(deadline) followed by a HYPERVISOR_sched_op(SCHEDOP_block,0 ). In the new code, I'm doing the following (based on what Mini-OS seems to be doing for HVM): solo5_time_t deadline = Int64_val(v_deadline); if (solo5_clock_monotonic() < deadline) { hypercall_set_timer_op(deadline); __asm__ __volatile__ ("hlt" : : : "memory"); /* XXX: cancel timer_op here if woken up early? */ } Again, is this the right thing to do for PVH? As the comment says, do I need to cancel the timer_op? I understood the semantics to be "fire once at/after the time deadline is reached", if that is indeed the case then with my current VIRQ_TIMER handler which does nothing in the interrupt context and has no side effects I should be fine. I can also post the code that does the actual demuxing of events from Xen (i.e. the upcall handler), but I've read through that several times now and I don't think the problem is there; adding diagnostic prints to both the low-level C event channel code and higher-level OCaml Activations code confirms that received events are being mapped to their ports correctly. Any advice much appreciated, Thanks, Martin From mirageos-devel-bounces@lists.xenproject.org Mon Jun 15 15:04:14 2020 Return-path: Envelope-to: archives@lists.xenproject.org Delivery-date: Mon, 15 Jun 2020 15:04:14 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1jkqeV-0005ur-6T; Mon, 15 Jun 2020 15:03:59 +0000 Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1jkqeT-0005u1-Sx for mirageos-devel@lists.xenproject.org; Mon, 15 Jun 2020 15:03:57 +0000 X-Inumbo-ID: 6bdfef6a-af19-11ea-b807-12813bfff9fa Received: from esa5.hc3370-68.iphmx.com (unknown [216.71.155.168]) by us1-amaz-eas2.inumbo.com (Halon) with ESMTPS id 6bdfef6a-af19-11ea-b807-12813bfff9fa; Mon, 15 Jun 2020 15:03:51 +0000 (UTC) Authentication-Results: esa5.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none IronPort-SDR: 4lB6PxmJeYWm+XlMP7xLA7fOZ65fLTIYKq2IWee2/NV0h8GNDq8KuKRB0tw0ExAun4d0xt99+Y JIqAujQMNtXqgN6yl/5kju6fa+ZH+5uQdEsf4vss3b/3nUkMQm6TsKgdCTQJ+bl2Xb7cLEBCyH 8J/vatIiDd1csX3H43MUT143iOopgZrSCJjJwLnHSSFnEtmMmZuERLWg0ttNCX4CC/OF3hXZVp 1mEsoCFz9tEv8Ffm/tI6pnNVGNUTEwFVxcT9fe75Q3H4krSOky7EFDPt/AA0GrbT9SE+uMUU6y jgs= X-SBRS: 2.7 X-MesageID: 20305325 X-Ironport-Server: esa5.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.73,514,1583211600"; d="scan'208";a="20305325" Date: Mon, 15 Jun 2020 17:03:44 +0200 From: Roger Pau =?utf-8?B?TW9ubsOp?= To: , , Subject: Re: Event delivery and "domain blocking" on PVHv2 Message-ID: <20200615150344.GG735@Air-de-Roger> References: <62479d08f7650c22678d7a86851eafc4@lucina.net> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Disposition: inline In-Reply-To: <62479d08f7650c22678d7a86851eafc4@lucina.net> X-ClientProxiedBy: AMSPEX02CAS02.citrite.net (10.69.22.113) To AMSPEX02CL02.citrite.net (10.69.22.126) X-BeenThere: mirageos-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Developer list for MirageOS List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: mirageos-devel-bounces@lists.xenproject.org Sender: "MirageOS-devel" On Mon, Jun 15, 2020 at 04:25:57PM +0200, Martin Lucina wrote: > Hi, > > puzzle time: In my continuing explorations of the PVHv2 ABIs for the new > MirageOS Xen stack, I've run into some issues with what looks like missed > deliveries of events on event channels. That would be weird, as other OSes using PVHv2 seem to be fine. > While a simple unikernel that only uses the Xen console and effectively does > for (1..5) { printf("foo"); sleep(1); } works fine, once I plug in the > existing OCaml Xenstore and Netfront code, the behaviour I see is that the > unikernel hangs in random places, blocking as if an event that should have > been delivered has been missed. > > Multiple runs of the unikernel have it block at different places during > Netfront setup, and sometimes it will run as far as a fully setup Netfront, > and then wait for network packets. However, even if it gets that far, > packets are not actually being delivered: > > Solo5: Xen console: port 0x2, ring @0x00000000FEFFF000 > | ___| > __| _ \ | _ \ __ \ > \__ \ ( | | ( | ) | > ____/\___/ _|\___/____/ > Solo5: Bindings version v0.6.5-6-gf4b47d11 > Solo5: Memory map: 256 MB addressable: > Solo5: reserved @ (0x0 - 0xfffff) > Solo5: text @ (0x100000 - 0x28ffff) > Solo5: rodata @ (0x290000 - 0x2e0fff) > Solo5: data @ (0x2e1000 - 0x3fafff) > Solo5: heap >= 0x3fb000 < stack < 0x10000000 > gnttab_init(): pages=1 entries=256 > 2020-06-15 13:42:08 -00:00: INF [net-xen frontend] connect 0 > > > > > Sometimes we hang here > 2020-06-15 13:42:08 -00:00: INF [net-xen frontend] create: id=0 domid=0 > 2020-06-15 13:42:08 -00:00: INF [net-xen frontend] sg:true gso_tcpv4:true > rx_copy:true rx_flip:false smart_poll:false > 2020-06-15 13:42:08 -00:00: INF [net-xen frontend] MAC: 00:16:3e:30:49:52 > > > > > Or here > gnttab_grant_access(): ref=0x8, domid=0x0, addr=0x8f9000, readonly=0 > gnttab_grant_access(): ref=0x9, domid=0x0, addr=0x8fb000, readonly=0 > evtchn_alloc_unbound(remote=0x0) = 0x4 > 2020-06-15 13:42:08 -00:00: INF [ethernet] Connected Ethernet interface > 00:16:3e:30:49:52 > 2020-06-15 13:42:08 -00:00: INF [ARP] Sending gratuitous ARP for 10.0.0.2 > (00:16:3e:30:49:52) > gnttab_grant_access(): ref=0xa, domid=0x0, addr=0x8fd000, readonly=1 > 2020-06-15 13:42:08 -00:00: INF [udp] UDP interface connected on 10.0.0.2 > 2020-06-15 13:42:08 -00:00: INF [tcpip-stack-direct] stack assembled: > mac=00:16:3e:30:49:52,ip=10.0.0.2 > Gntref.get(): Waiting for free grant > Gntref.get(): Waiting for free grant > > > > > The above are also rather odd, but not related to event > > > > > channel delivery, so one problem at a time... > > > > > Once we get this far, packets should be flowing but aren't > > > > > (either way). However, Xenstore is obviously working, as we > > > > > wouldn't get through Netfront setup without it. > > Given that I've essentially re-written the low-level event channel C code, > I'd like to verify that the mechanisms I'm using for event delivery are > indeed the right thing to do on PVHv2. > > For event delivery, I'm registering the upcall with Xen as follows: > > uint64_t val = 32ULL; > val |= (uint64_t)HVM_PARAM_CALLBACK_TYPE_VECTOR << 56; > int rc = hypercall_hvm_set_param(HVM_PARAM_CALLBACK_IRQ, val); > assert(rc == 0); > > i.e. upcalls are to be delivered via IDT vector. This way of event channel injection is slitgly hackish, and I would recommend using HVMOP_set_evtchn_upcall_vector, that way vectors will be properly routed using the lapic. Using HVM_PARAM_CALLBACK_TYPE_VECTOR vectors are injected without setting the IRR/ISR bit in the lapic registers. > Questions: > > 1. Being based on the Solo5 virtio code, the low-level setup code is doing > the "usual" i8259 PIC setup, to remap the PIC IRQs to vectors 32 and above. > Should I be doing this initialisation for Xen PVH at all? Hm, there are no IO-APICs (or legacy PICs) on a PVH domU, so there's not much to route. If Solo5 is thinking it's somehow configuring them it's likely writing to some hole in memory, or to some RAM. IO-APIC presence is signaled on the ACPI MADT table on PVH domU. > I'm not interested > in using the PIC for anything, and all interrupts will be delivered via Xen > event channels. > > 2. Related to the above, the IRQ handler code is ACKing the interrupt after > the handler runs. Should I be doing that? Does ACKing "IRQ" 0 on the PIC > have any interactions with Xen's view of event channels/pending upcalls? Which kind of ACking it's doing? Is it writing to the lapic EOI register? If so that would be OK when using HVMOP_set_evtchn_upcall_vector. If using HVM_PARAM_CALLBACK_TYPE_VECTOR there's nothing to Ack I think. > Next, for a PVHv2, uniprocessor only guest, is the following flow sufficient > to unmask an event channel? > > struct shared_info *s = SHARED_INFO(); > int pending = 0; > > atomic_sync_btc(port, &s->evtchn_mask[0]); > pending = sync_bt(port, &s->evtchn_mask[0]); You should check for pending interrupts on evtchn_pending, not evtchn_mask. > if (pending) { > /* > * Slow path: > * > * If pending is set here, then there was a race, and we lost the > * upcall. Mask the port again and force an upcall via a call to > * hyperspace. > * > * This should be sufficient for HVM/PVHv2 based on my understanding > of > * Linux drivers/xen/events/events_2l.c. > */ > atomic_sync_bts(port, &s->evtchn_mask[0]); > hypercall_evtchn_unmask(port); > } FWIW, I use the hypercall unconditionally on FreeBSD because I didn't see a performance difference when compared to this method. > Lastly, the old PV-only Mini-OS based stack would do delays ("block the > domain") by doing a HYPERVISOR_set_timer_op(deadline) followed by a > HYPERVISOR_sched_op(SCHEDOP_block,0 ). In the new code, I'm doing the > following (based on what Mini-OS seems to be doing for HVM): > > solo5_time_t deadline = Int64_val(v_deadline); > > if (solo5_clock_monotonic() < deadline) { > hypercall_set_timer_op(deadline); > __asm__ __volatile__ ("hlt" : : : "memory"); > /* XXX: cancel timer_op here if woken up early? */ > } > > Again, is this the right thing to do for PVH? hlt will trap into the hypervisor, so it's fine to use. > As the comment says, do I need to cancel the timer_op? I'm not sure. Keep in mind that a new call to hypercall_set_timer_op will overwrite the previous timer, and hence should be fine I think as long as you are using the one-shot timer. > I understood the > semantics to be "fire once at/after the time deadline is reached", if that > is indeed the case then with my current VIRQ_TIMER handler which does > nothing in the interrupt context and has no side effects I should be fine. I have no idea how Solo5 works, maybe you should re-set the timer to the next deadline in the handler? Or that's fine because the timer is always set before blocking. > I can also post the code that does the actual demuxing of events from Xen > (i.e. the upcall handler), but I've read through that several times now and > I don't think the problem is there; adding diagnostic prints to both the > low-level C event channel code and higher-level OCaml Activations code > confirms that received events are being mapped to their ports correctly. I can take a look at the event channel handler if you want, as you wish. The only weird think I've noticed is the error in the unmask where you seem to use evtchn_mask instead of evtchn_pending. I would also recommend using HVMOP_set_evtchn_upcall_vector instead of HVM_PARAM_CALLBACK_TYPE_VECTOR. In order to make some tools happy just set HVM_PARAM_CALLBACK_TYPE_VECTOR to 1. You can take a look at how Xen does it when running as a guest: http://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=xen/arch/x86/guest/xen/xen.c;h=2ff63d370a8a12fef166677e2ded7ed82a628ce8;hb=HEAD#l205 Roger. From mirageos-devel-bounces@lists.xenproject.org Mon Jun 15 15:52:05 2020 Return-path: Envelope-to: archives@lists.xenproject.org Delivery-date: Mon, 15 Jun 2020 15:52:05 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1jkrOy-00027Z-UV; Mon, 15 Jun 2020 15:52:00 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1jkrOx-00026z-Mq for mirageos-devel@lists.xenproject.org; Mon, 15 Jun 2020 15:51:59 +0000 X-Inumbo-ID: 21272040-af20-11ea-bb8b-bc764e2007e4 Received: from smtp.lucina.net (unknown [62.176.169.44]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id 21272040-af20-11ea-bb8b-bc764e2007e4; Mon, 15 Jun 2020 15:51:53 +0000 (UTC) Received: from webmail.moloch.sk (w3-s.a.lucina.net [62.176.169.73]) by smtp.lucina.net (Postfix) with ESMTPSA id 6DDC5122804; Mon, 15 Jun 2020 17:51:51 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=lucina.net; s=dkim-201811; t=1592236311; bh=/FpB91+u9v9wBFgEkB147d5HxiDmF56rYU3flrWJCVU=; h=Date:From:To:Cc:Subject:In-Reply-To:References:From; b=kTh5y8BamtZt5tmZe83ebu+xnp6Dw+TZROg6BfUYP6g3hQqNHcuiKAmXSyCpFZKL6 8DuuaJeq8IZ2UYPEhhRstwyNWoMO086F+wrr91b0++VqNUvyHVa7IXmhFoC7Ewo5ei fKadqJ9IsNk1XUfYwIVBQ9gmhH0nr3DiSLnCveAH4Gz/yEOcSkmpnSuj1UzvefBEdX e/6R2DunG5goRu/VttgSD7zEoDtc8NN0fXX1RcGKLr89iLQSeue6sScfi4m1WpUbWi RMWMfhuS413z8TrQnZ4y0MtC+XXzpM7rJhpC+yoFOMn27opZYxZovCYeJ4g9ONJvGK YszBJCPolTJ/A== MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Date: Mon, 15 Jun 2020 17:51:51 +0200 From: Martin Lucina To: =?UTF-8?Q?Roger_Pau_Monn=C3=A9?= Subject: Re: Event delivery and "domain blocking" on PVHv2 In-Reply-To: <20200615150344.GG735@Air-de-Roger> References: <62479d08f7650c22678d7a86851eafc4@lucina.net> <20200615150344.GG735@Air-de-Roger> Message-ID: <4a0bb4fa4dca2732feae4e2f825eb2a6@lucina.net> X-Sender: martin@lucina.net User-Agent: Roundcube Webmail/1.3.3 X-BeenThere: mirageos-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Developer list for MirageOS List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: xen-devel@lists.xenproject.org, mirageos-devel@lists.xenproject.org Errors-To: mirageos-devel-bounces@lists.xenproject.org Sender: "MirageOS-devel" On 2020-06-15 17:03, Roger Pau MonnĂ© wrote: > This way of event channel injection is slitgly hackish, and I would > recommend using HVMOP_set_evtchn_upcall_vector, that way vectors will > be properly routed using the lapic. > > Using HVM_PARAM_CALLBACK_TYPE_VECTOR vectors are injected without > setting the IRR/ISR bit in the lapic registers. I picked HVM_PARAM_CALLBACK_TYPE_VECTOR since that seemed like the easiest option for a uniprocessor, PVH-only guest. What does mean in the context of HVMOP_set_evtchn_upcall_vector? If it's an APIC vector number (sorry, not too familiar with the post-legacy i8259 world), does that imply that if I use HVMOP_set_evtchn_upcall_vector I need to do some APIC initialisation / programming? All I need for Solo5/Mirage purposes is to have the upcall land on IDT vector #32 or above. > >> Questions: >> >> 1. Being based on the Solo5 virtio code, the low-level setup code is >> doing >> the "usual" i8259 PIC setup, to remap the PIC IRQs to vectors 32 and >> above. >> Should I be doing this initialisation for Xen PVH at all? > > Hm, there are no IO-APICs (or legacy PICs) on a PVH domU, so there's > not much to route. If Solo5 is thinking it's somehow configuring them > it's likely writing to some hole in memory, or to some RAM. Solo5 only has a very primitive understanding of hardware interrupts, so it's only configuring the legacy PICs via port IO. > > IO-APIC presence is signaled on the ACPI MADT table on PVH domU. > Hmm, it'd be very unfortunate if I had to deal with ACPI, so here's hoping that I don't actually need to touch the APIC. >> I'm not interested >> in using the PIC for anything, and all interrupts will be delivered >> via Xen >> event channels. >> >> 2. Related to the above, the IRQ handler code is ACKing the interrupt >> after >> the handler runs. Should I be doing that? Does ACKing "IRQ" 0 on the >> PIC >> have any interactions with Xen's view of event channels/pending >> upcalls? > > Which kind of ACking it's doing? Is it writing to the lapic EOI > register? If so that would be OK when using > HVMOP_set_evtchn_upcall_vector. If using > HVM_PARAM_CALLBACK_TYPE_VECTOR there's nothing to Ack I think. Legacy PIC EOI via port IO. > >> Next, for a PVHv2, uniprocessor only guest, is the following flow >> sufficient >> to unmask an event channel? >> >> struct shared_info *s = SHARED_INFO(); >> int pending = 0; >> >> atomic_sync_btc(port, &s->evtchn_mask[0]); >> pending = sync_bt(port, &s->evtchn_mask[0]); > > You should check for pending interrupts on evtchn_pending, not > evtchn_mask. Ah, thanks for spotting that! Fixed, but just that change by itself does not seem to change the observed behaviour in any way. > >> if (pending) { >> /* >> * Slow path: >> * >> * If pending is set here, then there was a race, and we lost >> the >> * upcall. Mask the port again and force an upcall via a call >> to >> * hyperspace. >> * >> * This should be sufficient for HVM/PVHv2 based on my >> understanding >> of >> * Linux drivers/xen/events/events_2l.c. >> */ >> atomic_sync_bts(port, &s->evtchn_mask[0]); >> hypercall_evtchn_unmask(port); >> } > > FWIW, I use the hypercall unconditionally on FreeBSD because I didn't > see a performance difference when compared to this method. > >> Lastly, the old PV-only Mini-OS based stack would do delays ("block >> the >> domain") by doing a HYPERVISOR_set_timer_op(deadline) followed by a >> HYPERVISOR_sched_op(SCHEDOP_block,0 ). In the new code, I'm doing the >> following (based on what Mini-OS seems to be doing for HVM): >> >> solo5_time_t deadline = Int64_val(v_deadline); >> >> if (solo5_clock_monotonic() < deadline) { >> hypercall_set_timer_op(deadline); >> __asm__ __volatile__ ("hlt" : : : "memory"); >> /* XXX: cancel timer_op here if woken up early? */ >> } >> >> Again, is this the right thing to do for PVH? > > hlt will trap into the hypervisor, so it's fine to use. > Thanks for confirming. >> As the comment says, do I need to cancel the timer_op? > > I'm not sure. Keep in mind that a new call to hypercall_set_timer_op > will overwrite the previous timer, and hence should be fine I think as > long as you are using the one-shot timer. Is there something other than a one-shot timer? hypercall_set_timer_op is just a single-argument hypercall with an uint64_t deadline, right? (And not documented in xen.h either ...) > >> I understood the >> semantics to be "fire once at/after the time deadline is reached", if >> that >> is indeed the case then with my current VIRQ_TIMER handler which does >> nothing in the interrupt context and has no side effects I should be >> fine. > > I have no idea how Solo5 works, maybe you should re-set the timer to > the next deadline in the handler? > > Or that's fine because the timer is always set before blocking. Yes, it's always set before blocking, and we always unblock after the first interrupt (i.e. some event) is received, so should be fine. > >> I can also post the code that does the actual demuxing of events from >> Xen >> (i.e. the upcall handler), but I've read through that several times >> now and >> I don't think the problem is there; adding diagnostic prints to both >> the >> low-level C event channel code and higher-level OCaml Activations code >> confirms that received events are being mapped to their ports >> correctly. > > I can take a look at the event channel handler if you want, as you > wish. The only weird think I've noticed is the error in the unmask > where you seem to use evtchn_mask instead of evtchn_pending. Thanks for the offer, this stuff is fairly subtle. In the Mirage/Xen scenario, there are two parts to the upcall handler. The top half is executed in interrupt context and basically does nothing except acknowledge the upcall: int solo5__xen_evtchn_vector_handler(void *arg __attribute__((unused))) { struct vcpu_info *vi = VCPU0_INFO(); /* * All work is done outside of interrupt context by evtchn_demux_pending(), * so all we need to do here is acknowledge the upcall from Xen. */ vi->evtchn_upcall_pending = 0; return 1; } The bottom half is then called periodically (and always before blocking) by the OCaml code: static bool evtchn_demux_pending(void) { struct shared_info *s = SHARED_INFO(); struct vcpu_info *vi = VCPU0_INFO(); bool some_pending = false; vi->evtchn_upcall_pending = 0; /* * Demux events received from Xen. * * pending_l1 is the "outer" per-VCPU selector (evtchn_pending_sel). * pending_l2 is the "inner" system-wide word (evtchn_pending[l1i]). */ xen_ulong_t pending_l1, pending_l2; atomic_sync_xchg(&vi->evtchn_pending_sel, 0, &pending_l1); while (pending_l1 != 0) { xen_ulong_t l1i = ffs(pending_l1); /* * Masking pending_l2 with ~evtchn_mask[l1i] is necessary to get the * *current* masked events; otherwise races like the following * can occur: * * 1. X is generated, upcall scheduled by Xen. * 2. X is masked. * 3. Upcall is delivered. * 4. X fires despite now being masked. */ while ((pending_l2 = (s->evtchn_pending[l1i] & ~s->evtchn_mask[l1i])) != 0) { xen_ulong_t l2i = ffs(pending_l2); evtchn_port_t port = (l1i * (sizeof(xen_ulong_t) * 8)) + l2i; /* * Mark as pending on the OCaml side. */ evtchn_callback_ml[port] = 1; some_pending = true; atomic_sync_btc(l2i, &s->evtchn_pending[l1i]); } pending_l1 &= ~(1UL << l1i); } return some_pending; } > > I would also recommend using HVMOP_set_evtchn_upcall_vector instead of > HVM_PARAM_CALLBACK_TYPE_VECTOR. In order to make some tools happy just > set HVM_PARAM_CALLBACK_TYPE_VECTOR to 1. You can take a look at how > Xen does it when running as a guest: > > http://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=xen/arch/x86/guest/xen/xen.c;h=2ff63d370a8a12fef166677e2ded7ed82a628ce8;hb=HEAD#l205 Thanks for the pointer. As I write above, if I can use HVMOP_set_evtchn_upcall_vector w/o doing too much "extra work" on the guest side then I will go with that. > > Roger. -mato From mirageos-devel-bounces@lists.xenproject.org Mon Jun 15 16:53:08 2020 Return-path: Envelope-to: archives@lists.xenproject.org Delivery-date: Mon, 15 Jun 2020 16:53:08 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1jksLx-000885-C7; Mon, 15 Jun 2020 16:52:57 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1jksLw-000880-Nw for mirageos-devel@lists.xenproject.org; Mon, 15 Jun 2020 16:52:56 +0000 X-Inumbo-ID: a7bbb898-af28-11ea-bca7-bc764e2007e4 Received: from esa6.hc3370-68.iphmx.com (unknown [216.71.155.175]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id a7bbb898-af28-11ea-bca7-bc764e2007e4; Mon, 15 Jun 2020 16:52:54 +0000 (UTC) Authentication-Results: esa6.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none IronPort-SDR: ogwVSP6sgRf3vISs9oDIA72rmuVO3ykqi0ng0XugkFMlFYpWvSszEQrgg5rnUQlX/r3HyRrAws 79Um+qmAsa/DJznWc5iTyejkKz9/1lWGI9etbKo4Dk1vXmSBqHpj5Qnq/y1uBFKfs61C8eJxFw QaPpOiVAUbx3q0IytCYwC8sovLU6KAVqBGhB8EVDU2w0ybqG5L9x8p6LyPEQy8l/OjyWloGXsE mlbyjVVY/vjNt0Oy4ckUKC6KtLsCqsCzlnO9m7kPQoFklgHRfGdnxhjM+oIgBNAipsKP015T6k REE= X-SBRS: 2.7 X-MesageID: 20427979 X-Ironport-Server: esa6.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.73,515,1583211600"; d="scan'208";a="20427979" Date: Mon, 15 Jun 2020 18:52:44 +0200 From: Roger Pau =?utf-8?B?TW9ubsOp?= To: Martin Lucina Subject: Re: Event delivery and "domain blocking" on PVHv2 Message-ID: <20200615165101.GJ735@Air-de-Roger> References: <62479d08f7650c22678d7a86851eafc4@lucina.net> <20200615150344.GG735@Air-de-Roger> <4a0bb4fa4dca2732feae4e2f825eb2a6@lucina.net> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <4a0bb4fa4dca2732feae4e2f825eb2a6@lucina.net> X-ClientProxiedBy: AMSPEX02CAS02.citrite.net (10.69.22.113) To AMSPEX02CL02.citrite.net (10.69.22.126) X-BeenThere: mirageos-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Developer list for MirageOS List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: xen-devel@lists.xenproject.org, mirageos-devel@lists.xenproject.org Errors-To: mirageos-devel-bounces@lists.xenproject.org Sender: "MirageOS-devel" On Mon, Jun 15, 2020 at 05:51:51PM +0200, Martin Lucina wrote: > On 2020-06-15 17:03, Roger Pau MonnĂ© wrote: > > This way of event channel injection is slitgly hackish, and I would > > recommend using HVMOP_set_evtchn_upcall_vector, that way vectors will > > be properly routed using the lapic. > > > > Using HVM_PARAM_CALLBACK_TYPE_VECTOR vectors are injected without > > setting the IRR/ISR bit in the lapic registers. > > I picked HVM_PARAM_CALLBACK_TYPE_VECTOR since that seemed like the easiest > option for a uniprocessor, PVH-only guest. > > What does mean in the context of HVMOP_set_evtchn_upcall_vector? If > it's an APIC vector number (sorry, not too familiar with the post-legacy > i8259 world), does that imply that if I use HVMOP_set_evtchn_upcall_vector I > need to do some APIC initialisation / programming? > > All I need for Solo5/Mirage purposes is to have the upcall land on IDT > vector #32 or above. Oh, OK. It was reported that HVM_PARAM_CALLBACK_TYPE_VECTOR doesn't work well with migration and when using hardware APIC virtualization IIRC, because of the fact that interrupts are not signaled in the lapic and directly injected. > > > Questions: > > > > > > 1. Being based on the Solo5 virtio code, the low-level setup code is > > > doing > > > the "usual" i8259 PIC setup, to remap the PIC IRQs to vectors 32 and > > > above. > > > Should I be doing this initialisation for Xen PVH at all? > > > > Hm, there are no IO-APICs (or legacy PICs) on a PVH domU, so there's > > not much to route. If Solo5 is thinking it's somehow configuring them > > it's likely writing to some hole in memory, or to some RAM. > > Solo5 only has a very primitive understanding of hardware interrupts, so > it's only configuring the legacy PICs via port IO. Oh, then there's definitely no legacy PIC at all. Writes to the PIC IO ports will be dropped/ignored, and reads will return ~0. I guess you could implement some check based on that in order to avoid doing any initialization? I don't think it should be harmful in any way, but you are just likely spending a bunch of time trapping into the hypervisor to handle those reads/writes for no good reason. > > > > IO-APIC presence is signaled on the ACPI MADT table on PVH domU. > > > > Hmm, it'd be very unfortunate if I had to deal with ACPI, so here's hoping > that I don't actually need to touch the APIC. If you don't do any IO-APIC configuration at all then that's completely fine. > > > I'm not interested > > > in using the PIC for anything, and all interrupts will be delivered > > > via Xen > > > event channels. > > > > > > 2. Related to the above, the IRQ handler code is ACKing the > > > interrupt after > > > the handler runs. Should I be doing that? Does ACKing "IRQ" 0 on the > > > PIC > > > have any interactions with Xen's view of event channels/pending > > > upcalls? > > > > Which kind of ACking it's doing? Is it writing to the lapic EOI > > register? If so that would be OK when using > > HVMOP_set_evtchn_upcall_vector. If using > > HVM_PARAM_CALLBACK_TYPE_VECTOR there's nothing to Ack I think. > > Legacy PIC EOI via port IO. No need to do that. There's no legacy PIC on PVH, and then event channel interrupts are not routed through the PIC when using HVM_PARAM_CALLBACK_TYPE_VECTOR. > > I'm not sure. Keep in mind that a new call to hypercall_set_timer_op > > will overwrite the previous timer, and hence should be fine I think as > > long as you are using the one-shot timer. > > Is there something other than a one-shot timer? hypercall_set_timer_op is > just a single-argument hypercall with an uint64_t deadline, right? (And not > documented in xen.h either ...) There's also a periodic timer (see VCPUOP_{set/stop}_periodic_timer), which was enabled by default at 100Hz for PV guests. This is not the case for PVH. > > > I can also post the code that does the actual demuxing of events > > > from Xen > > > (i.e. the upcall handler), but I've read through that several times > > > now and > > > I don't think the problem is there; adding diagnostic prints to both > > > the > > > low-level C event channel code and higher-level OCaml Activations code > > > confirms that received events are being mapped to their ports > > > correctly. > > > > I can take a look at the event channel handler if you want, as you > > wish. The only weird think I've noticed is the error in the unmask > > where you seem to use evtchn_mask instead of evtchn_pending. > > Thanks for the offer, this stuff is fairly subtle. > > In the Mirage/Xen scenario, there are two parts to the upcall handler. The > top half is executed in interrupt context and basically does nothing except > acknowledge the upcall: > > int solo5__xen_evtchn_vector_handler(void *arg __attribute__((unused))) > { > struct vcpu_info *vi = VCPU0_INFO(); > > /* > * All work is done outside of interrupt context by > evtchn_demux_pending(), > * so all we need to do here is acknowledge the upcall from Xen. > */ > vi->evtchn_upcall_pending = 0; I think you should best avoid clearing evtchn_upcall_pending here, as Xen will then inject more vector callbacks if a new event is signaled, even when you have not processed the previous one? > return 1; > } > > The bottom half is then called periodically (and always before blocking) by > the OCaml code: > > static bool evtchn_demux_pending(void) > { > struct shared_info *s = SHARED_INFO(); > struct vcpu_info *vi = VCPU0_INFO(); > bool some_pending = false; > > vi->evtchn_upcall_pending = 0; > > /* > * Demux events received from Xen. > * > * pending_l1 is the "outer" per-VCPU selector (evtchn_pending_sel). > * pending_l2 is the "inner" system-wide word (evtchn_pending[l1i]). > */ > xen_ulong_t pending_l1, pending_l2; > atomic_sync_xchg(&vi->evtchn_pending_sel, 0, &pending_l1); > while (pending_l1 != 0) { > xen_ulong_t l1i = ffs(pending_l1); > > /* > * Masking pending_l2 with ~evtchn_mask[l1i] is necessary to get the > * *current* masked events; otherwise races like the following > * can occur: > * > * 1. X is generated, upcall scheduled by Xen. > * 2. X is masked. > * 3. Upcall is delivered. > * 4. X fires despite now being masked. > */ > while ((pending_l2 = > (s->evtchn_pending[l1i] & ~s->evtchn_mask[l1i])) != 0) { > xen_ulong_t l2i = ffs(pending_l2); > > evtchn_port_t port = (l1i * (sizeof(xen_ulong_t) * 8)) + l2i; > /* > * Mark as pending on the OCaml side. > */ > evtchn_callback_ml[port] = 1; How is this cleared? It must be cleared before the handler is run, or else you will likely miss interrupts. Also, I think you could mask the port before setting evtchn_callback_ml and unmask it when evtchn_callback_ml is cleared? > some_pending = true; > > atomic_sync_btc(l2i, &s->evtchn_pending[l1i]); > } > > pending_l1 &= ~(1UL << l1i); > } > > return some_pending; > } If you can dump the event channel numbers and their usage from Solo5 then you can check against the 'e' debug key from Xen in order to check if there are indeed pending events to be delivered on some of them. Just checking the output from the 'e' debug key will tell you if there's anything pending and if there are any channels masked. Roger. From mirageos-devel-bounces@lists.xenproject.org Mon Jun 15 16:58:50 2020 Return-path: Envelope-to: archives@lists.xenproject.org Delivery-date: Mon, 15 Jun 2020 16:58:50 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1jksRc-0008NP-HH; Mon, 15 Jun 2020 16:58:48 +0000 Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1jksRb-0008MO-S6 for mirageos-devel@lists.xenproject.org; Mon, 15 Jun 2020 16:58:47 +0000 X-Inumbo-ID: 76887684-af29-11ea-b81b-12813bfff9fa Received: from esa1.hc3370-68.iphmx.com (unknown [216.71.145.142]) by us1-amaz-eas2.inumbo.com (Halon) with ESMTPS id 76887684-af29-11ea-b81b-12813bfff9fa; Mon, 15 Jun 2020 16:58:41 +0000 (UTC) Authentication-Results: esa1.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none IronPort-SDR: EXtZe8DLeKVW4NdhUXHJmGzS59nbanjNfuh7yl5Z6MYZ/Kz9FQVS9HiRIleskPNxIm72kqgted AOGB8s90YLICbqr6FjalVEdRHyP4F79zci3LGleFoo9/t2Lo7GEq6+DUYDLS5pyyGl3KAXJpAK uthQbNTH8WWi950siBHikKGEDc4Zoaq7mP3/iaobTRbg+UD9ootzT0VIvv52jWVx97Zn+heRu5 iLZ9UkXvVuxmwXpEdzTlR9bW+gDx2sAAlUxKh3Qc3DHDLyn0YJhOC05c+LXpLpUyQuSb55w/TF pjg= X-SBRS: 2.7 X-MesageID: 20383002 X-Ironport-Server: esa1.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.73,515,1583211600"; d="scan'208";a="20383002" Subject: Re: Event delivery and "domain blocking" on PVHv2 To: , , References: <62479d08f7650c22678d7a86851eafc4@lucina.net> From: Andrew Cooper Message-ID: <5865159c-4190-e841-8020-7a4f3cf0fc24@citrix.com> Date: Mon, 15 Jun 2020 17:58:36 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.8.0 MIME-Version: 1.0 In-Reply-To: <62479d08f7650c22678d7a86851eafc4@lucina.net> Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 8bit Content-Language: en-GB X-ClientProxiedBy: AMSPEX02CAS02.citrite.net (10.69.22.113) To AMSPEX02CL02.citrite.net (10.69.22.126) X-BeenThere: mirageos-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Developer list for MirageOS List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: mirageos-devel-bounces@lists.xenproject.org Sender: "MirageOS-devel" On 15/06/2020 15:25, Martin Lucina wrote: > Hi, > > puzzle time: In my continuing explorations of the PVHv2 ABIs for the > new MirageOS Xen stack, I've run into some issues with what looks like > missed deliveries of events on event channels. > > While a simple unikernel that only uses the Xen console and > effectively does for (1..5) { printf("foo"); sleep(1); } works fine, > once I plug in the existing OCaml Xenstore and Netfront code, the > behaviour I see is that the unikernel hangs in random places, blocking > as if an event that should have been delivered has been missed. You can see what is going on, event channel wise, with the 'e' debug-key.  This will highlight cases such as the event channel being masked and pending, which is a common guest bug ending up in this state. > > > Given that I've essentially re-written the low-level event channel C > code, I'd like to verify that the mechanisms I'm using for event > delivery are indeed the right thing to do on PVHv2. > > For event delivery, I'm registering the upcall with Xen as follows: > >     uint64_t val = 32ULL; >     val |= (uint64_t)HVM_PARAM_CALLBACK_TYPE_VECTOR << 56; >     int rc = hypercall_hvm_set_param(HVM_PARAM_CALLBACK_IRQ, val); >     assert(rc == 0); > > i.e. upcalls are to be delivered via IDT vector. Don't use HVM_PARAM_CALLBACK_TYPE_VECTOR.  It is conceptually broken, as it bypasses all queueing and IRR logic in the LAPIC. At some point, I'm going to have to figure out a compatible way to deal with all the guests still using this mechanism, because it is incompatible with using hardware accelerated APIC support in IvyBridge/Zen+ hardware. Use HVMOP_set_evtchn_upcall_vector instead, which does the same thing, but actually behaves like a real vector as far as the rest of the LAPIC is concerned. > > Questions: > > 1. Being based on the Solo5 virtio code, the low-level setup code is > doing the "usual" i8259 PIC setup, to remap the PIC IRQs to vectors 32 > and above. Should I be doing this initialisation for Xen PVH at all? > I'm not interested in using the PIC for anything, and all interrupts > will be delivered via Xen event channels. PVH guests don't get a PIC by default.  Xen will just be swallowing all your setup and doing nothing with it. "plain" PVH guests also don't get an IO-APIC by default.  Unless you're wanting to get PVH dom0 support working, (or PCI Passthrough in the future), don't worry about the IO-APIC either. > > 2. Related to the above, the IRQ handler code is ACKing the interrupt > after the handler runs. Should I be doing that? Does ACKing "IRQ" 0 on > the PIC have any interactions with Xen's view of event > channels/pending upcalls? There's no PIC to begin with, but even then, talking to the PIC/IO-APIC would only be correct for type INTX/GSI. TYPE_VECTOR shouldn't have an ack at the LAPIC (it is this properly which makes it incompatible with hardware acceleration), while HVMOP_set_evtchn_upcall_vector should be acked at the LAPIC. ~Andrew From mirageos-devel-bounces@lists.xenproject.org Thu Jun 18 10:13:52 2020 Return-path: Envelope-to: archives@lists.xenproject.org Delivery-date: Thu, 18 Jun 2020 10:13:52 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1jlrY6-0003RO-9m; Thu, 18 Jun 2020 10:13:34 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1jlrY4-0003RI-Tg for mirageos-devel@lists.xenproject.org; Thu, 18 Jun 2020 10:13:32 +0000 X-Inumbo-ID: 5c04d526-b14c-11ea-bb8b-bc764e2007e4 Received: from smtp.lucina.net (unknown [62.176.169.44]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id 5c04d526-b14c-11ea-bb8b-bc764e2007e4; Thu, 18 Jun 2020 10:13:31 +0000 (UTC) Received: from nodbug.lucina.net (78-141-76-187.dynamic.orange.sk [78.141.76.187]) by smtp.lucina.net (Postfix) with ESMTPSA id 6F51B122804; Thu, 18 Jun 2020 12:13:30 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=lucina.net; s=dkim-201811; t=1592475210; bh=zRQRVQN0mJzMxDkmlI6cM+DDPW8ivpPSJOLJe/aNXLU=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=ec8nG3maHRRkKIfLa+G76YcY9firEBI7k5QNVpH6ud61OIUtvEbpzFK88zN5iuPfF 7NyxcGfkjt4+diE10XfI9VoDICYjKl0/8GxnMZeugNbdM2WitrV1t80d/yh3XK/fdD 1fC0DQTrwTCc//zfXT89CIEnsS5yXXW5INYVd2kZTurmqXVWzTn5fdnnqufaRqV6PQ 8n9CDqkNWuHcGmxrB9diLJd9iIfu78kcX+mMET87dlGlrDe9dDkbIX4Q8a3DONQGGc BFa+AmhOSK2udgfMTPfWbBoHF30tV0IxW7pKys+phAZvuQf8bm9Nx3519yDxx2ma79 S4DS3Q6S/Nrtw== Received: by nodbug.lucina.net (Postfix, from userid 1000) id 588A8265E722; Thu, 18 Jun 2020 12:13:30 +0200 (CEST) Date: Thu, 18 Jun 2020 12:13:30 +0200 From: Martin Lucina To: Andrew Cooper , Roger Pau =?iso-8859-1?Q?Monn=E9?= Subject: Re: Event delivery and "domain blocking" on PVHv2 Message-ID: <20200618101330.GB10330@nodbug.lucina.net> Mail-Followup-To: Martin Lucina , Andrew Cooper , Roger Pau =?iso-8859-1?Q?Monn=E9?= , xen-devel@lists.xenproject.org, mirageos-devel@lists.xenproject.org References: <62479d08f7650c22678d7a86851eafc4@lucina.net> <5865159c-4190-e841-8020-7a4f3cf0fc24@citrix.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <5865159c-4190-e841-8020-7a4f3cf0fc24@citrix.com> User-Agent: Mutt/1.10.1 (2018-07-13) X-BeenThere: mirageos-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Developer list for MirageOS List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: xen-devel@lists.xenproject.org, mirageos-devel@lists.xenproject.org Errors-To: mirageos-devel-bounces@lists.xenproject.org Sender: "MirageOS-devel" On Monday, 15.06.2020 at 17:58, Andrew Cooper wrote: > On 15/06/2020 15:25, Martin Lucina wrote: > > Hi, > > > > puzzle time: In my continuing explorations of the PVHv2 ABIs for the > > new MirageOS Xen stack, I've run into some issues with what looks like > > missed deliveries of events on event channels. > > > > While a simple unikernel that only uses the Xen console and > > effectively does for (1..5) { printf("foo"); sleep(1); } works fine, > > once I plug in the existing OCaml Xenstore and Netfront code, the > > behaviour I see is that the unikernel hangs in random places, blocking > > as if an event that should have been delivered has been missed. > > You can see what is going on, event channel wise, with the 'e' > debug-key.  This will highlight cases such as the event channel being > masked and pending, which is a common guest bug ending up in this state. Ok, based on your and Roger's suggestions I've made some changes: 1. I've dropped all the legacy PIC initialisation code from the Solo5 parts, written some basic APIC initialisation code and switched to using HVMOP_set_evtchn_upcall_vector for upcall registration, along with setting HVM_PARAM_CALLBACK_IRQ to 1 as suggested by Roger and done by Xen when running as a guest. Commit at [1], nothing controversial there. 2. I've re-worked the "bottom half" of the event channel upcall handler to mask the event when marking it as pending in the OCaml-side "shadow" structure, and unmask it immediately before an OCaml-side handler would be run, i.e. when doing a "test and clear port" operation on the OCaml-side structure. However, none of this seems to have fundamentally changed the behaviour. The unikernel still gets "stuck" at random points during netfront set-up. Now that I've extended the grant table size, *sometimes* get as far as a fully functioning netfront and packets will flow, but things will eventually freeze up (pretty much immediately if I do something like ping -f). When the unikernel is in the "frozen" state, the domain is blocked (so we think we're waiting for events), and the "e" debug key shows the relevant event channels (Xenstore or netfront) as pending but unmasked: Example - when hung during netfront bring-up: (XEN) Event channel information for domain 27: (XEN) Polling vCPUs: {} (XEN) port [p/m/s] (XEN) 1 [1/0/1]: s=3 n=0 x=0 d=0 p=33 (XEN) 2 [1/1/1]: s=3 n=0 x=0 d=0 p=34 (XEN) 3 [1/0/1]: s=5 n=0 x=0 v=0 (1 is Xenstore, 2 is console which we don't care about, 3 is VIRQ_TIMER). When hung after hammering with "ping -f": (XEN) Event channel information for domain 28: (XEN) Polling vCPUs: {} (XEN) port [p/m/s] (XEN) 1 [0/0/1]: s=3 n=0 x=0 d=0 p=33 (XEN) 2 [1/1/1]: s=3 n=0 x=0 d=0 p=34 (XEN) 3 [1/0/1]: s=5 n=0 x=0 v=0 (XEN) 4 [1/0/1]: s=3 n=0 x=0 d=0 p=35 (as above, 4 is netfront) Some more questions about the "bottom half" of the event channel upcall, called periodically by OCaml outside of interrupt context: static bool evtchn_demux_pending(void) { struct shared_info *s = SHARED_INFO(); struct vcpu_info *vi = VCPU0_INFO(); bool some_pending = false; vi->evtchn_upcall_pending = 0; >>>> Based on Roger's suggestion, this is now only done here and no longer >>>> in the "top half" in interrupt context, which now only ACKs the vector >>>> on the APIC. /* * Demux events received from Xen. * * pending_l1 is the "outer" per-VCPU selector (evtchn_pending_sel). * pending_l2 is the "inner" system-wide word (evtchn_pending[l1i]). */ xen_ulong_t pending_l1, pending_l2; atomic_sync_xchg(&vi->evtchn_pending_sel, 0, &pending_l1); while (pending_l1 != 0) { xen_ulong_t l1i = ffs(pending_l1); pending_l1 &= ~(1UL << l1i); /* * Masking pending_l2 with ~evtchn_mask[l1i] is necessary to get the * *current* masked events; otherwise races like the following * can occur: * * 1. X is generated, upcall scheduled by Xen. * 2. X is masked. * 3. Upcall is delivered. * 4. X fires despite now being masked. */ while ((pending_l2 = (s->evtchn_pending[l1i] & ~s->evtchn_mask[l1i])) != 0) { xen_ulong_t l2i = ffs(pending_l2); pending_l2 &= ~(1UL << l2i); >>>> Do I need the above? It doesn't make a difference and seems redundant >>>> since pending_l2 is always reassigned in the loop, but Mini-OS and >>>> others are doing it...? evtchn_port_t port = (l1i * (sizeof(xen_ulong_t) * 8)) + l2i; /* * Mark as pending on the OCaml side and mask the event until * just before OCaml gets around to handling it. Also, clear * the pending bit on the Xen side. */ evtchn_callback_ml[port] = 1; atomic_sync_bts(l2i, &s->evtchn_mask[l1i]); >>>> Mask added at Roger's suggestion, not in the original Mini-OS PV-based >>>> implementation. atomic_sync_btc(l2i, &s->evtchn_pending[l1i]); some_pending = true; } } return some_pending; } The OCaml code essentially calls the above periodically, and, if it returns true, then calls into the following "test and clear" operation for all ports: static inline bool evtchn_test_and_clear(evtchn_port_t port) { assert(port < EVTCHN_2L_NR_CHANNELS); if (evtchn_callback_ml[port] > 0) { evtchn_callback_ml[port] = 0; evtchn_unmask(port); >>>> Unmask added at Roger's suggestion, not in the original Mini-OS >>>> PV-based implementation. I'm not entirely happy about this, since >>>> it'll probably result in a lot more hypercalls when the event channel >>>> is busy? return true; } else { return false; } } If this returns true, then the event handler will get run immediately after returning from here, and before any further call to evtchn_demux_pending(). At this point I don't really have a clear idea of how to progress, comparing my implementation side-by-side with the original PV Mini-OS-based implementation doesn't show up any differences I can see. AFAICT the OCaml code I've also not changed in any material way, and that has been running in production on PV for years, so I'd be inclined to think the problem is in my reimplementation of the C parts, but where...? Martin [1] https://github.com/mato/solo5/commit/42afe3a2177b9caf63d0df435391ae03a6684ac8 From mirageos-devel-bounces@lists.xenproject.org Thu Jun 18 11:46:41 2020 Return-path: Envelope-to: archives@lists.xenproject.org Delivery-date: Thu, 18 Jun 2020 11:46:41 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1jlt00-0002ds-9d; Thu, 18 Jun 2020 11:46:28 +0000 Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1jlszy-0002dn-Or for mirageos-devel@lists.xenproject.org; Thu, 18 Jun 2020 11:46:26 +0000 X-Inumbo-ID: 561f377a-b159-11ea-ba7a-12813bfff9fa Received: from esa6.hc3370-68.iphmx.com (unknown [216.71.155.175]) by us1-amaz-eas2.inumbo.com (Halon) with ESMTPS id 561f377a-b159-11ea-ba7a-12813bfff9fa; Thu, 18 Jun 2020 11:46:25 +0000 (UTC) Authentication-Results: esa6.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none IronPort-SDR: sy/Tg8xTzfK+X5o0GbCh3t2mEwwRBxT7c/ucS7IEu41qVuoXUWhCmruU4w4/GFpz7BjSniz20p tr3fEW1ZfeMdDywwHNKeFNjWANYEa6arIDzMkGzEnqcZi+FvXWCxwQVebTayKMcyXMlpO62y9m KnC3KYbUryRbJeK5mU47Q1gdA137rXHU+NaFrmOoMXNE5zgZHDcpJatK4WKo5HhjpuuglDTpMj c69T+fJs5F3+aNeBRvaGKLSiNg+SNurZotH4ZqFmegQ7wz2pMeyivC/wWCQ47187U9RbaEk6yZ lzw= X-SBRS: 2.7 X-MesageID: 20705463 X-Ironport-Server: esa6.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.73,526,1583211600"; d="scan'208";a="20705463" Date: Thu, 18 Jun 2020 13:46:17 +0200 From: Roger Pau =?utf-8?B?TW9ubsOp?= To: Martin Lucina Subject: Re: Event delivery and "domain blocking" on PVHv2 Message-ID: <20200618114617.GJ735@Air-de-Roger> References: <62479d08f7650c22678d7a86851eafc4@lucina.net> <5865159c-4190-e841-8020-7a4f3cf0fc24@citrix.com> <20200618101330.GB10330@nodbug.lucina.net> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <20200618101330.GB10330@nodbug.lucina.net> X-ClientProxiedBy: AMSPEX02CAS01.citrite.net (10.69.22.112) To AMSPEX02CL02.citrite.net (10.69.22.126) X-BeenThere: mirageos-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Developer list for MirageOS List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Andrew Cooper , mirageos-devel@lists.xenproject.org, xen-devel@lists.xenproject.org Errors-To: mirageos-devel-bounces@lists.xenproject.org Sender: "MirageOS-devel" On Thu, Jun 18, 2020 at 12:13:30PM +0200, Martin Lucina wrote: > On Monday, 15.06.2020 at 17:58, Andrew Cooper wrote: > > On 15/06/2020 15:25, Martin Lucina wrote: > > > Hi, > > > > > > puzzle time: In my continuing explorations of the PVHv2 ABIs for the > > > new MirageOS Xen stack, I've run into some issues with what looks like > > > missed deliveries of events on event channels. > > > > > > While a simple unikernel that only uses the Xen console and > > > effectively does for (1..5) { printf("foo"); sleep(1); } works fine, > > > once I plug in the existing OCaml Xenstore and Netfront code, the > > > behaviour I see is that the unikernel hangs in random places, blocking > > > as if an event that should have been delivered has been missed. > > > > You can see what is going on, event channel wise, with the 'e' > > debug-key.  This will highlight cases such as the event channel being > > masked and pending, which is a common guest bug ending up in this state. > > Ok, based on your and Roger's suggestions I've made some changes: > > 1. I've dropped all the legacy PIC initialisation code from the Solo5 > parts, written some basic APIC initialisation code and switched to using > HVMOP_set_evtchn_upcall_vector for upcall registration, along with setting > HVM_PARAM_CALLBACK_IRQ to 1 as suggested by Roger and done by Xen when > running as a guest. Commit at [1], nothing controversial there. > > 2. I've re-worked the "bottom half" of the event channel upcall handler to > mask the event when marking it as pending in the OCaml-side "shadow" > structure, and unmask it immediately before an OCaml-side handler would be > run, i.e. when doing a "test and clear port" operation on the OCaml-side > structure. As long as the unmask happens after you having cleared the bit on the ocaml-side structure I think it should be fine, however I would unmask the event channel once you have finished running the ocaml handler for it. > > However, none of this seems to have fundamentally changed the behaviour. > The unikernel still gets "stuck" at random points during netfront set-up. > Now that I've extended the grant table size, *sometimes* get as far as a > fully functioning netfront and packets will flow, but things will > eventually freeze up (pretty much immediately if I do something like ping > -f). > > When the unikernel is in the "frozen" state, the domain is blocked (so we > think we're waiting for events), and the "e" debug key shows the relevant > event channels (Xenstore or netfront) as pending but unmasked: > > Example - when hung during netfront bring-up: > > (XEN) Event channel information for domain 27: > (XEN) Polling vCPUs: {} > (XEN) port [p/m/s] > (XEN) 1 [1/0/1]: s=3 n=0 x=0 d=0 p=33 > (XEN) 2 [1/1/1]: s=3 n=0 x=0 d=0 p=34 > (XEN) 3 [1/0/1]: s=5 n=0 x=0 v=0 > > (1 is Xenstore, 2 is console which we don't care about, 3 is VIRQ_TIMER). > > When hung after hammering with "ping -f": > > (XEN) Event channel information for domain 28: > (XEN) Polling vCPUs: {} > (XEN) port [p/m/s] > (XEN) 1 [0/0/1]: s=3 n=0 x=0 d=0 p=33 > (XEN) 2 [1/1/1]: s=3 n=0 x=0 d=0 p=34 > (XEN) 3 [1/0/1]: s=5 n=0 x=0 v=0 > (XEN) 4 [1/0/1]: s=3 n=0 x=0 d=0 p=35 > > (as above, 4 is netfront) So events are pending but somehow not injected. > > Some more questions about the "bottom half" of the event channel upcall, > called periodically by OCaml outside of interrupt context: > > static bool evtchn_demux_pending(void) > { > struct shared_info *s = SHARED_INFO(); > struct vcpu_info *vi = VCPU0_INFO(); > bool some_pending = false; > > vi->evtchn_upcall_pending = 0; > >>>> Based on Roger's suggestion, this is now only done here and no longer > >>>> in the "top half" in interrupt context, which now only ACKs the vector > >>>> on the APIC. > > /* > * Demux events received from Xen. > * > * pending_l1 is the "outer" per-VCPU selector (evtchn_pending_sel). > * pending_l2 is the "inner" system-wide word (evtchn_pending[l1i]). > */ > xen_ulong_t pending_l1, pending_l2; > atomic_sync_xchg(&vi->evtchn_pending_sel, 0, &pending_l1); > while (pending_l1 != 0) { > xen_ulong_t l1i = ffs(pending_l1); > pending_l1 &= ~(1UL << l1i); > > /* > * Masking pending_l2 with ~evtchn_mask[l1i] is necessary to get the > * *current* masked events; otherwise races like the following > * can occur: > * > * 1. X is generated, upcall scheduled by Xen. > * 2. X is masked. > * 3. Upcall is delivered. > * 4. X fires despite now being masked. > */ > while ((pending_l2 = > (s->evtchn_pending[l1i] & ~s->evtchn_mask[l1i])) != 0) { > xen_ulong_t l2i = ffs(pending_l2); > pending_l2 &= ~(1UL << l2i); > >>>> Do I need the above? It doesn't make a difference and seems redundant > >>>> since pending_l2 is always reassigned in the loop, but Mini-OS and > >>>> others are doing it...? No, pending_l2 AFAICT it's only used to get the event channel to process, so there's no point in clearing it on the local variable. What you care about is clearing it in evtchn_pending[l1i]. > > evtchn_port_t port = (l1i * (sizeof(xen_ulong_t) * 8)) + l2i; > /* > * Mark as pending on the OCaml side and mask the event until > * just before OCaml gets around to handling it. Also, clear > * the pending bit on the Xen side. > */ > evtchn_callback_ml[port] = 1; > atomic_sync_bts(l2i, &s->evtchn_mask[l1i]); > >>>> Mask added at Roger's suggestion, not in the original Mini-OS PV-based > >>>> implementation. > atomic_sync_btc(l2i, &s->evtchn_pending[l1i]); > some_pending = true; > } > } > return some_pending; > } > > The OCaml code essentially calls the above periodically, and, if > it returns true, then calls into the following "test and clear" operation > for all ports: > > static inline bool evtchn_test_and_clear(evtchn_port_t port) > { > assert(port < EVTCHN_2L_NR_CHANNELS); > if (evtchn_callback_ml[port] > 0) { > evtchn_callback_ml[port] = 0; > evtchn_unmask(port); > >>>> Unmask added at Roger's suggestion, not in the original Mini-OS > >>>> PV-based implementation. I'm not entirely happy about this, since > >>>> it'll probably result in a lot more hypercalls when the event channel > >>>> is busy? OTOH you will likely continue to get interrupts from it if you don't mask it, so you might receive several interrupts (because you clear the pending bit) without having executed the handler. IMO it would be better to do the unmask after the handler has run. > return true; > } > else { > return false; > } > } > > If this returns true, then the event handler will get run immediately after > returning from here, and before any further call to evtchn_demux_pending(). > > At this point I don't really have a clear idea of how to progress, > comparing my implementation side-by-side with the original PV Mini-OS-based > implementation doesn't show up any differences I can see. > > AFAICT the OCaml code I've also not changed in any material way, and that > has been running in production on PV for years, so I'd be inclined to think > the problem is in my reimplementation of the C parts, but where...? A good start would be to print the ISR and IRR lapic registers when blocked, to assert there are no pending vectors there. Can you apply the following patch to your Xen, rebuild and check the output of the 'l' debug key? Also add the output of the 'v' key. Roger. ---8<--- diff --git a/xen/arch/x86/hvm/vlapic.c b/xen/arch/x86/hvm/vlapic.c index 7b5c633033..45b195cc05 100644 --- a/xen/arch/x86/hvm/vlapic.c +++ b/xen/arch/x86/hvm/vlapic.c @@ -23,6 +23,7 @@ #include #include #include +#include #include #include #include @@ -1662,6 +1663,34 @@ void vlapic_destroy(struct vcpu *v) free_domheap_page(vlapic->regs_page); } +static void print_lapic(unsigned char key) +{ + const struct domain *d; + + for_each_domain ( d ) + { + const struct vcpu *v; + + if ( !has_vlapic(d) ) + continue; + + for_each_vcpu ( d, v ) + { + const struct vlapic *vlapic = vcpu_vlapic(v); + + printk("%pv IRR: %*pb\n", v, 256, &vlapic->regs->data[APIC_IRR]); + printk("%pv ISR: %*pb\n", v, 256, &vlapic->regs->data[APIC_ISR]); + } + } +} + +static int __init print_lapic_init(void) +{ + register_keyhandler('l', print_lapic, "dump local APIC info", 1); + return 0; +} +__initcall(print_lapic_init); + /* * Local variables: * mode: C From mirageos-devel-bounces@lists.xenproject.org Thu Jun 18 23:44:15 2020 Return-path: Envelope-to: archives@lists.xenproject.org Delivery-date: Thu, 18 Jun 2020 23:44:15 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1jm4CP-0001ui-Fd; Thu, 18 Jun 2020 23:44:01 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1jm4CO-0001uX-Ni for mirageos-devel@lists.xenproject.org; Thu, 18 Jun 2020 23:44:00 +0000 X-Inumbo-ID: 94ae3310-b1bd-11ea-bb8b-bc764e2007e4 Received: from esa5.hc3370-68.iphmx.com (unknown [216.71.155.168]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id 94ae3310-b1bd-11ea-bb8b-bc764e2007e4; Thu, 18 Jun 2020 23:43:59 +0000 (UTC) Authentication-Results: esa5.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none IronPort-SDR: uFy2yF6A3Sz1dmKVsQj8ZhJG1SABuqaEaoPwHhB8Y6/iNzyUA/U/xV0ZpN9d06cQS2bdfVIWUl bJauUP46LSy9HaJd9th/lLF/rusg4AEYsYC3H3U3qZSnhLmyUsXEBzhX60YNVoA7JT9g+2tbAY aXLXo3bBlfCVkM2VNolaXd54+MZl/cAY7DOx+9NcGqeV/paOHuf3GYs+ehwff2bdptf4xVmY3U Ni3Mx7ShySsIofO22N9sHbMdVgGtEUW1RpTfVDBJf0ZkQW48nJALoxJRXZFaqQFu/5mc+nlXXH geE= X-SBRS: 2.7 X-MesageID: 20650521 X-Ironport-Server: esa5.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.75,253,1589256000"; d="scan'208";a="20650521" Subject: Re: Event delivery and "domain blocking" on PVHv2 To: Martin Lucina , =?UTF-8?Q?Roger_Pau_Monn=c3=a9?= , , References: <62479d08f7650c22678d7a86851eafc4@lucina.net> <5865159c-4190-e841-8020-7a4f3cf0fc24@citrix.com> <20200618101330.GB10330@nodbug.lucina.net> From: Andrew Cooper Message-ID: <2f5c8fbc-0153-17b7-4a44-8f8ba0e3179f@citrix.com> Date: Fri, 19 Jun 2020 00:43:55 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.8.0 MIME-Version: 1.0 In-Reply-To: <20200618101330.GB10330@nodbug.lucina.net> Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 8bit Content-Language: en-GB X-ClientProxiedBy: AMSPEX02CAS02.citrite.net (10.69.22.113) To AMSPEX02CL02.citrite.net (10.69.22.126) X-BeenThere: mirageos-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Developer list for MirageOS List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: mirageos-devel-bounces@lists.xenproject.org Sender: "MirageOS-devel" On 18/06/2020 11:13, Martin Lucina wrote: > On Monday, 15.06.2020 at 17:58, Andrew Cooper wrote: >> On 15/06/2020 15:25, Martin Lucina wrote: >>> Hi, >>> >>> puzzle time: In my continuing explorations of the PVHv2 ABIs for the >>> new MirageOS Xen stack, I've run into some issues with what looks like >>> missed deliveries of events on event channels. >>> >>> While a simple unikernel that only uses the Xen console and >>> effectively does for (1..5) { printf("foo"); sleep(1); } works fine, >>> once I plug in the existing OCaml Xenstore and Netfront code, the >>> behaviour I see is that the unikernel hangs in random places, blocking >>> as if an event that should have been delivered has been missed. >> You can see what is going on, event channel wise, with the 'e' >> debug-key.  This will highlight cases such as the event channel being >> masked and pending, which is a common guest bug ending up in this state. > Ok, based on your and Roger's suggestions I've made some changes: > > 1. I've dropped all the legacy PIC initialisation code from the Solo5 > parts, written some basic APIC initialisation code and switched to using > HVMOP_set_evtchn_upcall_vector for upcall registration, along with setting > HVM_PARAM_CALLBACK_IRQ to 1 as suggested by Roger and done by Xen when > running as a guest. Commit at [1], nothing controversial there. Well...     uint64_t apic_base = rdmsrq(MSR_IA32_APIC_BASE);     wrmsrq(MSR_IA32_APIC_BASE,             apic_base | (APIC_BASE << 4) | MSR_IA32_APIC_BASE_ENABLE);     apic_base = rdmsrq(MSR_IA32_APIC_BASE);     if (!(apic_base & MSR_IA32_APIC_BASE_ENABLE)) {         log(ERROR, "Solo5: Could not enable APIC or not present\n");         assert(false);     } The only reason Xen doesn't crash your guest on that WRMSR is because 0xfee00080ull | (0xfee00000u << 4) == 0xfee00080ull, due to truncation and 0xfe | 0xee == 0xfe. Either way, the logic isn't correct. Xen doesn't support moving the APIC MMIO window (and almost certainly never will, because the only thing which changes it is malware).  You can rely on the default state being correct, because it is architecturally specified. ~Andrew From mirageos-devel-bounces@lists.xenproject.org Fri Jun 19 08:41:27 2020 Return-path: Envelope-to: archives@lists.xenproject.org Delivery-date: Fri, 19 Jun 2020 08:41:27 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1jmCaF-0006rS-7A; Fri, 19 Jun 2020 08:41:11 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1jmCaD-0006rB-Uq for mirageos-devel@lists.xenproject.org; Fri, 19 Jun 2020 08:41:09 +0000 X-Inumbo-ID: 9ed8c7d8-b208-11ea-b7bb-bc764e2007e4 Received: from smtp.lucina.net (unknown [62.176.169.44]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id 9ed8c7d8-b208-11ea-b7bb-bc764e2007e4; Fri, 19 Jun 2020 08:41:08 +0000 (UTC) Received: from webmail.moloch.sk (w3-s.a.lucina.net [62.176.169.73]) by smtp.lucina.net (Postfix) with ESMTPSA id BDF71122804; Fri, 19 Jun 2020 10:41:07 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=lucina.net; s=dkim-201811; t=1592556067; bh=asZAW8ytMFfTsgxokf5mQeT7kTk0KwF+C+X+TX76kis=; h=Date:From:To:Cc:Subject:In-Reply-To:References:From; b=gBdOW0isyKQbxhes8TGpBinbxv+ekQpFt73NxNCMIZPWZ9Ie3ZjdNKotH+UEW1Z2A rz58VGU6FO00Lew2vgwzvRq9WO7ji5SmxHzSCxWdUdmLkTWWNX8FtVeyb9jq/RUU5N uV4qP+5LM68PlEWHg17BnLJ2c/iH9nhQoPXVbxQZ5dtta+2dFdcjzOU4iBS3Gpgp/0 haDWVqD6a07/Ti4brGxWiK+9PDb1MILdv1URQj2u6eGivUX2DPj3NkvP6upOpV+j/v 9bA9gW6rODd8c8rnJC6cd5P4QsHMamUO7GJMp4WeyhU6yCMN2oZ66c6YRlee6wBHk9 K8PwGr+DrBf+w== MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Date: Fri, 19 Jun 2020 10:41:07 +0200 From: Martin Lucina To: Andrew Cooper Subject: Re: Event delivery and "domain blocking" on PVHv2 In-Reply-To: <2f5c8fbc-0153-17b7-4a44-8f8ba0e3179f@citrix.com> References: <62479d08f7650c22678d7a86851eafc4@lucina.net> <5865159c-4190-e841-8020-7a4f3cf0fc24@citrix.com> <20200618101330.GB10330@nodbug.lucina.net> <2f5c8fbc-0153-17b7-4a44-8f8ba0e3179f@citrix.com> Message-ID: X-Sender: martin@lucina.net User-Agent: Roundcube Webmail/1.3.3 X-BeenThere: mirageos-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Developer list for MirageOS List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: xen-devel@lists.xenproject.org, mirageos-devel@lists.xenproject.org, =?UTF-8?Q?Roger_Pau_Monn=C3=A9?= Errors-To: mirageos-devel-bounces@lists.xenproject.org Sender: "MirageOS-devel" On 2020-06-19 01:43, Andrew Cooper wrote: > On 18/06/2020 11:13, Martin Lucina wrote: >> On Monday, 15.06.2020 at 17:58, Andrew Cooper wrote: >>> On 15/06/2020 15:25, Martin Lucina wrote: >>>> Hi, >>>> >>>> puzzle time: In my continuing explorations of the PVHv2 ABIs for the >>>> new MirageOS Xen stack, I've run into some issues with what looks >>>> like >>>> missed deliveries of events on event channels. >>>> >>>> While a simple unikernel that only uses the Xen console and >>>> effectively does for (1..5) { printf("foo"); sleep(1); } works fine, >>>> once I plug in the existing OCaml Xenstore and Netfront code, the >>>> behaviour I see is that the unikernel hangs in random places, >>>> blocking >>>> as if an event that should have been delivered has been missed. >>> You can see what is going on, event channel wise, with the 'e' >>> debug-key.  This will highlight cases such as the event channel being >>> masked and pending, which is a common guest bug ending up in this >>> state. >> Ok, based on your and Roger's suggestions I've made some changes: >> >> 1. I've dropped all the legacy PIC initialisation code from the Solo5 >> parts, written some basic APIC initialisation code and switched to >> using >> HVMOP_set_evtchn_upcall_vector for upcall registration, along with >> setting >> HVM_PARAM_CALLBACK_IRQ to 1 as suggested by Roger and done by Xen when >> running as a guest. Commit at [1], nothing controversial there. > > Well... > >     uint64_t apic_base = rdmsrq(MSR_IA32_APIC_BASE); >     wrmsrq(MSR_IA32_APIC_BASE, >             apic_base | (APIC_BASE << 4) | MSR_IA32_APIC_BASE_ENABLE); >     apic_base = rdmsrq(MSR_IA32_APIC_BASE); >     if (!(apic_base & MSR_IA32_APIC_BASE_ENABLE)) { >         log(ERROR, "Solo5: Could not enable APIC or not present\n"); >         assert(false); >     } > > The only reason Xen doesn't crash your guest on that WRMSR is because > 0xfee00080ull | (0xfee00000u << 4) == 0xfee00080ull, due to truncation > and 0xfe | 0xee == 0xfe. > > Either way, the logic isn't correct. Oh, thanks. Don't you wish C had a "strict" mode where you could disable/warn on implicit type promotion? I certainly do. > > Xen doesn't support moving the APIC MMIO window (and almost certainly > never will, because the only thing which changes it is malware).  You > can rely on the default state being correct, because it is > architecturally specified. Noted. I'll change the code to just verify that APIC_BASE is indeed FEE00000 at start of day and that the enable operation succeeded -- I like to keep the code robust, e.g. against cut-n-pasting to somewhere else that might be used in a non-Xen context later where the precondition may not hold. Martin > > ~Andrew From mirageos-devel-bounces@lists.xenproject.org Fri Jun 19 10:29:05 2020 Return-path: Envelope-to: archives@lists.xenproject.org Delivery-date: Fri, 19 Jun 2020 10:29:05 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1jmEGZ-0008UH-HK; Fri, 19 Jun 2020 10:28:59 +0000 Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1jmEGY-0008Tf-Od for mirageos-devel@lists.xenproject.org; Fri, 19 Jun 2020 10:28:58 +0000 X-Inumbo-ID: ab136fc6-b217-11ea-bb5e-12813bfff9fa Received: from smtp.lucina.net (unknown [62.176.169.44]) by us1-amaz-eas2.inumbo.com (Halon) with ESMTPS id ab136fc6-b217-11ea-bb5e-12813bfff9fa; Fri, 19 Jun 2020 10:28:51 +0000 (UTC) Received: from webmail.moloch.sk (w3-s.a.lucina.net [62.176.169.73]) by smtp.lucina.net (Postfix) with ESMTPSA id DDE30122804; Fri, 19 Jun 2020 12:28:50 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=lucina.net; s=dkim-201811; t=1592562530; bh=n/UcbhmynjYkE8I4QJZoH1EVb+BPwxBhePbHkQIhJ9U=; h=Date:From:To:Cc:Subject:In-Reply-To:References:From; b=GyaCk2uHNYTD3+9cf+WBXPBbTmyvFMy8yrwKd0q9pDQaOQiZ96NCAgfmvQ3q1KRQw KiSkwWeSj6FLwui58hlSi7Xmy61cHuRxiJC5FHP5KHmq7+EnfQfDZXd3/Dw1JBm9uv SUeZgqDPdLp2G+3wdvk2G/dx2dJCBguPZHPwdK2Smq1vImhLfafaaqNdTHnBN+LW49 Ej5Htdagf/MUwDFWDMweTQIFOrk6CTMxsM7YRhOk7YIlJC/3p/liwBt1cqLOBZeeoR cdLvP62mds4GcvfGpaAeU/kZjnk/t/O91WLhvq61fiySBUpNdZFud0RaU9LaeVpVJg q5mvmn3LUUn9w== MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Date: Fri, 19 Jun 2020 12:28:50 +0200 From: Martin Lucina To: =?UTF-8?Q?Roger_Pau_Monn=C3=A9?= Subject: Re: Event delivery and "domain blocking" on PVHv2 In-Reply-To: <20200618114617.GJ735@Air-de-Roger> References: <62479d08f7650c22678d7a86851eafc4@lucina.net> <5865159c-4190-e841-8020-7a4f3cf0fc24@citrix.com> <20200618101330.GB10330@nodbug.lucina.net> <20200618114617.GJ735@Air-de-Roger> Message-ID: <17deb17cec442f96cc7aba98ef4c047c@lucina.net> X-Sender: martin@lucina.net User-Agent: Roundcube Webmail/1.3.3 X-BeenThere: mirageos-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Developer list for MirageOS List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Andrew Cooper , mirageos-devel@lists.xenproject.org, xen-devel@lists.xenproject.org Errors-To: mirageos-devel-bounces@lists.xenproject.org Sender: "MirageOS-devel" On 2020-06-18 13:46, Roger Pau MonnĂ© wrote: > On Thu, Jun 18, 2020 at 12:13:30PM +0200, Martin Lucina wrote: >> At this point I don't really have a clear idea of how to progress, >> comparing my implementation side-by-side with the original PV >> Mini-OS-based >> implementation doesn't show up any differences I can see. >> >> AFAICT the OCaml code I've also not changed in any material way, and >> that >> has been running in production on PV for years, so I'd be inclined to >> think >> the problem is in my reimplementation of the C parts, but where...? > > A good start would be to print the ISR and IRR lapic registers when > blocked, to assert there are no pending vectors there. > > Can you apply the following patch to your Xen, rebuild and check the > output of the 'l' debug key? > > Also add the output of the 'v' key. Had to fight the Xen Debian packages a bit as I wanted to patch the exact same Xen (there are some failures when building on a system that has Xen installed due to following symlinks when fixing shebangs). Here you go, when stuck during netfront setup, after allocating its event channel, presumably waiting on Xenstore: 'e': (XEN) Event channel information for domain 3: (XEN) Polling vCPUs: {} (XEN) port [p/m/s] (XEN) 1 [1/0/1]: s=3 n=0 x=0 d=0 p=33 (XEN) 2 [1/1/1]: s=3 n=0 x=0 d=0 p=34 (XEN) 3 [1/0/1]: s=5 n=0 x=0 v=0 (XEN) 4 [0/1/1]: s=2 n=0 x=0 d=0 'l': (XEN) d3v0 IRR: ffff8301732dc200b (XEN) d3v0 ISR: ffff8301732dc100b 'v': (XEN) *********** VMCS Areas ************** (XEN) (XEN) >>> Domain 3 <<< (XEN) VCPU 0 (XEN) *** Guest State *** (XEN) CR0: actual=0x0000000080000033, shadow=0x0000000080000033, gh_mask=ffffffffffffffff (XEN) CR4: actual=0x0000000000002660, shadow=0x0000000000000020, gh_mask=ffffffffffc8f860 (XEN) CR3 = 0x00000000002e9000 (XEN) RSP = 0x000000000ffffec0 (0x000000000ffffec0) RIP = 0x0000000000209997 (0x0000000000209998) (XEN) RFLAGS=0x00000297 (0x00000297) DR7 = 0x0000000000000400 (XEN) Sysenter RSP=0000000000000000 CS:RIP=0000:0000000000000000 (XEN) sel attr limit base (XEN) CS: 0008 0a099 ffffffff 0000000000000000 (XEN) DS: 0010 0c093 ffffffff 0000000000000000 (XEN) SS: 0010 0c093 ffffffff 0000000000000000 (XEN) ES: 0010 0c093 ffffffff 0000000000000000 (XEN) FS: 0000 1c000 ffffffff 0000000000000000 (XEN) GS: 0000 1c000 ffffffff 0000000000000000 (XEN) GDTR: 00000027 00000000003e13c0 (XEN) LDTR: 0000 10000 00000000 0000000000000000 (XEN) IDTR: 000002ff 00000000003e10a8 (XEN) TR: 0018 0008b 00000068 00000000003e1040 (XEN) EFER = 0x0000000000000000 PAT = 0x0007040600070406 (XEN) PreemptionTimer = 0x00000000 SM Base = 0x00000000 (XEN) DebugCtl = 0x0000000000000000 DebugExceptions = 0x0000000000000000 (XEN) Interruptibility = 00000000 ActivityState = 00000000 (XEN) *** Host State *** (XEN) RIP = 0xffff82d08031dd30 (vmx_asm_vmexit_handler) RSP = 0xffff83009c057f70 (XEN) CS=e008 SS=0000 DS=0000 ES=0000 FS=0000 GS=0000 TR=e040 (XEN) FSBase=0000000000000000 GSBase=0000000000000000 TRBase=ffff82d08055e000 (XEN) GDTBase=ffff82d08042e000 IDTBase=ffff82d080559000 (XEN) CR0=0000000080050033 CR3=0000000172a67000 CR4=00000000003526e0 (XEN) Sysenter RSP=ffff83009c057fa0 CS:RIP=e008:ffff82d0803654b0 (XEN) EFER = 0x0000000000000000 PAT = 0x0000050100070406 (XEN) *** Control State *** (XEN) PinBased=0000003f CPUBased=b6a065fa SecondaryExec=000014eb (XEN) EntryControls=000053ff ExitControls=000fefff (XEN) ExceptionBitmap=00060002 PFECmask=00000000 PFECmatch=00000000 (XEN) VMEntry: intr_info=00000020 errcode=00000000 ilen=00000000 (XEN) VMExit: intr_info=00000000 errcode=00000000 ilen=00000001 (XEN) reason=0000000c qualification=0000000000000000 (XEN) IDTVectoring: info=00000000 errcode=00000000 (XEN) TSC Offset = 0xffffffcf8453199b TSC Multiplier = 0x0000000000000000 (XEN) TPR Threshold = 0x00 PostedIntrVec = 0x00 (XEN) EPT pointer = 0x0000000172a0b01e EPTP index = 0x0000 (XEN) PLE Gap=00000080 Window=00001000 (XEN) Virtual processor ID = 0x000e VMfunc controls = 0000000000000000 (XEN) ************************************** RIP 0x209997 is the 'hlt' instruction in mirage_xen_evtchn_block_domain() so we are indeed blocking waiting for events to show up. Martin From mirageos-devel-bounces@lists.xenproject.org Fri Jun 19 11:21:41 2020 Return-path: Envelope-to: archives@lists.xenproject.org Delivery-date: Fri, 19 Jun 2020 11:21:41 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1jmF5U-0005SW-Ef; Fri, 19 Jun 2020 11:21:36 +0000 Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1jmF5T-0005SA-Cb for mirageos-devel@lists.xenproject.org; Fri, 19 Jun 2020 11:21:35 +0000 X-Inumbo-ID: 0507e2f8-b21f-11ea-bb68-12813bfff9fa Received: from esa6.hc3370-68.iphmx.com (unknown [216.71.155.175]) by us1-amaz-eas2.inumbo.com (Halon) with ESMTPS id 0507e2f8-b21f-11ea-bb68-12813bfff9fa; Fri, 19 Jun 2020 11:21:29 +0000 (UTC) Authentication-Results: esa6.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none IronPort-SDR: 0wOIVdkT3G2h9ju7T5EzUsx6SBGvXUlgnK5eFMP+qBhY5WNfDQ/REVj9kYLvuRG/sdPXzUx31n RbhZmvqLcCnMXoLAWB5ZuDPC9EdvrtZhoDu/GErkMf4tITCnpa2zC1oFEypJM9IgktVXdUk+aR 5WfoKQuVIFIPvVFcE/pSRVA2HUO57TlDBrX0N7TQr1Wy9TAmSc+w5nlsbDr7QneoQXMjn4b1Ty Dm7au3d1WHSfCPmJpub+MIjbw7nU4cmg0+Xx/72eDnFH5YdFaWZMJyppxb0NuABRT1YY32uLZ1 Re0= X-SBRS: 2.7 X-MesageID: 20806862 X-Ironport-Server: esa6.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.75,255,1589256000"; d="scan'208";a="20806862" Date: Fri, 19 Jun 2020 13:21:19 +0200 From: Roger Pau =?utf-8?B?TW9ubsOp?= To: Martin Lucina Subject: Re: Event delivery and "domain blocking" on PVHv2 Message-ID: <20200619112119.GY735@Air-de-Roger> References: <62479d08f7650c22678d7a86851eafc4@lucina.net> <5865159c-4190-e841-8020-7a4f3cf0fc24@citrix.com> <20200618101330.GB10330@nodbug.lucina.net> <20200618114617.GJ735@Air-de-Roger> <17deb17cec442f96cc7aba98ef4c047c@lucina.net> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <17deb17cec442f96cc7aba98ef4c047c@lucina.net> X-ClientProxiedBy: AMSPEX02CAS02.citrite.net (10.69.22.113) To AMSPEX02CL02.citrite.net (10.69.22.126) X-BeenThere: mirageos-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Developer list for MirageOS List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Andrew Cooper , mirageos-devel@lists.xenproject.org, xen-devel@lists.xenproject.org Errors-To: mirageos-devel-bounces@lists.xenproject.org Sender: "MirageOS-devel" On Fri, Jun 19, 2020 at 12:28:50PM +0200, Martin Lucina wrote: > On 2020-06-18 13:46, Roger Pau MonnĂ© wrote: > > On Thu, Jun 18, 2020 at 12:13:30PM +0200, Martin Lucina wrote: > > > At this point I don't really have a clear idea of how to progress, > > > comparing my implementation side-by-side with the original PV > > > Mini-OS-based > > > implementation doesn't show up any differences I can see. > > > > > > AFAICT the OCaml code I've also not changed in any material way, and > > > that > > > has been running in production on PV for years, so I'd be inclined > > > to think > > > the problem is in my reimplementation of the C parts, but where...? > > > > A good start would be to print the ISR and IRR lapic registers when > > blocked, to assert there are no pending vectors there. > > > > Can you apply the following patch to your Xen, rebuild and check the > > output of the 'l' debug key? > > > > Also add the output of the 'v' key. > > Had to fight the Xen Debian packages a bit as I wanted to patch the exact > same Xen (there are some failures when building on a system that has Xen > installed due to following symlinks when fixing shebangs). > > Here you go, when stuck during netfront setup, after allocating its event > channel, presumably waiting on Xenstore: > > 'e': > > (XEN) Event channel information for domain 3: > (XEN) Polling vCPUs: {} > (XEN) port [p/m/s] > (XEN) 1 [1/0/1]: s=3 n=0 x=0 d=0 p=33 > (XEN) 2 [1/1/1]: s=3 n=0 x=0 d=0 p=34 > (XEN) 3 [1/0/1]: s=5 n=0 x=0 v=0 > (XEN) 4 [0/1/1]: s=2 n=0 x=0 d=0 > > 'l': > > (XEN) d3v0 IRR: > ffff8301732dc200b > (XEN) d3v0 ISR: > ffff8301732dc100b Which version of Xen is this? AFAICT it doesn't have the support to print a bitmap. Do you think you could also pick commit 8cd9500958d818e3deabdd0d4164ea6fe1623d7c [0] and rebuild? (and print the info again). Thanks, Roger. [0] http://xenbits.xen.org/gitweb/?p=xen.git;a=patch;h=8cd9500958d818e3deabdd0d4164ea6fe1623d7c From mirageos-devel-bounces@lists.xenproject.org Fri Jun 19 11:21:54 2020 Return-path: Envelope-to: archives@lists.xenproject.org Delivery-date: Fri, 19 Jun 2020 11:21:54 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1jmF5m-0005Um-Hv; Fri, 19 Jun 2020 11:21:54 +0000 Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1jmF5m-0005Ud-7h for mirageos-devel@lists.xenproject.org; Fri, 19 Jun 2020 11:21:54 +0000 X-Inumbo-ID: 12e54d3f-b21f-11ea-bb68-12813bfff9fa Received: from esa6.hc3370-68.iphmx.com (unknown [216.71.155.175]) by us1-amaz-eas2.inumbo.com (Halon) with ESMTPS id 12e54d3f-b21f-11ea-bb68-12813bfff9fa; Fri, 19 Jun 2020 11:21:53 +0000 (UTC) Authentication-Results: esa6.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none IronPort-SDR: m/UIk+IX4OhBIYj0b83yvLEjPBBNyDvazVVdFlb1mUvnvNLYLj5hciS1LySpTayJnnxhEfJYAK lAumgPgX9GEfhTqBMr67BWay63HtuSzqaUIUp+XFu8mI3eStJxuwiPtv02Ri6OJOeYGvV1+uqd HMNZHPbFo5bTeegevcF/Tw9TNrsRtQgVEZO9qefDDB3fC/CscQVJnl/w17ptzdFMACZH6zC6wt ZRfyvthrtzHQ/JJRYYHziUezpaybO8W7Hbg0q8fGNvPGFXP2iklufgGYHQ70hobc5VQA9M/bdd Nv8= X-SBRS: 2.7 X-MesageID: 20806877 X-Ironport-Server: esa6.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.75,255,1589256000"; d="scan'208";a="20806877" Subject: Re: Event delivery and "domain blocking" on PVHv2 To: Martin Lucina , =?UTF-8?Q?Roger_Pau_Monn=c3=a9?= References: <62479d08f7650c22678d7a86851eafc4@lucina.net> <5865159c-4190-e841-8020-7a4f3cf0fc24@citrix.com> <20200618101330.GB10330@nodbug.lucina.net> <20200618114617.GJ735@Air-de-Roger> <17deb17cec442f96cc7aba98ef4c047c@lucina.net> From: Andrew Cooper Message-ID: <0ebf1417-49e5-d9b2-81b0-b02c7e6cf833@citrix.com> Date: Fri, 19 Jun 2020 12:21:48 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.8.0 MIME-Version: 1.0 In-Reply-To: <17deb17cec442f96cc7aba98ef4c047c@lucina.net> Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 8bit Content-Language: en-GB X-ClientProxiedBy: AMSPEX02CAS01.citrite.net (10.69.22.112) To AMSPEX02CL02.citrite.net (10.69.22.126) X-BeenThere: mirageos-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Developer list for MirageOS List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: xen-devel@lists.xenproject.org, mirageos-devel@lists.xenproject.org Errors-To: mirageos-devel-bounces@lists.xenproject.org Sender: "MirageOS-devel" On 19/06/2020 11:28, Martin Lucina wrote: > On 2020-06-18 13:46, Roger Pau MonnĂ© wrote: >> On Thu, Jun 18, 2020 at 12:13:30PM +0200, Martin Lucina wrote: >>> At this point I don't really have a clear idea of how to progress, >>> comparing my implementation side-by-side with the original PV >>> Mini-OS-based >>> implementation doesn't show up any differences I can see. >>> >>> AFAICT the OCaml code I've also not changed in any material way, and >>> that >>> has been running in production on PV for years, so I'd be inclined >>> to think >>> the problem is in my reimplementation of the C parts, but where...? >> >> A good start would be to print the ISR and IRR lapic registers when >> blocked, to assert there are no pending vectors there. >> >> Can you apply the following patch to your Xen, rebuild and check the >> output of the 'l' debug key? >> >> Also add the output of the 'v' key. > > Had to fight the Xen Debian packages a bit as I wanted to patch the > exact same Xen (there are some failures when building on a system that > has Xen installed due to following symlinks when fixing shebangs). > > Here you go, when stuck during netfront setup, after allocating its > event channel, presumably waiting on Xenstore: > > 'e': > > (XEN) Event channel information for domain 3: > (XEN) Polling vCPUs: {} > (XEN)     port [p/m/s] > (XEN)        1 [1/0/1]: s=3 n=0 x=0 d=0 p=33 > (XEN)        2 [1/1/1]: s=3 n=0 x=0 d=0 p=34 > (XEN)        3 [1/0/1]: s=5 n=0 x=0 v=0 > (XEN)        4 [0/1/1]: s=2 n=0 x=0 d=0 > > 'l': > > (XEN) d3v0 > IRR:                                                          >                                                                          >                                                                          >                                      ffff8301732dc200b > (XEN) d3v0 > ISR:                                                          >                                                                          >                                                                          >                                      ffff8301732dc100b > > 'v': > > (XEN) *********** VMCS Areas ************** > (XEN) > (XEN) >>> Domain 3 <<< > (XEN)   VCPU 0 > (XEN) *** Guest State *** > (XEN) CR0: actual=0x0000000080000033, shadow=0x0000000080000033, > gh_mask=ffffffffffffffff > (XEN) CR4: actual=0x0000000000002660, shadow=0x0000000000000020, > gh_mask=ffffffffffc8f860 > (XEN) CR3 = 0x00000000002e9000 > (XEN) RSP = 0x000000000ffffec0 (0x000000000ffffec0)  RIP = > 0x0000000000209997 (0x0000000000209998) > (XEN) RFLAGS=0x00000297 (0x00000297)  DR7 = 0x0000000000000400 > (XEN) Sysenter RSP=0000000000000000 CS:RIP=0000:0000000000000000 > (XEN)        sel  attr  limit   base > (XEN)   CS: 0008 0a099 ffffffff 0000000000000000 > (XEN)   DS: 0010 0c093 ffffffff 0000000000000000 > (XEN)   SS: 0010 0c093 ffffffff 0000000000000000 > (XEN)   ES: 0010 0c093 ffffffff 0000000000000000 > (XEN)   FS: 0000 1c000 ffffffff 0000000000000000 > (XEN)   GS: 0000 1c000 ffffffff 0000000000000000 > (XEN) GDTR:            00000027 00000000003e13c0 > (XEN) LDTR: 0000 10000 00000000 0000000000000000 > (XEN) IDTR:            000002ff 00000000003e10a8 > (XEN)   TR: 0018 0008b 00000068 00000000003e1040 > (XEN) EFER = 0x0000000000000000  PAT = 0x0007040600070406 > (XEN) PreemptionTimer = 0x00000000  SM Base = 0x00000000 > (XEN) DebugCtl = 0x0000000000000000  DebugExceptions = 0x0000000000000000 > (XEN) Interruptibility = 00000000  ActivityState = 00000000 > (XEN) *** Host State *** > (XEN) RIP = 0xffff82d08031dd30 (vmx_asm_vmexit_handler)  RSP = > 0xffff83009c057f70 > (XEN) CS=e008 SS=0000 DS=0000 ES=0000 FS=0000 GS=0000 TR=e040 > (XEN) FSBase=0000000000000000 GSBase=0000000000000000 > TRBase=ffff82d08055e000 > (XEN) GDTBase=ffff82d08042e000 IDTBase=ffff82d080559000 > (XEN) CR0=0000000080050033 CR3=0000000172a67000 CR4=00000000003526e0 > (XEN) Sysenter RSP=ffff83009c057fa0 CS:RIP=e008:ffff82d0803654b0 > (XEN) EFER = 0x0000000000000000  PAT = 0x0000050100070406 > (XEN) *** Control State *** > (XEN) PinBased=0000003f CPUBased=b6a065fa SecondaryExec=000014eb > (XEN) EntryControls=000053ff ExitControls=000fefff > (XEN) ExceptionBitmap=00060002 PFECmask=00000000 PFECmatch=00000000 > (XEN) VMEntry: intr_info=00000020 errcode=00000000 ilen=00000000 > (XEN) VMExit: intr_info=00000000 errcode=00000000 ilen=00000001 > (XEN)         reason=0000000c qualification=0000000000000000 > (XEN) IDTVectoring: info=00000000 errcode=00000000 > (XEN) TSC Offset = 0xffffffcf8453199b  TSC Multiplier = > 0x0000000000000000 > (XEN) TPR Threshold = 0x00  PostedIntrVec = 0x00 > (XEN) EPT pointer = 0x0000000172a0b01e  EPTP index = 0x0000 > (XEN) PLE Gap=00000080 Window=00001000 > (XEN) Virtual processor ID = 0x000e VMfunc controls = 0000000000000000 > (XEN) ************************************** > > RIP 0x209997 is the 'hlt' instruction in > mirage_xen_evtchn_block_domain() so we are indeed blocking waiting for > events to show up. I can't find this in the code, but it might be an x86-ism you're falling over here. Its not safe to use hlt with interrupts enabled, unless it is exactly `sti; hlt` where the STI instruction transitions the interrupt flag from clear to set (i.e. you had interrupts disabled beforehand). Otherwise you can take the interrupt intended to wake you on the before the hlt is executed. ~Andrew From mirageos-devel-bounces@lists.xenproject.org Fri Jun 19 16:41:39 2020 Return-path: Envelope-to: archives@lists.xenproject.org Delivery-date: Fri, 19 Jun 2020 16:41:39 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1jmK53-00029d-H2; Fri, 19 Jun 2020 16:41:29 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1jmK53-00029A-4l for mirageos-devel@lists.xenproject.org; Fri, 19 Jun 2020 16:41:29 +0000 X-Inumbo-ID: b51ab1b2-b24b-11ea-b7bb-bc764e2007e4 Received: from smtp.lucina.net (unknown [62.176.169.44]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id b51ab1b2-b24b-11ea-b7bb-bc764e2007e4; Fri, 19 Jun 2020 16:41:22 +0000 (UTC) Received: from webmail.moloch.sk (w3-s.a.lucina.net [62.176.169.73]) by smtp.lucina.net (Postfix) with ESMTPSA id 6F49D122804; Fri, 19 Jun 2020 18:41:21 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=lucina.net; s=dkim-201811; t=1592584881; bh=ZBT3d1lzWFughmSUjt+Pfhm1LcKzzY40X07feeXVQYQ=; h=Date:From:To:Cc:Subject:In-Reply-To:References:From; b=YEFKAPIeZQrrKhxVKabTImMkOs2UrMzLrhNwHG+5MC1yJVRAJW7J7ejxu8g5G0PiX KBjUryeBhCWb5N/npRtW88r77gARJeFsdwcXr18iiiYIS9RwULlmZ7qCaG2WVGmDQh cqqiU/p+FTZr8WbkuhIKEuxigMGBBukLoODIGMU4Joxx94PVPLWM5NDb3tyFFBOeGE o31zJs4bFNcJOLtxxfYNjXexpvoyGeWGL7kaswCXRjNHafUjuedmAI6mgvDyJKK9DK UHe+L3SH1csrJrObmCA+silUKDUpwHq7NG2OcQwvolZ4S1cXTYAbDbP+FVDjEoF02/ q+bw9ReQHAHuw== MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Date: Fri, 19 Jun 2020 18:41:21 +0200 From: Martin Lucina To: =?UTF-8?Q?Roger_Pau_Monn=C3=A9?= Subject: Re: Event delivery and "domain blocking" on PVHv2 In-Reply-To: <20200619112119.GY735@Air-de-Roger> References: <62479d08f7650c22678d7a86851eafc4@lucina.net> <5865159c-4190-e841-8020-7a4f3cf0fc24@citrix.com> <20200618101330.GB10330@nodbug.lucina.net> <20200618114617.GJ735@Air-de-Roger> <17deb17cec442f96cc7aba98ef4c047c@lucina.net> <20200619112119.GY735@Air-de-Roger> Message-ID: X-Sender: martin@lucina.net User-Agent: Roundcube Webmail/1.3.3 X-BeenThere: mirageos-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Developer list for MirageOS List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Andrew Cooper , mirageos-devel@lists.xenproject.org, xen-devel@lists.xenproject.org Errors-To: mirageos-devel-bounces@lists.xenproject.org Sender: "MirageOS-devel" On 2020-06-19 13:21, Roger Pau MonnĂ© wrote: > On Fri, Jun 19, 2020 at 12:28:50PM +0200, Martin Lucina wrote: >> On 2020-06-18 13:46, Roger Pau MonnĂ© wrote: >> > On Thu, Jun 18, 2020 at 12:13:30PM +0200, Martin Lucina wrote: >> > > At this point I don't really have a clear idea of how to progress, >> > > comparing my implementation side-by-side with the original PV >> > > Mini-OS-based >> > > implementation doesn't show up any differences I can see. >> > > >> > > AFAICT the OCaml code I've also not changed in any material way, and >> > > that >> > > has been running in production on PV for years, so I'd be inclined >> > > to think >> > > the problem is in my reimplementation of the C parts, but where...? >> > >> > A good start would be to print the ISR and IRR lapic registers when >> > blocked, to assert there are no pending vectors there. >> > >> > Can you apply the following patch to your Xen, rebuild and check the >> > output of the 'l' debug key? >> > >> > Also add the output of the 'v' key. >> >> Had to fight the Xen Debian packages a bit as I wanted to patch the >> exact >> same Xen (there are some failures when building on a system that has >> Xen >> installed due to following symlinks when fixing shebangs). >> >> Here you go, when stuck during netfront setup, after allocating its >> event >> channel, presumably waiting on Xenstore: >> >> 'e': >> >> (XEN) Event channel information for domain 3: >> (XEN) Polling vCPUs: {} >> (XEN) port [p/m/s] >> (XEN) 1 [1/0/1]: s=3 n=0 x=0 d=0 p=33 >> (XEN) 2 [1/1/1]: s=3 n=0 x=0 d=0 p=34 >> (XEN) 3 [1/0/1]: s=5 n=0 x=0 v=0 >> (XEN) 4 [0/1/1]: s=2 n=0 x=0 d=0 >> >> 'l': >> >> (XEN) d3v0 IRR: >> ffff8301732dc200b >> (XEN) d3v0 ISR: >> ffff8301732dc100b > > Which version of Xen is this? AFAICT it doesn't have the support to > print a bitmap. That in Debian 10 (stable): ii xen-hypervisor-4.11-amd64 4.11.3+24-g14b62ab3e5-1~deb10u1.2 amd64 Xen Hypervisor on AMD64 xen_major : 4 xen_minor : 11 xen_extra : .4-pre xen_version : 4.11.4-pre > > Do you think you could also pick commit > 8cd9500958d818e3deabdd0d4164ea6fe1623d7c [0] and rebuild? (and print > the info again). Done, here you go: (XEN) Event channel information for domain 3: (XEN) Polling vCPUs: {} (XEN) port [p/m/s] (XEN) 1 [1/0/1]: s=3 n=0 x=0 d=0 p=33 (XEN) 2 [1/1/1]: s=3 n=0 x=0 d=0 p=34 (XEN) 3 [1/0/1]: s=5 n=0 x=0 v=0 (XEN) 4 [0/1/1]: s=3 n=0 x=0 d=0 p=35 (XEN) d3v0 IRR: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000 (XEN) d3v0 ISR: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000 (XEN) *********** VMCS Areas ************** (XEN) (XEN) >>> Domain 3 <<< (XEN) VCPU 0 (XEN) *** Guest State *** (XEN) CR0: actual=0x0000000080000033, shadow=0x0000000080000033, gh_mask=ffffffffffffffff (XEN) CR4: actual=0x0000000000002660, shadow=0x0000000000000020, gh_mask=ffffffffffc8f860 (XEN) CR3 = 0x00000000002e9000 (XEN) RSP = 0x000000000ffffec0 (0x000000000ffffec0) RIP = 0x0000000000209997 (0x0000000000209998) (XEN) RFLAGS=0x00000297 (0x00000297) DR7 = 0x0000000000000400 (XEN) Sysenter RSP=0000000000000000 CS:RIP=0000:0000000000000000 (XEN) sel attr limit base (XEN) CS: 0008 0a099 ffffffff 0000000000000000 (XEN) DS: 0010 0c093 ffffffff 0000000000000000 (XEN) SS: 0010 0c093 ffffffff 0000000000000000 (XEN) ES: 0010 0c093 ffffffff 0000000000000000 (XEN) FS: 0000 1c000 ffffffff 0000000000000000 (XEN) GS: 0000 1c000 ffffffff 0000000000000000 (XEN) GDTR: 00000027 00000000003e13c0 (XEN) LDTR: 0000 10000 00000000 0000000000000000 (XEN) IDTR: 000002ff 00000000003e10a8 (XEN) TR: 0018 0008b 00000068 00000000003e1040 (XEN) EFER = 0x0000000000000000 PAT = 0x0007040600070406 (XEN) PreemptionTimer = 0x00000000 SM Base = 0x00000000 (XEN) DebugCtl = 0x0000000000000000 DebugExceptions = 0x0000000000000000 (XEN) Interruptibility = 00000000 ActivityState = 00000000 (XEN) *** Host State *** (XEN) RIP = 0xffff82d08031df40 (vmx_asm_vmexit_handler) RSP = 0xffff83009c057f70 (XEN) CS=e008 SS=0000 DS=0000 ES=0000 FS=0000 GS=0000 TR=e040 (XEN) FSBase=0000000000000000 GSBase=0000000000000000 TRBase=ffff82d08055e000 (XEN) GDTBase=ffff82d08042f000 IDTBase=ffff82d080559000 (XEN) CR0=0000000080050033 CR3=0000000171d9b000 CR4=00000000003526e0 (XEN) Sysenter RSP=ffff83009c057fa0 CS:RIP=e008:ffff82d0803664b0 (XEN) EFER = 0x0000000000000000 PAT = 0x0000050100070406 (XEN) *** Control State *** (XEN) PinBased=0000003f CPUBased=b6a065fa SecondaryExec=000014eb (XEN) EntryControls=000053ff ExitControls=000fefff (XEN) ExceptionBitmap=00060002 PFECmask=00000000 PFECmatch=00000000 (XEN) VMEntry: intr_info=00000020 errcode=00000000 ilen=00000000 (XEN) VMExit: intr_info=00000000 errcode=00000000 ilen=00000001 (XEN) reason=0000000c qualification=0000000000000000 (XEN) IDTVectoring: info=00000000 errcode=00000000 (XEN) TSC Offset = 0xffffffd2984c98e6 TSC Multiplier = 0x0000000000000000 (XEN) TPR Threshold = 0x00 PostedIntrVec = 0x00 (XEN) EPT pointer = 0x000000016e5ac01e EPTP index = 0x0000 (XEN) PLE Gap=00000080 Window=00001000 (XEN) Virtual processor ID = 0x000e VMfunc controls = 0000000000000000 (XEN) ************************************** Martin > > Thanks, Roger. > > [0] > http://xenbits.xen.org/gitweb/?p=xen.git;a=patch;h=8cd9500958d818e3deabdd0d4164ea6fe1623d7c From mirageos-devel-bounces@lists.xenproject.org Fri Jun 19 16:46:25 2020 Return-path: Envelope-to: archives@lists.xenproject.org Delivery-date: Fri, 19 Jun 2020 16:46:25 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1jmK9o-0002OR-VQ; Fri, 19 Jun 2020 16:46:24 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1jmK9n-0002OM-MZ for mirageos-devel@lists.xenproject.org; Fri, 19 Jun 2020 16:46:23 +0000 X-Inumbo-ID: 6835d63c-b24c-11ea-bb8b-bc764e2007e4 Received: from smtp.lucina.net (unknown [62.176.169.44]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id 6835d63c-b24c-11ea-bb8b-bc764e2007e4; Fri, 19 Jun 2020 16:46:23 +0000 (UTC) Received: from webmail.moloch.sk (w3-s.a.lucina.net [62.176.169.73]) by smtp.lucina.net (Postfix) with ESMTPSA id 0E521122804; Fri, 19 Jun 2020 18:46:22 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=lucina.net; s=dkim-201811; t=1592585182; bh=oAGUPnrn4Qyr0dXzpsV9TWyYeXe15zTyF+cf1o6lts0=; h=Date:From:To:Cc:Subject:In-Reply-To:References:From; b=mWEOVhcL6UHDibFbWtvYEO8ki/QNwXfxWJZV7HgRAw0kerywsa3ieRyu7m0ekcRGQ WVb5/9Sg6WopcXdclVeyVAl0pbdZSKs4Z2A93DW3lgFg0bf1xxv/ox1drRuXLYit2S 8uztEoM8mGWOAX/8CzwivdCyVNR5hWPhFjr1FBkQEo9J1y0QcdFAzeio6DmO901ps8 ciCHkgJvVCAui/riDLanE9Elyv6pLURJpHQIJ5sW+gy1sVEcdFhyW2zHWBITQTKKVG fH9H9KpjKFvDWwu1Ozs1F0k8XTxJsVs4qeZ2vIihqmNPwn5Io3WHzH2CM9/EzXmXHX GW+5O/4cg94gg== MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Transfer-Encoding: 7bit Date: Fri, 19 Jun 2020 18:46:22 +0200 From: Martin Lucina To: Andrew Cooper Subject: Re: Event delivery and "domain blocking" on PVHv2 In-Reply-To: <0ebf1417-49e5-d9b2-81b0-b02c7e6cf833@citrix.com> References: <62479d08f7650c22678d7a86851eafc4@lucina.net> <5865159c-4190-e841-8020-7a4f3cf0fc24@citrix.com> <20200618101330.GB10330@nodbug.lucina.net> <20200618114617.GJ735@Air-de-Roger> <17deb17cec442f96cc7aba98ef4c047c@lucina.net> <0ebf1417-49e5-d9b2-81b0-b02c7e6cf833@citrix.com> Message-ID: <89c4e6e5bc1b986989e549a467eed439@lucina.net> X-Sender: martin@lucina.net User-Agent: Roundcube Webmail/1.3.3 X-BeenThere: mirageos-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Developer list for MirageOS List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: xen-devel@lists.xenproject.org, mirageos-devel@lists.xenproject.org, =?UTF-8?Q?Roger_Pau_Monn=C3=A9?= Errors-To: mirageos-devel-bounces@lists.xenproject.org Sender: "MirageOS-devel" On 2020-06-19 13:21, Andrew Cooper wrote: > On 19/06/2020 11:28, Martin Lucina wrote: >> RIP 0x209997 is the 'hlt' instruction in >> mirage_xen_evtchn_block_domain() so we are indeed blocking waiting for >> events to show up. > > I can't find this in the code, but it might be an x86-ism you're > falling > over here. Solo5 only contains the lowest-level bits, and only then those parts that "fit" the responsibility of that layer. The rest is here (WIP, not cleaned up yet): https://github.com/mato/mirage-xen/tree/xen-pvh-via-solo5 The event channel code, including the function that blocks the domain, is in lib/bindings/evtchn_stubs.c. > > Its not safe to use hlt with interrupts enabled, unless it is exactly > `sti; hlt` where the STI instruction transitions the interrupt flag > from > clear to set (i.e. you had interrupts disabled beforehand). > > Otherwise you can take the interrupt intended to wake you on the before > the hlt is executed. Hmm, so what is the right thing to do when blocking on PVHv2 in this scenario? Just use SCHEDOP_block? "cli; sti; hlt"? (Tried the latter, doesn't help). Martin From mirageos-devel-bounces@lists.xenproject.org Fri Jun 19 16:54:41 2020 Return-path: Envelope-to: archives@lists.xenproject.org Delivery-date: Fri, 19 Jun 2020 16:54:41 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1jmKHl-0003HO-Sw; Fri, 19 Jun 2020 16:54:37 +0000 Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1jmKHk-0003HJ-SR for mirageos-devel@lists.xenproject.org; Fri, 19 Jun 2020 16:54:36 +0000 X-Inumbo-ID: 8d3fba32-b24d-11ea-bbbd-12813bfff9fa Received: from esa1.hc3370-68.iphmx.com (unknown [216.71.145.142]) by us1-amaz-eas2.inumbo.com (Halon) with ESMTPS id 8d3fba32-b24d-11ea-bbbd-12813bfff9fa; Fri, 19 Jun 2020 16:54:34 +0000 (UTC) Authentication-Results: esa1.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none IronPort-SDR: J4TTlQT1ID5jKu5v1esKXrwor2/zAp+QL4MzFP8/Stzr6ep8JzIq6TBWC+2B+1wu//P/e5inS4 Mn+DHxXvKIzOl357XnfHqoDs0xpYy0xl5mx3Qzfp4Y2nJCSwldpSHki+p2eq7mkHJ1b861/8/I GyMOC9YGppDuxYiIY4JPgX7QbJeksDSYAaHK046KnsDVsZCVQVBLjNXD9JApWmG51ra658sgN6 Zurgw2NkuBv/NTad0SpeRA77u6mgdvgfkLmAzs9QpfMXKvk2Mt7j2xXW0PNB1w0LeFWck5Li3s tyI= X-SBRS: 2.7 X-MesageID: 20791138 X-Ironport-Server: esa1.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.75,256,1589256000"; d="scan'208";a="20791138" Date: Fri, 19 Jun 2020 18:54:26 +0200 From: Roger Pau =?utf-8?B?TW9ubsOp?= To: Martin Lucina Subject: Re: Event delivery and "domain blocking" on PVHv2 Message-ID: <20200619165426.GD735@Air-de-Roger> References: <62479d08f7650c22678d7a86851eafc4@lucina.net> <5865159c-4190-e841-8020-7a4f3cf0fc24@citrix.com> <20200618101330.GB10330@nodbug.lucina.net> <20200618114617.GJ735@Air-de-Roger> <17deb17cec442f96cc7aba98ef4c047c@lucina.net> <20200619112119.GY735@Air-de-Roger> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-ClientProxiedBy: AMSPEX02CAS02.citrite.net (10.69.22.113) To AMSPEX02CL02.citrite.net (10.69.22.126) X-BeenThere: mirageos-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Developer list for MirageOS List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Andrew Cooper , mirageos-devel@lists.xenproject.org, xen-devel@lists.xenproject.org Errors-To: mirageos-devel-bounces@lists.xenproject.org Sender: "MirageOS-devel" On Fri, Jun 19, 2020 at 06:41:21PM +0200, Martin Lucina wrote: > On 2020-06-19 13:21, Roger Pau MonnĂ© wrote: > > On Fri, Jun 19, 2020 at 12:28:50PM +0200, Martin Lucina wrote: > > > On 2020-06-18 13:46, Roger Pau MonnĂ© wrote: > > > > On Thu, Jun 18, 2020 at 12:13:30PM +0200, Martin Lucina wrote: > > > > > At this point I don't really have a clear idea of how to progress, > > > > > comparing my implementation side-by-side with the original PV > > > > > Mini-OS-based > > > > > implementation doesn't show up any differences I can see. > > > > > > > > > > AFAICT the OCaml code I've also not changed in any material way, and > > > > > that > > > > > has been running in production on PV for years, so I'd be inclined > > > > > to think > > > > > the problem is in my reimplementation of the C parts, but where...? > > > > > > > > A good start would be to print the ISR and IRR lapic registers when > > > > blocked, to assert there are no pending vectors there. > > > > > > > > Can you apply the following patch to your Xen, rebuild and check the > > > > output of the 'l' debug key? > > > > > > > > Also add the output of the 'v' key. > > > > > > Had to fight the Xen Debian packages a bit as I wanted to patch the > > > exact > > > same Xen (there are some failures when building on a system that has > > > Xen > > > installed due to following symlinks when fixing shebangs). > > > > > > Here you go, when stuck during netfront setup, after allocating its > > > event > > > channel, presumably waiting on Xenstore: > > > > > > 'e': > > > > > > (XEN) Event channel information for domain 3: > > > (XEN) Polling vCPUs: {} > > > (XEN) port [p/m/s] > > > (XEN) 1 [1/0/1]: s=3 n=0 x=0 d=0 p=33 > > > (XEN) 2 [1/1/1]: s=3 n=0 x=0 d=0 p=34 > > > (XEN) 3 [1/0/1]: s=5 n=0 x=0 v=0 > > > (XEN) 4 [0/1/1]: s=2 n=0 x=0 d=0 > > > > > > 'l': > > > > > > (XEN) d3v0 IRR: > > > ffff8301732dc200b > > > (XEN) d3v0 ISR: > > > ffff8301732dc100b > > > > Which version of Xen is this? AFAICT it doesn't have the support to > > print a bitmap. > > That in Debian 10 (stable): > > ii xen-hypervisor-4.11-amd64 4.11.3+24-g14b62ab3e5-1~deb10u1.2 > amd64 Xen Hypervisor on AMD64 > > xen_major : 4 > xen_minor : 11 > xen_extra : .4-pre > xen_version : 4.11.4-pre > > > > > Do you think you could also pick commit > > 8cd9500958d818e3deabdd0d4164ea6fe1623d7c [0] and rebuild? (and print > > the info again). > > Done, here you go: > > (XEN) Event channel information for domain 3: > (XEN) Polling vCPUs: {} > (XEN) port [p/m/s] > (XEN) 1 [1/0/1]: s=3 n=0 x=0 d=0 p=33 > (XEN) 2 [1/1/1]: s=3 n=0 x=0 d=0 p=34 > (XEN) 3 [1/0/1]: s=5 n=0 x=0 v=0 > (XEN) 4 [0/1/1]: s=3 n=0 x=0 d=0 p=35 > > > (XEN) d3v0 IRR: > 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000 > (XEN) d3v0 ISR: > 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000 So there's nothing pending on the lapic. Can you assert that you will always execute evtchn_demux_pending after you have received an event channel interrupt (ie: executed solo5__xen_evtchn_vector_handler)? I think this would be simpler if you moved evtchn_demux_pending into solo5__xen_evtchn_vector_handler? As there would be less asynchronous processing, and thus likely less races? Roger. From mirageos-devel-bounces@lists.xenproject.org Fri Jun 19 17:42:31 2020 Return-path: Envelope-to: archives@lists.xenproject.org Delivery-date: Fri, 19 Jun 2020 17:42:31 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1jmL1y-0007f0-3N; Fri, 19 Jun 2020 17:42:22 +0000 Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1jmL1x-0007ev-GZ for mirageos-devel@lists.xenproject.org; Fri, 19 Jun 2020 17:42:21 +0000 X-Inumbo-ID: 39954076-b254-11ea-bbd2-12813bfff9fa Received: from esa4.hc3370-68.iphmx.com (unknown [216.71.155.144]) by us1-amaz-eas2.inumbo.com (Halon) with ESMTPS id 39954076-b254-11ea-bbd2-12813bfff9fa; Fri, 19 Jun 2020 17:42:20 +0000 (UTC) Authentication-Results: esa4.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none IronPort-SDR: HM6kqYKEN36aCLijVPw3284hmcR2Na5SOQ+OAAXLJbsd9Nic+H94ffrxC5fn7opoNSiRrdq/cN rNob5PWveZm6aYImuegajhC2XkEK8nXcubPXaXLxbbTVRKDWjnZR7STIHIsAA+ZP3trP92Qj56 XHb7l8yy4YgO8pU1HNK/B1WGA5KUdaA1w0cvOGjCeH4qJV9BuyQn/rDHmdDSCFjPMpxvl9bU/t tfNrxGJvBLVAe9UGKCZTEWKfcpgwRp9SIXe6Uw47C3jc+GFuNW9R1LaSSG5aQ0pOwPMrgfY2aM 8uQ= X-SBRS: 2.7 X-MesageID: 21282996 X-Ironport-Server: esa4.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.75,256,1589256000"; d="scan'208";a="21282996" Date: Fri, 19 Jun 2020 19:42:13 +0200 From: Roger Pau =?utf-8?B?TW9ubsOp?= To: Martin Lucina Subject: Re: Event delivery and "domain blocking" on PVHv2 Message-ID: <20200619174143.GE735@Air-de-Roger> References: <62479d08f7650c22678d7a86851eafc4@lucina.net> <5865159c-4190-e841-8020-7a4f3cf0fc24@citrix.com> <20200618101330.GB10330@nodbug.lucina.net> <20200618114617.GJ735@Air-de-Roger> <17deb17cec442f96cc7aba98ef4c047c@lucina.net> <20200619112119.GY735@Air-de-Roger> <20200619165426.GD735@Air-de-Roger> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <20200619165426.GD735@Air-de-Roger> X-ClientProxiedBy: AMSPEX02CAS02.citrite.net (10.69.22.113) To AMSPEX02CL02.citrite.net (10.69.22.126) X-BeenThere: mirageos-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Developer list for MirageOS List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Andrew Cooper , mirageos-devel@lists.xenproject.org, xen-devel@lists.xenproject.org Errors-To: mirageos-devel-bounces@lists.xenproject.org Sender: "MirageOS-devel" On Fri, Jun 19, 2020 at 06:54:26PM +0200, Roger Pau MonnĂ© wrote: > On Fri, Jun 19, 2020 at 06:41:21PM +0200, Martin Lucina wrote: > > On 2020-06-19 13:21, Roger Pau MonnĂ© wrote: > > > On Fri, Jun 19, 2020 at 12:28:50PM +0200, Martin Lucina wrote: > > > > On 2020-06-18 13:46, Roger Pau MonnĂ© wrote: > > > > > On Thu, Jun 18, 2020 at 12:13:30PM +0200, Martin Lucina wrote: > > > > > > At this point I don't really have a clear idea of how to progress, > > > > > > comparing my implementation side-by-side with the original PV > > > > > > Mini-OS-based > > > > > > implementation doesn't show up any differences I can see. > > > > > > > > > > > > AFAICT the OCaml code I've also not changed in any material way, and > > > > > > that > > > > > > has been running in production on PV for years, so I'd be inclined > > > > > > to think > > > > > > the problem is in my reimplementation of the C parts, but where...? > > > > > > > > > > A good start would be to print the ISR and IRR lapic registers when > > > > > blocked, to assert there are no pending vectors there. > > > > > > > > > > Can you apply the following patch to your Xen, rebuild and check the > > > > > output of the 'l' debug key? > > > > > > > > > > Also add the output of the 'v' key. > > > > > > > > Had to fight the Xen Debian packages a bit as I wanted to patch the > > > > exact > > > > same Xen (there are some failures when building on a system that has > > > > Xen > > > > installed due to following symlinks when fixing shebangs). > > > > > > > > Here you go, when stuck during netfront setup, after allocating its > > > > event > > > > channel, presumably waiting on Xenstore: > > > > > > > > 'e': > > > > > > > > (XEN) Event channel information for domain 3: > > > > (XEN) Polling vCPUs: {} > > > > (XEN) port [p/m/s] > > > > (XEN) 1 [1/0/1]: s=3 n=0 x=0 d=0 p=33 > > > > (XEN) 2 [1/1/1]: s=3 n=0 x=0 d=0 p=34 > > > > (XEN) 3 [1/0/1]: s=5 n=0 x=0 v=0 > > > > (XEN) 4 [0/1/1]: s=2 n=0 x=0 d=0 > > > > > > > > 'l': > > > > > > > > (XEN) d3v0 IRR: > > > > ffff8301732dc200b > > > > (XEN) d3v0 ISR: > > > > ffff8301732dc100b > > > > > > Which version of Xen is this? AFAICT it doesn't have the support to > > > print a bitmap. > > > > That in Debian 10 (stable): > > > > ii xen-hypervisor-4.11-amd64 4.11.3+24-g14b62ab3e5-1~deb10u1.2 > > amd64 Xen Hypervisor on AMD64 > > > > xen_major : 4 > > xen_minor : 11 > > xen_extra : .4-pre > > xen_version : 4.11.4-pre > > > > > > > > Do you think you could also pick commit > > > 8cd9500958d818e3deabdd0d4164ea6fe1623d7c [0] and rebuild? (and print > > > the info again). > > > > Done, here you go: > > > > (XEN) Event channel information for domain 3: > > (XEN) Polling vCPUs: {} > > (XEN) port [p/m/s] > > (XEN) 1 [1/0/1]: s=3 n=0 x=0 d=0 p=33 > > (XEN) 2 [1/1/1]: s=3 n=0 x=0 d=0 p=34 > > (XEN) 3 [1/0/1]: s=5 n=0 x=0 v=0 > > (XEN) 4 [0/1/1]: s=3 n=0 x=0 d=0 p=35 > > > > > > (XEN) d3v0 IRR: > > 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000 > > (XEN) d3v0 ISR: > > 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000 > > So there's nothing pending on the lapic. Can you assert that you will > always execute evtchn_demux_pending after you have received an event > channel interrupt (ie: executed solo5__xen_evtchn_vector_handler)? > > I think this would be simpler if you moved evtchn_demux_pending into > solo5__xen_evtchn_vector_handler? As there would be less asynchronous > processing, and thus likely less races? Having though about this, I think this model of not demuxing in solo5__xen_evtchn_vector_handler is always racy, as it's not possible to assert that you would always call evtchn_demux_pending after solo5__xen_evtchn_vector_handler? Ie: if you receive an interrupt just before going to sleep (after the sti and before the hlt) you will execute solo5__xen_evtchn_vector_handler and EOI the vector, but then evtchn_demux_pending will never get called, and thus the interrupts will stay indefinitely pending? Roger. From mirageos-devel-bounces@lists.xenproject.org Sat Jun 20 18:24:53 2020 Return-path: Envelope-to: archives@lists.xenproject.org Delivery-date: Sat, 20 Jun 2020 18:24:53 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1jmiAL-0003b3-0H; Sat, 20 Jun 2020 18:24:33 +0000 Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1jmiAJ-0003ay-Pq for mirageos-devel@lists.xenproject.org; Sat, 20 Jun 2020 18:24:31 +0000 X-Inumbo-ID: 4889e26a-b323-11ea-bcc5-12813bfff9fa Received: from sonic303-3.consmr.mail.bf2.yahoo.com (unknown [74.6.131.42]) by us1-amaz-eas2.inumbo.com (Halon) with ESMTPS id 4889e26a-b323-11ea-bcc5-12813bfff9fa; Sat, 20 Jun 2020 18:24:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1592677470; bh=lDV/R5/YhWEaTiXSl7P+649Q1EL6JBhrAIozU/U2lfw=; h=Date:From:Reply-To:To:Subject:References:From:Subject; b=duoj2nagctXo0kz5Mlsx83DjZNrpZWyKm8Lr6aYv3Mhk72TXlwb/KW9VVsOpFh1RmYuBI+H7WdJheRDpLl0LH1T5WKycauZYPsM6pU0Fbx1TvLPauuvyJnu0nic0AwLnkYXQJGIBdhzfw4B4y5DDNegeOaCRLy2gapWcN9+q17EO3gFKdDACQM0k718Lw8MUM7q7UbJzw0LRZ6quLd0LGrJsfhD6Qma9Ik3S/GTEqGHbUv5OC4PVqqI9xyFicuJQoIVpiJQEHYpLUqOmnu0WibU8EbKgPMjKLIVvLeeaBOgtJmwaVNZtwGstpNu7KEG4s/jS6SBVpCWgt1Pv3pieLg== X-YMail-OSG: wxyc3LkVM1mEv6Y7TdlfuTaqGCp9K5DWSVsMKJfihiHWxdDP3PQTEw3LZgu_Df2 cxXWK_K2oebie.NYFe4MxYtTN5h_S6tAetHWKi8cNTfxuCn7tJtuk.di5NmlsYDt77FsIIzvRPc1 lKLo_8S3ANq6UTyMmP7C9DhQmY0UVnraL58FI_3iWWHiV3GsnTi7i_JW0QGD5FcHyXGBFU1uVf9G dZLO2gCYZ0SlLM5zC6EIE37_xfLfumvMF.Y24I3.iNhyN_hJeiFfiaoZbzk9tm2B4w2NKs36aN2a ja7yyUqZkAPZk._wN5NR9yGsGiiHV3tyBcQYwDEUvzVoowZlL.pzCISFr3S0tTLO06rviOTow_wt fUZX7MDV4f9m2piYSIw6PjyYfn3ZHcE1IjqWo0Fl_SaN_e0eJBDkFDWzoBsRFJm50xSMJUWOKmrj 5n4X7izcsSepdUkHtO1VpBNkEtzWXtNS5itEwkdbppyqQejk3n55LgNASIo6s1JonRxXgT6QFk0U szifzjRw2YMi9RbCRLWNF5yi6x4NC9n3dRrBTLqoT2Ir7VG5N5G4.2tSB0M2d6GQV_pgtABtOgoT vHaOfq05fVmIKXgmzb3IOaULQs0iOASoqsC_wZUy_l1JLySQZ8EkYzLQHDZcic5qJhLUvLn0SPYj wWvg_rexc.gfHCldGqpnAph9hxnIRrBqy3DuEUhlMwdpqZxFfkEgPPkqXUTyzYIFOXku72j43mY3 zSxbWjE8yfa_BpUSuF7FXa8281W7tFPMiBB3d7YDXXBVHazY46RR46jIkfHuZoQx4kKVUEAZgJgP sYehjqnNrsztYqs2a6rCJ6Wd9Jsr0wMmNXK19KFyk2_y1Vwp4Zgy8iXoSBYGdZfdEb_tkvo9I4gc 65L.f7w-- Received: from sonic.gate.mail.ne1.yahoo.com by sonic303.consmr.mail.bf2.yahoo.com with HTTP; Sat, 20 Jun 2020 18:24:30 +0000 Date: Sat, 20 Jun 2020 18:24:28 +0000 (UTC) From: Jason Long To: "mirageos-devel@lists.xenproject.org" Message-ID: <494205866.856353.1592677468181@mail.yahoo.com> Subject: I'm a new user and wants to learn more about the Mirage OS. MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_856352_379731036.1592677468181" References: <494205866.856353.1592677468181.ref@mail.yahoo.com> X-Mailer: WebService/1.1.16138 YahooMailAndroidMobile YMobile/1.0 (com.yahoo.mobile.client.android.mail/6.8.2; Android/7.1.1; NMF26F; bbc100; BlackBerry; BBC100-1; 5.16; 1184x720; ) Content-Length: 1342 X-BeenThere: mirageos-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Developer list for MirageOS List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Reply-To: "hack3rcon@yahoo.com" Errors-To: mirageos-devel-bounces@lists.xenproject.org Sender: "MirageOS-devel" ------=_Part_856352_379731036.1592677468181 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Hello,I don't know this address is OK for this kind of questions or not. Sorry, if I'm in a wrong list.I want to know what is the benefit of the Mirage OS?Which companies using it and for what?How can I download and install it?Why it using Ocalm programming language? Thank you. ------=_Part_856352_379731036.1592677468181 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 7bit
Hello,
I don't know this address is OK for this kind of questions or not. Sorry, if I'm in a wrong list.
I want to know what is the benefit of the Mirage OS?
Which companies using it and for what?
How can I download and install it?
Why it using Ocalm programming language?

Thank you.

------=_Part_856352_379731036.1592677468181-- From mirageos-devel-bounces@lists.xenproject.org Mon Jun 22 10:59:01 2020 Return-path: Envelope-to: archives@lists.xenproject.org Delivery-date: Mon, 22 Jun 2020 10:59:01 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1jnK9x-0001wG-Ry; Mon, 22 Jun 2020 10:58:41 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1jnK9w-0001wB-P7 for mirageos-devel@lists.xenproject.org; Mon, 22 Jun 2020 10:58:40 +0000 X-Inumbo-ID: 53956c2e-b477-11ea-bb8b-bc764e2007e4 Received: from smtp.lucina.net (unknown [62.176.169.44]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id 53956c2e-b477-11ea-bb8b-bc764e2007e4; Mon, 22 Jun 2020 10:58:39 +0000 (UTC) Received: from webmail.moloch.sk (w3-s.a.lucina.net [62.176.169.73]) by smtp.lucina.net (Postfix) with ESMTPSA id 06159122804; Mon, 22 Jun 2020 12:58:38 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=lucina.net; s=dkim-201811; t=1592823518; bh=AjldR7lSzE4srHcYluGJ3oH/ie3BZmsR0Ph46rZZm5g=; h=Date:From:To:Cc:Subject:In-Reply-To:References:From; b=qltgW80XAzY/26uZa6rBcCiVvcxFVpmI0xHD3h8HuPyQA5LT+qW/llhBSj/EZk4V2 AcfdeOuyKX8nZreZGbfwl9RteRy3B83kMx6l+tMdmT/9eJsOh7yqMBKHiH3KM4Tv3E Zvru9vfe2zDLGMUBHUg9U88dVlWupRTy/BnxrTNvmukmnMmrbwPr5ASAJYxPbHG5SQ nkG+lXhNaSczpPvwx1vo113WnafmFEwoDMeHhxliADZWy2ri728Gb3OgJzUCh03VNa V7ivlI3ihgJkuBPFyvSZRMwjhLq/AOy8qLr5Va6WpLxKSvW0X9pDC90XuAatUAXnpR EF1JUM15gCx4w== MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Date: Mon, 22 Jun 2020 12:58:37 +0200 From: Martin Lucina To: =?UTF-8?Q?Roger_Pau_Monn=C3=A9?= Subject: Re: Event delivery and "domain blocking" on PVHv2 In-Reply-To: <20200619174143.GE735@Air-de-Roger> References: <62479d08f7650c22678d7a86851eafc4@lucina.net> <5865159c-4190-e841-8020-7a4f3cf0fc24@citrix.com> <20200618101330.GB10330@nodbug.lucina.net> <20200618114617.GJ735@Air-de-Roger> <17deb17cec442f96cc7aba98ef4c047c@lucina.net> <20200619112119.GY735@Air-de-Roger> <20200619165426.GD735@Air-de-Roger> <20200619174143.GE735@Air-de-Roger> Message-ID: <7ed4a5f98b3002f3233e02d5ce803ef0@lucina.net> X-Sender: martin@lucina.net User-Agent: Roundcube Webmail/1.3.3 X-BeenThere: mirageos-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Developer list for MirageOS List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Andrew Cooper , mirageos-devel@lists.xenproject.org, xen-devel@lists.xenproject.org Errors-To: mirageos-devel-bounces@lists.xenproject.org Sender: "MirageOS-devel" On 2020-06-19 19:42, Roger Pau Monné wrote: > On Fri, Jun 19, 2020 at 06:54:26PM +0200, Roger Pau Monné wrote: >> On Fri, Jun 19, 2020 at 06:41:21PM +0200, Martin Lucina wrote: >> > On 2020-06-19 13:21, Roger Pau Monné wrote: >> > > On Fri, Jun 19, 2020 at 12:28:50PM +0200, Martin Lucina wrote: >> > > > On 2020-06-18 13:46, Roger Pau Monné wrote: >> > > > > On Thu, Jun 18, 2020 at 12:13:30PM +0200, Martin Lucina wrote: >> > > > > > At this point I don't really have a clear idea of how to progress, >> > > > > > comparing my implementation side-by-side with the original PV >> > > > > > Mini-OS-based >> > > > > > implementation doesn't show up any differences I can see. >> > > > > > >> > > > > > AFAICT the OCaml code I've also not changed in any material way, and >> > > > > > that >> > > > > > has been running in production on PV for years, so I'd be inclined >> > > > > > to think >> > > > > > the problem is in my reimplementation of the C parts, but where...? >> > > > > >> > > > > A good start would be to print the ISR and IRR lapic registers when >> > > > > blocked, to assert there are no pending vectors there. >> > > > > >> > > > > Can you apply the following patch to your Xen, rebuild and check the >> > > > > output of the 'l' debug key? >> > > > > >> > > > > Also add the output of the 'v' key. >> > > > >> > > > Had to fight the Xen Debian packages a bit as I wanted to patch the >> > > > exact >> > > > same Xen (there are some failures when building on a system that has >> > > > Xen >> > > > installed due to following symlinks when fixing shebangs). >> > > > >> > > > Here you go, when stuck during netfront setup, after allocating its >> > > > event >> > > > channel, presumably waiting on Xenstore: >> > > > >> > > > 'e': >> > > > >> > > > (XEN) Event channel information for domain 3: >> > > > (XEN) Polling vCPUs: {} >> > > > (XEN) port [p/m/s] >> > > > (XEN) 1 [1/0/1]: s=3 n=0 x=0 d=0 p=33 >> > > > (XEN) 2 [1/1/1]: s=3 n=0 x=0 d=0 p=34 >> > > > (XEN) 3 [1/0/1]: s=5 n=0 x=0 v=0 >> > > > (XEN) 4 [0/1/1]: s=2 n=0 x=0 d=0 >> > > > >> > > > 'l': >> > > > >> > > > (XEN) d3v0 IRR: >> > > > ffff8301732dc200b >> > > > (XEN) d3v0 ISR: >> > > > ffff8301732dc100b >> > > >> > > Which version of Xen is this? AFAICT it doesn't have the support to >> > > print a bitmap. >> > >> > That in Debian 10 (stable): >> > >> > ii xen-hypervisor-4.11-amd64 4.11.3+24-g14b62ab3e5-1~deb10u1.2 >> > amd64 Xen Hypervisor on AMD64 >> > >> > xen_major : 4 >> > xen_minor : 11 >> > xen_extra : .4-pre >> > xen_version : 4.11.4-pre >> > >> > > >> > > Do you think you could also pick commit >> > > 8cd9500958d818e3deabdd0d4164ea6fe1623d7c [0] and rebuild? (and print >> > > the info again). >> > >> > Done, here you go: >> > >> > (XEN) Event channel information for domain 3: >> > (XEN) Polling vCPUs: {} >> > (XEN) port [p/m/s] >> > (XEN) 1 [1/0/1]: s=3 n=0 x=0 d=0 p=33 >> > (XEN) 2 [1/1/1]: s=3 n=0 x=0 d=0 p=34 >> > (XEN) 3 [1/0/1]: s=5 n=0 x=0 v=0 >> > (XEN) 4 [0/1/1]: s=3 n=0 x=0 d=0 p=35 >> > >> > >> > (XEN) d3v0 IRR: >> > 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000 >> > (XEN) d3v0 ISR: >> > 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000 >> >> So there's nothing pending on the lapic. Can you assert that you will >> always execute evtchn_demux_pending after you have received an event >> channel interrupt (ie: executed solo5__xen_evtchn_vector_handler)? >> >> I think this would be simpler if you moved evtchn_demux_pending into >> solo5__xen_evtchn_vector_handler? As there would be less asynchronous >> processing, and thus likely less races? > > Having though about this, I think this model of not demuxing in > solo5__xen_evtchn_vector_handler is always racy, as it's not possible > to assert that you would always call evtchn_demux_pending after > solo5__xen_evtchn_vector_handler? > > Ie: if you receive an interrupt just before going to sleep (after the > sti and before the hlt) you will execute > solo5__xen_evtchn_vector_handler and EOI the vector, but then > evtchn_demux_pending will never get called, and thus the interrupts > will stay indefinitely pending? Aha! Thank you for pointing this out. I think you may be right, but this should be possible without doing the demuxing in interrupt context. How about this arrangement, which appears to work for me; no hangs I can see so far and domU survives ping -f fine with no packet loss: CAMLprim value mirage_xen_evtchn_block_domain(value v_deadline) { struct vcpu_info *vi = VCPU0_INFO(); solo5_time_t deadline = Int64_val(v_deadline); if (solo5_clock_monotonic() < deadline) { __asm__ __volatile__ ("cli" : : : "memory"); if (vi->evtchn_upcall_pending) { __asm__ __volatile__ ("sti"); } else { hypercall_set_timer_op(deadline); __asm__ __volatile__ ("sti; hlt"); } } return Val_unit; } i.e. Always go to sleep with interrupts disabled, but before doing so re-check that no events have become pending since the last time evtchn_demux_pending() was called. This holds, since the only thing that sets vi->evtchn_upcall_pending is Xen, and the only thing that clears it is evtchn_demux_pending(). Right? In an attempt to understand why the original PV code worked I re-read the PV Mini-OS block_domain code again and realised that I had entirely missed one part of its behaviour, which is that it intends[*] to run with interrupts/upcalls disabled *all* the time and relies on SCHEDOP_block atomically re-enabling them and triggering an upcall before returning (PV) or "briefly enabling interrupts to allow handlers to run" (HVM). We're doing the inverse, but our behaviour matches my mental model of how things should work. [*] AFAICT there's a bug in Mini-OS as ASSERT(irqs_disabled) is a no-op, and block_domain is called with upcalls/interrupts enabled the first time round. But I'm not 100% sure, and that code is a twisty little maze of #ifdefs all alike. Martin > Roger. From mirageos-devel-bounces@lists.xenproject.org Mon Jun 22 13:59:21 2020 Return-path: Envelope-to: archives@lists.xenproject.org Delivery-date: Mon, 22 Jun 2020 13:59:21 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1jnMyb-00023K-1k; Mon, 22 Jun 2020 13:59:09 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1jnMya-00022B-En for mirageos-devel@lists.xenproject.org; Mon, 22 Jun 2020 13:59:08 +0000 X-Inumbo-ID: 85dc6c64-b490-11ea-8496-bc764e2007e4 Received: from esa6.hc3370-68.iphmx.com (unknown [216.71.155.175]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id 85dc6c64-b490-11ea-8496-bc764e2007e4; Mon, 22 Jun 2020 13:59:00 +0000 (UTC) Authentication-Results: esa6.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none IronPort-SDR: 7y4rcKvZcox4JFp72HsObqCBchKdRBPFQ+txFZgp5PYBmgG+CpJa2dAZJs5k057cPFD9L0p+9v bssuV7vrAiO89TodoThg07a2b9QUBMrdkiS3JZLO1ifQtw6hZE05Zw1kq/bEAOQuK0emmgKdqN LEdrw7WS8Lo4S05270JNP23KAQ2CR1Pt5iQttOoV26p+SQwHIKFCzns5zwUagRl2HspJApiJmQ 3pgBxfsMdCIbovvJEsHAYblfqYcBBUa/FWRhV8okz+PMlVo11BUjLFw+EJHkFoTxlYhcCtZDNk wGU= X-SBRS: 2.7 X-MesageID: 20968042 X-Ironport-Server: esa6.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.75,267,1589256000"; d="scan'208";a="20968042" Date: Mon, 22 Jun 2020 15:58:53 +0200 From: Roger Pau =?utf-8?B?TW9ubsOp?= To: Martin Lucina Subject: Re: Event delivery and "domain blocking" on PVHv2 Message-ID: <20200622135853.GK735@Air-de-Roger> References: <62479d08f7650c22678d7a86851eafc4@lucina.net> <5865159c-4190-e841-8020-7a4f3cf0fc24@citrix.com> <20200618101330.GB10330@nodbug.lucina.net> <20200618114617.GJ735@Air-de-Roger> <17deb17cec442f96cc7aba98ef4c047c@lucina.net> <20200619112119.GY735@Air-de-Roger> <20200619165426.GD735@Air-de-Roger> <20200619174143.GE735@Air-de-Roger> <7ed4a5f98b3002f3233e02d5ce803ef0@lucina.net> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <7ed4a5f98b3002f3233e02d5ce803ef0@lucina.net> X-ClientProxiedBy: AMSPEX02CAS02.citrite.net (10.69.22.113) To AMSPEX02CL02.citrite.net (10.69.22.126) X-BeenThere: mirageos-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Developer list for MirageOS List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Andrew Cooper , mirageos-devel@lists.xenproject.org, xen-devel@lists.xenproject.org Errors-To: mirageos-devel-bounces@lists.xenproject.org Sender: "MirageOS-devel" On Mon, Jun 22, 2020 at 12:58:37PM +0200, Martin Lucina wrote: > On 2020-06-19 19:42, Roger Pau Monné wrote: > > On Fri, Jun 19, 2020 at 06:54:26PM +0200, Roger Pau Monné wrote: > > > On Fri, Jun 19, 2020 at 06:41:21PM +0200, Martin Lucina wrote: > > > > On 2020-06-19 13:21, Roger Pau Monné wrote: > > > > > On Fri, Jun 19, 2020 at 12:28:50PM +0200, Martin Lucina wrote: > > > > > > On 2020-06-18 13:46, Roger Pau Monné wrote: > > > > > > > On Thu, Jun 18, 2020 at 12:13:30PM +0200, Martin Lucina wrote: > > > > > > > > At this point I don't really have a clear idea of how to progress, > > > > > > > > comparing my implementation side-by-side with the original PV > > > > > > > > Mini-OS-based > > > > > > > > implementation doesn't show up any differences I can see. > > > > > > > > > > > > > > > > AFAICT the OCaml code I've also not changed in any material way, and > > > > > > > > that > > > > > > > > has been running in production on PV for years, so I'd be inclined > > > > > > > > to think > > > > > > > > the problem is in my reimplementation of the C parts, but where...? > > > > > > > > > > > > > > A good start would be to print the ISR and IRR lapic registers when > > > > > > > blocked, to assert there are no pending vectors there. > > > > > > > > > > > > > > Can you apply the following patch to your Xen, rebuild and check the > > > > > > > output of the 'l' debug key? > > > > > > > > > > > > > > Also add the output of the 'v' key. > > > > > > > > > > > > Had to fight the Xen Debian packages a bit as I wanted to patch the > > > > > > exact > > > > > > same Xen (there are some failures when building on a system that has > > > > > > Xen > > > > > > installed due to following symlinks when fixing shebangs). > > > > > > > > > > > > Here you go, when stuck during netfront setup, after allocating its > > > > > > event > > > > > > channel, presumably waiting on Xenstore: > > > > > > > > > > > > 'e': > > > > > > > > > > > > (XEN) Event channel information for domain 3: > > > > > > (XEN) Polling vCPUs: {} > > > > > > (XEN) port [p/m/s] > > > > > > (XEN) 1 [1/0/1]: s=3 n=0 x=0 d=0 p=33 > > > > > > (XEN) 2 [1/1/1]: s=3 n=0 x=0 d=0 p=34 > > > > > > (XEN) 3 [1/0/1]: s=5 n=0 x=0 v=0 > > > > > > (XEN) 4 [0/1/1]: s=2 n=0 x=0 d=0 > > > > > > > > > > > > 'l': > > > > > > > > > > > > (XEN) d3v0 IRR: > > > > > > ffff8301732dc200b > > > > > > (XEN) d3v0 ISR: > > > > > > ffff8301732dc100b > > > > > > > > > > Which version of Xen is this? AFAICT it doesn't have the support to > > > > > print a bitmap. > > > > > > > > That in Debian 10 (stable): > > > > > > > > ii xen-hypervisor-4.11-amd64 4.11.3+24-g14b62ab3e5-1~deb10u1.2 > > > > amd64 Xen Hypervisor on AMD64 > > > > > > > > xen_major : 4 > > > > xen_minor : 11 > > > > xen_extra : .4-pre > > > > xen_version : 4.11.4-pre > > > > > > > > > > > > > > Do you think you could also pick commit > > > > > 8cd9500958d818e3deabdd0d4164ea6fe1623d7c [0] and rebuild? (and print > > > > > the info again). > > > > > > > > Done, here you go: > > > > > > > > (XEN) Event channel information for domain 3: > > > > (XEN) Polling vCPUs: {} > > > > (XEN) port [p/m/s] > > > > (XEN) 1 [1/0/1]: s=3 n=0 x=0 d=0 p=33 > > > > (XEN) 2 [1/1/1]: s=3 n=0 x=0 d=0 p=34 > > > > (XEN) 3 [1/0/1]: s=5 n=0 x=0 v=0 > > > > (XEN) 4 [0/1/1]: s=3 n=0 x=0 d=0 p=35 > > > > > > > > > > > > (XEN) d3v0 IRR: > > > > 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000 > > > > (XEN) d3v0 ISR: > > > > 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000 > > > > > > So there's nothing pending on the lapic. Can you assert that you will > > > always execute evtchn_demux_pending after you have received an event > > > channel interrupt (ie: executed solo5__xen_evtchn_vector_handler)? > > > > > > I think this would be simpler if you moved evtchn_demux_pending into > > > solo5__xen_evtchn_vector_handler? As there would be less asynchronous > > > processing, and thus likely less races? > > > > Having though about this, I think this model of not demuxing in > > solo5__xen_evtchn_vector_handler is always racy, as it's not possible > > to assert that you would always call evtchn_demux_pending after > > solo5__xen_evtchn_vector_handler? > > > > Ie: if you receive an interrupt just before going to sleep (after the > > sti and before the hlt) you will execute > > solo5__xen_evtchn_vector_handler and EOI the vector, but then > > evtchn_demux_pending will never get called, and thus the interrupts > > will stay indefinitely pending? > > Aha! Thank you for pointing this out. I think you may be right, but this > should be possible without doing the demuxing in interrupt context. If you don't do the demuxing in the interrupt context (ie: making the interrupt handler a noop), then you don't likely need such interrupt anyway? > How about this arrangement, which appears to work for me; no hangs I can see > so far and domU survives ping -f fine with no packet loss: > > CAMLprim value > mirage_xen_evtchn_block_domain(value v_deadline) > { > struct vcpu_info *vi = VCPU0_INFO(); > solo5_time_t deadline = Int64_val(v_deadline); > > if (solo5_clock_monotonic() < deadline) { > __asm__ __volatile__ ("cli" : : : "memory"); > if (vi->evtchn_upcall_pending) { > __asm__ __volatile__ ("sti"); > } > else { > hypercall_set_timer_op(deadline); What if you set a deadline so close that evtchn_upcall_pending gets set by Xen here and the interrupt is injected? You would execute the noop handler and just hlt, and could likely end up in the same blocked situation as before? > __asm__ __volatile__ ("sti; hlt"); > } > } > return Val_unit; > } > > i.e. Always go to sleep with interrupts disabled, but before doing so > re-check that no events have become pending since the last time > evtchn_demux_pending() was called. This holds, since the only thing that > sets vi->evtchn_upcall_pending is Xen, and the only thing that clears it is > evtchn_demux_pending(). > > Right? TBH this is a hard model to get right, I think your best bet at attempting something along this lines is to forget about using the event channel interrupt and instead use SCHEDOP_poll. You could do something like (written in pure C as I have no idea of the ocaml bindings): int mirage_xen_evtchn_block_domain(uint64_t timeout) { evtchn_port_t ports[MAX_PORTS]; struct sched_poll poll = { .timeout = timeout, .nr_ports = 0, }; set_xen_guest_handle(poll.ports, ports); /* Fill ports you care about (ie: all event channel ports in use) */ ports[poll.nr_ports++] = port_1; ports[poll.nr_ports++] = port_2; [...] /* Check that you don't overrun MAX_PORTS */ /* On return demux events and call timer handler if timeout expired. */ return hypercall_sched_op(SCHEDOP_poll, &poll); } Doing something like the above you could forget about setting up the event channel interrupt and the timer. > __asm__ __volatile__ ("sti; hlt"); > } > } > return Val_unit; > } > > In an attempt to understand why the original PV code worked I re-read the PV > Mini-OS block_domain code again and realised that I had entirely missed one > part of its behaviour, which is that it intends[*] to run with > interrupts/upcalls disabled *all* the time and relies on SCHEDOP_block > atomically re-enabling them and triggering an upcall before returning (PV) > or "briefly enabling interrupts to allow handlers to run" (HVM). We're doing > the inverse, but our behaviour matches my mental model of how things should > work. Not really IMO, as SCHEDOP_block is a single 'instruction' from a guest PoV that does the enabling of interrupts and returns if there are pending ones. Also SCHEDOP_block is not exactly the same on HVM, as it just checks for pending vectors to inject, but not for pending event channels. On HVM you cannot call hlt with interrupts disabled, or the vCPU will be taken down. There are quite a lot of subtle differences between PV and HVM in this regard, and I think the best approach would be to use SCHEDOP_poll in order to implement the kind of model you describe. Roger. From mirageos-devel-bounces@lists.xenproject.org Mon Jun 22 15:31:21 2020 Return-path: Envelope-to: archives@lists.xenproject.org Delivery-date: Mon, 22 Jun 2020 15:31:21 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1jnOPZ-0003Y7-Aq; Mon, 22 Jun 2020 15:31:05 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1jnOPY-0003X1-49 for mirageos-devel@lists.xenproject.org; Mon, 22 Jun 2020 15:31:04 +0000 X-Inumbo-ID: 60416c0e-b49d-11ea-bca7-bc764e2007e4 Received: from smtp.lucina.net (unknown [62.176.169.44]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id 60416c0e-b49d-11ea-bca7-bc764e2007e4; Mon, 22 Jun 2020 15:31:01 +0000 (UTC) Received: from webmail.moloch.sk (w3-s.a.lucina.net [62.176.169.73]) by smtp.lucina.net (Postfix) with ESMTPSA id 2B5BA122804; Mon, 22 Jun 2020 17:31:00 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=lucina.net; s=dkim-201811; t=1592839860; bh=Xws8g1Wi5dE+ZjqV+LiybhYjyilm/86GEW8v+QkfRc4=; h=Date:From:To:Cc:Subject:In-Reply-To:References:From; b=pnLlAt8Do/2GfKL/HAydudr90+ZlKTYedk1C+x8MHzL7kIkx6dvtQC+5eJKDqQG64 3hX1OUCi3hgX/4ZbWmnq2e4mlw8XR0rn87KW2c9sfo5JLObxYf/VdkkdZwsBLEO6F4 cLxMpoIXApI2U87il2HjNj5Z+nD4pSc1fen+3IA9OJeH6b6Shugx0PId04RnY7vOEW GmpyQptkfw64HDbSFgopbszgq6rXhAIMywSGDsFXA02mlFm4vlyjBNdtacgJa2uigp 041BCTREpK2FXfdqmO/nk0zpKVxEF6atSVqfcfHKT5EF8nbkDz+nzSsR7jDXMcZy6C 7PymPZbi1Altw== MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Date: Mon, 22 Jun 2020 17:31:00 +0200 From: Martin Lucina To: =?UTF-8?Q?Roger_Pau_Monn=C3=A9?= Subject: Re: Event delivery and "domain blocking" on PVHv2 In-Reply-To: <20200622135853.GK735@Air-de-Roger> References: <62479d08f7650c22678d7a86851eafc4@lucina.net> <5865159c-4190-e841-8020-7a4f3cf0fc24@citrix.com> <20200618101330.GB10330@nodbug.lucina.net> <20200618114617.GJ735@Air-de-Roger> <17deb17cec442f96cc7aba98ef4c047c@lucina.net> <20200619112119.GY735@Air-de-Roger> <20200619165426.GD735@Air-de-Roger> <20200619174143.GE735@Air-de-Roger> <7ed4a5f98b3002f3233e02d5ce803ef0@lucina.net> <20200622135853.GK735@Air-de-Roger> Message-ID: <745e4ac251b146480b2c6d6afbf5f34a@lucina.net> X-Sender: martin@lucina.net User-Agent: Roundcube Webmail/1.3.3 X-BeenThere: mirageos-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Developer list for MirageOS List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Andrew Cooper , mirageos-devel@lists.xenproject.org, xen-devel@lists.xenproject.org Errors-To: mirageos-devel-bounces@lists.xenproject.org Sender: "MirageOS-devel" On 2020-06-22 15:58, Roger Pau Monné wrote: > On Mon, Jun 22, 2020 at 12:58:37PM +0200, Martin Lucina wrote: >> Aha! Thank you for pointing this out. I think you may be right, but >> this >> should be possible without doing the demuxing in interrupt context. > > If you don't do the demuxing in the interrupt context (ie: making the > interrupt handler a noop), then you don't likely need such interrupt > anyway? I need the/an interrupt to wake the VCPU from HLT state if we went to sleep. > >> How about this arrangement, which appears to work for me; no hangs I >> can see >> so far and domU survives ping -f fine with no packet loss: >> >> CAMLprim value >> mirage_xen_evtchn_block_domain(value v_deadline) >> { >> struct vcpu_info *vi = VCPU0_INFO(); >> solo5_time_t deadline = Int64_val(v_deadline); >> >> if (solo5_clock_monotonic() < deadline) { >> __asm__ __volatile__ ("cli" : : : "memory"); >> if (vi->evtchn_upcall_pending) { >> __asm__ __volatile__ ("sti"); >> } >> else { >> hypercall_set_timer_op(deadline); > > What if you set a deadline so close that evtchn_upcall_pending gets > set by Xen here and the interrupt is injected? You would execute the > noop handler and just hlt, and could likely end up in the same blocked > situation as before? Why would an interrupt be injected here? Doesn't the immediately preceding "cli" disable that? Or perhaps I need to do a PV/HVM hybrid and set vi->evtchn_upcall_mask just before the cli, and clear it after the sti? >> i.e. Always go to sleep with interrupts disabled, but before doing so >> re-check that no events have become pending since the last time >> evtchn_demux_pending() was called. This holds, since the only thing >> that >> sets vi->evtchn_upcall_pending is Xen, and the only thing that clears >> it is >> evtchn_demux_pending(). >> >> Right? > > TBH this is a hard model to get right, I think your best bet at > attempting something along this lines is to forget about using the > event channel interrupt and instead use SCHEDOP_poll. You could do > something like (written in pure C as I have no idea of the ocaml > bindings): > [SCHEDOP_poll code snipped] Thanks for the suggestion. This brings us full-circle -- I found [1] and [2] way back from 2013 when Mirage/Xen was initially using SCHEDOP_poll and then switched to the current interrupts + SCHEDOP_block approach. Part of the motivation for the change at the time was to allow waiting on/servicing more than 128 ports (the limit for SCHEDOP_poll). I doubt anyone wants to do that these days, but it still makes me a bit reluctant to change back to SCHEDOP_poll. >> In an attempt to understand why the original PV code worked I re-read >> the PV >> Mini-OS block_domain code again and realised that I had entirely >> missed one >> part of its behaviour, which is that it intends[*] to run with >> interrupts/upcalls disabled *all* the time and relies on SCHEDOP_block >> atomically re-enabling them and triggering an upcall before returning >> (PV) >> or "briefly enabling interrupts to allow handlers to run" (HVM). We're >> doing >> the inverse, but our behaviour matches my mental model of how things >> should >> work. > > Not really IMO, as SCHEDOP_block is a single 'instruction' from a > guest PoV that does the enabling of interrupts and returns if there > are pending ones. Indeed and this is exactly the operation I need in the HVM world with the current model. > Also SCHEDOP_block is not exactly the same on HVM, as it just checks > for pending vectors to inject, but not for pending event channels. ... well, I don't want to use SCHEDOP_block anyway since that is not possible on ARM, and I would like to keep the code at least "portable with not too much effort". So there should be a non-racy "HVM way" to do this? > HVM you cannot call hlt with interrupts disabled, or the vCPU will be > taken down. > > There are quite a lot of subtle differences between PV and HVM in this > regard, and I think the best approach would be to use SCHEDOP_poll in > order to implement the kind of model you describe. I can see that now. The interactions between the "virtual hardware" (interrupt delivery, VCPU IF) and "PV" parts (event delivery, masking) are hard to understand for me, yet the two are intertwined in the way HVM works :-( Coming back to your earlier suggestion of moving the event demuxing (but not the handling) into the upcall interrupt handler itself, perhaps that approach is still worth investigating in combination with re-working the OCaml side array view of pending events, or at least ensuring that atomic operations are used since it would now be updated asynchronously. Martin [1] https://lists.cam.ac.uk/pipermail/cl-mirage/2013-September/msg00053.html [2] https://github.com/mirage/mirage-platform/pull/58 From mirageos-devel-bounces@lists.xenproject.org Mon Jun 22 15:36:10 2020 Return-path: Envelope-to: archives@lists.xenproject.org Delivery-date: Mon, 22 Jun 2020 15:36:10 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1jnOUR-0003oe-PH; Mon, 22 Jun 2020 15:36:07 +0000 Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1jnOUQ-0003oZ-Oo for mirageos-devel@lists.xenproject.org; Mon, 22 Jun 2020 15:36:06 +0000 X-Inumbo-ID: 15868a2c-b49e-11ea-bea5-12813bfff9fa Received: from mx2.suse.de (unknown [195.135.220.15]) by us1-amaz-eas2.inumbo.com (Halon) with ESMTPS id 15868a2c-b49e-11ea-bea5-12813bfff9fa; Mon, 22 Jun 2020 15:36:05 +0000 (UTC) X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay2.suse.de (unknown [195.135.221.27]) by mx2.suse.de (Postfix) with ESMTP id DE694C1D3; Mon, 22 Jun 2020 15:36:03 +0000 (UTC) Subject: Re: Event delivery and "domain blocking" on PVHv2 To: Martin Lucina References: <62479d08f7650c22678d7a86851eafc4@lucina.net> <5865159c-4190-e841-8020-7a4f3cf0fc24@citrix.com> <20200618101330.GB10330@nodbug.lucina.net> <20200618114617.GJ735@Air-de-Roger> <17deb17cec442f96cc7aba98ef4c047c@lucina.net> <20200619112119.GY735@Air-de-Roger> <20200619165426.GD735@Air-de-Roger> <20200619174143.GE735@Air-de-Roger> <7ed4a5f98b3002f3233e02d5ce803ef0@lucina.net> <20200622135853.GK735@Air-de-Roger> <745e4ac251b146480b2c6d6afbf5f34a@lucina.net> From: Jan Beulich Message-ID: <7667cd25-238f-9822-f560-9f507baab5c3@suse.com> Date: Mon, 22 Jun 2020 17:36:03 +0200 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:68.0) Gecko/20100101 Thunderbird/68.9.0 MIME-Version: 1.0 In-Reply-To: <745e4ac251b146480b2c6d6afbf5f34a@lucina.net> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit X-BeenThere: mirageos-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Developer list for MirageOS List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Andrew Cooper , xen-devel@lists.xenproject.org, mirageos-devel@lists.xenproject.org, =?UTF-8?Q?Roger_Pau_Monn=c3=a9?= Errors-To: mirageos-devel-bounces@lists.xenproject.org Sender: "MirageOS-devel" On 22.06.2020 17:31, Martin Lucina wrote: > On 2020-06-22 15:58, Roger Pau Monné wrote: >> On Mon, Jun 22, 2020 at 12:58:37PM +0200, Martin Lucina wrote: >>> How about this arrangement, which appears to work for me; no hangs I >>> can see >>> so far and domU survives ping -f fine with no packet loss: >>> >>> CAMLprim value >>> mirage_xen_evtchn_block_domain(value v_deadline) >>> { >>> struct vcpu_info *vi = VCPU0_INFO(); >>> solo5_time_t deadline = Int64_val(v_deadline); >>> >>> if (solo5_clock_monotonic() < deadline) { >>> __asm__ __volatile__ ("cli" : : : "memory"); >>> if (vi->evtchn_upcall_pending) { >>> __asm__ __volatile__ ("sti"); >>> } >>> else { >>> hypercall_set_timer_op(deadline); >> >> What if you set a deadline so close that evtchn_upcall_pending gets >> set by Xen here and the interrupt is injected? You would execute the >> noop handler and just hlt, and could likely end up in the same blocked >> situation as before? > > Why would an interrupt be injected here? Doesn't the immediately > preceding > "cli" disable that? > > Or perhaps I need to do a PV/HVM hybrid and set vi->evtchn_upcall_mask > just > before the cli, and clear it after the sti? evtchn_upcall_mask is a strictly PV-only thing. See e.g. the code comment in hvm_set_callback_irq_level(). Jan From mirageos-devel-bounces@lists.xenproject.org Mon Jun 22 16:09:35 2020 Return-path: Envelope-to: archives@lists.xenproject.org Delivery-date: Mon, 22 Jun 2020 16:09:35 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1jnP0m-0007JS-7a; Mon, 22 Jun 2020 16:09:32 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1jnP0l-0007IU-F4 for mirageos-devel@lists.xenproject.org; Mon, 22 Jun 2020 16:09:31 +0000 X-Inumbo-ID: bd055ae0-b4a2-11ea-bb8b-bc764e2007e4 Received: from esa2.hc3370-68.iphmx.com (unknown [216.71.145.153]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id bd055ae0-b4a2-11ea-bb8b-bc764e2007e4; Mon, 22 Jun 2020 16:09:24 +0000 (UTC) Authentication-Results: esa2.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none IronPort-SDR: tDwExkz/82tywebeocbKJhBdFN3tfWzBkUEjqD0yrjau6Dgh72pmX88KugfvDvdOjp7SHWf2MZ DjE0YoP9SpfZ2YIIj4q+jIBvoZMLpS+qnEF4WLe1//t1s+UzNqmfgUaQC0x/Gf1si2lGDUo6Kh fsrbykdyUi5wlIrRUBxugyjoLSDc1Gf84PoP8UQx0hqD3VoD6rkImYBRzfPv0PDkWW9OoeyWvN 9Eqlwo5235qYcwh060Q5TVEOI+4k9xuK+CYExK3wgMgnbMtBqte8iZVASFZOlvvtY+uEjo2SC6 jmw= X-SBRS: 2.7 X-MesageID: 20648870 X-Ironport-Server: esa2.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.75,267,1589256000"; d="scan'208";a="20648870" Date: Mon, 22 Jun 2020 18:09:16 +0200 From: Roger Pau =?utf-8?B?TW9ubsOp?= To: Martin Lucina Subject: Re: Event delivery and "domain blocking" on PVHv2 Message-ID: <20200622160916.GM735@Air-de-Roger> References: <20200618101330.GB10330@nodbug.lucina.net> <20200618114617.GJ735@Air-de-Roger> <17deb17cec442f96cc7aba98ef4c047c@lucina.net> <20200619112119.GY735@Air-de-Roger> <20200619165426.GD735@Air-de-Roger> <20200619174143.GE735@Air-de-Roger> <7ed4a5f98b3002f3233e02d5ce803ef0@lucina.net> <20200622135853.GK735@Air-de-Roger> <745e4ac251b146480b2c6d6afbf5f34a@lucina.net> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <745e4ac251b146480b2c6d6afbf5f34a@lucina.net> X-ClientProxiedBy: AMSPEX02CAS02.citrite.net (10.69.22.113) To AMSPEX02CL02.citrite.net (10.69.22.126) X-BeenThere: mirageos-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Developer list for MirageOS List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Andrew Cooper , mirageos-devel@lists.xenproject.org, xen-devel@lists.xenproject.org Errors-To: mirageos-devel-bounces@lists.xenproject.org Sender: "MirageOS-devel" On Mon, Jun 22, 2020 at 05:31:00PM +0200, Martin Lucina wrote: > On 2020-06-22 15:58, Roger Pau Monné wrote: > > On Mon, Jun 22, 2020 at 12:58:37PM +0200, Martin Lucina wrote: > > > Aha! Thank you for pointing this out. I think you may be right, but > > > this > > > should be possible without doing the demuxing in interrupt context. > > > > If you don't do the demuxing in the interrupt context (ie: making the > > interrupt handler a noop), then you don't likely need such interrupt > > anyway? > > I need the/an interrupt to wake the VCPU from HLT state if we went to sleep. > > > > > > How about this arrangement, which appears to work for me; no hangs I > > > can see > > > so far and domU survives ping -f fine with no packet loss: > > > > > > CAMLprim value > > > mirage_xen_evtchn_block_domain(value v_deadline) > > > { > > > struct vcpu_info *vi = VCPU0_INFO(); > > > solo5_time_t deadline = Int64_val(v_deadline); > > > > > > if (solo5_clock_monotonic() < deadline) { > > > __asm__ __volatile__ ("cli" : : : "memory"); > > > if (vi->evtchn_upcall_pending) { > > > __asm__ __volatile__ ("sti"); > > > } > > > else { > > > hypercall_set_timer_op(deadline); > > > > What if you set a deadline so close that evtchn_upcall_pending gets > > set by Xen here and the interrupt is injected? You would execute the > > noop handler and just hlt, and could likely end up in the same blocked > > situation as before? > > Why would an interrupt be injected here? Doesn't the immediately preceding > "cli" disable that? Well, I mean between the sti and the hlt instruction. I think there's always a window for a race here, and that's the reason for having SCHEDOP_block (see comment referring to avoiding a "wakeup waiting" race). > Or perhaps I need to do a PV/HVM hybrid and set vi->evtchn_upcall_mask just > before the cli, and clear it after the sti? I think SCHEDOP_block is broken on HVM, as as Jan points out evtchn_upcall_mask is not used on HVM (we should really have a comment about this in xen.h). So that hypercall is no longer useful to avoid wakeup waiting races. > > > i.e. Always go to sleep with interrupts disabled, but before doing so > > > re-check that no events have become pending since the last time > > > evtchn_demux_pending() was called. This holds, since the only thing > > > that > > > sets vi->evtchn_upcall_pending is Xen, and the only thing that > > > clears it is > > > evtchn_demux_pending(). > > > > > > Right? > > > > TBH this is a hard model to get right, I think your best bet at > > attempting something along this lines is to forget about using the > > event channel interrupt and instead use SCHEDOP_poll. You could do > > something like (written in pure C as I have no idea of the ocaml > > bindings): > > [SCHEDOP_poll code snipped] > > Thanks for the suggestion. This brings us full-circle -- I found [1] and > [2] way back from 2013 when Mirage/Xen was initially using SCHEDOP_poll > and then switched to the current interrupts + SCHEDOP_block approach. > > Part of the motivation for the change at the time was to allow waiting > on/servicing more than 128 ports (the limit for SCHEDOP_poll). I doubt > anyone wants to do that these days, but it still makes me a bit reluctant > to change back to SCHEDOP_poll. Right, Mirage/Xen being single processor it's not likely to use more than 128 event channels, but I can understand the desire to not be limited by that amount. > > > In an attempt to understand why the original PV code worked I > > > re-read the PV > > > Mini-OS block_domain code again and realised that I had entirely > > > missed one > > > part of its behaviour, which is that it intends[*] to run with > > > interrupts/upcalls disabled *all* the time and relies on SCHEDOP_block > > > atomically re-enabling them and triggering an upcall before > > > returning (PV) > > > or "briefly enabling interrupts to allow handlers to run" (HVM). > > > We're doing > > > the inverse, but our behaviour matches my mental model of how things > > > should > > > work. > > > > Not really IMO, as SCHEDOP_block is a single 'instruction' from a > > guest PoV that does the enabling of interrupts and returns if there > > are pending ones. > > Indeed and this is exactly the operation I need in the HVM world with the > current model. Another option would be to consider re-purposing SCHEDOP_block for HVM so it does a sti on behalf of the guest and then checks for pending interrupts to inject. > > Also SCHEDOP_block is not exactly the same on HVM, as it just checks > > for pending vectors to inject, but not for pending event channels. > > ... well, I don't want to use SCHEDOP_block anyway since that is not > possible > on ARM, and I would like to keep the code at least "portable with not too > much effort". So there should be a non-racy "HVM way" to do this? One solution (albeit it would be slightly crappy IMO) is to make sure the timer is always fully handled in interrupt context, so that you _never_ call hlt with a timer event pending. That way you are always guaranteed to be woken up. > > HVM you cannot call hlt with interrupts disabled, or the vCPU will be > > taken down. > > > > There are quite a lot of subtle differences between PV and HVM in this > > regard, and I think the best approach would be to use SCHEDOP_poll in > > order to implement the kind of model you describe. > > I can see that now. The interactions between the "virtual hardware" > (interrupt delivery, VCPU IF) and "PV" parts (event delivery, masking) are > hard to understand for me, yet the two are intertwined in the way HVM > works :-( > > Coming back to your earlier suggestion of moving the event demuxing (but not > the handling) into the upcall interrupt handler itself, perhaps that > approach > is still worth investigating in combination with re-working the OCaml side > array > view of pending events, or at least ensuring that atomic operations are used > since it would now be updated asynchronously. I assume you must be doing something similar for KVM in Solo5, where you handle (at least certain) interrupts in interrupt context, or else the same issues would arise? Roger. From mirageos-devel-bounces@lists.xenproject.org Mon Jun 22 16:20:42 2020 Return-path: Envelope-to: archives@lists.xenproject.org Delivery-date: Mon, 22 Jun 2020 16:20:42 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1jnPBX-0000dr-Dl; Mon, 22 Jun 2020 16:20:39 +0000 Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1jnPBW-0000dk-86 for mirageos-devel@lists.xenproject.org; Mon, 22 Jun 2020 16:20:38 +0000 X-Inumbo-ID: 4e02af10-b4a4-11ea-beaa-12813bfff9fa Received: from mx2.suse.de (unknown [195.135.220.15]) by us1-amaz-eas2.inumbo.com (Halon) with ESMTPS id 4e02af10-b4a4-11ea-beaa-12813bfff9fa; Mon, 22 Jun 2020 16:20:36 +0000 (UTC) X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay2.suse.de (unknown [195.135.221.27]) by mx2.suse.de (Postfix) with ESMTP id B120DC220; Mon, 22 Jun 2020 16:20:35 +0000 (UTC) Subject: Re: Event delivery and "domain blocking" on PVHv2 To: =?UTF-8?Q?Roger_Pau_Monn=c3=a9?= References: <20200618101330.GB10330@nodbug.lucina.net> <20200618114617.GJ735@Air-de-Roger> <17deb17cec442f96cc7aba98ef4c047c@lucina.net> <20200619112119.GY735@Air-de-Roger> <20200619165426.GD735@Air-de-Roger> <20200619174143.GE735@Air-de-Roger> <7ed4a5f98b3002f3233e02d5ce803ef0@lucina.net> <20200622135853.GK735@Air-de-Roger> <745e4ac251b146480b2c6d6afbf5f34a@lucina.net> <20200622160916.GM735@Air-de-Roger> From: Jan Beulich Message-ID: Date: Mon, 22 Jun 2020 18:20:34 +0200 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:68.0) Gecko/20100101 Thunderbird/68.9.0 MIME-Version: 1.0 In-Reply-To: <20200622160916.GM735@Air-de-Roger> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit X-BeenThere: mirageos-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Developer list for MirageOS List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Andrew Cooper , Martin Lucina , mirageos-devel@lists.xenproject.org, xen-devel@lists.xenproject.org Errors-To: mirageos-devel-bounces@lists.xenproject.org Sender: "MirageOS-devel" On 22.06.2020 18:09, Roger Pau Monné wrote: > On Mon, Jun 22, 2020 at 05:31:00PM +0200, Martin Lucina wrote: >> On 2020-06-22 15:58, Roger Pau Monné wrote: >>> On Mon, Jun 22, 2020 at 12:58:37PM +0200, Martin Lucina wrote: >>>> Aha! Thank you for pointing this out. I think you may be right, but >>>> this >>>> should be possible without doing the demuxing in interrupt context. >>> >>> If you don't do the demuxing in the interrupt context (ie: making the >>> interrupt handler a noop), then you don't likely need such interrupt >>> anyway? >> >> I need the/an interrupt to wake the VCPU from HLT state if we went to sleep. >> >>> >>>> How about this arrangement, which appears to work for me; no hangs I >>>> can see >>>> so far and domU survives ping -f fine with no packet loss: >>>> >>>> CAMLprim value >>>> mirage_xen_evtchn_block_domain(value v_deadline) >>>> { >>>> struct vcpu_info *vi = VCPU0_INFO(); >>>> solo5_time_t deadline = Int64_val(v_deadline); >>>> >>>> if (solo5_clock_monotonic() < deadline) { >>>> __asm__ __volatile__ ("cli" : : : "memory"); >>>> if (vi->evtchn_upcall_pending) { >>>> __asm__ __volatile__ ("sti"); >>>> } >>>> else { >>>> hypercall_set_timer_op(deadline); >>> >>> What if you set a deadline so close that evtchn_upcall_pending gets >>> set by Xen here and the interrupt is injected? You would execute the >>> noop handler and just hlt, and could likely end up in the same blocked >>> situation as before? >> >> Why would an interrupt be injected here? Doesn't the immediately preceding >> "cli" disable that? > > Well, I mean between the sti and the hlt instruction. When EFLAGS.IF was clear before STI, then the first point at which an interrupt can get injected is when HLT is already executed (i.e. to wake from this HLT). That's the so called "STI shadow". Jan From mirageos-devel-bounces@lists.xenproject.org Mon Jun 22 16:26:55 2020 Return-path: Envelope-to: archives@lists.xenproject.org Delivery-date: Mon, 22 Jun 2020 16:26:55 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1jnPHY-000168-VZ; Mon, 22 Jun 2020 16:26:52 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1jnPHX-00014w-JW for mirageos-devel@lists.xenproject.org; Mon, 22 Jun 2020 16:26:51 +0000 X-Inumbo-ID: 29cbd8a0-b4a5-11ea-bb8b-bc764e2007e4 Received: from smtp.lucina.net (unknown [62.176.169.44]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id 29cbd8a0-b4a5-11ea-bb8b-bc764e2007e4; Mon, 22 Jun 2020 16:26:45 +0000 (UTC) Received: from webmail.moloch.sk (w3-s.a.lucina.net [62.176.169.73]) by smtp.lucina.net (Postfix) with ESMTPSA id DC2A0122804; Mon, 22 Jun 2020 18:26:44 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=lucina.net; s=dkim-201811; t=1592843204; bh=mQQXPxHXOMJnHmwl/M96HjVCwVd2JTdxTuAjUdi23H8=; h=Date:From:To:Cc:Subject:In-Reply-To:References:From; b=S4ACGWs+Kz/7HOI8guY2BTY+7HMQp7b4VKVLow5o4N2MPfwD6hy7J3EGLdOTG/Nv8 DtqY1StSPlGWGG+Ir9ntH8NyUt5FdfyOE7b6gRZdz0qnulOVH8N1pods77LEdJ5qZU ACSg+YEXiuchewfwx5g+8kesVQbzYoRC0oiY4YryqtwpM2sppkf71euzhQOBFPpEpr O3lhwhaqrS9bfpEGw/RdYjlQQDV3CqaReZwqn3XDdSXRFuMNTUUNUHfQRBm2pO9xfk 7Ibi8QD5HRhLKRcV+qxadw+0DZdWWiNEv5/QE/8IsxV8UeCmW434xtCxELL0QdtZY2 pN2A936vVhByA== MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Date: Mon, 22 Jun 2020 18:26:44 +0200 From: Martin Lucina To: Jan Beulich Subject: Re: Event delivery and "domain blocking" on PVHv2 In-Reply-To: References: <20200618101330.GB10330@nodbug.lucina.net> <20200618114617.GJ735@Air-de-Roger> <17deb17cec442f96cc7aba98ef4c047c@lucina.net> <20200619112119.GY735@Air-de-Roger> <20200619165426.GD735@Air-de-Roger> <20200619174143.GE735@Air-de-Roger> <7ed4a5f98b3002f3233e02d5ce803ef0@lucina.net> <20200622135853.GK735@Air-de-Roger> <745e4ac251b146480b2c6d6afbf5f34a@lucina.net> <20200622160916.GM735@Air-de-Roger> Message-ID: <7db6ae6c0270fde4d417e3c6134f3dbc@lucina.net> X-Sender: martin@lucina.net User-Agent: Roundcube Webmail/1.3.3 X-BeenThere: mirageos-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Developer list for MirageOS List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Andrew Cooper , xen-devel@lists.xenproject.org, mirageos-devel@lists.xenproject.org, =?UTF-8?Q?Roger_Pau_Monn=C3=A9?= Errors-To: mirageos-devel-bounces@lists.xenproject.org Sender: "MirageOS-devel" On 2020-06-22 18:20, Jan Beulich wrote: > On 22.06.2020 18:09, Roger Pau Monné wrote: >> On Mon, Jun 22, 2020 at 05:31:00PM +0200, Martin Lucina wrote: >>> On 2020-06-22 15:58, Roger Pau Monné wrote: >>>> On Mon, Jun 22, 2020 at 12:58:37PM +0200, Martin Lucina wrote: >>>>> Aha! Thank you for pointing this out. I think you may be right, but >>>>> this >>>>> should be possible without doing the demuxing in interrupt context. >>>> >>>> If you don't do the demuxing in the interrupt context (ie: making >>>> the >>>> interrupt handler a noop), then you don't likely need such interrupt >>>> anyway? >>> >>> I need the/an interrupt to wake the VCPU from HLT state if we went to >>> sleep. >>> >>>> >>>>> How about this arrangement, which appears to work for me; no hangs >>>>> I >>>>> can see >>>>> so far and domU survives ping -f fine with no packet loss: >>>>> >>>>> CAMLprim value >>>>> mirage_xen_evtchn_block_domain(value v_deadline) >>>>> { >>>>> struct vcpu_info *vi = VCPU0_INFO(); >>>>> solo5_time_t deadline = Int64_val(v_deadline); >>>>> >>>>> if (solo5_clock_monotonic() < deadline) { >>>>> __asm__ __volatile__ ("cli" : : : "memory"); >>>>> if (vi->evtchn_upcall_pending) { >>>>> __asm__ __volatile__ ("sti"); >>>>> } >>>>> else { >>>>> hypercall_set_timer_op(deadline); >>>> >>>> What if you set a deadline so close that evtchn_upcall_pending gets >>>> set by Xen here and the interrupt is injected? You would execute the >>>> noop handler and just hlt, and could likely end up in the same >>>> blocked >>>> situation as before? >>> >>> Why would an interrupt be injected here? Doesn't the immediately >>> preceding >>> "cli" disable that? >> >> Well, I mean between the sti and the hlt instruction. > > When EFLAGS.IF was clear before STI, then the first point at which > an interrupt can get injected is when HLT is already executed (i.e. > to wake from this HLT). That's the so called "STI shadow". Indeed, that's what the Intel SDM says, and Andrew already mentioned earlier in this thread in a different context, here: https://lists.xenproject.org/archives/html/mirageos-devel/2020-06/msg00021.html . So it would seem that my latest approach is race-free? Martin From mirageos-devel-bounces@lists.xenproject.org Mon Jun 22 16:36:34 2020 Return-path: Envelope-to: archives@lists.xenproject.org Delivery-date: Mon, 22 Jun 2020 16:36:34 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1jnPQs-0002D6-5Y; Mon, 22 Jun 2020 16:36:30 +0000 Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1jnPQr-0002D1-Fd for mirageos-devel@lists.xenproject.org; Mon, 22 Jun 2020 16:36:29 +0000 X-Inumbo-ID: 84e521c8-b4a6-11ea-beaf-12813bfff9fa Received: from esa1.hc3370-68.iphmx.com (unknown [216.71.145.142]) by us1-amaz-eas2.inumbo.com (Halon) with ESMTPS id 84e521c8-b4a6-11ea-beaf-12813bfff9fa; Mon, 22 Jun 2020 16:36:28 +0000 (UTC) Authentication-Results: esa1.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none IronPort-SDR: zenzgTPE0gwVQjxT8qMs0u6lkcHLDtZd0ulrGRJqKik6SlE3UsmONSxc6EJJHX2OsTNN4ZJjDO zQKRbX6kGGU4IaiYNrRIYYxXh1eWDB4oltec5YfhTjmbghoUd8Xo9WCEWbrq6zY165QjRx4MV6 vPQumjzDD0loK3L8huzC+MozOEk+e2SIG71LbLsTjb9n9LJGf8TVQ0y0oJeXhaK1x7CsF11054 Jfkz9g+voi5w+jLQD2+/O8PUgCh4vG6zVdN92tC9qurBytHXFziDltxhSs6mXQG0Jc72ERiDII QqU= X-SBRS: 2.7 X-MesageID: 20944777 X-Ironport-Server: esa1.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.75,267,1589256000"; d="scan'208";a="20944777" Date: Mon, 22 Jun 2020 18:36:20 +0200 From: Roger Pau =?utf-8?B?TW9ubsOp?= To: Jan Beulich , Martin Lucina Subject: Re: Event delivery and "domain blocking" on PVHv2 Message-ID: <20200622163620.GO735@Air-de-Roger> References: <17deb17cec442f96cc7aba98ef4c047c@lucina.net> <20200619112119.GY735@Air-de-Roger> <20200619165426.GD735@Air-de-Roger> <20200619174143.GE735@Air-de-Roger> <7ed4a5f98b3002f3233e02d5ce803ef0@lucina.net> <20200622135853.GK735@Air-de-Roger> <745e4ac251b146480b2c6d6afbf5f34a@lucina.net> <20200622160916.GM735@Air-de-Roger> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-ClientProxiedBy: AMSPEX02CAS02.citrite.net (10.69.22.113) To AMSPEX02CL02.citrite.net (10.69.22.126) X-BeenThere: mirageos-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Developer list for MirageOS List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Andrew Cooper , mirageos-devel@lists.xenproject.org, xen-devel@lists.xenproject.org Errors-To: mirageos-devel-bounces@lists.xenproject.org Sender: "MirageOS-devel" On Mon, Jun 22, 2020 at 06:20:34PM +0200, Jan Beulich wrote: > On 22.06.2020 18:09, Roger Pau Monné wrote: > > On Mon, Jun 22, 2020 at 05:31:00PM +0200, Martin Lucina wrote: > >> On 2020-06-22 15:58, Roger Pau Monné wrote: > >>> On Mon, Jun 22, 2020 at 12:58:37PM +0200, Martin Lucina wrote: > >>>> Aha! Thank you for pointing this out. I think you may be right, but > >>>> this > >>>> should be possible without doing the demuxing in interrupt context. > >>> > >>> If you don't do the demuxing in the interrupt context (ie: making the > >>> interrupt handler a noop), then you don't likely need such interrupt > >>> anyway? > >> > >> I need the/an interrupt to wake the VCPU from HLT state if we went to sleep. > >> > >>> > >>>> How about this arrangement, which appears to work for me; no hangs I > >>>> can see > >>>> so far and domU survives ping -f fine with no packet loss: > >>>> > >>>> CAMLprim value > >>>> mirage_xen_evtchn_block_domain(value v_deadline) > >>>> { > >>>> struct vcpu_info *vi = VCPU0_INFO(); > >>>> solo5_time_t deadline = Int64_val(v_deadline); > >>>> > >>>> if (solo5_clock_monotonic() < deadline) { > >>>> __asm__ __volatile__ ("cli" : : : "memory"); > >>>> if (vi->evtchn_upcall_pending) { > >>>> __asm__ __volatile__ ("sti"); > >>>> } > >>>> else { > >>>> hypercall_set_timer_op(deadline); > >>> > >>> What if you set a deadline so close that evtchn_upcall_pending gets > >>> set by Xen here and the interrupt is injected? You would execute the > >>> noop handler and just hlt, and could likely end up in the same blocked > >>> situation as before? > >> > >> Why would an interrupt be injected here? Doesn't the immediately preceding > >> "cli" disable that? > > > > Well, I mean between the sti and the hlt instruction. > > When EFLAGS.IF was clear before STI, then the first point at which > an interrupt can get injected is when HLT is already executed (i.e. > to wake from this HLT). That's the so called "STI shadow". Oh, so then what Martin does seems to be fine, as there's no race, by the fact that evtchn_upcall_pending is checked with interrupts disabled. Thanks, Roger. From mirageos-devel-bounces@lists.xenproject.org Sat Jun 27 20:23:36 2020 Return-path: Envelope-to: archives@lists.xenproject.org Delivery-date: Sat, 27 Jun 2020 20:23:36 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1jpHM6-0000BL-NB; Sat, 27 Jun 2020 20:23:18 +0000 Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1jpHM4-0000BG-Vx for mirageos-devel@lists.xenproject.org; Sat, 27 Jun 2020 20:23:17 +0000 X-Inumbo-ID: 070f9f46-b8b4-11ea-83fd-12813bfff9fa Received: from orbitalfox.eu (unknown [95.172.232.202]) by us1-amaz-eas2.inumbo.com (Halon) with ESMTPS id 070f9f46-b8b4-11ea-83fd-12813bfff9fa; Sat, 27 Jun 2020 20:23:14 +0000 (UTC) Received: from [192.168.88.5] (unknown [192.168.88.5]) by orbitalfox.eu (Postfix) with ESMTPSA id 5D47FAA102F for ; Sat, 27 Jun 2020 21:23:13 +0100 (BST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=orbitalfox.eu; s=orbitalfox; t=1593289393; bh=5b1EsHpxkv8SyHiXLQcm/CLsMD85QwpnfY1s7MsM9h8=; h=Subject:To:References:From:Date:In-Reply-To; b=XyLZ1kde+IuR+UegDVjk9ThQuIt8sYtyDYH/ZMr97619RQopzucaVlgTR5Dx8mNHF oa01i+XyoFCOpdzIz+1ZlEaoBP70hWuuMGjmWyzhdSinvLAg0Zlk1QCmLaMzytWH1f MIoFU5ZU690P3VW9Uzcm/JxIrHc9Wl0/CuaphoWQp5XVP8eNUcaUCL4gJTDgDxN/Wx jZQxYCso/oAGqCmlxQ3zF5t0xufe4hxpOhXDiSjqRaYsjXQ1JruTgd93SpSpmAssR1 sVDeusBqx5Tz0JJB6wwHlSd1h3Bs5THzcrtAGMdUqSkO52DipKLS3wplxe6NfsLfeM qI/AleoFa/PrA== Subject: Re: I'm a new user and wants to learn more about the Mirage OS. To: mirageos-devel@lists.xenproject.org References: <494205866.856353.1592677468181.ref@mail.yahoo.com> <494205866.856353.1592677468181@mail.yahoo.com> From: =?UTF-8?B?b3JiaWZ4IPCfpoo=?= Message-ID: Date: Sat, 27 Jun 2020 21:23:11 +0100 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:68.0) Gecko/20100101 Thunderbird/68.9.0 MIME-Version: 1.0 In-Reply-To: <494205866.856353.1592677468181@mail.yahoo.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-GB Content-Transfer-Encoding: 7bit X-BeenThere: mirageos-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Developer list for MirageOS List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: mirageos-devel-bounces@lists.xenproject.org Sender: "MirageOS-devel" Jason Long, 2020/06/20: > I don't know this address is OK for this kind of questions or not. It should be fine, given how quiet the list is. > I want to know what is the benefit of the Mirage OS? Have you read this? https://mirage.io > Which companies using it and for what? I don't know. > How can I download and install it? [Docs :: Installation](https://mirage.io/wiki/install) > Why it using Ocalm programming language? Because OCaml rocks.