|
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [Xen-devel] [RFC][v2][PATCH 01/14] tools: introduce some new parameters to set rdm policy
This patch introduces user configurable parameters to specify RDM
resource and according policies,
Global RDM parameter:
rdm = [ 'type=none/host, reserve=strict/relaxed' ]
Per-device RDM parameter:
pci = [ 'sbdf, rdm_reserve=strict/relaxed' ]
Global RDM parameter, "type", allows user to specify reserved regions
explicitly, e.g. using 'host' to include all reserved regions reported
on this platform which is good to handle hotplug scenario. In the future
this parameter may be further extended to allow specifying random regions,
e.g. even those belonging to another platform as a preparation for live
migration with passthrough devices. Instead, 'none' means we have nothing
to do all reserved regions and ignore all policies, so guest work as before.
'strict/relaxed' policy decides how to handle conflict when reserving RDM
regions in pfn space. If conflict exists, 'strict' means an immediate error
so VM will be killed, while 'relaxed' allows moving forward with a warning
message thrown out.
Default per-device RDM policy is 'strict', while default global RDM policy
is 'relaxed'. When both policies are specified on a given region, 'strict' is
always preferred.
Signed-off-by: Tiejun Chen <tiejun.chen@xxxxxxxxx>
---
docs/man/xl.cfg.pod.5 | 57 +++++++++++++++++++++++++++
docs/misc/vtd.txt | 24 ++++++++++++
tools/libxl/libxl_create.c | 13 +++++++
tools/libxl/libxl_internal.h | 2 +
tools/libxl/libxl_pci.c | 2 +
tools/libxl/libxl_types.idl | 18 +++++++++
tools/libxl/libxlu_pci.c | 92 ++++++++++++++++++++++++++++++++++++++++++++
tools/libxl/libxlutil.h | 4 ++
tools/libxl/xl_cmdimpl.c | 10 +++++
9 files changed, 222 insertions(+)
diff --git a/docs/man/xl.cfg.pod.5 b/docs/man/xl.cfg.pod.5
index 8e4154f..12c34c4 100644
--- a/docs/man/xl.cfg.pod.5
+++ b/docs/man/xl.cfg.pod.5
@@ -645,6 +645,49 @@ assigned slave device.
=back
+=item B<rdm= "RDM_RESERVE_STRING" >
+
+(HVM/x86 only) Specifies the information about Reserved Device Memory (RDM),
+which is necessary to enable robust device passthrough usage. One example of
+RDM is reported through ACPI Reserved Memory Region Reporting (RMRR)
+structure on x86 platform.
+
+B<RDM_RESERVE_STRING> has the form C<[KEY=VALUE,KEY=VALUE,...> where:
+
+=over 4
+
+=item B<KEY=VALUE>
+
+Possible B<KEY>s are:
+
+=over 4
+
+=item B<type="STRING">
+
+Currently we just have two types:
+
+"host" means all reserved device memory on this platform should be reserved
+in this VM's pfn space. This global RDM parameter allows user to specify
+reserved regions explicitly. And using "host" to include all reserved regions
+reported on this platform which is good to handle hotplug scenario. In the
+future this parameter may be further extended to allow specifying random
+regions, e.g. even those belonging to another platform as a preparation
+for live migration with passthrough devices.
+
+"none" means we have nothing to do all reserved regions and ignore all
policies,
+so guest work as before.
+
+=over 4
+
+=item B<reserve="STRING">
+
+Conflict may be detected when reserving reserved device memory in gfn space.
+"strict" means an unsolved conflict leads to immediate VM crash, while
+"relaxed" allows VM moving forward with a warning message thrown out. "relaxed"
+is default.
+
+Note this may be overrided by another sub item, rdm_reserve, in pci device.
+
=item B<pci=[ "PCI_SPEC_STRING", "PCI_SPEC_STRING", ... ]>
Specifies the host PCI devices to passthrough to this guest. Each
B<PCI_SPEC_STRING>
@@ -707,6 +750,20 @@ dom0 without confirmation. Please use with care.
D0-D3hot power management states for the PCI device. False (0) by
default.
+=item B<rdm_reserv="STRING">
+
+(HVM/x86 only) Specifies the information about Reserved Device Memory (RDM),
+which is necessary to enable robust device passthrough usage. One example of
+RDM is reported through ACPI Reserved Memory Region Reporting (RMRR)
+structure on x86 platform.
+
+Conflict may be detected when reserving reserved device memory in gfn space.
+"strict" means an unsolved conflict leads to immediate VM crash, while
+"relaxed" allows VM moving forward with a warning message thrown out. "strict"
+is default.
+
+Note this would override global B<rdm> option.
+
=back
=back
diff --git a/docs/misc/vtd.txt b/docs/misc/vtd.txt
index 9af0e99..7d63c47 100644
--- a/docs/misc/vtd.txt
+++ b/docs/misc/vtd.txt
@@ -111,6 +111,30 @@ in the config file:
To override for a specific device:
pci = [ '01:00.0,msitranslate=0', '03:00.0' ]
+RDM, 'reserved device memory', for PCI Device Passthrough
+---------------------------------------------------------
+
+There are some devices the BIOS controls, for e.g. USB devices to perform
+PS2 emulation. The regions of memory used for these devices are marked
+reserved in the e820 map. When we turn on DMA translation, DMA to those
+regions will fail. Hence BIOS uses RMRR to specify these regions along with
+devices that need to access these regions. OS is expected to setup
+identity mappings for these regions for these devices to access these regions.
+
+While creating a VM we should reserve them in advance, and avoid any conflicts.
+So we introduce user configurable parameters to specify RDM resource and
+according policies,
+
+To enable this globally, add "rdm" in the config file:
+
+ rdm = "type=host, reserve=relaxed" (default policy is "relaxed")
+
+Or just for a specific device:
+
+ pci = [ '01:00.0,rdm_reserve=relaxed', '03:00.0,rdm_reserve=strict' ]
+
+For all the options available to RDM, see xl.cfg(5).
+
Caveat on Conventional PCI Device Passthrough
---------------------------------------------
diff --git a/tools/libxl/libxl_create.c b/tools/libxl/libxl_create.c
index f0da7dc..d649ead 100644
--- a/tools/libxl/libxl_create.c
+++ b/tools/libxl/libxl_create.c
@@ -100,6 +100,12 @@ static int sched_params_valid(libxl__gc *gc,
return 1;
}
+void libxl__rdm_setdefault(libxl__gc *gc, libxl_domain_build_info *b_info)
+{
+ b_info->rdm.type = LIBXL_RDM_RESERVE_TYPE_NONE;
+ b_info->rdm.reserve = LIBXL_RDM_RESERVE_FLAG_RELAXED;
+}
+
int libxl__domain_build_info_setdefault(libxl__gc *gc,
libxl_domain_build_info *b_info)
{
@@ -410,6 +416,8 @@ int libxl__domain_build_info_setdefault(libxl__gc *gc,
libxl_domain_type_to_string(b_info->type));
return ERROR_INVAL;
}
+
+ libxl__rdm_setdefault(gc, b_info);
return 0;
}
@@ -1439,6 +1447,11 @@ static void domcreate_attach_pci(libxl__egc *egc,
libxl__multidev *multidev,
}
for (i = 0; i < d_config->num_pcidevs; i++) {
+ /*
+ * If the rdm global policy is 'force' we should override each device.
+ */
+ if (d_config->b_info.rdm.reserve == LIBXL_RDM_RESERVE_FLAG_STRICT)
+ d_config->pcidevs[i].rdm_reserve = LIBXL_RDM_RESERVE_FLAG_STRICT;
ret = libxl__device_pci_add(gc, domid, &d_config->pcidevs[i], 1);
if (ret < 0) {
LIBXL__LOG(ctx, LIBXL__LOG_ERROR,
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 8aaa1ad..3a2f6ec 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -1108,6 +1108,8 @@ _hidden int libxl__device_vtpm_setdefault(libxl__gc *gc,
libxl_device_vtpm *vtpm
_hidden int libxl__device_vfb_setdefault(libxl__gc *gc, libxl_device_vfb *vfb);
_hidden int libxl__device_vkb_setdefault(libxl__gc *gc, libxl_device_vkb *vkb);
_hidden int libxl__device_pci_setdefault(libxl__gc *gc, libxl_device_pci *pci);
+_hidden void libxl__rdm_setdefault(libxl__gc *gc,
+ libxl_domain_build_info *b_info);
_hidden const char *libxl__device_nic_devname(libxl__gc *gc,
uint32_t domid,
diff --git a/tools/libxl/libxl_pci.c b/tools/libxl/libxl_pci.c
index e0743f8..07e84f2 100644
--- a/tools/libxl/libxl_pci.c
+++ b/tools/libxl/libxl_pci.c
@@ -1039,6 +1039,8 @@ static int libxl__device_pci_reset(libxl__gc *gc,
unsigned int domain, unsigned
int libxl__device_pci_setdefault(libxl__gc *gc, libxl_device_pci *pci)
{
+ /* We'd like to force reserve rdm specific to a device by default.*/
+ pci->rdm_reserve = LIBXL_RDM_RESERVE_FLAG_STRICT;
return 0;
}
diff --git a/tools/libxl/libxl_types.idl b/tools/libxl/libxl_types.idl
index 65d479f..f1acd13 100644
--- a/tools/libxl/libxl_types.idl
+++ b/tools/libxl/libxl_types.idl
@@ -73,6 +73,17 @@ libxl_domain_type = Enumeration("domain_type", [
(2, "PV"),
], init_val = "LIBXL_DOMAIN_TYPE_INVALID")
+libxl_rdm_reserve_type = Enumeration("rdm_reserve_type", [
+ (0, "none"),
+ (1, "host"),
+ ])
+
+libxl_rdm_reserve_flag = Enumeration("rdm_reserve_flag", [
+ (-1, "invalid"),
+ (0, "strict"),
+ (1, "relaxed"),
+ ], init_val = "LIBXL_RDM_RESERVE_FLAG_INVALID")
+
libxl_channel_connection = Enumeration("channel_connection", [
(0, "UNKNOWN"),
(1, "PTY"),
@@ -366,6 +377,11 @@ libxl_vnode_info = Struct("vnode_info", [
("vcpus", libxl_bitmap), # vcpus in this node
])
+libxl_rdm_reserve = Struct("rdm_reserve", [
+ ("type", libxl_rdm_reserve_type),
+ ("reserve", libxl_rdm_reserve_flag),
+ ])
+
libxl_domain_build_info = Struct("domain_build_info",[
("max_vcpus", integer),
("avail_vcpus", libxl_bitmap),
@@ -413,6 +429,7 @@ libxl_domain_build_info = Struct("domain_build_info",[
("kernel", string),
("cmdline", string),
("ramdisk", string),
+ ("rdm", libxl_rdm_reserve),
("u", KeyedUnion(None, libxl_domain_type, "type",
[("hvm", Struct(None, [("firmware", string),
("bios", libxl_bios_type),
@@ -533,6 +550,7 @@ libxl_device_pci = Struct("device_pci", [
("power_mgmt", bool),
("permissive", bool),
("seize", bool),
+ ("rdm_reserve", libxl_rdm_reserve_flag),
])
libxl_device_vtpm = Struct("device_vtpm", [
diff --git a/tools/libxl/libxlu_pci.c b/tools/libxl/libxlu_pci.c
index 26fb143..9255878 100644
--- a/tools/libxl/libxlu_pci.c
+++ b/tools/libxl/libxlu_pci.c
@@ -42,6 +42,9 @@ static int pcidev_struct_fill(libxl_device_pci *pcidev,
unsigned int domain,
#define STATE_OPTIONS_K 6
#define STATE_OPTIONS_V 7
#define STATE_TERMINAL 8
+#define STATE_TYPE 9
+#define STATE_RDM_TYPE 10
+#define STATE_RESERVE_FLAG 11
int xlu_pci_parse_bdf(XLU_Config *cfg, libxl_device_pci *pcidev, const char
*str)
{
unsigned state = STATE_DOMAIN;
@@ -143,6 +146,17 @@ int xlu_pci_parse_bdf(XLU_Config *cfg, libxl_device_pci
*pcidev, const char *str
pcidev->permissive = atoi(tok);
}else if ( !strcmp(optkey, "seize") ) {
pcidev->seize = atoi(tok);
+ }else if ( !strcmp(optkey, "rdm_reserve") ) {
+ if ( !strcmp(tok, "strict") ) {
+ pcidev->rdm_reserve = LIBXL_RDM_RESERVE_FLAG_STRICT;
+ } else if ( !strcmp(tok, "relaxed") ) {
+ pcidev->rdm_reserve = LIBXL_RDM_RESERVE_FLAG_RELAXED;
+ } else {
+ XLU__PCI_ERR(cfg, "%s is not an valid PCI RDM property"
+ " flag: 'strict' or 'relaxed'.",
+ tok);
+ goto parse_error;
+ }
}else{
XLU__PCI_ERR(cfg, "Unknown PCI BDF option: %s", optkey);
}
@@ -167,6 +181,84 @@ parse_error:
return ERROR_INVAL;
}
+int xlu_rdm_parse(XLU_Config *cfg, libxl_rdm_reserve *rdm, const char *str)
+{
+ unsigned state = STATE_TYPE;
+ char *buf2, *tok, *ptr, *end;
+
+ if (NULL == (buf2 = ptr = strdup(str)))
+ return ERROR_NOMEM;
+
+ for (tok = ptr, end = ptr + strlen(ptr) + 1; ptr < end; ptr++) {
+ switch(state) {
+ case STATE_TYPE:
+ if (*ptr == '=') {
+ state = STATE_RDM_TYPE;
+ *ptr = '\0';
+ if (strcmp(tok, "type")) {
+ XLU__PCI_ERR(cfg, "Unknown RDM state option: %s", tok);
+ goto parse_error;
+ }
+ tok = ptr + 1;
+ }
+ break;
+ case STATE_RDM_TYPE:
+ if (*ptr == '\0' || *ptr == ',') {
+ state = STATE_RESERVE_FLAG;
+ *ptr = '\0';
+ if (!strcmp(tok, "host")) {
+ rdm->type = LIBXL_RDM_RESERVE_TYPE_HOST;
+ } else if (!strcmp(tok, "none")) {
+ rdm->type = LIBXL_RDM_RESERVE_TYPE_NONE;
+ } else {
+ XLU__PCI_ERR(cfg, "Unknown RDM type option: %s", tok);
+ goto parse_error;
+ }
+ tok = ptr + 1;
+ }
+ break;
+ case STATE_RESERVE_FLAG:
+ if (*ptr == '=') {
+ state = STATE_OPTIONS_V;
+ *ptr = '\0';
+ if (strcmp(tok, "reserve")) {
+ XLU__PCI_ERR(cfg, "Unknown RDM property value: %s", tok);
+ goto parse_error;
+ }
+ tok = ptr + 1;
+ }
+ break;
+ case STATE_OPTIONS_V:
+ if (*ptr == ',' || *ptr == '\0') {
+ state = STATE_TERMINAL;
+ *ptr = '\0';
+ if (!strcmp(tok, "strict")) {
+ rdm->reserve = LIBXL_RDM_RESERVE_FLAG_STRICT;
+ } else if (!strcmp(tok, "relaxed")) {
+ rdm->reserve = LIBXL_RDM_RESERVE_FLAG_RELAXED;
+ } else {
+ XLU__PCI_ERR(cfg, "Unknown RDM property flag value: %s",
+ tok);
+ goto parse_error;
+ }
+ tok = ptr + 1;
+ }
+ default:
+ break;
+ }
+ }
+
+ free(buf2);
+
+ if (tok != ptr || state != STATE_TERMINAL)
+ goto parse_error;
+
+ return 0;
+
+parse_error:
+ return ERROR_INVAL;
+}
+
/*
* Local variables:
* mode: C
diff --git a/tools/libxl/libxlutil.h b/tools/libxl/libxlutil.h
index 989605a..e81b644 100644
--- a/tools/libxl/libxlutil.h
+++ b/tools/libxl/libxlutil.h
@@ -106,6 +106,10 @@ int xlu_disk_parse(XLU_Config *cfg, int nspecs, const char
*const *specs,
*/
int xlu_pci_parse_bdf(XLU_Config *cfg, libxl_device_pci *pcidev, const char
*str);
+/*
+ * RDM parsing
+ */
+int xlu_rdm_parse(XLU_Config *cfg, libxl_rdm_reserve *rdm, const char *str);
/*
* Vif rate parsing.
diff --git a/tools/libxl/xl_cmdimpl.c b/tools/libxl/xl_cmdimpl.c
index 373aa37..5b8cf2b 100644
--- a/tools/libxl/xl_cmdimpl.c
+++ b/tools/libxl/xl_cmdimpl.c
@@ -1919,6 +1919,14 @@ skip_vfb:
xlu_cfg_get_defbool(config, "e820_host", &b_info->u.pv.e820_host, 0);
}
+ if (!xlu_cfg_get_string(config, "rdm", &buf, 0)) {
+ libxl_rdm_reserve rdm;
+ if (!xlu_rdm_parse(config, &rdm, buf)) {
+ b_info->rdm.type = rdm.type;
+ b_info->rdm.reserve = rdm.reserve;
+ }
+ }
+
if (!xlu_cfg_get_list (config, "pci", &pcis, 0, 0)) {
d_config->num_pcidevs = 0;
d_config->pcidevs = NULL;
@@ -1933,6 +1941,8 @@ skip_vfb:
pcidev->power_mgmt = pci_power_mgmt;
pcidev->permissive = pci_permissive;
pcidev->seize = pci_seize;
+ /* We'd like to force reserve rdm specific to a device by
default.*/
+ pcidev->rdm_reserve = LIBXL_RDM_RESERVE_FLAG_STRICT;
if (!xlu_pci_parse_bdf(config, pcidev, buf))
d_config->num_pcidevs++;
}
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel
|
 |
Lists.xenproject.org is hosted with RackSpace, monitoring our |