[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[xen-unstable test] 106979: regressions - FAIL



flight 106979 xen-unstable real [real]
http://logs.test-lab.xenproject.org/osstest/logs/106979/

Regressions :-(

Tests which did not succeed and are blocking,
including tests which could not be run:
 test-armhf-armhf-xl-xsm     15 guest-start/debian.repeat fail REGR. vs. 106959

Regressions which are regarded as allowable (not blocking):
 test-armhf-armhf-libvirt-xsm 13 saverestore-support-check    fail  like 106959
 test-armhf-armhf-libvirt     13 saverestore-support-check    fail  like 106959
 test-amd64-amd64-xl-qemut-win7-amd64 16 guest-stop            fail like 106959
 test-amd64-i386-xl-qemuu-win7-amd64 16 guest-stop             fail like 106959
 test-amd64-i386-xl-qemut-win7-amd64 16 guest-stop             fail like 106959
 test-amd64-amd64-xl-qemuu-win7-amd64 16 guest-stop            fail like 106959
 test-armhf-armhf-libvirt-raw 12 saverestore-support-check    fail  like 106959
 test-amd64-amd64-xl-rtds      9 debian-install               fail  like 106959

Tests which did not succeed, but are not blocking:
 test-arm64-arm64-libvirt-xsm  1 build-check(1)               blocked  n/a
 test-arm64-arm64-xl           1 build-check(1)               blocked  n/a
 build-arm64-libvirt           1 build-check(1)               blocked  n/a
 test-arm64-arm64-libvirt-qcow2  1 build-check(1)               blocked  n/a
 test-arm64-arm64-libvirt      1 build-check(1)               blocked  n/a
 test-arm64-arm64-xl-credit2   1 build-check(1)               blocked  n/a
 test-arm64-arm64-xl-rtds      1 build-check(1)               blocked  n/a
 test-arm64-arm64-xl-multivcpu  1 build-check(1)               blocked  n/a
 test-arm64-arm64-xl-xsm       1 build-check(1)               blocked  n/a
 test-amd64-i386-libvirt-xsm  12 migrate-support-check        fail   never pass
 test-amd64-i386-libvirt      12 migrate-support-check        fail   never pass
 test-amd64-amd64-libvirt     12 migrate-support-check        fail   never pass
 test-amd64-amd64-libvirt-xsm 12 migrate-support-check        fail   never pass
 build-arm64                   5 xen-build                    fail   never pass
 build-arm64-xsm               5 xen-build                    fail   never pass
 test-amd64-amd64-libvirt-qemuu-debianhvm-amd64-xsm 10 migrate-support-check 
fail never pass
 test-amd64-i386-libvirt-qemuu-debianhvm-amd64-xsm 10 migrate-support-check 
fail never pass
 build-arm64-pvops             5 kernel-build                 fail   never pass
 test-armhf-armhf-xl-arndale  12 migrate-support-check        fail   never pass
 test-armhf-armhf-xl-arndale  13 saverestore-support-check    fail   never pass
 test-amd64-amd64-libvirt-vhd 11 migrate-support-check        fail   never pass
 test-armhf-armhf-xl-xsm      12 migrate-support-check        fail   never pass
 test-armhf-armhf-xl-xsm      13 saverestore-support-check    fail   never pass
 test-amd64-amd64-qemuu-nested-amd 16 debian-hvm-install/l1/l2  fail never pass
 test-armhf-armhf-xl-multivcpu 12 migrate-support-check        fail  never pass
 test-armhf-armhf-libvirt-xsm 12 migrate-support-check        fail   never pass
 test-armhf-armhf-libvirt     12 migrate-support-check        fail   never pass
 test-armhf-armhf-xl-multivcpu 13 saverestore-support-check    fail  never pass
 test-armhf-armhf-xl          12 migrate-support-check        fail   never pass
 test-armhf-armhf-xl          13 saverestore-support-check    fail   never pass
 test-armhf-armhf-xl-credit2  12 migrate-support-check        fail   never pass
 test-armhf-armhf-xl-credit2  13 saverestore-support-check    fail   never pass
 test-armhf-armhf-xl-cubietruck 12 migrate-support-check        fail never pass
 test-armhf-armhf-xl-cubietruck 13 saverestore-support-check    fail never pass
 test-armhf-armhf-xl-rtds     12 migrate-support-check        fail   never pass
 test-armhf-armhf-xl-rtds     13 saverestore-support-check    fail   never pass
 test-armhf-armhf-libvirt-raw 11 migrate-support-check        fail   never pass
 test-armhf-armhf-xl-vhd      11 migrate-support-check        fail   never pass
 test-armhf-armhf-xl-vhd      12 saverestore-support-check    fail   never pass

version targeted for testing:
 xen                  68a08e12c44435eb86600072b9e725e2387ce163
baseline version:
 xen                  ac9ff74f39a734756af90ccbb7184551f7b1e22a

Last test of basis   106959  2017-03-28 09:14:17 Z    1 days
Testing same since   106979  2017-03-29 16:19:31 Z    0 days    1 attempts

------------------------------------------------------------
People who touched revisions under test:
  Ian Jackson <Ian.Jackson@xxxxxxxxxxxxx>
  Jonathan Davies <jonathan.davies@xxxxxxxxxx>
  Thomas Sanders <thomas.sanders@xxxxxxxxxx>

jobs:
 build-amd64-xsm                                              pass    
 build-arm64-xsm                                              fail    
 build-armhf-xsm                                              pass    
 build-i386-xsm                                               pass    
 build-amd64-xtf                                              pass    
 build-amd64                                                  pass    
 build-arm64                                                  fail    
 build-armhf                                                  pass    
 build-i386                                                   pass    
 build-amd64-libvirt                                          pass    
 build-arm64-libvirt                                          blocked 
 build-armhf-libvirt                                          pass    
 build-i386-libvirt                                           pass    
 build-amd64-oldkern                                          pass    
 build-i386-oldkern                                           pass    
 build-amd64-prev                                             pass    
 build-i386-prev                                              pass    
 build-amd64-pvops                                            pass    
 build-arm64-pvops                                            fail    
 build-armhf-pvops                                            pass    
 build-i386-pvops                                             pass    
 build-amd64-rumprun                                          pass    
 build-i386-rumprun                                           pass    
 test-xtf-amd64-amd64-1                                       pass    
 test-xtf-amd64-amd64-2                                       pass    
 test-xtf-amd64-amd64-3                                       pass    
 test-xtf-amd64-amd64-4                                       pass    
 test-xtf-amd64-amd64-5                                       pass    
 test-amd64-amd64-xl                                          pass    
 test-arm64-arm64-xl                                          blocked 
 test-armhf-armhf-xl                                          pass    
 test-amd64-i386-xl                                           pass    
 test-amd64-amd64-xl-qemut-debianhvm-amd64-xsm                pass    
 test-amd64-i386-xl-qemut-debianhvm-amd64-xsm                 pass    
 test-amd64-amd64-libvirt-qemuu-debianhvm-amd64-xsm           pass    
 test-amd64-i386-libvirt-qemuu-debianhvm-amd64-xsm            pass    
 test-amd64-amd64-xl-qemuu-debianhvm-amd64-xsm                pass    
 test-amd64-i386-xl-qemuu-debianhvm-amd64-xsm                 pass    
 test-amd64-amd64-xl-qemut-stubdom-debianhvm-amd64-xsm        pass    
 test-amd64-i386-xl-qemut-stubdom-debianhvm-amd64-xsm         pass    
 test-amd64-amd64-libvirt-xsm                                 pass    
 test-arm64-arm64-libvirt-xsm                                 blocked 
 test-armhf-armhf-libvirt-xsm                                 pass    
 test-amd64-i386-libvirt-xsm                                  pass    
 test-amd64-amd64-xl-xsm                                      pass    
 test-arm64-arm64-xl-xsm                                      blocked 
 test-armhf-armhf-xl-xsm                                      fail    
 test-amd64-i386-xl-xsm                                       pass    
 test-amd64-amd64-qemuu-nested-amd                            fail    
 test-amd64-amd64-xl-pvh-amd                                  pass    
 test-amd64-i386-qemut-rhel6hvm-amd                           pass    
 test-amd64-i386-qemuu-rhel6hvm-amd                           pass    
 test-amd64-amd64-xl-qemut-debianhvm-amd64                    pass    
 test-amd64-i386-xl-qemut-debianhvm-amd64                     pass    
 test-amd64-amd64-xl-qemuu-debianhvm-amd64                    pass    
 test-amd64-i386-xl-qemuu-debianhvm-amd64                     pass    
 test-amd64-i386-freebsd10-amd64                              pass    
 test-amd64-amd64-xl-qemuu-ovmf-amd64                         pass    
 test-amd64-i386-xl-qemuu-ovmf-amd64                          pass    
 test-amd64-amd64-rumprun-amd64                               pass    
 test-amd64-amd64-xl-qemut-win7-amd64                         fail    
 test-amd64-i386-xl-qemut-win7-amd64                          fail    
 test-amd64-amd64-xl-qemuu-win7-amd64                         fail    
 test-amd64-i386-xl-qemuu-win7-amd64                          fail    
 test-armhf-armhf-xl-arndale                                  pass    
 test-amd64-amd64-xl-credit2                                  pass    
 test-arm64-arm64-xl-credit2                                  blocked 
 test-armhf-armhf-xl-credit2                                  pass    
 test-armhf-armhf-xl-cubietruck                               pass    
 test-amd64-i386-freebsd10-i386                               pass    
 test-amd64-i386-rumprun-i386                                 pass    
 test-amd64-amd64-qemuu-nested-intel                          pass    
 test-amd64-amd64-xl-pvh-intel                                pass    
 test-amd64-i386-qemut-rhel6hvm-intel                         pass    
 test-amd64-i386-qemuu-rhel6hvm-intel                         pass    
 test-amd64-amd64-libvirt                                     pass    
 test-arm64-arm64-libvirt                                     blocked 
 test-armhf-armhf-libvirt                                     pass    
 test-amd64-i386-libvirt                                      pass    
 test-amd64-amd64-migrupgrade                                 pass    
 test-amd64-i386-migrupgrade                                  pass    
 test-amd64-amd64-xl-multivcpu                                pass    
 test-arm64-arm64-xl-multivcpu                                blocked 
 test-armhf-armhf-xl-multivcpu                                pass    
 test-amd64-amd64-pair                                        pass    
 test-amd64-i386-pair                                         pass    
 test-amd64-amd64-libvirt-pair                                pass    
 test-amd64-i386-libvirt-pair                                 pass    
 test-amd64-amd64-amd64-pvgrub                                pass    
 test-amd64-amd64-i386-pvgrub                                 pass    
 test-amd64-amd64-pygrub                                      pass    
 test-arm64-arm64-libvirt-qcow2                               blocked 
 test-amd64-amd64-xl-qcow2                                    pass    
 test-armhf-armhf-libvirt-raw                                 pass    
 test-amd64-i386-xl-raw                                       pass    
 test-amd64-amd64-xl-rtds                                     fail    
 test-arm64-arm64-xl-rtds                                     blocked 
 test-armhf-armhf-xl-rtds                                     pass    
 test-amd64-i386-xl-qemut-winxpsp3-vcpus1                     pass    
 test-amd64-i386-xl-qemuu-winxpsp3-vcpus1                     pass    
 test-amd64-amd64-libvirt-vhd                                 pass    
 test-armhf-armhf-xl-vhd                                      pass    
 test-amd64-amd64-xl-qemut-winxpsp3                           pass    
 test-amd64-i386-xl-qemut-winxpsp3                            pass    
 test-amd64-amd64-xl-qemuu-winxpsp3                           pass    
 test-amd64-i386-xl-qemuu-winxpsp3                            pass    


------------------------------------------------------------
sg-report-flight on osstest.test-lab.xenproject.org
logs: /home/logs/logs
images: /home/logs/images

Logs, config files, etc. are available at
    http://logs.test-lab.xenproject.org/osstest/logs

Explanation of these reports, and of osstest in general, is at
    http://xenbits.xen.org/gitweb/?p=osstest.git;a=blob;f=README.email;hb=master
    http://xenbits.xen.org/gitweb/?p=osstest.git;a=blob;f=README;hb=master

Test harness code can be found at
    http://xenbits.xen.org/gitweb?p=osstest.git;a=summary


Not pushing.

------------------------------------------------------------
commit 68a08e12c44435eb86600072b9e725e2387ce163
Merge: ac9ff74 6d969f1
Author: Ian Jackson <ian.jackson@xxxxxxxxxxxxx>
Date:   Tue Mar 28 15:46:25 2017 +0100

    Merge XSA 206 branch

commit 6d969f128364509271773d6883694a432b8d961d
Author: Thomas Sanders <thomas.sanders@xxxxxxxxxx>
Date:   Mon Mar 27 14:36:34 2017 +0100

    oxenstored transaction conflicts: improve logging
    
    For information related to transaction conflicts, potentially frequent
    logging at "info" priority has been changed to "debug" priority, and
    once per two minutes there is an "info" priority summary.
    
    Additional detailed logging has been added at "debug" priority.
    
    Signed-off-by: Thomas Sanders <thomas.sanders@xxxxxxxxxx>

commit 4bcfc6881e9e2e58a29e9dcbcc16b67aff8e9f88
Author: Thomas Sanders <thomas.sanders@xxxxxxxxxx>
Date:   Fri Mar 24 19:55:03 2017 +0000

    oxenstored: don't wake to issue no conflict-credit
    
    In the main loop, when choosing the timeout for the select function
    call, we were setting it so as to wake up to issue conflict-credit to
    any domains that could accept it. When xenstore is idle, this would
    mean waking up every 50ms (by default) to do no work. With this
    commit, we check whether any domain is below its cap, and if not then
    we set the timeout for longer (the same timeout as before the
    conflict-protection feature was added).
    
    Signed-off-by: Thomas Sanders <thomas.sanders@xxxxxxxxxx>
    Reviewed-by: Jonathan Davies <jonathan.davies@xxxxxxxxxx>

commit b2a4105820807beb85cfeeaf2dc3c8bb93cc6d0c
Author: Thomas Sanders <thomas.sanders@xxxxxxxxxx>
Date:   Fri Mar 24 16:16:10 2017 +0000

    oxenstored: do not commit read-only transactions
    
    The packet telling us to end the transaction has always carried an
    argument telling us whether to commit.
    
    If the transaction made no modifications to the tree, now we ignore
    that argument and do not commit: it is just a waste of effort.
    
    This makes read-only transactions immune to conflicts, and means that
    we do not need to store any of their details in the history that is
    used for assigning blame for conflicts.
    
    We count a transaction as a read-only transaction only if it contains
    no operations that modified the tree.
    
    This means that (for example) a transaction that creates a new node
    then deletes it would NOT count as read-only, even though it makes no
    change overall. A more sophisticated algorithm could judge the
    transaction based on comparison of its initial and final states, but
    this would add complexity and computational cost.
    
    Signed-off-by: Thomas Sanders <thomas.sanders@xxxxxxxxxx>
    Reviewed-by: Jonathan Davies <jonathan.davies@xxxxxxxxxx>

commit 467c9de412f2ba80dc40a82b049abefc7bf8126a
Author: Thomas Sanders <thomas.sanders@xxxxxxxxxx>
Date:   Thu Mar 23 19:06:54 2017 +0000

    oxenstored: allow self-conflicts
    
    We already avoid inter-domain conflicts but now allow intra-domain
    conflicts.  Although there are no known practical examples of a domain
    that might perform operations that conflict with its own transactions,
    this is conceivable, so here we avoid changing those semantics
    unnecessarily.
    
    When a transaction commit fails with a conflict and we look through
    the history of commits to see which connection(s) to blame, ignore
    historical commits that were made by the same connection as the
    failing commit.
    
    Signed-off-by: Thomas Sanders <thomas.sanders@xxxxxxxxxx>
    Reviewed-by: Jonathan Davies <jonathan.davies@xxxxxxxxxx>

commit 8ddbce0426ca4bea0ebc80479aa14f22f590e7af
Author: Jonathan Davies <jonathan.davies@xxxxxxxxxx>
Date:   Thu Mar 23 14:28:16 2017 +0000

    oxenstored: blame the connection that caused a transaction conflict
    
    Blame each connection found to have made a commit that would cause this
    transaction to fail. Each blamed connection is penalised by having its
    conflict-credit decremented.
    
    Note the change in semantics for the replay function: we no longer stop 
after
    finding the first operation that can't be replayed. This allows us to 
identify
    all operations that conflicted with this transaction, not just the one that
    conflicted first.
    
    Signed-off-by: Jonathan Davies <jonathan.davies@xxxxxxxxxx>
    Signed-off-by: Thomas Sanders <thomas.sanders@xxxxxxxxxx>
    v1 Reviewed-by: Christian Lindig <christian.lindig@xxxxxxxxxx>
    
    Changes since v1:
     * use correct log levels for informational messages
    Changes since v2:
     * fix the blame algorithm and improve logging
       (fix was reviewed by Jonathan Davies)
    
    Signed-off-by: Thomas Sanders <thomas.sanders@xxxxxxxxxx>

commit 57718f2e14b7cf81515b91199d8cfead644c3197
Author: Jonathan Davies <jonathan.davies@xxxxxxxxxx>
Date:   Mon Mar 27 08:58:29 2017 +0000

    oxenstored: track commit history
    
    Since the list of historic activity cannot grow without bound, it is safe 
to use
    this to track commits.
    
    Signed-off-by: Jonathan Davies <jonathan.davies@xxxxxxxxxx>
    Reviewed-by: Thomas Sanders <thomas.sanders@xxxxxxxxxx>

commit cfd8bd7d403e2cad706f220a90383922e4ef2f61
Author: Thomas Sanders <thomas.sanders@xxxxxxxxxx>
Date:   Thu Mar 23 14:25:16 2017 +0000

    oxenstored: discard old commit-history on txn end
    
    The history of commits is to be used for working out which historical
    commit(s) (including atomic writes) caused conflicts with a
    currently-failing commit of a transaction. Any commit that was made
    before the current transaction started cannot be relevant. Therefore
    we never need to keep history from before the start of the
    longest-running transaction that is open at any given time: whenever a
    transaction ends (with or without a commit) then if it was the
    longest-running open transaction we can delete history up until start
    of the the next-longest-running open transaction.
    
    Some transactions might stay open for a very long time, so if any
    transaction exceeds conflict_max_history_seconds then we remove it
    from consideration in this context, and will not guarantee to keep
    remembering about historical commits made during such a transaction.
    
    We implement this by keeping a list of all open transactions that have
    not been open too long. When a transaction ends, we remove it from the
    list, along with any that have been open longer than the maximum; then
    we delete any history from before the start of the longest-running
    transaction remaining in the list.
    
    Signed-off-by: Thomas Sanders <thomas.sanders@xxxxxxxxxx>
    Reviewed-by: Jonathan Davies <jonathan.davies@xxxxxxxxxx>
    Reviewed-by: Christian Lindig <christian.lindig@xxxxxxxxxx>

commit 00c7d98d2df18e0d138bc6ef59e9e07bfc756d34
Author: Jonathan Davies <jonathan.davies@xxxxxxxxxx>
Date:   Thu Mar 23 14:20:33 2017 +0000

    oxenstored: only record operations with side-effects in history
    
    There is no need to record "read" operations as they will never cause 
another
    transaction to fail.
    
    Signed-off-by: Jonathan Davies <jonathan.davies@xxxxxxxxxx>
    Reviewed-by: Thomas Sanders <thomas.sanders@xxxxxxxxxx>
    
    Forward port to xen-unstable:
     * Remove Xenbus.Xb.Op.Restrict
    
    Signed-off-by: Ian Jackson <ian.jackson@xxxxxxxxxxxxx>

commit 83ee94f39cf51bb9bcf1affabfbee8257c559655
Author: Jonathan Davies <jonathan.davies@xxxxxxxxxx>
Date:   Tue Mar 14 13:20:07 2017 +0000

    oxenstored: support commit history tracking
    
    Add ability to track xenstore tree operations -- either non-transactional
    operations or committed transactions.
    
    For now, the call to actually retain commits is commented out because 
history
    can grow without bound.
    
    For now, we call record_commit for all non-transactional operations. A
    subsequent patch will make it retain only the ones with side-effects.
    
    Signed-off-by: Jonathan Davies <jonathan.davies@xxxxxxxxxx>
    Signed-off-by: Thomas Sanders <thomas.sanders@xxxxxxxxxx>
    Reviewed-by: Christian Lindig <christian.lindig@xxxxxxxxxx>

commit 85b1b18582011f3b80e78aa4c2439e2de5cd9419
Author: Jonathan Davies <jonathan.davies@xxxxxxxxxx>
Date:   Tue Mar 14 12:17:38 2017 +0000

    oxenstored: add transaction info relevant to history-tracking
    
    Specifically:
     * retain the original store (not just the root) in full transactions
     * store commit count at the time of the start of the transaction
    
    Signed-off-by: Jonathan Davies <jonathan.davies@xxxxxxxxxx>
    Reviewed-by: Thomas Sanders <thomas.sanders@xxxxxxxxxx>
    Reviewed-by: Ian Jackson <ian.jackson@xxxxxxxxxxxxx>
    Reviewed-by: Christian Lindig <christian.lindig@xxxxxxxxxx>

commit 8242b508cac8c2021c6683c510774e3330d97220
Author: Thomas Sanders <thomas.sanders@xxxxxxxxxx>
Date:   Tue Mar 14 12:15:52 2017 +0000

    oxenstored: ignore domains with no conflict-credit
    
    When processing connections, skip those from domains with no remaining
    conflict-credit.
    
    Also, issue a point of conflict-credit at regular intervals, the
    period being set by the configuration option "conflict-max-history-
    seconds".  When issuing conflict-credit, we give a point either to
    every domain at once (one each) or only to the single domain at the
    front of the queue, depending on the configuration option
    "conflict-rate-limit-is-aggregate".
    
    Signed-off-by: Thomas Sanders <thomas.sanders@xxxxxxxxxx>
    Reviewed-by: Jonathan Davies <jonathan.davies@xxxxxxxxxx>
    Reviewed-by: Christian Lindig <christian.lindig@xxxxxxxxxx>

commit 3efdade7a419a2d9e6a5c64af26df1993732eab4
Author: Thomas Sanders <thomas.sanders@xxxxxxxxxx>
Date:   Tue Mar 14 12:15:52 2017 +0000

    oxenstored: handling of domain conflict-credit
    
    This commit gives each domain a conflict-credit variable, which will
    later be used for limiting how often a domain can cause other domain's
    transaction-commits to fail.
    
    This commit also provides functions and data for manipulating domains
    and their conflict-credit, and checking whether they have credit.
    
    Signed-off-by: Thomas Sanders <thomas.sanders@xxxxxxxxxx>
    Reviewed-by: Jonathan Davies <jonathan.davies@xxxxxxxxxx>
    Reviewed-by: Christian Lindig <christian.lindig@xxxxxxxxxx>

commit 312d362398036cb4204330ef1cbf5f48ed9dfe05
Author: Thomas Sanders <thomas.sanders@xxxxxxxxxx>
Date:   Tue Mar 14 12:15:52 2017 +0000

    oxenstored: comments explaining some variables
    
    It took a while of reading and reasoning to work out what these are
    for, so here are comments to make life easier for everyone reading
    this code in future.
    
    Signed-off-by: Thomas Sanders <thomas.sanders@xxxxxxxxxx>
    Reviewed-by: Jonathan Davies <jonathan.davies@xxxxxxxxxx>
    Reviewed-by: Ian Jackson <ian.jackson@xxxxxxxxxxxxx>
    Reviewed-by: Christian Lindig <christian.lindig@xxxxxxxxxx>

commit 6234eae81e969aa2b9a0efadb6c57979a4bc2447
Author: Ian Jackson <ian.jackson@xxxxxxxxxxxxx>
Date:   Tue Mar 7 16:09:13 2017 +0000

    xenstored: Log when the write transaction rate limit bites
    
    Signed-off-by: Ian Jackson <ian.jackson@xxxxxxxxxxxxx>

commit 083996c46b2d3953177cd9d6b58ba38634fce545
Author: Ian Jackson <ian.jackson@xxxxxxxxxxxxx>
Date:   Tue Mar 7 16:09:12 2017 +0000

    xenstored: apply a write transaction rate limit
    
    This avoids a rogue client being about to stall another client (eg the
    toolstack) indefinitely.
    
    This is XSA-206.
    
    Signed-off-by: Ian Jackson <Ian.Jackson@xxxxxxxxxxxxx>
(qemu changes not included)

_______________________________________________
osstest-output mailing list
osstest-output@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/cgi-bin/mailman/listinfo/osstest-output

 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.