[Xen-devel] [OSSTEST PATCH 00/26] Reduce hard power cycles

[Ian Jackson <ian.jackson@xxxxxxxxxxxxx>]

We have been having trouble with PDU port relays sticking.  We vaguely
suspect that this is happening in Massachusetts due to the
lower-voltage (120V, and therefore higher-current) electricity
(compared to Cambridge, 240V, where the PDUs remain reliable).

To mitigate this issue (and generally prolong hardware life), try to
avoid hard power cycling test boxes when we can.  Specifically, try a
reboot with ssh first, and if that fails try the IMPI if provided.  If
the IMPI doesn't work either or is not provided, hard power cycle.

Patches 01-25 are ready I think, and have been tested.  There is a
difficulty with the FreeBSD host install, which is discussed in patch
26 which is definitely not ready in this form.

Ian Jackson (26):
  power: Osstest::PDU::*: drop Exporter blocks
  power: Osstest::PDU::*: use parent ::unsupported
  power: ipmi_try: Use `use parent' instead of Exporter
  power: Osstest::PDU::*: reuse default new method where applicable
  power handling: Break out power_cycle_parse_method
  power handling: Introduce power_reboot_attempts
  power: PowerApproaches replaces $ho->{PowerMethobjs}
  power: power_reboot_attempts: Try multiple approaches
  power: Provide `try_off' pdu method; deprecate ipmi_try
  power: New ILOM/PDU arrangements - try just IPMI
  power: Do not sleep between power off and power on if not needed
  power: Provide `ssh' power method
  power: Try rebooting hosts with ssh first
  power: Document Power* host properties and power methods
  power: Honour approach_re in power_cycle and mg-hosts power
  ts-host-powercycle: Use a lighter-weight method if available
  power: Rename target_reboot_force from target_reboot_hard
  power: power_reboot_attempts: Honour an $approach_re
  ts-logs-capture: power: try ILOM first
  power: Use `Combined' as Name for PDU+ILOM approach
  power: Record approach used for power cycles in runvars
  Executive: Break out span_colour
  Executive: Export $grey_* with :colours
  sg-report-host-history: Move SET LOCAL into transaction
  sg-report-host-history: Show used power approach(es)
  DO NOT APPLY power: ts-freebsd-host-install: Use power_reboot_attempts

 Osstest/Executive.pm       |  11 +++-
 Osstest/PDU/eth008.pm      |  11 +---
 Osstest/PDU/guest.pm       |  16 ++---
 Osstest/PDU/ipmi.pm        |  11 +---
 Osstest/PDU/ipmi_try.pm    |  14 +----
 Osstest/PDU/ipmiextra.pm   |  11 +---
 Osstest/PDU/manual.pm      |  16 +----
 Osstest/PDU/msw.pm         |  11 +---
 Osstest/PDU/pause.pm       |  11 +---
 Osstest/PDU/ssh.pm         |  74 ++++++++++++++++++++++
 Osstest/PDU/statedb.pm     |   2 +
 Osstest/PDU/try_off.pm     |  70 +++++++++++++++++++++
 Osstest/PDU/unsupported.pm |  16 ++---
 Osstest/PDU/xenuse.pm      |  16 +----
 Osstest/TestSupport.pm     | 154 +++++++++++++++++++++++++++++++++++++++------
 README                     |  77 +++++++++++++++++++++++
 README.dev                 |   4 +-
 mg-hosts                   |  14 +++--
 sg-report-host-history     |  67 ++++++++++++++++++--
 ts-freebsd-host-install    |  16 ++---
 ts-host-install            |  19 +++---
 ts-host-powercycle         |   6 +-
 ts-logs-capture            |   2 +-
 23 files changed, 482 insertions(+), 167 deletions(-)
 create mode 100644 Osstest/PDU/ssh.pm
 create mode 100644 Osstest/PDU/try_off.pm

-- 
2.11.0


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel