Re: [Wg-test-framework] colo outage

["Paul L. George" <pgeorge@xxxxxxxxxx>]

Ian,

We were there on Friday afternoon doing prep fwork for the rimava 0/1 
installation.  

We were in front of the rack early in the day moving a set of rails for the 
rimava servers. They are 4U away from Newcastle. I can assure you,  no power 
button was pushed.  

I am not crazy about how the power cords fit in the PDU's so we always check to 
see that all the power lights are on before we leave.  They looked like they 
were all on when we left.

We were there around that time, so it is possible that a power cord got nudged. 
 But Newcastle's plug is in outlet 20 and rimava is in outlet 13.  A completely 
different bank.

I will see if APC makes some kind of retention bar or cord lock down.

It could have been us but as I said all lights were on before we left and we 
did not hear any startup fan noise while we were working.  Believe me you can 
tell if a server power cycles while you are there.

Sent from my iPhone
Paul l. George
508-450-9332

PLG Enterprises, Inc.
42 East Main Street
Westborough, MA 01581


> On Apr 29, 2015, at 7:57 AM, Ian Jackson <Ian.Jackson@xxxxxxxxxxxxx> wrote:
> 
> Thanks, Ian, for picking up the pieces.
> 
> Ian Campbell writes ("colo outage"):
>> I noticed over the weekend that newcastle (the primary of the two vm
>> servers in the test lab) was not accessible via ssh, ping, etc.
>> Therefore neither was the main osstest vm and most of the service VMs
> 
> I have an xterm here which says:
> 
>  Broadcast message from root@newcastle (Fri Apr 24 19:06:25 2015):
>  Power button pressed
>  The system is going down for system halt NOW!
> 
>> Anyone want to fess up? 
> 
> It is entirely unacceptable that things like this are happening and we
> are only finding out why and how by forensics after the fact.
> 
> Lars, please can you find out from the colo admin staff who was
> signed in for physical access at 15:06 EDT on Friday.
> 
> Also, I would like to impose a moratorium on making any changes on
> Fridays.[1]  So: no rack installation work, and no software
> reconfiguration work.
> 
> Thanks,
> Ian.

_______________________________________________
Wg-test-framework mailing list
Wg-test-framework@xxxxxxxxxxxxxxxxxxxx
http://lists.xenproject.org/cgi-bin/mailman/listinfo/wg-test-framework