[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Wg-openstack] Minutes of Feb 10, 2016 Openstack WG call

== Attendees ==

Lars Kurth, Stefano S, Bob B, Anthony P (Citrix)

Michael Glasgow (Oracle)

Jim Fehlig (Suse)

== Agenda ==

Outstanding Actions ...

ACTION (open): Lars will make mods to docs and upstream 

Lars: have I missed the deadlines already

Bob: Still OK. 

Lars: Do you have a sense on urgency

Bob: By mid march should should be fine

Lars: Used oxygen XML trial license, which has run out and is not looking forward to using text edit

Bob: There may be a recommended open source XML text edit tool

ACTION(Lars): ask on openstack docs IRC for advice

=== Updates ===


* Yesterday, there was an issue with the CI loop logger. Installer stopped working and had to change

a python package. 

ACTION(Anthony): to sync with Bob to make sure we can recreate the setup


* No updates. Pushed peer-2-peer migration patches last night in libvirt 1.3.2, which improves migration

* Statistics patches from virtual interfaces are now pushed (Jim has not double checked whether everything has been pushed). 

* Release of 1.3.2 should be end of February

Lars: Should these changes be mentioned in the docs

Jim: not sure, but Joao has the details

Jim: migration out of the box did not used to work without special tweaks, but should work now

ACTION (Joao): verify and let Lars know of any docs changes

=== Security Breach in CI Loop ===

Summary: due to an increase in monthly cost, we tracked down an increase of network traffic, which has shown that the host has been compromised through Jenkins on Jan 7th. Unfortunately Bob, deleted the Jenkins VM by accident such that we can do further forensics. A rogue process was running that was known to an abusive IP address.

Snippet from previous email exchange

Total bandwidth last month was 910GB

Total bandwidth this month was 15,970GB




One further piece of informationâ


root@libvirtxen-jenkins:/var/log# ifconfig eth0

eth0      Link encap:Ethernet  HWaddr bc:76:4e:20:92:de

          inet addr:  Bcast:  Mask:

          inet6 addr: 2001:4802:7803:101:be76:4eff:fe20:92de/64 Scope:Global

          inet6 addr: fe80::be76:4eff:fe20:92de/64 Scope:Link


          RX packets:350541818 errors:0 dropped:0 overruns:0 frame:0

          TX packets:482460237992 errors:0 dropped:0 overruns:0 carrier:0

          collisions:0 txqueuelen:1000

          RX bytes:224242958424 (224.2 GB)  TX bytes:29211038447450 (29.2 TB)


It seems the vast majority of traffic has been transmitted traffic â there should be very little transmitted traffic from the VM.

I wonder if the VM has been compromised and was being used for a DDoS.

root@libvirtxen-jenkins:/var/log# netstat -n | grep -v 127 | less

Active Internet connections (w/o servers)

Proto Recv-Q Send-Q Local Address           Foreign Address         State

tcp        0      1   SYN_SENT    1579/6b4bc32b47474c




Iâm pretty confident the host has been compromised now on Jan 7th; I canât think of any other reason /tmp/su is a new symlink to /usr/sbin/sshd; Jenkins wouldnât do that.


root@libvirtxen-jenkins:/tmp# ls -altr

-rwxrwxrwx  1 jenkins jenkins 1599477 Jan  7 11:16 6b4bc32b47474c58

lrwxrwxrwx  1 jenkins jenkins      14 Jan  7 20:45 su -> /usr/sbin/sshd


Someone got in through the Jenkins process and has added this 6b4bc32b47474c58 program as a backdoor to get back in.


Iâve shutdown the VM.  Clearly a full redeploy and checking of Jenkins security is needed.

Lars: is there some monitoring we could do to avoid running up bills accidentally

Bob: Sure can be done

Lars: We used Credativ to monitor some of our other infrastructure. Could add this.

ACTION(Lars): Discuss with Ian and on the next call with Credativ 

=== April OpenStack Summit ===

Bob will be going Mon-Wed

Michael lives in Austin and will go. 

Lars will be in Austin for OSCON in mid May and could meet Michael then

Michael: anyone presenting?

Lars: No

Michael: We are trying with one paper ...

ACTION(Michael): send link to submission for voting

Lars: would also be happy to promote on Xen Project social media


Wg-openstack mailing list



Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.