|
== Attendees ==
Lars Kurth, Stefano S, Bob B, Anthony P (Citrix)
Michael Glasgow (Oracle)
Jim Fehlig (Suse)
== Agenda ==
Outstanding Actions ...
ACTION (open): Lars will make mods to docs and upstream
Lars: have I missed the deadlines already
Bob: Still OK.
Lars: Do you have a sense on urgency
Bob: By mid march should should be fine
Lars: Used oxygen XML trial license, which has run out and is not looking forward to using text edit
Bob: There may be a recommended open source XML text edit tool
ACTION(Lars): ask on openstack docs IRC for advice
=== Updates ===
Anthony:
* Yesterday, there was an issue with the CI loop logger. Installer stopped working and had to change
a python package.
ACTION(Anthony): to sync with Bob to make sure we can recreate the setup
Jim:
* No updates. Pushed peer-2-peer migration patches last night in libvirt 1.3.2, which improves migration
* Statistics patches from virtual interfaces are now pushed (Jim has not double checked whether everything has been pushed).
* Release of 1.3.2 should be end of February
Lars: Should these changes be mentioned in the docs
Jim: not sure, but Joao has the details
Jim: migration out of the box did not used to work without special tweaks, but should work now
ACTION (Joao): verify and let Lars know of any docs changes
=== Security Breach in CI Loop ===
Summary: due to an increase in monthly cost, we tracked down an increase of network traffic, which has shown that the host has been compromised through Jenkins on Jan 7th. Unfortunately Bob, deleted
the Jenkins VM by accident such that we can do further forensics. A rogue process was running that was known to an abusive IP address.
Snippet from previous email exchange
Total bandwidth last month was 910GB
Total bandwidth this month was 15,970GB
One further piece of informationâ
root@libvirtxen-jenkins:/var/log# ifconfig eth0
eth0 Link encap:Ethernet HWaddr bc:76:4e:20:92:de
inet addr:104.239.169.124 Bcast:104.239.169.255 Mask:255.255.255.0
inet6 addr: 2001:4802:7803:101:be76:4eff:fe20:92de/64 Scope:Global
inet6 addr: fe80::be76:4eff:fe20:92de/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:350541818 errors:0 dropped:0 overruns:0 frame:0
TX packets:482460237992 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:224242958424 (224.2 GB) TX bytes:29211038447450 (29.2 TB)
It seems the vast majority of traffic has been transmitted traffic â there should be very little transmitted traffic from the VM.
I wonder if the VM has been compromised and was being used for a DDoS.
root@libvirtxen-jenkins:/var/log# netstat -n | grep -v 127 | less
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 1 104.239.169.124:44831 104.217.216.169:20582 SYN_SENT 1579/6b4bc32b47474c
http://www.abuseipdb.com/report-history/104.217.216.169
Iâm pretty confident the host has been compromised now on Jan 7th; I canât think of any other reason /tmp/su is a new symlink
to /usr/sbin/sshd; Jenkins wouldnât do that.
root@libvirtxen-jenkins:/tmp# ls -altr
-rwxrwxrwx 1 jenkins jenkins 1599477 Jan 7 11:16 6b4bc32b47474c58
lrwxrwxrwx 1 jenkins jenkins 14 Jan 7 20:45 su -> /usr/sbin/sshd
Someone got in through the Jenkins process and has added this 6b4bc32b47474c58 program as a backdoor to get back in.
Iâve shutdown the VM. Clearly a full redeploy and checking of Jenkins security is needed.
Lars: is there some monitoring we could do to avoid running up bills accidentally
Bob: Sure can be done
Lars: We used Credativ to monitor some of our other infrastructure. Could add this.
ACTION(Lars): Discuss with Ian and on the next call with Credativ
=== April OpenStack Summit ===
Bob will be going Mon-Wed
Michael lives in Austin and will go.
Lars will be in Austin for OSCON in mid May and could meet Michael then
Michael: anyone presenting?
Lars: No
Michael: We are trying with one paper ...
ACTION(Michael): send link to submission for voting
Lars: would also be happy to promote on Xen Project social media
| 
