Re: [Wg-openstack] Minutes of Feb 10, 2016 Openstack WG call

[Joao Martins <joao.m.martins@xxxxxxxxxx>]

On 02/10/2016 05:32 PM, Lars Kurth wrote:
> == Attendees ==
> 
> Lars Kurth, Stefano S, Bob B, Anthony P (Citrix)
> 
> Michael Glasgow (Oracle)
> 
> Jim Fehlig (Suse)
> 
> 
> == Agenda ==
> 
> Outstanding Actions ...
> 
> 
> ACTION (open): Lars will make mods to docs and upstream 
> 
> Lars: have I missed the deadlines already
> 
> Bob: Still OK. 
> 
> Lars: Do you have a sense on urgency
> 
> Bob: By mid march should should be fine
> 
> Lars: Used oxygen XML trial license, which has run out and is not looking
> forward to using text edit
> 
> Bob: There may be a recommended open source XML text edit tool
> 
> 
> ACTION(Lars): ask on openstack docs IRC for advice
> 
> 
> === Updates ===
> 
> Anthony:
> 
> * Yesterday, there was an issue with the CI loop logger. Installer stopped
> working and had to change
> 
> a python package. 
> 
> 
> ACTION(Anthony): to sync with Bob to make sure we can recreate the setup
> 
> 
> Jim:
> 
> * No updates. Pushed peer-2-peer migration patches last night in libvirt 
> 1.3.2,
> which improves migration
> 
> * Statistics patches from virtual interfaces are now pushed (Jim has not 
> double
> checked whether everything has been pushed). 
> 
> * Release of 1.3.2 should be end of February
> 
> 
> Lars: Should these changes be mentioned in the docs
> 
> Jim: not sure, but Joao has the details
> 
> Jim: migration out of the box did not used to work without special tweaks, but
> should work now
> 
> 
> ACTION (Joao): verify and let Lars know of any docs changes
> 
Good that you folks pointed out about the docs. I just sent a followup on the
changes required
(http://lists.xen.org/archives/html/wg-openstack/2016-02/msg00004.html).

Thanks!

> 
> === Security Breach in CI Loop ===
> 
> Summary: due to an increase in monthly cost, we tracked down an increase of
> network traffic, which has shown that the host has been compromised through
> Jenkins on Jan 7th. Unfortunately Bob, deleted the Jenkins VM by accident such
> that we can do further forensics. A rogue process was running that was known 
> to
> an abusive IP address.
> 
> 
> Snippet from previous email exchange
> 
> 
> Total bandwidth last month was 910GB
> 
> Total bandwidth this month was 15,970GB
> 
> image001.png
> 
>  
> 
> image002.png
> 
> 
> One further piece of information…
> 
>  
> 
> root@libvirtxen-jenkins:/var/log# ifconfig eth0
> 
> eth0      Link encap:Ethernet  HWaddr bc:76:4e:20:92:de
> 
>           inet addr:104.239.169.124  Bcast:104.239.169.255  Mask:255.255.255.0
> 
>           inet6 addr: 2001:4802:7803:101:be76:4eff:fe20:92de/64 Scope:Global
> 
>           inet6 addr: fe80::be76:4eff:fe20:92de/64 Scope:Link
> 
>           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
> 
>           RX packets:350541818 errors:0 dropped:0 overruns:0 frame:0
> 
>           TX packets:482460237992 errors:0 dropped:0 overruns:0 carrier:0
> 
>           collisions:0 txqueuelen:1000
> 
>           RX bytes:224242958424 (224.2 GB)  TX bytes:29211038447450 (29.2 TB)
> 
>  
> 
> It seems the vast majority of traffic has been transmitted traffic – there
> should be very little transmitted traffic from the VM.
> 
> I wonder if the VM has been compromised and was being used for a DDoS.
> 
> 
> root@libvirtxen-jenkins:/var/log# netstat -n | grep -v 127 | less
> 
> Active Internet connections (w/o servers)
> 
> Proto Recv-Q Send-Q Local Address           Foreign Address         State
> 
> tcp        0      1 104.239.169.124:44831   104.217.216.169:20582   SYN_SENT  
>  
> 1579/6b4bc32b47474c
> 
>  
> 
> http://www.abuseipdb.com/report-history/104.217.216.169
> 
>  
> 
> I’m pretty confident the host has been compromised now on Jan 7^th ; I can’t
> think of any other reason /tmp/su is a new symlink to /usr/sbin/sshd; Jenkins
> wouldn’t do that.
> 
>  
> 
> root@libvirtxen-jenkins:/tmp# ls -altr
> 
> -rwxrwxrwx  1 jenkins jenkins 1599477 Jan  7 11:16 6b4bc32b47474c58
> 
> lrwxrwxrwx  1 jenkins jenkins      14 Jan  7 20:45 su -> /usr/sbin/sshd
> 
>  
> 
> Someone got in through the Jenkins process and has added this 6b4bc32b47474c58
> program as a backdoor to get back in.
> 
>  
> 
> I’ve shutdown the VM.  Clearly a full redeploy and checking of Jenkins 
> security
> is needed.
> 
> 
> Lars: is there some monitoring we could do to avoid running up bills 
> accidentally
> 
> Bob: Sure can be done
> 
> Lars: We used Credativ to monitor some of our other infrastructure. Could add 
> this.
> 
> 
> ACTION(Lars): Discuss with Ian and on the next call with Credativ 
> 
> 
> === April OpenStack Summit ===
> 
> Bob will be going Mon-Wed
> 
> Michael lives in Austin and will go. 
> 
> Lars will be in Austin for OSCON in mid May and could meet Michael then
> 
> 
> Michael: anyone presenting?
> 
> Lars: No
> 
> Michael: We are trying with one paper ...
> 
> 
> ACTION(Michael): send link to submission for voting
> 
> 
> Lars: would also be happy to promote on Xen Project social media
> 
>  
> 
> 
> 
> 
> 
> _______________________________________________
> Wg-openstack mailing list
> Wg-openstack@xxxxxxxxxxxxxxxxxxxx
> http://lists.xenproject.org/cgi-bin/mailman/listinfo/wg-openstack
> 

_______________________________________________
Wg-openstack mailing list
Wg-openstack@xxxxxxxxxxxxxxxxxxxx
http://lists.xenproject.org/cgi-bin/mailman/listinfo/wg-openstack