[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[SECURITY] Albatross console out of memory



Dear everyone,

please find a (fixed) security issue for albatross-console, discovered and reported by Edwin Török.


Have a nice weekend,

Hannes

Background
==========

Albatross-console reads the console output from multiple unikernel tenders (solo5-hvt). This console output can be retrieved using albatross-client.

The console protocol is fairly simple: the unikernel invokes a PUTS hypercall, which sends arbitrary bytes of given length to the unikernel tender (host, typically solo5-hvt), which writes them to a file descriptor.

albatross_console reads this output, and assumes it will be newline delimited, and keeps at most 1024 lines.


Problem description
===================

This helps guard against unlimited memory usage from runaway/long running unikernels, however it doesn't guard against malicious (or buggy) unikernels.

The problem is this line in albatross_console:

```
let rec loop () =
        Lwt_io.read_line channel
```

Unfortunately Lwt_io.read_line doesn't take a parameter to limit the size of the line that is read, so it is very easy for a unikernel to exhaust the memory of albatross_console: all it needs to do is to write a lot of bytes without ever writing a newline.

Tested with the Debian packages, but confirmed the above code to be present in latest master too:
```
$ /usr/libexec/albatross/albatross-console --version
version v2.3.0-9-g5b14787 protocol version 5
```

Impact
======

A typical attack will look like this in albatross_console's logs:
```
Jun 07 16:41:18 ubuntu22 albatross-console[13721]: albatross-console:
[WARNING] disconnected
Jun 07 16:41:42 ubuntu22 albatross-console[13721]: albatross-console:
[ERROR] exception Unix.Unix_error(Unix.EBADF, "check_descriptor", "")
while writing
Jun 07 16:42:09 ubuntu22 albatross-console[13721]: albatross-console:
[WARNING] disconnected
Jun 07 16:44:07 ubuntu22 albatross-console[13721]: albatross-console:
[ERROR] :mine error while reading Out of memory
Jun 07 16:55:43 ubuntu22 albatross-console[13721]: albatross-console:
[ERROR] exception Unix.Unix_error(Unix.EBADF, "check_descriptor", "")
while writing
Jun 07 16:57:20 ubuntu22 albatross-console[13721]: albatross-console:
[ERROR] exception Unix.Unix_error(Unix.EPIPE, "send", "") while writing
Jun 07 16:57:20 ubuntu22 albatross-console[13721]: albatross-console:
[ERROR] exception Unix.Unix_error(Unix.EPIPE, "send", "") while writing
```

While the attack is happening albatross_console will also be very slow to react to the console of other unikernels, and will use increasing amounts of memory until an out of memory exception is raised.

Albatross_console stays running in my tests (obviously the bad unikernel will no longer have a functioning console, but that is to be expected).

However using so much memory can have an effect on other services running on the host.

Workaround
==========

Set MemoryMax=1G in the service file of albatross_console to limit the amount of memory a runaway console can use, so at least it fails sooner.

However in practice this just keeps albatross_console using 100% CPU without raising an Out of Memory exception, although it stays within 1G of memory.

There is no known workaround for the DoS attack, in theory Lwt should be switching promises when another one becomes runnable, but it doesn't provide fairness mechanisms.

Solution
========

Use albatross in version 2.5.0 or above. Another solution is to apply the patch dhttps://github.com/robur-coop/albatross/commit/d01805796ec710691a701c01ed3f0e4cd284a161 manually.

Binary builds of version 2.5.0 are available from https://builds.robur.coop

Timeline
========

2025-06-07: issue discovered by Edwin Török as part of the HACKSAT25
challenge
2025-06-07: initial email sent to hacksat@xxxxxxxxxxxx
2025-06-11: initial email sent to albatross maintainer
2025-08-15: albatross 2.5.0 released with the patch



 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.