From mirageos-devel-bounces@lists.xenproject.org Fri Aug 15 14:56:10 2025 Return-path: Envelope-to: archives@lists.xenproject.org Delivery-date: Fri, 15 Aug 2025 14:56:10 +0000 Received: from list by lists.xenproject.org with outflank-mailman.1083575.1443154 (Exim 4.92) (envelope-from ) id 1umvqW-000325-31; Fri, 15 Aug 2025 14:55:56 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 1083575.1443154; Fri, 15 Aug 2025 14:55:56 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1umvqV-00031y-Vc; Fri, 15 Aug 2025 14:55:55 +0000 Received: by outflank-mailman (input) for mailman id 1083575; Fri, 15 Aug 2025 14:55:55 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1umvqU-00031r-Qh for mirageos-devel@lists.xenproject.org; Fri, 15 Aug 2025 14:55:55 +0000 Received: from mail.mehnert.org (mail.mehnert.org [213.73.89.200]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id f0a11aa3-79e7-11f0-a328-13f23c93f187; Fri, 15 Aug 2025 16:55:53 +0200 (CEST) Received: from [192.168.42.80] (i5C74C0FA.versanet.de [92.116.192.250]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "hannes@mehnert.org", Issuer "mehnert root CA" (not verified)) by mail.mehnert.org (Postfix) with ESMTPS id 3363C26402; Fri, 15 Aug 2025 16:55:52 +0200 (CEST) X-BeenThere: mirageos-devel@lists.xenproject.org List-Id: Developer list for MirageOS List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: mirageos-devel-bounces@lists.xenproject.org Precedence: list Sender: "MirageOS-devel" X-Inumbo-ID: f0a11aa3-79e7-11f0-a328-13f23c93f187 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mehnert.org; s=mail; t=1755269752; bh=Fk+bpwJAKClGC4rUtJmhEgjKAzhFIcWwzjkKur+O7VQ=; h=Date:To:Cc:From:Subject; b=QDyk6cDvBr8PX21KDSlEZFEug574he0EwOhFpC4RUlqhqg6tJXqnZSPdBzR7pA6Os lh6rWwZV6k4IoSNGEFWuK9PS8+63XSPnnoVDbQZyHPaiM50bt6Mm1x7rWyBGPGJEaJ 20G101/HRp1ixNN5MmxIP8+Q8HeisW9LuVPH8mv1mPeUcKHV2+GtkrrWx/cLFVECeX dppeMyfk6JjboYeSDvoBiqmt9s0urBfQPJaFGABGsxHNX1x1bMCnuEY74UP3iLhapM 38fJ0B8B0dQopz5QL+0ixcqwofmjB23mMM0ZhCbqE+/uqkh6utdsTMR1FtOWpOKwpB MSbCEPxkFJKQw== Message-ID: <5ad44c02-7ce2-452b-ac56-7ea8c70fb03b@mehnert.org> Date: Fri, 15 Aug 2025 16:55:51 +0200 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird To: Mirageos Devel Content-Language: en-US Cc: =?UTF-8?B?RWR3aW4gVMO2csO2aw==?= From: Hannes Mehnert Subject: [SECURITY] Albatross console out of memory Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Dear everyone, please find a (fixed) security issue for albatross-console, discovered and reported by Edwin Török. Have a nice weekend, Hannes Background ========== Albatross-console reads the console output from multiple unikernel tenders (solo5-hvt). This console output can be retrieved using albatross-client. The console protocol is fairly simple: the unikernel invokes a PUTS hypercall, which sends arbitrary bytes of given length to the unikernel tender (host, typically solo5-hvt), which writes them to a file descriptor. albatross_console reads this output, and assumes it will be newline delimited, and keeps at most 1024 lines. Problem description =================== This helps guard against unlimited memory usage from runaway/long running unikernels, however it doesn't guard against malicious (or buggy) unikernels. The problem is this line in albatross_console: ``` let rec loop () = Lwt_io.read_line channel ``` Unfortunately Lwt_io.read_line doesn't take a parameter to limit the size of the line that is read, so it is very easy for a unikernel to exhaust the memory of albatross_console: all it needs to do is to write a lot of bytes without ever writing a newline. Tested with the Debian packages, but confirmed the above code to be present in latest master too: ``` $ /usr/libexec/albatross/albatross-console --version version v2.3.0-9-g5b14787 protocol version 5 ``` Impact ====== A typical attack will look like this in albatross_console's logs: ``` Jun 07 16:41:18 ubuntu22 albatross-console[13721]: albatross-console: [WARNING] disconnected Jun 07 16:41:42 ubuntu22 albatross-console[13721]: albatross-console: [ERROR] exception Unix.Unix_error(Unix.EBADF, "check_descriptor", "") while writing Jun 07 16:42:09 ubuntu22 albatross-console[13721]: albatross-console: [WARNING] disconnected Jun 07 16:44:07 ubuntu22 albatross-console[13721]: albatross-console: [ERROR] :mine error while reading Out of memory Jun 07 16:55:43 ubuntu22 albatross-console[13721]: albatross-console: [ERROR] exception Unix.Unix_error(Unix.EBADF, "check_descriptor", "") while writing Jun 07 16:57:20 ubuntu22 albatross-console[13721]: albatross-console: [ERROR] exception Unix.Unix_error(Unix.EPIPE, "send", "") while writing Jun 07 16:57:20 ubuntu22 albatross-console[13721]: albatross-console: [ERROR] exception Unix.Unix_error(Unix.EPIPE, "send", "") while writing ``` While the attack is happening albatross_console will also be very slow to react to the console of other unikernels, and will use increasing amounts of memory until an out of memory exception is raised. Albatross_console stays running in my tests (obviously the bad unikernel will no longer have a functioning console, but that is to be expected). However using so much memory can have an effect on other services running on the host. Workaround ========== Set MemoryMax=1G in the service file of albatross_console to limit the amount of memory a runaway console can use, so at least it fails sooner. However in practice this just keeps albatross_console using 100% CPU without raising an Out of Memory exception, although it stays within 1G of memory. There is no known workaround for the DoS attack, in theory Lwt should be switching promises when another one becomes runnable, but it doesn't provide fairness mechanisms. Solution ======== Use albatross in version 2.5.0 or above. Another solution is to apply the patch dhttps://github.com/robur-coop/albatross/commit/d01805796ec710691a701c01ed3f0e4cd284a161 manually. Binary builds of version 2.5.0 are available from https://builds.robur.coop Timeline ======== 2025-06-07: issue discovered by Edwin Török as part of the HACKSAT25 challenge 2025-06-07: initial email sent to hacksat@parsimoni.co 2025-06-11: initial email sent to albatross maintainer 2025-08-15: albatross 2.5.0 released with the patch From mirageos-devel-bounces@lists.xenproject.org Thu Aug 28 07:29:56 2025 Return-path: Envelope-to: archives@lists.xenproject.org Delivery-date: Thu, 28 Aug 2025 07:29:56 +0000 Received: from list by lists.xenproject.org with outflank-mailman.1097689.1451926 (Exim 4.92) (envelope-from ) id 1urX4r-0000DX-VK; Thu, 28 Aug 2025 07:29:45 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 1097689.1451926; Thu, 28 Aug 2025 07:29:45 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1urX4r-0000DQ-SI; Thu, 28 Aug 2025 07:29:45 +0000 Received: by outflank-mailman (input) for mailman id 1097689; Thu, 28 Aug 2025 07:29:44 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1urX4p-0008Ri-Kj for mirageos-devel@lists.xenproject.org; Thu, 28 Aug 2025 07:29:44 +0000 Received: from mail.mehnert.org (mail.mehnert.org [213.73.89.200]) by se1-gles-flk1.inumbo.com (Halon) with ESMTPS id c1e31430-83e0-11f0-ae26-e363de0e7a9e; Thu, 28 Aug 2025 09:29:41 +0200 (CEST) Received: from [192.168.42.80] (i5C74C367.versanet.de [92.116.195.103]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "hannes@mehnert.org", Issuer "mehnert root CA" (not verified)) by mail.mehnert.org (Postfix) with ESMTPS id 3E6181B2E1 for ; Thu, 28 Aug 2025 09:29:38 +0200 (CEST) X-BeenThere: mirageos-devel@lists.xenproject.org List-Id: Developer list for MirageOS List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: mirageos-devel-bounces@lists.xenproject.org Precedence: list Sender: "MirageOS-devel" X-Inumbo-ID: c1e31430-83e0-11f0-ae26-e363de0e7a9e DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mehnert.org; s=mail; t=1756366178; bh=Bk1hD9Uyvp8/6WH4SBNQsC4bvB0Mgh+19c3SL15HOs8=; h=Date:To:From:Subject; b=SxJLNnFWsRjP7ZGmYZcyOJsDrPrJ+wASKpoywCNRpKK+U3MZu72ggkYwrrvJJ4Se8 RpiXFFPrMMKlw+Rc+m9iNBEbCfYFSIJoRUFx6YZ0//OPS5h8wZB/RpoV2uiZ3X8WsC 4FwokfC0axfeELxNbPR5HSxLbfz1vtAdrEhERGtGXhHUS4iT6bNfexZazYdFGARPUD g7DznGXXnviy6qQhln4eSFGj4Tynv2+2pUgfm+LYYnTJhQ6dESdiFIUXWo8bQTUHrS eeo9l3qjdpPdOOJmPR5P10DNgHMI+NG2HBliY7aJiGqBLh0eKyz8RNaAwwGsxVcDhU Ncho+yLeygeUA== Message-ID: <7478e5b1-2ed7-4963-81be-6a5e66911a9e@mehnert.org> Date: Thu, 28 Aug 2025 09:29:37 +0200 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird To: Mirageos Devel Content-Language: en-US From: Hannes Mehnert Subject: MirageOS community meeting on Mon Sep 1st 10 - 12 CEST Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Dear everyone, I hope you had a great summer break. It is my pleasure to invite you to the upcoming MirageOS community meeting on Monday Sep 1st between 10:00 and 12:00 CEST. We meet as usual at https://meet.jit.si/MirageOS -- the agenda and notes will be discussed and transcribed at https://pad.data.coop/To6IOSeNSOK9kFVlgo7XWw?both# Looking forward to see you soon, Hannes