[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [MirageOS-devel] Systematic crash on create_bounce_frame when hitting specific data allocation threshold
Hi Thomas,
I've tried a few things:
- `Gc.full_major()` unfortunately doesn't help.
- Looking at the address pointed by the RIP at the moment of the
exception, I can see this instruction:
25605f: e8 7c ad ff ff callq 250de0 <memcpy>
I don't know how useful can it be, considering that I can trigger the
same crash by actually changing the code and, in this case, the
references instruction would be something totally different (like a
movel, push). Maybe the instruction type is not much related to the
crash itself? I feel like it doesn't make much sense..
- Regarding in-lining the raw data in the code, I'm still working on it.
Actually I don't fully understand what you mean, are you suggesting
de-structuring the JSON format and insert into my code directly a
list/array of values? Or copying the JSON output directly inside my code
as a static variable? I've tried the latter and the error persists. I
will build the list of static values and see what happens.
Anyway, whatever I do with the retrieved JSON (even List.iter with an
empty function body), the unikernel crashes. I have the impression that
as soon as I try to access the variable containing the JSON value the
system crash is triggered.
Best regards,
Vittorio
Il 14/12/2016 13:45, Thomas Leonard ha scritto:
On 14 December 2016 at 11:35, Vittorio Cozzolino
<vittorio.cozzolino@xxxxxxxxx> wrote:
Hi,
I'm running a unikernel on XEN that basically accesses a remote DB, fetches
and computes some data, sends out the result. Apparently, if I try to fetch
and parse a JSON response greater than a empirically found threshold
(details at the bottom of the email), the PVM XEN unikernel just crashes and
this is wait I see when running sudo xl dmesg:
(XEN) Pagetable walk from 00000000002c9ff8:
(XEN) L4[0x000] = 00000010b5f67067 0000000000000567
(XEN) L3[0x000] = 00000010b5f68067 0000000000000568
(XEN) L2[0x001] = 00000010b5f6a067 000000000000056a
(XEN) L1[0x0c9] = 00100010b1ac9025 00000000000002c9
(XEN) domain_crash_sync called from entry.S: fault at ffff82d0802261be
create_bounce_frame+0x66/0x13a
(XEN) Domain 23 (vcpu#0) crashed on cpu#17:
(XEN) ----[ Xen-4.6.0 x86_64 debug=n Not tainted ]----
(XEN) CPU: 17
(XEN) RIP: e033:[<0000000000258cf4>]
(XEN) RFLAGS: 0000000000010206 EM: 1 CONTEXT: pv guest (d23v0)
(XEN) rax: 0000000000258cf0 rbx: 0000000000000000 rcx: 0000000000000073
(XEN) rdx: 0000000000442528 rsi: 0000000000000000 rdi: 00000000002ca018
(XEN) rbp: 00000000002ca1e8 rsp: 00000000002ca000 r8: 0000000000000002
(XEN) r9: 0000000000000007 r10: 0000000000000007 r11: 0000000000000000
(XEN) r12: 00000000002ca118 r13: 0000000000000000 r14: 00000011238fa000
(XEN) r15: 0000000000000074 cr0: 0000000080050033 cr4: 00000000001526e0
(XEN) cr3: 00000010b5f66000 cr2: 00000000002c9ff8
(XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: e02b cs: e033
(XEN) Guest stack trace from rsp=00000000002ca000:
(XEN) 00000000002ca118 0000000000000000 000000000025933f 0000000000000074
(XEN) 00000011238fa000 0000000000000000 00000000002ca118 00000000002ca1e8
(XEN) 0000000000000000 0000000000000000 0000000000000007 0000000000000007
(XEN) 0000000000000002 ffff800000000000 0000000000000073 0000000000442528
(XEN) 00000000002ca118 0000000000000000 ffffffffffffffff 0000000000256708
(XEN) 000000010000e030 0000000000010006 00000000002ca0c8 000000000000e02b
(XEN) 0000000000000ffc 3736353433323130 4645444342413938 4e4d4c4b4a494847
(XEN) 00000000002ca18b 00000000002ca1e8 00000000002ca18a 0000000000000074
(XEN) 00000000002566a0 00000000002ca118 00000000002561bc 7561662065676150
(XEN) 696c20746120746c 646461207261656e 3062642073736572 706972202c306433
(XEN) 2c38303736353220 3030207367657220 3030303030303030 202c383333616332
(XEN) 6533616332207073 735f72756f202c38 3030303030302070 3261633230303030
(XEN) 65646f63202c3866 ffffffff0a0d3020 0000000000000bfc 61665f686374614d
(XEN) 0200006572756c69 0000000000000073 0000000000000000 ffffffffffffffef
(XEN) 0000000000000000 00000000002ca2e8 0000000000000000 00000011238fa000
(XEN) 0000000000000074 00000000002ca338 000000000025630a 636f6c625f737953
(XEN) 0000003000000030 00000000002ca2e0 00000000002ca218 ffffffffffffffeb
(XEN) 0000000000db03d0 0000000000256708 00000000002ca338 00000000002ca3e8
(XEN) 00000000002ca2f8 ffffffffffffffe9 00000000000013fc 656e696665646e55
(XEN) 7372756365725f64 75646f6d5f657669 050000000000656c 00000000003df368
I've tried to destroy/create multiple times the same unikernel and I always
receive the same error. When running on Unix I don't bump into this issue,
even when fetching and parsing multiple MB of data.
By filling my code with logs, I figured out where exactly the unikernel
stops. Specifically during the JSON response parsing (I'm using the YoJson
library):
let directExtractionn rawJson =
Log.info (fun f -> f "Initializing direct extraction");
let json = Yojson.Basic.from_string rawJson in
let result = [json] |> filter_member "results" |> flatten |>
filter_member "series"
|> flatten |> filter_member "values" |> flatten in
List.map (
fun item ->
let datapoint = match item |> index 1 with
| `String a -> a
| `Float f -> string_of_float f
| `Int i -> string_of_float
(float_of_int i)
| `Bool b -> string_of_bool b
in
datapoint
) result |> computeAverage >>= fun aver ->
log_lwt ~inject:(fun f -> f "Result %f" aver)
I know that probably my code is not really optimized and clean but I'm quite
shocked to see that my unikernel crashes when it has to extract roughly 3500
datapoints (it's more or less the threshold at which it crashes). The
function computeAverage is not even called. If I run the same code on Unix I
can parse and process up to a 1M datapoints in less than a second. I've also
tried to increase the number of vcpus and memory, but nothing changed (16
vcpus and 4GB of memory).
I would like to add that this threshold changes depending on the host
machine:
- Machine A (Ubuntu 14.04, Xen 4.6.0, 32 Cores, 128 GB RAM, 10 GB Network
Interface) -> Threshold is around 107Kb
- Machine B (Debian 8.5, Xen 4.4.1, 4 cores, 8 GB RAM, 1GB Network
Interface) -> Threshold is around 33Kb
Can you simplify the case? For example, instead of fetching the JSON,
what if you in-line the raw data in your code and parse that?
Does adding a `Gc.full_major ()` just before the crash help? That
might indicate we're running out of memory and failing to run the GC
for some reason.
You could also use `objdump -d` or similar on the unikernel image and
see what the addresses in the stack trace correspond to.
--
Vittorio Cozzolino, M.Eng.
Technische Universität München - Institut für Informatik
Office 01.05.041
Boltzmannstr. 3, 85748 Garching, Germany
Tel: +49 89 289-17356
http://www.cm.in.tum.de/en/research-group/vittorio-cozzolino
Attachment:
smime.p7s
Description: Firma crittografica S/MIME
_______________________________________________
MirageOS-devel mailing list
MirageOS-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/cgi-bin/mailman/listinfo/mirageos-devel
|
