[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [MirageOS-devel] Systematic crash on create_bounce_frame when hitting specific data allocation threshold
Hi Anil,
thanks for the answer. On both my test machine I have Ocaml 4.02.3 and
Mirage 2.9.1.
Best regards,
Vittorio
Il 14/12/2016 12:46, Anil Madhavapeddy ha scritto:
On 14 Dec 2016, at 11:35, Vittorio Cozzolino <vittorio.cozzolino@xxxxxxxxx>
wrote:
Hi,
I'm running a unikernel on XEN that basically accesses a remote DB, fetches and
computes some data, sends out the result. Apparently, if I try to fetch and
parse a JSON response greater than a empirically found threshold (details at
the bottom of the email), the PVM XEN unikernel just crashes and this is wait I
see when running sudo xl dmesg:
(XEN) Pagetable walk from 00000000002c9ff8:
(XEN) L4[0x000] = 00000010b5f67067 0000000000000567
(XEN) L3[0x000] = 00000010b5f68067 0000000000000568
(XEN) L2[0x001] = 00000010b5f6a067 000000000000056a
(XEN) L1[0x0c9] = 00100010b1ac9025 00000000000002c9
(XEN) domain_crash_sync called from entry.S: fault at ffff82d0802261be
create_bounce_frame+0x66/0x13a
(XEN) Domain 23 (vcpu#0) crashed on cpu#17:
(XEN) ----[ Xen-4.6.0 x86_64 debug=n Not tainted ]----
(XEN) CPU: 17
(XEN) RIP: e033:[<0000000000258cf4>]
(XEN) RFLAGS: 0000000000010206 EM: 1 CONTEXT: pv guest (d23v0)
(XEN) rax: 0000000000258cf0 rbx: 0000000000000000 rcx: 0000000000000073
(XEN) rdx: 0000000000442528 rsi: 0000000000000000 rdi: 00000000002ca018
(XEN) rbp: 00000000002ca1e8 rsp: 00000000002ca000 r8: 0000000000000002
(XEN) r9: 0000000000000007 r10: 0000000000000007 r11: 0000000000000000
(XEN) r12: 00000000002ca118 r13: 0000000000000000 r14: 00000011238fa000
(XEN) r15: 0000000000000074 cr0: 0000000080050033 cr4: 00000000001526e0
(XEN) cr3: 00000010b5f66000 cr2: 00000000002c9ff8
(XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: e02b cs: e033
(XEN) Guest stack trace from rsp=00000000002ca000:
(XEN) 00000000002ca118 0000000000000000 000000000025933f 0000000000000074
(XEN) 00000011238fa000 0000000000000000 00000000002ca118 00000000002ca1e8
(XEN) 0000000000000000 0000000000000000 0000000000000007 0000000000000007
(XEN) 0000000000000002 ffff800000000000 0000000000000073 0000000000442528
(XEN) 00000000002ca118 0000000000000000 ffffffffffffffff 0000000000256708
(XEN) 000000010000e030 0000000000010006 00000000002ca0c8 000000000000e02b
(XEN) 0000000000000ffc 3736353433323130 4645444342413938 4e4d4c4b4a494847
(XEN) 00000000002ca18b 00000000002ca1e8 00000000002ca18a 0000000000000074
(XEN) 00000000002566a0 00000000002ca118 00000000002561bc 7561662065676150
(XEN) 696c20746120746c 646461207261656e 3062642073736572 706972202c306433
(XEN) 2c38303736353220 3030207367657220 3030303030303030 202c383333616332
(XEN) 6533616332207073 735f72756f202c38 3030303030302070 3261633230303030
(XEN) 65646f63202c3866 ffffffff0a0d3020 0000000000000bfc 61665f686374614d
(XEN) 0200006572756c69 0000000000000073 0000000000000000 ffffffffffffffef
(XEN) 0000000000000000 00000000002ca2e8 0000000000000000 00000011238fa000
(XEN) 0000000000000074 00000000002ca338 000000000025630a 636f6c625f737953
(XEN) 0000003000000030 00000000002ca2e0 00000000002ca218 ffffffffffffffeb
(XEN) 0000000000db03d0 0000000000256708 00000000002ca338 00000000002ca3e8
(XEN) 00000000002ca2f8 ffffffffffffffe9 00000000000013fc 656e696665646e55
(XEN) 7372756365725f64 75646f6d5f657669 050000000000656c 00000000003df368
I've tried to destroy/create multiple times the same unikernel and I always
receive the same error. When running on Unix I don't bump into this issue, even
when fetching and parsing multiple MB of data.
By filling my code with logs, I figured out where exactly the unikernel stops.
Specifically during the JSON response parsing (I'm using the YoJson library):
let directExtractionn rawJson =
Log.info (fun f -> f "Initializing direct extraction");
let json = Yojson.Basic.from_string rawJson in
let result = [json] |> filter_member "results" |> flatten |> filter_member
"series"
|> flatten |> filter_member "values" |> flatten in
List.map (
fun item ->
let datapoint = match item |> index 1 with
| `String a -> a
| `Float f -> string_of_float f
| `Int i -> string_of_float (float_of_int
i)
| `Bool b -> string_of_bool b
in
datapoint
) result |> computeAverage >>= fun aver ->
log_lwt ~inject:(fun f -> f "Result %f" aver)
I know that probably my code is not really optimized and clean but I'm quite
shocked to see that my unikernel crashes when it has to extract roughly 3500
datapoints (it's more or less the threshold at which it crashes). The function
computeAverage is not even called. If I run the same code on Unix I can parse
and process up to a 1M datapoints in less than a second. I've also tried to
increase the number of vcpus and memory, but nothing changed (16 vcpus and 4GB
of memory).
I would like to add that this threshold changes depending on the host machine:
- Machine A (Ubuntu 14.04, Xen 4.6.0, 32 Cores, 128 GB RAM, 10 GB Network
Interface) -> Threshold is around 107Kb
- Machine B (Debian 8.5, Xen 4.4.1, 4 cores, 8 GB RAM, 1GB Network Interface)
-> Threshold is around 33Kb
Hi Vittorio,
This is clearly a bug :-) To help us diagnose it, which version of Mirage were
you building with? The released Mirage 2.9.1, or the development branch on
https://github.com/mirage/mirage-dev? Also which version of OCaml was in use?
regards,
Anil
--
Vittorio Cozzolino, M.Eng.
Technische Universität München - Institut für Informatik
Office 01.05.041
Boltzmannstr. 3, 85748 Garching, Germany
Tel: +49 89 289-17356
http://www.cm.in.tum.de/en/research-group/vittorio-cozzolino
Attachment:
smime.p7s
Description: Firma crittografica S/MIME
_______________________________________________
MirageOS-devel mailing list
MirageOS-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/cgi-bin/mailman/listinfo/mirageos-devel
|
