[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [MirageOS-devel] Systematic crash on create_bounce_frame when hitting specific data allocation threshold
All set! Thanks for the help!
Vittorio
Il 14/12/2016 17:18, Thomas Leonard ha scritto:
On 14 December 2016 at 16:13, Vittorio Cozzolino
<vittorio.cozzolino@xxxxxxxxx> wrote:
Ok,
I've built a "lightweight" version of my original Unikernel, there is
basically only the code necessary to trigger the crash.
Do I copy-paste the code here or somewhere else? I have already an issue
open here https://github.com/mirage/mirage/issues/731, should I updated it
and copy there the unikernel code?
Great! You can either add it to the issue or, if you want more space /
multiple files, there's a "Gist" link at the top of the GitHub page
that can be a convenient place to paste stuff.
Best regards,
Vittorio
Il 14/12/2016 16:36, Thomas Leonard ha scritto:
On 14 December 2016 at 15:12, Vittorio Cozzolino
<vittorio.cozzolino@xxxxxxxxx> wrote:
Hi Thomas,
I've tried a few things:
- `Gc.full_major()` unfortunately doesn't help.
- Looking at the address pointed by the RIP at the moment of the
exception,
I can see this instruction:
25605f: e8 7c ad ff ff callq 250de0 <memcpy>
I don't know how useful can it be, considering that I can trigger the
same
crash by actually changing the code and, in this case, the references
instruction would be something totally different (like a movel, push).
Maybe
the instruction type is not much related to the crash itself? I feel like
it
doesn't make much sense..
It would be more interesting to know the caller of this function, etc.
It's possible that it branched to an invalid address and started
executing random code at some point, so the actual location of the
crash might not help but things further up the stack might be useful.
- Regarding in-lining the raw data in the code, I'm still working on it.
Actually I don't fully understand what you mean, are you suggesting
de-structuring the JSON format and insert into my code directly a
list/array
of values? Or copying the JSON output directly inside my code as a static
variable? I've tried the latter and the error persists. I will build the
list of static values and see what happens.
Yes, I mean putting the json in your code, as
let raw_json = "..."
If it still crashes with this, you can remove the database call. If it
still crashes, you can remove networking completely from your
unikernel. You can eliminate a lot of code quickly this way.
If you can get a unikernel that just parses a JSON string and crashes,
other people can try it too and it should be easy to find the cause.
Anyway, whatever I do with the retrieved JSON (even List.iter with an
empty
function body), the unikernel crashes. I have the impression that as soon
as
I try to access the variable containing the JSON value the system crash
is
triggered.
Best regards,
Vittorio
Il 14/12/2016 13:45, Thomas Leonard ha scritto:
On 14 December 2016 at 11:35, Vittorio Cozzolino
<vittorio.cozzolino@xxxxxxxxx> wrote:
Hi,
I'm running a unikernel on XEN that basically accesses a remote DB,
fetches
and computes some data, sends out the result. Apparently, if I try to
fetch
and parse a JSON response greater than a empirically found threshold
(details at the bottom of the email), the PVM XEN unikernel just
crashes
and
this is wait I see when running sudo xl dmesg:
(XEN) Pagetable walk from 00000000002c9ff8:
(XEN) L4[0x000] = 00000010b5f67067 0000000000000567
(XEN) L3[0x000] = 00000010b5f68067 0000000000000568
(XEN) L2[0x001] = 00000010b5f6a067 000000000000056a
(XEN) L1[0x0c9] = 00100010b1ac9025 00000000000002c9
(XEN) domain_crash_sync called from entry.S: fault at ffff82d0802261be
create_bounce_frame+0x66/0x13a
(XEN) Domain 23 (vcpu#0) crashed on cpu#17:
(XEN) ----[ Xen-4.6.0 x86_64 debug=n Not tainted ]----
(XEN) CPU: 17
(XEN) RIP: e033:[<0000000000258cf4>]
(XEN) RFLAGS: 0000000000010206 EM: 1 CONTEXT: pv guest (d23v0)
(XEN) rax: 0000000000258cf0 rbx: 0000000000000000 rcx:
0000000000000073
(XEN) rdx: 0000000000442528 rsi: 0000000000000000 rdi:
00000000002ca018
(XEN) rbp: 00000000002ca1e8 rsp: 00000000002ca000 r8:
0000000000000002
(XEN) r9: 0000000000000007 r10: 0000000000000007 r11:
0000000000000000
(XEN) r12: 00000000002ca118 r13: 0000000000000000 r14:
00000011238fa000
(XEN) r15: 0000000000000074 cr0: 0000000080050033 cr4:
00000000001526e0
(XEN) cr3: 00000010b5f66000 cr2: 00000000002c9ff8
(XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: e02b cs: e033
(XEN) Guest stack trace from rsp=00000000002ca000:
(XEN) 00000000002ca118 0000000000000000 000000000025933f
0000000000000074
(XEN) 00000011238fa000 0000000000000000 00000000002ca118
00000000002ca1e8
(XEN) 0000000000000000 0000000000000000 0000000000000007
0000000000000007
(XEN) 0000000000000002 ffff800000000000 0000000000000073
0000000000442528
(XEN) 00000000002ca118 0000000000000000 ffffffffffffffff
0000000000256708
(XEN) 000000010000e030 0000000000010006 00000000002ca0c8
000000000000e02b
(XEN) 0000000000000ffc 3736353433323130 4645444342413938
4e4d4c4b4a494847
(XEN) 00000000002ca18b 00000000002ca1e8 00000000002ca18a
0000000000000074
(XEN) 00000000002566a0 00000000002ca118 00000000002561bc
7561662065676150
(XEN) 696c20746120746c 646461207261656e 3062642073736572
706972202c306433
(XEN) 2c38303736353220 3030207367657220 3030303030303030
202c383333616332
(XEN) 6533616332207073 735f72756f202c38 3030303030302070
3261633230303030
(XEN) 65646f63202c3866 ffffffff0a0d3020 0000000000000bfc
61665f686374614d
(XEN) 0200006572756c69 0000000000000073 0000000000000000
ffffffffffffffef
(XEN) 0000000000000000 00000000002ca2e8 0000000000000000
00000011238fa000
(XEN) 0000000000000074 00000000002ca338 000000000025630a
636f6c625f737953
(XEN) 0000003000000030 00000000002ca2e0 00000000002ca218
ffffffffffffffeb
(XEN) 0000000000db03d0 0000000000256708 00000000002ca338
00000000002ca3e8
(XEN) 00000000002ca2f8 ffffffffffffffe9 00000000000013fc
656e696665646e55
(XEN) 7372756365725f64 75646f6d5f657669 050000000000656c
00000000003df368
I've tried to destroy/create multiple times the same unikernel and I
always
receive the same error. When running on Unix I don't bump into this
issue,
even when fetching and parsing multiple MB of data.
By filling my code with logs, I figured out where exactly the unikernel
stops. Specifically during the JSON response parsing (I'm using the
YoJson
library):
let directExtractionn rawJson =
Log.info (fun f -> f "Initializing direct extraction");
let json = Yojson.Basic.from_string rawJson in
let result = [json] |> filter_member "results" |> flatten
|>
filter_member "series"
|> flatten |> filter_member "values" |> flatten in
List.map (
fun item ->
let datapoint = match item |> index 1
with
| `String a -> a
| `Float f -> string_of_float f
| `Int i -> string_of_float
(float_of_int i)
| `Bool b -> string_of_bool b
in
datapoint
) result |> computeAverage >>= fun aver ->
log_lwt ~inject:(fun f -> f "Result %f" aver)
I know that probably my code is not really optimized and clean but I'm
quite
shocked to see that my unikernel crashes when it has to extract roughly
3500
datapoints (it's more or less the threshold at which it crashes). The
function computeAverage is not even called. If I run the same code on
Unix I
can parse and process up to a 1M datapoints in less than a second. I've
also
tried to increase the number of vcpus and memory, but nothing changed
(16
vcpus and 4GB of memory).
I would like to add that this threshold changes depending on the host
machine:
- Machine A (Ubuntu 14.04, Xen 4.6.0, 32 Cores, 128 GB RAM, 10 GB
Network
Interface) -> Threshold is around 107Kb
- Machine B (Debian 8.5, Xen 4.4.1, 4 cores, 8 GB RAM, 1GB Network
Interface) -> Threshold is around 33Kb
Can you simplify the case? For example, instead of fetching the JSON,
what if you in-line the raw data in your code and parse that?
Does adding a `Gc.full_major ()` just before the crash help? That
might indicate we're running out of memory and failing to run the GC
for some reason.
You could also use `objdump -d` or similar on the unikernel image and
see what the addresses in the stack trace correspond to.
--
Vittorio Cozzolino, M.Eng.
Technische Universität München - Institut für Informatik
Office 01.05.041
Boltzmannstr. 3, 85748 Garching, Germany
Tel: +49 89 289-17356
http://www.cm.in.tum.de/en/research-group/vittorio-cozzolino
--
Vittorio Cozzolino, M.Eng.
Technische Universität München - Institut für Informatik
Office 01.05.041
Boltzmannstr. 3, 85748 Garching, Germany
Tel: +49 89 289-17356
http://www.cm.in.tum.de/en/research-group/vittorio-cozzolino
--
Vittorio Cozzolino, M.Eng.
Technische Universität München - Institut für Informatik
Office 01.05.041
Boltzmannstr. 3, 85748 Garching, Germany
Tel: +49 89 289-17356
http://www.cm.in.tum.de/en/research-group/vittorio-cozzolino
Attachment:
smime.p7s
Description: Firma crittografica S/MIME
_______________________________________________
MirageOS-devel mailing list
MirageOS-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/cgi-bin/mailman/listinfo/mirageos-devel
|
