Xen project Mailing List

Re: [Xen-devel] [PATCH 2/2] call ao_how callback explicitly

To: "Ian Jackson" <Ian.Jackson@xxxxxxxxxxxxx>

From: "Bamvor Jian Zhang" <bjzhang@xxxxxxxx>

Date: Fri, 08 Nov 2013 06:17:40 -0700

Cc: Jim Fehlig <JFEHLIG@xxxxxxxx>, xen-devel@xxxxxxxxxxxxx, ian.campbell@xxxxxxxxxx, stefano.stabellini@xxxxxxxxxxxxx

Delivery-date: Fri, 08 Nov 2013 13:17:53 +0000

List-id: Xen developer discussion <xen-devel.lists.xen.org>

>>>Ian Jackson <Ian.Jackson@xxxxxxxxxxxxx> wrote: > Bamvor Jian Zhang writes ("Re: [Xen-devel] [PATCH 2/2] call ao_how callback > explicitly"): > ... > > > I'm not sure what you mean. Under what conditions do you not see the > > > callback happen ? The existing machinery is supposed to take care of > > > that. Specifically, in the patch you're changing, > > > libxl__ao_complete_check_progress_reports puts the ao onto > > > egc->aos_for_callback. On the return path from the libxl event > > > function back to the application, we are supposed to call > > > libxl__egc_cleanup, which in turn calls egc_run_callbacks, which > > > should pick up the aos on aos_for_callback. libxl__egc_cleanup is > > > called from the event loops in libxl_event_wait and > > > libxl__ao_inprogress and from the macro EGC_FREE. It should be > > > impossible for a path to be missed because inside libxl one needs an > > > egc to call libxl__ao_complete. > ... > > the libvirt libxl driver register the event handler through > > libxl_event_register_callbacks, > > Right. > > > so the libxl_event_wait could not get the event. > > I understand from this that you're calling libxl_event_wait ? (Rather > than that you're saying that you're not calling libxl_event_wait, but > doing so wouldn't help.) i am talking about why i could not call libxl_event_wait. > > But, as I understand it, you're passing an ao_how which specifies a > callback, rather than the generation of an event. So I think > libxl_event_wait is irrelevant. agree. > > > and if the ao_how is used, the libxl__egc_cleanup in > > "if ( poller )" statement will not be called either. > > even if i could expose the libxl_egc_cleanup to libvirt, i still do not > > know when should i call it? > > No, you shouldn't need to call it explicitly and doing so won't help. > > > i do not know if there is a event triggered > > before the ao_how callback should be called. (is there a fd event when > > the async operation complete?). > > It's the other way around. The async operation completes, probably, > because of an fd event. So I think the call flow should be: > > 1. Your application notices the fd is readable (say) > 2. Your application calls libxl_osevent_occurred_fd. > 3. libxl_osevent_occurred_fd allocates an egc on entry. > 4. In libxl, libxl_osevent_occurred_fd calls whatever internal > functions are supposed to respond to the fd. If appropriate, > this will cause some ao to complete. All of these calls > will receive the same egc as created in step 3. I.e.: > 5. In libxl, something calls libxl__ao_complete. > 6. libxl__ao_complete puts the ao on the egc's aos_for_callback > list. > 7. The call stack unwinds inside libxl, eventually returning > to libxl_osevent_occurred_fd. > 8. libxl_osevent_occurred_fd uses EGC_FREE on exit, and EGC_FREE > contains a call to libxl__egc_cleanup. > 9. libxl__egc_cleanup calls egc_run_callbacks which finds the ao > on the aos_for_callback list and makes your application > callback. with log and gdb, i could see above flow except the ao in egc->aops_for_callback is empty. and from gdb i could know that the egc is different between libxl__ao_complete_check_progress_reports and fd event(here is watchfd_callback). libxl__ao_complete_check_progress_reports (gdb) p egc $1 = (libxl__egc *) 0x7fb1fd03f500 watchfd_callback (gdb) p egc $4 = (libxl__egc *) 0x7ffffb291960 after check the code in libxl_osevent_occurred_fd, the egc is created in watchfd_callback. so, EGC_FREE could not handle the ao added by libxl__ao_complete_check_progress_reports. the latter egc is created by AO_CREATE in do_domain_create. is it correct? if so, could we use the same egc all the time? here is the full libxl.log: libxl: debug: libxl_create.c:1230:do_domain_create: ao 0x7fb1d0001e50: create: how=0x7fb1f6448e80 callback=0x7fb1f6210fb0 poller=(nil) libxl: debug: libxl_device.c:257:libxl__device_disk_set_backend: Disk vdev=hda spec.backend=tap libxl: debug: libxl_create.c:675:initiate_domain_create: running bootloader libxl: debug: libxl_bootloader.c:321:libxl__bootloader_run: not a PV domain, skipping bootloader libxl: debug: libxl_event.c:608:libxl__ev_xswatch_deregister: watch w=0x7fb1d0003078: deregister unregistered libxl: debug: libxl_numa.c:475:libxl__get_numa_candidate: New best NUMA placement candidate found: nr_nodes=1, nr_cpus=8, nr_vcpus=9, free_memkb=908 libxl: detail: libxl_dom.c:195:numa_place_domain: NUMA placement candidate with 1 nodes, 8 cpus and 908 KB free selected xc: detail: elf_parse_binary: phdr: paddr=0x100000 memsz=0x9f0c8 xc: detail: elf_parse_binary: memory: 0x100000 -> 0x19f0c8 xc: info: VIRTUAL MEMORY ARRANGEMENT: Loader: 0000000000100000->000000000019f0c8 Modules: 0000000000000000->0000000000000000 TOTAL: 0000000000000000->000000001f800000 ENTRY ADDRESS: 0000000000100000 xc: info: PHYSICAL MEMORY ALLOCATION: 4KB PAGES: 0x0000000000000200 2MB PAGES: 0x00000000000000fb 1GB PAGES: 0x0000000000000000 xc: detail: elf_load_binary: phdr 0 at 0x7fb1f2f48000 -> 0x7fb1f2fddf4d libxl: debug: libxl_device.c:257:libxl__device_disk_set_backend: Disk vdev=hda spec.backend=tap libxl: debug: libxl_event.c:559:libxl__ev_xswatch_register: watch w=0x7fb1d0002558 wpath=/local/domain/0/backend/vbd/2/768/state token=3/0: register slotnum=3 libxl: debug: libxl_create.c:1243:do_domain_create: ao 0x7fb1d0001e50: inprogress: poller=(nil), flags=i libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x7fb1d0002558 wpath=/local/domain/0/backend/vbd/2/768/state token=3/0: event epath=/local/domain/0/backend/vbd/2/768/state libxl: debug: libxl_event.c:643:devstate_watch_callback: backend /local/domain/0/backend/vbd/2/768/state wanted state 2 ok libxl: debug: libxl_event.c:596:libxl__ev_xswatch_deregister: watch w=0x7fb1d0002558 wpath=/local/domain/0/backend/vbd/2/768/state token=3/0: deregister slotnum=3 libxl: debug: libxl_event.c:608:libxl__ev_xswatch_deregister: watch w=0x7fb1d0002558: deregister unregistered libxl: debug: libxl_device.c:959:device_hotplug: calling hotplug script: /etc/xen/scripts/block add libxl: debug: libxl_dm.c:1218:libxl__spawn_local_dm: Spawning device-model /usr/lib/xen/bin/qemu-system-i386 with arguments: libxl: debug: libxl_dm.c:1220:libxl__spawn_local_dm: /usr/lib/xen/bin/qemu-system-i386 libxl: debug: libxl_dm.c:1220:libxl__spawn_local_dm: -xen-domid libxl: debug: libxl_dm.c:1220:libxl__spawn_local_dm: 2 libxl: debug: libxl_dm.c:1220:libxl__spawn_local_dm: -chardev libxl: debug: libxl_dm.c:1220:libxl__spawn_local_dm: socket,id=libxl-cmd,path=/var/run/xen/qmp-libxl-2,server,nowait libxl: debug: libxl_dm.c:1220:libxl__spawn_local_dm: -mon libxl: debug: libxl_dm.c:1220:libxl__spawn_local_dm: chardev=libxl-cmd,mode=control libxl: debug: libxl_dm.c:1220:libxl__spawn_local_dm: -name libxl: debug: libxl_dm.c:1220:libxl__spawn_local_dm: bjz_04_sles11_sp2 libxl: debug: libxl_dm.c:1220:libxl__spawn_local_dm: -vnc libxl: debug: libxl_dm.c:1220:libxl__spawn_local_dm: 127.0.0.1:0,to=99 libxl: debug: libxl_dm.c:1220:libxl__spawn_local_dm: -global libxl: debug: libxl_dm.c:1220:libxl__spawn_local_dm: isa-fdc.driveA= libxl: debug: libxl_dm.c:1220:libxl__spawn_local_dm: -vga libxl: debug: libxl_dm.c:1220:libxl__spawn_local_dm: cirrus libxl: debug: libxl_dm.c:1220:libxl__spawn_local_dm: -global libxl: debug: libxl_dm.c:1220:libxl__spawn_local_dm: vga.vram_size_mb=8 libxl: debug: libxl_dm.c:1220:libxl__spawn_local_dm: -boot libxl: debug: libxl_dm.c:1220:libxl__spawn_local_dm: order=c libxl: debug: libxl_dm.c:1220:libxl__spawn_local_dm: -net libxl: debug: libxl_dm.c:1220:libxl__spawn_local_dm: none libxl: debug: libxl_dm.c:1220:libxl__spawn_local_dm: -M libxl: debug: libxl_dm.c:1220:libxl__spawn_local_dm: xenfv libxl: debug: libxl_dm.c:1220:libxl__spawn_local_dm: -m libxl: debug: libxl_dm.c:1220:libxl__spawn_local_dm: 504 libxl: debug: libxl_dm.c:1220:libxl__spawn_local_dm: -drive libxl: debug: libxl_dm.c:1220:libxl__spawn_local_dm: file=/var/lib/xen/images_2/bjz_04_sles11_sp2/disk0.raw,if=ide,index=0,media=disk,format=raw,cache=writeback libxl: debug: libxl_event.c:559:libxl__ev_xswatch_register: watch w=0x7fb1d00032b0 wpath=/local/domain/0/device-model/2/state token=3/1: register slotnum=3 libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x7fb1d00032b0 wpath=/local/domain/0/device-model/2/state token=3/1: event epath=/local/domain/0/device-model/2/state libxl: debug: libxl_event.c:596:libxl__ev_xswatch_deregister: watch w=0x7fb1d00032b0 wpath=/local/domain/0/device-model/2/state token=3/1: deregister slotnum=3 libxl: debug: libxl_event.c:472:watchfd_callback: watch epath=/local/domain/0/device-model/2/state token=3/1: empty slot libxl: debug: libxl_event.c:608:libxl__ev_xswatch_deregister: watch w=0x7fb1d00032b0: deregister unregistered libxl: debug: libxl_qmp.c:707:libxl__qmp_initialize: connected to /var/run/xen/qmp-libxl-2 libxl: debug: libxl_qmp.c:299:qmp_handle_response: message type: qmp libxl: debug: libxl_qmp.c:555:qmp_send_prepare: next qmp command: '{ "execute": "qmp_capabilities", "id": 1 } ' libxl: debug: libxl_qmp.c:299:qmp_handle_response: message type: return libxl: debug: libxl_qmp.c:555:qmp_send_prepare: next qmp command: '{ "execute": "query-chardev", "id": 2 } ' libxl: debug: libxl_qmp.c:299:qmp_handle_response: message type: return libxl: debug: libxl_qmp.c:555:qmp_send_prepare: next qmp command: '{ "execute": "query-vnc", "id": 3 } ' libxl: debug: libxl_qmp.c:299:qmp_handle_response: message type: return libxl: debug: libxl_event.c:559:libxl__ev_xswatch_register: watch w=0x7fb1d0007b58 wpath=/local/domain/0/backend/vif/2/0/state token=3/2: register slotnum=3 libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x7fb1d0007b58 wpath=/local/domain/0/backend/vif/2/0/state token=3/2: event epath=/local/domain/0/backend/vif/2/0/state libxl: debug: libxl_event.c:643:devstate_watch_callback: backend /local/domain/0/backend/vif/2/0/state wanted state 2 ok libxl: debug: libxl_event.c:596:libxl__ev_xswatch_deregister: watch w=0x7fb1d0007b58 wpath=/local/domain/0/backend/vif/2/0/state token=3/2: deregister slotnum=3 libxl: debug: libxl_event.c:608:libxl__ev_xswatch_deregister: watch w=0x7fb1d0007b58: deregister unregistered libxl: debug: libxl_device.c:959:device_hotplug: calling hotplug script: /etc/xen/scripts/vif-bridge online libxl: debug: libxl_event.c:472:watchfd_callback: watch epath=/local/domain/0/backend/vif/2/0/state token=3/2: empty slot libxl: debug: libxl_event.c:1737:libxl__ao_progress_report: ao 0x7fb1d0001e50: progress report: ignored libxl: debug: libxl_event.c:1569:libxl__ao_complete: ao 0x7fb1d0001e50: complete, rc=0 libxl: debug: libxl_event.c:1599:libxl__ao_complete_check_progress_reports: ao 0x7fb1d0001e50: complete for callback libxl: debug: libxl_event.c:1173:egc_run_callbacks: ao 0x7fb1d0001e50: completion callback libxl: debug: libxl_event.c:1541:libxl__ao__destroy: ao 0x7fb1d0001e50: destroy libxl: debug: libxl_event.c:559:libxl__ev_xswatch_register: watch w=0x7fb1d00029b0 wpath=@releaseDomain token=3/3: register slotnum=3 libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x7fb1d00029b0 wpath=@releaseDomain token=3/3: event epath=@releaseDomain libxl: debug: libxl.c:1000:domain_death_xswatch_callback: [evg=0x7fb1d0006330:2] from domid=2 nentries=1 rc=1 libxl: debug: libxl.c:1011:domain_death_xswatch_callback: [evg=0x7fb1d0006330:2] got=domaininfos[0] got->domain=2 libxl: debug: libxl.c:1038:domain_death_xswatch_callback: exists shutdown_reported=0 dominf.flags=ffff0022 libxl: debug: libxl.c:1004:domain_death_xswatch_callback: [evg=0] all reported libxl: debug: libxl.c:1068:domain_death_xswatch_callback: domain death search done xc: debug: hypercall buffer: total allocations:0 total releases:0 xc: debug: hypercall buffer: current allocations:0 maximum allocations:0 xc: debug: hypercall buffer: cache current size:0 xc: debug: hypercall buffer: cache hits:0 misses:0 toobig:0 best regards bamvor > The purpose of the "poller" machinery is so that if at step 5 we > decide to return an event structure, rather than making a callback, a > concurrent call to libxl_event_wait (on a different thread) can be > woken up. > > Can you turn on libxl's debugging and send me a trace of it losing the > event ? > > Ian. > > _______________________________________________ > Xen-devel mailing list > Xen-devel@xxxxxxxxxxxxx > http://lists.xen.org/xen-devel > > _______________________________________________ Xen-devel mailing list Xen-devel@xxxxxxxxxxxxx http://lists.xen.org/xen-devel

©2013 Xen Project, A Linux Foundation Collaborative Project. All Rights Reserved.
Linux Foundation is a registered trademark of The Linux Foundation.
Xen Project is a trademark of The Linux Foundation.