[Xen-devel] RFC/Proposal: Partial `libxenctrl` API/ABI stabilisation

[Ian Campbell <ian.campbell@xxxxxxxxxx>]

% Partial `libxenctrl` API/ABI stabilisation
% Ian Campbell <ian.campbell@xxxxxxxxxx>
% Draft A

# Introduction

The low-level `libxenctrl` library currently has an unstable API and
ABI and some of the hypervisor interfaces which it exposes are
similarly unstable.

However several external projects use some of these interfaces (at
least: qemu and kexec-tools), which presents problems for distros and
other consumers. In particular the need for spurious rebuilds of those
components against newer versions of Xen and difficulty supporting
parallel installation of different versions of Xen (useful during
upgrade).

This document considers whether parts of `libxenctrl` can be split out
into new libraries with more useful API and ABI guarantees.

XXX: I haven't yet done a full pass over the list of symbols in
`libxenctrl` to categorise them and decide where they belong. I
thought I would get some early feedback first and just picked a few
representative examples for each library.

# ABI/API Compatibility Classes

Compatibility opportunities:

* `LAPI` -- Library API
* `LABI` -- Library ABI
* `HABI` -- Hypervisor ABI (includes ioctls).

Each can either be Stable (`S`), Unstable (`U`) or don't care (`x`,
because made moot by higher level interface class, e.g. no real point
from an application PoV to a Stable HABI behind a unstable LABI).

Stable vs. Unstable is across major hypervisor version bump, always
aim to be stable across point releases.

For libraries Stable means `SONAME` major component, but forward
compatible only (i.e. old app on new library works, new app on old
library may not link due to e.g. new symbols. This is the normal
SONAME expectation). XXX find a link to the sort of scheme I mean.

`HABI` may include ioctls used to access those ABIs, typically these
are already required to be stable by the relevant OS maintainers.

A library interface may fall into one of these categories (I expect
there are others and we may not want any library to use some of even
these):

* Unstable `LAPI` (`Uxx`)
    * The Wild West
    * Current Examples: `libxenctrl`

* Stable `LAPI`, Unstable `LABI` (`SUx`)
    * Requires application rebuild for a new Xen version, but not
      application source code changes.
    * Current Examples: `libxenlight`

* Stable `LAPI`, Stable `LABI`, Unstable `HABI` (`SSU`)
    * Library can be switched out via dynamic linking across
      hypervisor upgrade (mechanism TBD, pos. distro specific, e.g.
      symlink switched on boot). Requires application/daemon restart
      but not rebuild (but changing hypervisor version involves a
      reboot anyway).
    * Current Examples: None??

* Stable `LAPI`, Stable `LABI`, Stable `HABI` (`SSS`)
    * Applications linked again a library will function against any
      hypervisor version.
    * Current Examples: None??

# Goal

Provide `SSU` or `SSS` interfaces for major external consumers of
current libxenctrl functionality.

Out of scope (for now): `SSU` or `SSS` interfaces for consumers of
`libxenlight`. Rationale: Lets focus on fixing external consumers of
libxenctrl first.

# Major External Consumers of `libxenctrl`

* qemu
* kexec tools
* in guest tools e.g. users of `libxenstore`, `libvchan`, and by
  extension `grant table` and `event channel` functionality. NB:
  `libxenstore` is already `SSU` or `SSS` (XXX?)

# `libxenctrl` symbols

Gathered by:

    nm tools/libxc/libxenctrl.so | grep ' [Tt] ' | cut -f 3 -d \ | sort -u

`libxenctrl` today exposes many symbols which look to be internal. We
should consider also reducing that set by using
`__attribute__((visibility("hidden")))`.

The following proposes some functional groupings via some proposed
split library names. In some cases we may also wish to consider
replacing an API with one which can be properly maintained going
forwards. e.g.:

- perhaps replacing domctl's used by qemu with new stable
  hypercall ABIs and reflecting that in new library APIs.
- perhaps exposing more constrained versions of some broad interfaces
  for external users.

XXX: Change `xc_*` namespacing as well as library names?

## `libxenhypercall`

Core open/close interface, "make a hypercall" functionality, hypercall
buffers.

All other libraries likely depend on this. Applications do as well in
order to access open/close interface at least.

    - xc_interface_close
    - xc_interface_is_fake (???)
    - xc_interface_open
    - xc_hypercall_buffer_array_create
    - xc_hypercall_buffer_array_destroy

## `libxenevtchn`

Interacting with `/dev/xen/evtchn`

    - xc_evtchn_alloc_unbound
    - xc_evtchn_bind_interdomain
    - xc_evtchn_bind_unbound_port
    - xc_evtchn_bind_virq
    - xc_evtchn_close
    - xc_evtchn_fd
    - xc_evtchn_notify
    - xc_evtchn_open
    - xc_evtchn_pending
    - xc_evtchn_reset
    - xc_evtchn_status
    - xc_evtchn_unbind
    - xc_evtchn_unmask

## `libxengnttab`

Interacting with `/dev/xen/gnt{shr,alloc}`

XXX two libs or one?

    - xc_gntshr_close
    - xc_gntshr_munmap
    - xc_gntshr_open
    - xc_gntshr_share_page_notify
    - xc_gntshr_share_pages
    - xc_gnttab_close
    - xc_gnttab_get_version
    - xc_gnttab_map_domain_grant_refs
    - xc_gnttab_map_grant_ref
    - xc_gnttab_map_grant_ref_notify
    - xc_gnttab_map_grant_refs
    - xc_gnttab_map_table_v1
    - xc_gnttab_map_table_v2
    - xc_gnttab_munmap
    - xc_gnttab_op
    - xc_gnttab_open
    - xc_gnttab_set_max_grants

## `libxendevicemodel`

Functions specifically for use by device model

## `libxenkexec`

Functions specifically for use by kexec tools
        
    - xc_kexec_exec
    - xc_kexec_get_range
    - xc_kexec_load
    - xc_kexec_unload

## `libxentoollog`

Logging stuff (already logically separate?)

    - xtl_createlogger_stdiostream
    - xtl_level_to_string
    - xtl_log
    - xtl_logger_destroy
    - xtl_logv
    - xtl_progress
    - xtl_stdiostream_adjust_flags
    - xtl_stdiostream_set_minlevel

## `libxenctrl` (Unclassified remains)

XXX Lots of this should be internal/hidden

    - bitmap_64_to_byte
    - bitmap_byte_to_64
    - call_gmon_start
    - deregister_tm_clones
    - discard_file_cache
    - do_domctl
    - do_evtchn_op
    - __do_global_dtors_aux
    - __do_global_dtors_aux_fini_array_entry
    - do_memory_op
    - do_multicall_op
    - do_physdev_op
    - do_platform_op
    - do_sysctl
    - do_sysctl_save
    - do_tmem_op
    - do_xen_hypercall
    - do_xen_version
    - elfnote_dump_core_header
    - elfnote_dump_format_version
    - elfnote_dump_none
    - elfnote_dump_xen_version
    - elfnote_fill_format_version
    - elfnote_fill_xen_version
    - elfnote_init
    - _fini
    - flush_mmu_updates
    - frame_dummy
    - __frame_dummy_init_array_entry
    - _gnttab_map_table
    - hypercall_buffer_cache_alloc
    - hypercall_buffer_cache_free
    - hypercall_buffer_cache_lock
    - hypercall_buffer_cache_unlock
    - _init
    - linux_evtchn_bind_interdomain
    - linux_evtchn_bind_unbound_port
    - linux_evtchn_bind_virq
    - linux_evtchn_close
    - linux_evtchn_fd
    - linux_evtchn_notify
    - linux_evtchn_open
    - linux_evtchn_pending
    - linux_evtchn_unbind
    - linux_evtchn_unmask
    - linux_gntshr_close
    - linux_gntshr_munmap
    - linux_gntshr_open
    - linux_gntshr_share_pages
    - linux_gnttab_close
    - linux_gnttab_grant_map
    - linux_gnttab_munmap
    - linux_gnttab_open
    - linux_gnttab_set_max_grants
    - linux_osdep_init
    - linux_privcmd_alloc_hypercall_buffer
    - linux_privcmd_close
    - linux_privcmd_free_hypercall_buffer
    - linux_privcmd_hypercall
    - linux_privcmd_map_foreign_batch
    - linux_privcmd_map_foreign_bulk
    - linux_privcmd_map_foreign_range
    - linux_privcmd_map_foreign_ranges
    - linux_privcmd_open
    - local_file_dump
    - modify_returncode
    - progress_erase
    - read_exact
    - register_tm_clones
    - retry_paged
    - stdiostream_destroy
    - stdiostream_message
    - stdiostream_progress
    - stdiostream_vmessage
    - tbuf_enable
    - VALGRIND_PRINTF
    - VALGRIND_PRINTF_BACKTRACE
    - write_exact
    - writev_exact
    - xc_add_mmu_update
    - xc_alloc_mmu_updates
    - xc_assign_device
    - xc_assign_dt_device
    - xc_availheap
    - _xc_clean_errbuf
    - xc_clear_domain_pages
    - xc_clear_last_error
    - xc_copy_to_domain_page
    - xc_core_arch_auto_translated_physmap
    - xc_core_arch_context_get_shdr
    - xc_core_arch_get_scratch_gpfn
    - xc_core_arch_gpfn_may_present
    - xc_core_arch_map_p2m
    - xc_core_arch_map_p2m_rw
    - xc_core_arch_map_p2m_writable
    - xc_core_arch_memory_map_get
    - xc_core_ehdr_init
    - xc_core_shdr_free
    - xc_core_shdr_get
    - xc_core_shdr_init
    - xc_core_shdr_set
    - xc_core_strtab_free
    - xc_core_strtab_get
    - xc_core_strtab_init
    - xc_cpumap_alloc
    - xc_cpumap_clearcpu
    - xc_cpumap_setcpu
    - xc_cpumap_testcpu
    - xc_cpu_offline
    - xc_cpu_online
    - xc_cpupool_addcpu
    - xc_cpupool_create
    - xc_cpupool_destroy
    - xc_cpupool_freeinfo
    - xc_cpupool_getinfo
    - xc_cpupool_infofree
    - xc_cpupool_movedomain
    - xc_cpupool_removecpu
    - xc_cputopoinfo
    - xc_deassign_device
    - xc_deassign_dt_device
    - xc_disable_turbo
    - xc_domain_add_to_physmap
    - xc_domain_bind_pt_irq
    - xc_domain_bind_pt_irq_int
    - xc_domain_bind_pt_isa_irq
    - xc_domain_bind_pt_pci_irq
    - xc_domain_bind_pt_spi_irq
    - xc_domain_cacheflush
    - xc_domain_claim_pages
    - xc_domain_create
    - xc_domain_create_config
    - xc_domain_debug_control
    - xc_domain_decrease_reservation
    - xc_domain_decrease_reservation_exact
    - xc_domain_destroy
    - xc_domain_disable_migrate
    - xc_domain_dumpcore
    - xc_domain_dumpcore_via_callback
    - xc_domain_get_cpu_usage
    - xc_domain_get_guest_width
    - xc_domain_getinfo
    - xc_domain_getinfolist
    - xc_domain_get_machine_address_size
    - xc_domain_get_pod_target
    - xc_domain_get_tsc_info
    - xc_domain_hvm_getcontext
    - xc_domain_hvm_getcontext_partial
    - xc_domain_hvm_setcontext
    - xc_domain_increase_reservation
    - xc_domain_increase_reservation_exact
    - xc_domain_iomem_permission
    - xc_domain_ioport_mapping
    - xc_domain_ioport_permission
    - xc_domain_irq_permission
    - xc_domain_maximum_gpfn
    - xc_domain_max_vcpus
    - xc_domain_memory_exchange_pages
    - xc_domain_memory_mapping
    - xc_domain_node_getaffinity
    - xc_domain_node_setaffinity
    - xc_domain_nr_gpfns
    - xc_domain_p2m_audit
    - xc_domain_pause
    - xc_domain_pin_memory_cacheattr
    - xc_domain_pod_target
    - xc_domain_populate_physmap
    - xc_domain_populate_physmap_exact
    - xc_domain_resume
    - xc_domain_resume_any
    - xc_domain_resume_cooperative
    - xc_domain_send_trigger
    - xc_domain_set_access_required
    - xc_domain_setdebugging
    - xc_domain_sethandle
    - xc_domain_set_machine_address_size
    - xc_domain_set_max_evtchn
    - xc_domain_setmaxmem
    - xc_domain_set_memmap_limit
    - xc_domain_set_memory_map
    - xc_domain_set_pod_target
    - xc_domain_set_target
    - xc_domain_set_time_offset
    - xc_domain_set_tsc_info
    - xc_domain_set_virq_handler
    - xc_domain_setvnuma
    - xc_domain_shutdown
    - xc_domain_subscribe_for_suspend
    - xc_domain_suppress_spurious_page_faults
    - xc_domain_unbind_msi_irq
    - xc_domain_unbind_pt_irq
    - xc_domain_unbind_pt_irq_int
    - xc_domain_unbind_pt_spi_irq
    - xc_domain_unpause
    - xc_domain_update_msi_irq
    - xc_domctl
    - xc_enable_turbo
    - xc_error_code_to_desc
    - xc_ffs16
    - xc_ffs32
    - xc_ffs64
    - xc_ffs8
    - xc_flask_access
    - xc_flask_add
    - xc_flask_add_device
    - xc_flask_add_iomem
    - xc_flask_add_ioport
    - xc_flask_add_pirq
    - xc_flask_avc_cachestats
    - xc_flask_avc_hashstats
    - xc_flask_context_to_sid
    - xc_flask_del
    - xc_flask_del_device
    - xc_flask_del_iomem
    - xc_flask_del_ioport
    - xc_flask_del_pirq
    - xc_flask_getavc_threshold
    - xc_flask_getbool_byid
    - xc_flask_getbool_byname
    - xc_flask_getenforce
    - xc_flask_load
    - xc_flask_op
    - xc_flask_policyvers
    - xc_flask_relabel_domain
    - xc_flask_setavc_threshold
    - xc_flask_setbool
    - xc_flask_setenforce
    - xc_flask_sid_to_context
    - xc_flush_mmu_updates
    - xc_get_cpufreq_avgfreq
    - xc_get_cpufreq_para
    - xc_get_cpuidle_max_cstate
    - xc_getcpuinfo
    - xc_get_cpumap_size
    - xc_get_device_group
    - xc_get_hvm_param
    - xc_get_last_error
    - xc_get_machine_memory_map
    - xc_get_max_cpus
    - xc_get_max_nodes
    - xc_get_mem_access
    - xc_get_nodemap_size
    - xc_get_online_cpus
    - xc_get_pfn_list
    - xc_get_pfn_type_batch
    - xc_get_tot_pages
    - xc_get_vcpu_migration_delay
    - xc_hvm_create_ioreq_server
    - xc_hvm_destroy_ioreq_server
    - xc_hvm_get_ioreq_server_info
    - xc_hvm_inject_msi
    - xc_hvm_inject_trap
    - xc_hvm_map_io_range_to_ioreq_server
    - xc_hvm_map_pcidev_to_ioreq_server
    - xc_hvm_modified_memory
    - xc_hvm_param_deprecated_check
    - xc_hvm_param_get
    - xc_hvm_param_set
    - xc_hvm_set_ioreq_server_state
    - xc_hvm_set_isa_irq_level
    - xc_hvm_set_mem_type
    - xc_hvm_set_pci_intx_level
    - xc_hvm_set_pci_link_route
    - xc_hvm_track_dirty_vram
    - xc_hvm_unmap_io_range_from_ioreq_server
    - xc_hvm_unmap_pcidev_from_ioreq_server
    - xc__hypercall_bounce_post
    - xc__hypercall_bounce_pre
    - xc__hypercall_buffer_alloc
    - xc__hypercall_buffer_alloc_pages
    - xc__hypercall_buffer_array_alloc
    - xc__hypercall_buffer_array_get
    - xc__hypercall_buffer_cache_release
    - xc__hypercall_buffer_free
    - xc__hypercall_buffer_free_pages
    - _xc_init_errbuf
    - xc_interface_close
    - xc_interface_close_common
    - xc_interface_is_fake
    - xc_interface_open
    - xc_interface_open_common
    - xc_lockprof_query
    - xc_lockprof_query_number
    - xc_lockprof_reset
    - xc_machphys_mfn_list
    - xc_make_page_below_4G
    - xc_map_domain_meminfo
    - xc_map_foreign_batch
    - xc_map_foreign_batch_single
    - xc_map_foreign_bulk
    - xc_map_foreign_bulk_compat
    - xc_map_foreign_pages
    - xc_map_foreign_range
    - xc_map_foreign_ranges
    - xc_maximum_ram_page
    - xc_mca_op
    - xc_mem_access_disable_emulate
    - xc_mem_access_enable_emulate
    - xc_memalign
    - xc_mem_paging_disable
    - xc_mem_paging_enable
    - xc_mem_paging_evict
    - xc_mem_paging_load
    - xc_mem_paging_memop
    - xc_mem_paging_nominate
    - xc_mem_paging_prep
    - xc_mem_paging_resume
    - xc_memshr_add_to_physmap
    - xc_memshr_audit
    - xc_memshr_control
    - xc_memshr_debug_gfn
    - xc_memshr_debug_gref
    - xc_memshr_domain_resume
    - xc_memshr_memop
    - xc_memshr_nominate_gfn
    - xc_memshr_nominate_gref
    - xc_memshr_ring_disable
    - xc_memshr_ring_enable
    - xc_memshr_share_gfns
    - xc_memshr_share_grefs
    - xc_mmuext_op
    - xc_monitor_disable
    - xc_monitor_enable
    - xc_monitor_mov_to_cr0
    - xc_monitor_mov_to_cr3
    - xc_monitor_mov_to_cr4
    - xc_monitor_mov_to_msr
    - xc_monitor_resume
    - xc_monitor_singlestep
    - xc_monitor_software_breakpoint
    - xc_nodemap_alloc
    - xc_numainfo
    - xc_osdep_get_info
    - xc_osdep_log
    - xc_osdep_put
    - xc_osdep_type_name
    - xc_pcitopoinfo
    - xc_perfc_query
    - xc_perfc_query_number
    - xc_perfc_reset
    - xc_pfn_to_mfn
    - xc_physdev_map_pirq
    - xc_physdev_map_pirq_msi
    - xc_physdev_pci_access_modify
    - xc_physdev_unmap_pirq
    - xc_physinfo
    - xc_pm_get_cxstat
    - xc_pm_get_max_cx
    - xc_pm_get_max_px
    - xc_pm_get_pxstat
    - xc_pm_reset_cxstat
    - xc_pm_reset_pxstat
    - xc_psr_cmt_attach
    - xc_psr_cmt_detach
    - xc_psr_cmt_enabled
    - xc_psr_cmt_get_data
    - xc_psr_cmt_get_domain_rmid
    - xc_psr_cmt_get_l3_cache_size
    - xc_psr_cmt_get_l3_event_mask
    - xc_psr_cmt_get_l3_upscaling_factor
    - xc_psr_cmt_get_total_rmid
    - xc_readconsolering
    - xc_report
    - xc_report_error
    - xc_report_progress_single
    - xc_report_progress_step
    - xc_reportv
    - xc_resource_op
    - xc_resource_op_multi
    - xc_resource_op_one
    - xc_sched_arinc653_schedule_get
    - xc_sched_arinc653_schedule_set
    - xc_sched_credit2_domain_get
    - xc_sched_credit2_domain_set
    - xc_sched_credit_domain_get
    - xc_sched_credit_domain_set
    - xc_sched_credit_params_get
    - xc_sched_credit_params_set
    - xc_sched_id
    - xc_sched_rtds_domain_get
    - xc_sched_rtds_domain_set
    - xc_sedf_domain_get
    - xc_sedf_domain_set
    - xc_send_debug_keys
    - xc_set_broken_page_p2m
    - xc_set_cpufreq_gov
    - xc_set_cpufreq_para
    - xc_set_cpuidle_max_cstate
    - xc_set_hvm_param
    - xc_set_mem_access
    - xc_set_progress_prefix
    - xc_set_sched_opt_smt
    - xc_set_vcpu_migration_delay
    - xc_shadow_control
    - xc_sharing_freed_pages
    - xc_sharing_used_frames
    - xc_strerror
    - xc_sysctl
    - xc_tbuf_disable
    - xc_tbuf_enable
    - xc_tbuf_get_size
    - xc_tbuf_set_cpu_mask
    - xc_tbuf_set_evt_mask
    - xc_tbuf_set_size
    - xc_test_assign_device
    - xc_test_assign_dt_device
    - xc_tmem_auth
    - xc_tmem_control
    - xc_tmem_control_oid
    - xc_tmem_restore
    - xc_tmem_restore_extra
    - xc_tmem_restore_new_pool
    - xc_tmem_save
    - xc_tmem_save_done
    - xc_tmem_save_extra
    - xc_tmem_uuid_parse
    - xc_translate_foreign_address
    - xc_unmap_domain_meminfo
    - xc_vcpu_getaffinity
    - xc_vcpu_getcontext
    - xc_vcpu_getinfo
    - xc_vcpu_setaffinity
    - xc_vcpu_setcontext
    - xc_version
    - xc_vm_event_control
    - xc_vm_event_enable
    - xc_watchdog
    - xc_interface_close_common
    - xc_interface_open_common

# Symbols used by qemu

    $ nm tools/qemu-xen-dir-remote/i386-softmmu/qemu-system-i386 | grep \\bU.xc_

    - xc_domain_add_to_physmap
    - xc_domain_bind_pt_pci_irq
    - xc_domain_create
    - xc_domain_destroy
    - xc_domain_getinfo
    - xc_domain_ioport_mapping
    - xc_domain_max_vcpus
    - xc_domain_memory_mapping
    - xc_domain_pin_memory_cacheattr
    - xc_domain_populate_physmap_exact
    - xc_domain_setmaxmem
    - xc_domain_shutdown
    - xc_domain_unbind_msi_irq
    - xc_domain_unbind_pt_irq
    - xc_domain_unpause
    - xc_domain_update_msi_irq
    - xc_evtchn_alloc_unbound
    - xc_evtchn_bind_interdomain
    - xc_evtchn_close
    - xc_evtchn_fd
    - xc_evtchn_notify
    - xc_evtchn_open
    - xc_evtchn_pending
    - xc_evtchn_unbind
    - xc_evtchn_unmask
    - xc_get_hvm_param
    - xc_gnttab_close
    - xc_gnttab_map_grant_ref
    - xc_gnttab_map_grant_refs
    - xc_gnttab_munmap
    - xc_gnttab_open
    - xc_gnttab_set_max_grants
    - xc_hvm_inject_msi
    - xc_hvm_modified_memory
    - xc_hvm_set_isa_irq_level
    - xc_hvm_set_mem_type
    - xc_hvm_set_pci_intx_level
    - xc_hvm_set_pci_link_route
    - xc_hvm_track_dirty_vram
    - xc_interface_close
    - xc_interface_open
    - xc_linux_build
    - xc_map_foreign_bulk
    - xc_map_foreign_pages
    - xc_map_foreign_range
    - xc_physdev_map_pirq
    - xc_physdev_map_pirq_msi
    - xc_physdev_unmap_pirq
    - xc_set_hvm_param

# Symbols used by kexec

    - xc__hypercall_buffer_array_alloc
    - xc_get_machine_memory_map
    - xc_get_max_cpus
    - xc_hypercall_buffer_array_create
    - xc_hypercall_buffer_array_destroy
    - xc_interface_close
    - xc_interface_open
    - xc_kexec_exec
    - xc_kexec_get_range
    - xc_kexec_load
    - xc_kexec_unload
    - xc_version



_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel