Design Session: Resource allocation for Guests

[Ayan Kumar Halder <ayankuma@xxxxxxx>]

Hi All,

Please find the meeting notes. Please feel free to correct anything.


Jan - A patch series got stuck because revieres say contridactory 
things. Do we allocate all the resources when a guest is created ? Or do 
we can allocate resources on demand (so first time a guest asks for a 
resource, it may fail) ?
Jan - The guest lives or gives a bug

George - It is good to know ahead of time that guest will not fail.

Jurgen - For Virtio, this allocation required a significant amount of 
memory. So do allocation on demand. The guest will know that it will 
allocate resource (grant). Why should the guest allocate extra resource 
in the beginning. It will be nice for the guest to say beforehand what 
memory is needed. Then hypervisor can adjust resources accordingly. Also 
tell hypervisor when it does not need a resource
George - Ok. Memory allocated to the guest at boot. The guest can be 
ballooned up to add resources
Jurgen - Guest can add to grant table.

Jan - What we have in guest config file. Is the memory used by the 
guest. It is the memory used on behalf of the guest (eg event channels)
George - We can’t do that now (would have been good if it were done 
before).XAPI does something to manage the resources for VM. Sometime, VM 
will fail.
Jurgen - Why would it fail

George - Xen can’t allocate the memory. The guest fails not during 
creation, but in its lifetime. One needs to have a way if the guest 
needs to use a feature, then it should be possible to free memory.
Jurgen - Dynamic resource allocation, give resource to hypervisor

Jan - Need an option in xl config

Bertrand - For FUSA, we need to determine the memory size of the guest 
beforehand .
George - How do we predict the memory need for a VM ?

Jan - Extra memory for guest will come from the guest.

George- Need a way to make sure that the guest has the memory it needs.

Jan - If you have separate pools per domain, how do you associate each 
pool to domain. Andrew had introduced dmalloc().
Bertrand - On Arm guest, this is solved. We have separate pools of 
memory. The x86 issues don’t apply.
Jan - Grant table / event channel in common code

Bertrand - Not very sure when allocation happens for common code. Using 
memory for guest if you push to Xen, means unmapping the pages for 
guest. Then you may need page tables for Xen. The problem may increase.
Jan - On x86, this might be a problem . map_domain_page() does not need 
a page table.
Bertrand - Security concerns. Need to cleanly unmap the page when it is 
passed from guest to Xen. When you create a guest, you need how much 
memory you need. For ballooning system, the guest cannot do ballooning 
after some time. Ballooning systems will need two heaps for memory.

Jan - Config file memory, guest sees as normal ram. It is under the 
control of guest. Guest can decide to use extra memory
Jurgen/George - Guest may say I need more grants. How much more memory 
do I need? Xen tells you. Guest ballons up in domain heap.
Bertrand - The system is not possible to know in advance the memory

Jan - For grant table, this is possible. A new grant table, will give 
you a ratio of the memory for each entry.
Jurgen - You can’t document.

Bertrand - Upto the guest to give more memory.

George - Guest fails until it gives all the memory.

Jan - Add a new grant table so query grant table.

Jurgen - Querying how much memory a guest needs. (If the free memory 
falls below 64K, then increase it to another 64 K. This is Linux approach).
Jan - Indicate how much memory was missing for last allocation.

Marek - Subsequent allocation need more pages. So you may need more retries.

Jan - Error handling code

George - I will make a guest with these params. Balloon the heap. Try if 
it works.
Bertrand - The code becomes complex where the guest tells how much 
memory it needs in advance.
George - Robust approach, is to to retry to get more memory .

Bertrand - It is not FUSA safe. How much memory a guest needs (min 
heap), then a have ballooning system. This should work for x86 and Arm.
Marek - Guest memory should be allocated upfront by tool stack (know 
what is the initial pool).
Jurgen - Old kernel do not use ballooning system

Bertrand - Add a new guest and it crashes. This is not acceptable.

Jan - Provide the memory (for both the heaps) in xen config file.

Bertrand - Keep this option as a Kconfig. Make the code generic enough.

George - Set as an experimental feature. This may not work. New feature 
may gracefully handle memory failure.
Jan - If a guest demands more memory (after converting grant table), 
then this need new configuration.
George - Set the number of grants (instead of pool).

Bertrand - Prove a debug console to say which guest is using which 
parameter. You need end up giving more memory.
Jan - Xen should report what was used in the lifetime of guest.

Bertrand - We cannot have a realistic view of the default memory for a 
guest.
Jurgen - The sizes are limited at guest start.

George - Permissive mode. Allocate first from private pool and then main 
pool. Like running SE linux in permissive mode.
Jan - Could work for permissive mode.

Roger - Why do we want to move the feature to tool stack ? Do not expose 
feature if the guest does not use it
Jan - It should be the hypervisor to determine.

Bert - If the user want to have a default value, store it in Xen.

Jur - Required for Qemu stuff for special cases.

Roger - If the guest exhausts memory, then what happens. Guest kernel 
does not use the pool
George - If the heap runs out of memory, then it is a vulnerability. If 
a user can cause kernel to exhaust memory on Xen, it is possible now.
Roger - The guest gets killed.

Jurgen - One guest should not make the host unusable

Bert - A guest can impact only itself not others (FUSA). The pool is 
preallocated on guest creation.
George - How to send feedback from Xen to guest telling it does not have 
memory. Guest frees memory
Roger - We need good ref counting for feature.

Jurgen - The grant table may not be used all together at the same time. 
Expand grant table for this feature.
Jan - As we do conversion between xmalloc and dmalloc, document 
everything to influence calculation.
Roger - does not work for dynamic allocation

Jan - By documenting the various need, avoid situation where guest runs 
out of pool
George - Write tests to allocate all pages. We have an agreement to use 
per guest pool.
Jan - Permissive option to fall back to main pool

George - We should make a pool, set aside what is need. With permissive 
option, throw error
Jan - Guest will log what is allocated from the pool. Maintain the max 
need of memory
George - Guest will write down what memory it has allocated.

Bertrand - Add a structure for the guest for counting. Some places need 
to hack
Bertrand - At some point, every allocation should happen from guest. Not 
by Xen
Jan - For PCI hot plug, allocate memory from Xen heap. This may be a 
problem when there is no dom0
Jan - Why would a dom0less system not see a new hot plug. May be 
dom0less does not detect hot plug.
Dom0less with ACPI was never considered.

Bertrand -  (DMALLOC) This has been done on Arm

George - Until dmalloc is used, allocate upfront.

Jan - Let’s not defer further allocation. Also, we are moving to per 
domain pool.
Greorge - Dynamic allocation increases chance of failure.

Jan - Replace runtime allocation to dmalloc().  Then switch over. Then 
convert static allocation can be converted to dmalloc().
Jurgen - Design interface to guest, so that linux can use it

Jan - The only interface need is to indicate ballooning (private pool). 
How large the private pool is ?
Jurgen - Easier for linux to buffer allocation memory and not retry 
every time. Performance is better.
Bert - You need one more page

Jan - Guest should report how many free pages.


Kind regards,

Ayan