[Xen-devel] X86 arch_domain ginormous, sizeof(struct domain) already == PAGE_SIZE

[Corneliu ZUZU <czuzu@xxxxxxxxxxxxxxx>]

I'm trying to refactor some arch-specific code into common code and was 
surprised to find out that the x86 domain structure already occupies 
PAGE_SIZE bytes, couldn't even add an unsigned short field in it w/o 
causing a compile-time error.
I'm using the master branch of git://xenbits.xenproject.org/xen.git, 
compiled Xen by simply running "./configure && make dist-xen" (on my 
x86_64 Ubuntu machine).
Concretely, the change that caused problems was the refactoring of the 
x86 arch_domain.monitor field [@ xen/include/asm-x86/domain.h], which 
originally was:
     struct {
        unsigned int write_ctrlreg_enabled       : 4;
        unsigned int write_ctrlreg_sync          : 4;
        unsigned int write_ctrlreg_onchangeonly  : 4;
        unsigned int mov_to_msr_enabled          : 1;
        unsigned int mov_to_msr_extended         : 1;
        unsigned int singlestep_enabled          : 1;
        unsigned int software_breakpoint_enabled : 1;
        unsigned int guest_request_enabled       : 1;
        unsigned int guest_request_sync          : 1;
     } monitor;

by leaving there only the x86-specific part, i.e.:
     struct {
        uint8_t mov_to_msr_enabled          : 1;
        uint8_t mov_to_msr_extended         : 1;
     } monitor;

and moving the rest directly into the domain structure, i.e. add @ the 
end of struct domain [@ xen/include/xen/sched.h]:
     struct {
        uint16_t write_ctrlreg_enabled       : 4;
        uint16_t write_ctrlreg_sync          : 4;
        uint16_t write_ctrlreg_onchangeonly  : 4;
        uint16_t singlestep_enabled          : 1;
        uint16_t software_breakpoint_enabled : 1;
        uint16_t guest_request_enabled       : 1;
        uint16_t guest_request_sync          : 1;
     } monitor;

After the above change, X86 compilation of [xen/arch/x86/domain.c] fails w/:
    error: static assertion failed: "!(sizeof(*d) > PAGE_SIZE)"
the line that caused the fail being:
    BUILD_BUG_ON(sizeof(*d) > PAGE_SIZE);

To investigate I compiled the unaltered domain.c & used the pahole tool 
(part of dwarves package) to obtain the complete layout of the domain 
structure (& its members).
What I obtained:
    * sizeof(struct domain) is already = 4096 bytes (= PAGE_SIZE)
    * sizeof(struct arch_domain) [x86] = sizeof(domain.arch) = 3328 
bytes (arch_domain is marked __cacheline_aligned, i.e. aligned to 128 bytes)
    * sizeof(domain.arch.hvm_domain) = 2224 bytes
    * sizeof(domain.arch.hvm_domain.pl_time) = 1088 bytes

=> overall, X86 timers-related information occupies the most.

One could shrink the domain structure by transforming some of its fields 
to pointers, e.g. I could transform the pl_time field into a pointer and 
dynamically allocate its data when domain_create is called.
Since the domain structure was designed to fit in a single page & 
arch_domain is marked __cacheline_aligned I presume such changes are 
sensible and should be done wisely.
How I should proceed?

Thank you,
Corneliu.

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel