My xen version is Xen3.4.0, and i use el5u2 pvhvm to have test.
The memory size is 6114. "xm info" dose not show any change on free_memory
after command "xm mem-set domid 4096". It seems free_memory will change after
i set memory size less than 3083M, such as: free memory will increase 1 after
command "xm mem-set domid 3082".
And this is the exact same behaviour you see in a PAE Linux HVM guest with
PV drivers also? It's weird, seems like maybe for some reason the balloon
driver thinks the guest has 3083M and therefore doesn't balloon down until
it sees a target lower than that, and even then only by the difference
between 3083M and the target.
I suggest you add tracing to your PV drivers and see what the balloon driver
thinks is going on. This configuration really should work, so hopefully only
something simple and stupid is going wroing.
Sorry the false alarm.