[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [Xen-devel] RE: TSC scaling and softtsc reprise, and PROPOSAL
> >> I've informally heard that certain version of the JVM and > >> Oracle Db have a habit of pounding rdtsc hard from user > >> space, but I don't know what rates. > > > > Indeed they do and they use it for timestamping > > events/transactions, so these are the very same > > apps that need to guarantee SMP timestamp ordering. > > Why would you expect host TSC consistency running on Xen to > be worse than > when running on a native OS? In short, it is because a new class of machine is emerging in the virtualization space that is really a NUMA machine, tries to look like a SMP (non-NUMA) machine by making memory access fast enough that NUMA-ness can be ignored, but for the purposes of time, is still a NUMA machine. Let's consider three physical platforms: SMALL = single socket (multi-core) MEDIUM = multiple sockets, same motherboard LARGE = multiple sockets, multiple motherboards The LARGE is becoming more widely available (e.g. HP DL785) because multiple motherboards are very convenient for field upgradeability (which has a major impact on support costs). They also make a very nice consolidation target for virtualizing a bunch of SMALL machines. However, SMALL and MEDIUM are much less expensive so much more prevalent (especially as development machines!). On SMALL, TSC is always consistent between cores (at least on all but the first dual-core processors). On MEDIUM, some claim that TSC is always consistent between cores on different sockets because the sockets share a motherboard crystal. I don't know if this is true; if it is true, MEDIUM can be considered the same as SMALL, if not MEDIUM can be considered the same as LARGE. So ignore MEDIUM as a subcase of one of the others. On LARGE, the motherboards are connected by HT or QPI, but neither has any form of clock synchronization. So, from a clock perspective, LARGE needs to be "partitioned"; OR there has to be sophisticated system software that does its best to synchronize TSC across all of the cores (which enterprise OS's like HP-UX and AIX have, Linux is working on, and Xen has... though it remains to be seen if any of these work "good enough"); OR TSC has to be abandoned altogether by all software that relies on it (OR TSC needs to be emulated). This problem on LARGE machines is obscure enough that software is developed (on SMALL machines) that has a hidden timebomb if TSC is not perfectly consistent. Admittedly, all such software should have a switch that abandons TSC altogether in favor of an OS "gettimeofday", but this either depends on TSC as well or on a verrryyy sllloooowwww platform timer that if used frequently probably has as bad or worse a performance impact as emulating TSC. So what is "good enough"? If Xen's existing algorithm works poorly on LARGE systems (or even on older SMALL systems), applications should abandon TSC. If Xen's existing algorithm works "well", then applications can and should use TSC. But unless "good enough" can be carefully defined and agreed upon between Xen and the applications AND Xen can communicate "YES this platform is good enough or NOT" to any software that cares, we are caught in a gray area. Unfortunately, neither is true: "good enough" is not defined, AND there is no clean way to communicate it even if it were. And living in the gray area means some very infrequent, very bizarre bugs can arise because sometimes, unbeknownst to that application, rarely and irreproducibly, time will appear to go backwards. And if timestamps are used, for example, to replay transactions, data corruption occurs. So the choices are: 1) Ignore the problem and hope it never happens (or if it does that Xen doesn't get blamed) 2) Tell all Xen users that TSC should not be used as a timestamp. (In other words, fix your apps or always turn on the app's TSC-is-bad option when running virtualized on a "bad" physical machine.) 3) Always emulate TSC and let the heavy TSC users pay the performance cost. Last, as Intel has pointed out, a related kind of issue occurs when live migration moves a running VM from a machine with one TSC rate to another machine with a different TSC rate (or if TSC rate varies on the same machine, i.e. for power-savings reasons). It would be nice if our choice (above) solves this problem too. _______________________________________________ Xen-devel mailing list Xen-devel@xxxxxxxxxxxxxxxxxxx http://lists.xensource.com/xen-devel
|
![]() |
Lists.xenproject.org is hosted with RackSpace, monitoring our |