RE: Guest-visible phys2mach part of Xen arch-neutral API? was:[Xen-devel] Uses of &frame_table[xfn]

[Kevin Fox <Kevin.Fox@xxxxxxx>]

On Fri, 2005-12-30 at 11:14 -0800, Magenheimer, Dan (HP Labs Fort
Collins) wrote:
> > > Yes, that's the current status: No way to see underlying 
> > > machine address in other domains and thus no way for driver domains. 
> 
> > Itanium cluster (~1000 nodes). I'm not sure if it would be possible to
> > use Xen on the cluster without driver domains though. Just food for
> 
> Just to clarify: This doesn't say that there can't be driver domains.
> Driver domains would need to be implemented p==m, same as domain0, except
> they would need to be given a different EFI memory map.  They have
> not been implemented on Xen/ia64 because they have not yet re-appeared
> in Xen/x86.

I am not really familiar with the internals of Xen. Just thought that
driver domains were being brushed off as unimportant again. Sorry. We do
a lot of high performance computing here and without something like
driver domains, the performance hit can't be justified.

> 
> But I am intrigued by your statement... are you assuming that
> Xen and domain0 are both single-system-image across the 1000+
> nodes?

That particular cluster is made up of dual processor HP RX2600's. It may
go to quad processors at some point. Each node has its own OS.

We'd really like to run two domU's on each node, one for jobs and
another for some services. The jobs need to be really close to the Elan4
cards since latency kills and the other domU needs to be close to disk.

Your right, having a Xen/Dom0 across a box that large would be
difficult. Largest box I know of that might be able to do that would be
NASA's alticies (What the heck is the plural of Altix anyway? :) which
are 512 processors. I have heard rumor of larger alticies in the pipe.
An Altix does have some hardware to allow slicing up a physical box into
a few smaller boxes, but it is rather basic. Having something like Xen
run on top of it could be useful.

>   I think one of the difficulties with implementing driver
> domains is that Xen (in the hypervisor) needs to discover and
> somehow partition all the devices.  This could be a real challenge
> for huge clusters.

Could not Dom0 detect the hardware and then pci hot remove it or
something and hand a reference over to Xen for handing to a DomU?

Linux already scales to ccNuma's of 512 processors. I believe SGI even
got the kernel changes into mainline to do so.

Kevin

> 
> Thanks,
> Dan
> 
> > -----Original Message-----
> > From: Kevin Fox [mailto:Kevin.Fox@xxxxxxx] 
> > Sent: Friday, December 30, 2005 9:35 AM
> > To: Tian, Kevin
> > Cc: Keir Fraser; Magenheimer, Dan (HP Labs Fort Collins); Xen 
> > Mailing List
> > Subject: RE: Guest-visible phys2mach part of Xen arch-neutral 
> > API? was:[Xen-devel] Uses of &frame_table[xfn]
> > 
> > There has been some discussion about some day using Xen on our large
> > Itanium cluster (~1000 nodes). I'm not sure if it would be possible to
> > use Xen on the cluster without driver domains though. Just food for
> > thought.
> > 
> > Kevin
> > 
> > PS. I'm fairly disappointed in the lack of focus that driver 
> > domains are
> > receiving in Xen. I use them on several machines (one in 
> > production) and
> > I think it counts among Xen's greatest features.
> > 
> > On Fri, 2005-12-30 at 10:16 +0800, Tian, Kevin wrote:
> > > >From: Keir Fraser [mailto:Keir.Fraser@xxxxxxxxxxxx]
> > > >Sent: 2005å12æ29æ 21:48
> > > >
> > > >
> > > >On 29 Dec 2005, at 01:59, Tian, Kevin wrote:
> > > >
> > > >>        IMO, I see the phys2mach mapping as a basic 
> > virtualization policy,
> > > >> instead of an architecture specific requirement. After adding
> > > >> phys2mach concept to XEN/IA64, we can reuse more common 
> > code without
> > > >> ifdef. Then correspondingly also need to add several 
> > necessary changes
> > > >> like x86: DMA, SWIOTLB, AGP, etc, to ensure legal machine address
> > > >> written into physical devices.
> > > >
> > > >This seems to make sense to me. How does ia64/xen work right now?
> > > >Machine addresses visible to domain0 and full virtualisation of
> > > >addresses exposed to other domains (with no way of seeing 
> > underlying
> > > >machine addresses)?
> > > >
> > > >  -- Keir
> > > 
> > > Yes, that's the current status: No way to see underlying 
> > machine address in other domains and thus no way for driver domains.
> > > 
> > > Thanks,
> > > Kevin
> > > 
> > > _______________________________________________
> > > Xen-devel mailing list
> > > Xen-devel@xxxxxxxxxxxxxxxxxxx
> > > http://lists.xensource.com/xen-devel
> > 

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel