[Cbe-oss-dev] Cell internal registers access

[Arnd Bergmann]

On Saturday 06 May 2006 00:38, Benjamin Herrenschmidt wrote:
> So I would add a get_xxx_FIR() for every xxx sub_module ? Hrm .. the RAS
> stuff is a bit annoying there. I don't know.. I don't beleive in too
> much abstractions. I'm ok keeping the actual definitions of the register
> spaces and associated constants in separate files, but I'm not sure we
> should absolutely keep them private to some "driver" for that module.
> 
> If I follow your logic, for RAS, just in order to get to the FIR's (and
> maybe a couple others), I would need to create a module for the MIC, a
> module for the IOC, a module for the BIU, etc... 
> 
> I think the actual "driver" modules in the system slightly overlap
> physical cell modules.

The problem is that we don't know which of the functional units will
show up in future CPUs, so it would be good not to rely on the
same set to be present as on Cell BE.

If that scheme doesn't work out for the FIR, we might need to come
up with something different, as you said. Do you already know which
areas you need to touch?

I already learned that for the oprofile code, we will need to
make significant changes to the the way that the IIC (interrupt.c)
handling works, since that needs to be acknowledged in per-chip
registers and is using external exception with some special bits
set instead of the 0xf00 performance monitor exception.

> > I had hoped that we could avoid a data structure specific to
> > a physical Cell BE and do all that with the existing CPU numbers
> > and NUMA node identifiers, but it seems we might need yet another
> > dimension for that.
> 
> I'm not sure what you mean in the above. There is a data structure for
> the PMD (bad name btw :), one for hte IIC (missing the IOC_FIR btw),
> there will be one for the MIC, etc... Then, my idea was to have a module
> that hands out those to clients. This module is the one to know the
> thread<->BE mapping etc... Clients call it with something like
> 
> struct cbe_xxx_regs __iomem *cbe_get_xxx_registers(int cpu);
> 
> That modules contains the thread<->BE mapping logic, the device-tree
> parsing (thus if we ever clean that device-tree and remove those hackish
> address properties, it's the only thing that will need to change) etc...
> 
> In fact, I'm tempted to also provide lower level BE<->thread mapping
> routines in there for the few cases that might be interested.

Yes, that is probably the best we can do. For the interrupt stuff
in particular, we have so far only used the per-SMT-thread data
(source, prio and generation), so it was an actual mapping between
smp_processor_id() and iomem area, which will not really fit
in your approach.

> The SMT <-> core mapping isn't quite well defined... Cell uses
> ibm,interrupt-server#s but I'm not sure I completely like it at this
> point... At least I'm not ready to recommend it to other embedded
> vendors. Still need to think about it. In the meantime, this is not a
> blocking issue as long as we don't have that logic copied in every
> module, which is the case today.

The advantage of ibm,interrupt-server#s is that it is the only one that
is really mentioned in OpenPAPR or any of the referenced documents.
 
> > > Note: I'm thinking about leaving the SPUs out of that since they are
> > > fairly separate and have already their own .h and exported accessors.
> > > It's more for IIC, BIU, IOC, PMD, etc...
> > 
> > As Christian Krafft now noticed while playing with some of the
> > thermal registers (you two should talk about that), we do need to
> > keep the SPEs in the picture, although in a different way: At least
> > the thermal registers have per-spe data in on of the mmio spaces
> > outside of the SPE-local (priv1/2- and problem-state) mmio areas.
> 
> They do but that isn't a problem. I'm really only talking about
> definition the register maps for the various modules and getting access
> to them. Regarding SPE thermal management, we simply don't know yet what
> we want to do. Wether the SPE layer will be the one to query the
> temperatures or wether we'll have a temperature management module that
> will call into the SPE layer. That is not a problem though. The SPE
> termperatures are referenced by physical SPE number on a given BE. The
> spu_base code already knows how to map between physical SPEs, BEs
> etc...  
>
> > So what is needed is, given a 'struct spu *', a way to find the
> > respective mmio area for PMD, and the index on the spu in that
> > area.
> 
> Well, if needed, we can have the "registers" module described above have
> an API that takes a physical BE number (since that's what SPU layer
> wants) and have a cbe_thread_to_be() call for callers who only have an
> smp_processor_id().

Well, the problem is that we still don't have a conclusion what an
identifier for one 'cell be' should be in general. I still think
it should be a platform independant concept of a node with (potentially)
more than once processors, and SOC bus and at the same time not
need to be a NUMA node on its own.

	Arnd <><