[RFC] DT affinity bindings/representing bus masters with DT

[Lorenzo Pieralisi]

Hi David,

thanks for your feedback.

On Wed, Mar 06, 2013 at 09:57:14PM +0000, David Gibson wrote:
> On Fri, Feb 15, 2013 at 05:21:02PM +0000, Lorenzo Pieralisi wrote:
> > Hi all,
> > 
> > in order to come up with a solid solution to the affinity bindings concept
> > we are facing in the ARM world, I am posting this RFC so that hopefully people
> > on the list can chime in and help us in this endeavour.
> > 
> > I tried to keep things simple on purpose and some statements are a bit of an
> > oversimplification, we can elaborate on those if needed.
> > 
> > Hereafter a summary of what we are trying to achieve.
> > 
> > Current device tree bindings allow to describe HW configurations of systems
> > in a bus like fashion, where each bus contains bus slaves and mapping
> > heuristics to translate address spaces across bus layers (AMBA -> PCI).
> > 
> > The device tree by definition represents the view of the system from the
> > perspective of the CPU(s). This means that all devices (but CPUs) present
> > in the device tree are to be seen as "slave" components, ie devices sitting on
> > a bus and accessible from the CPU with an addressing mode that can vary and it
> > is defined by the bus the device is sitting on.
> > 
> > There are specific cases in current SoCs though where resources belonging to
> > a slave device should be linked to a master in the SoC system hierarchy.
> > 
> > To the best of my knowledge, the mechanism used to implement this linkage is
> > not defined by any device tree binding; it probably can be a phandle in the
> > device tree syntax, but this has some implications that will be described as
> > follows.
> > 
> > A programmable device, let's call it "foo" for the sake of this discussion,
> > has several resources (ie memory spaces) that should be mapped to bus masters
> > in a SoC hierarchy (each resource "belongs" to a master). The only way this
> > can be currently done through a device tree is by linking the resource in
> > question to a device representing the master node through a phandle (keeping
> > in mind that the concept of master node does not exist in current DT
> > bindings).
> > 
> > An example is worth a thousand words (pseudo dts file):
> > 
> > / {
> > 	#address-cells = 1;
> > 	#size-cells = 1;
> > 
> > 	acme1: acme at 4000000 {
> > 		reg = <0x4000000 0x1000>;
> > 	};
> > 
> > 	acme2: acme at 5000000 {
> > 		reg = <0x5000000 0x1000>;
> > 	};
> > 
> > 	foo at 2000000 {
> > 		reg = <0x2000000 0x1000
> > 		       0x2001000 0x1000>;
> > 		affinity = <&acme1 &acme2>;
> > 	};
> > };
> > 
> > where the "affinity" property contains a number of phandles equal to the number
> > of resources (ie reg properties) in the foo at 2000000 node. The "affinity"
> > property maps a reg property to a device tree node, one phandle per "reg"
> > property.
> > 
> > acme1 and acme2 are two bus masters in the system (eg a DMA and a GPU).
> > 
> > Each foo at 2000000 reg property maps to a device that represents a bus master
> > (to make it clearer, a foo at 2000000 reg property defines an address space that
> > belongs to a bus master, ie the address space represents a programming
> > interface specific to that master; in the bindings above address 0x2000000 is
> > the address at which acme1 device can programme its "foo" interface, address
> > 0x2001000 is the address at which acme2 device can programme its "foo"
> > interface).
> 
> Ok.  I think annotating the existing reg property like this is a very
> bad idea.  I haven't seen all the previous discussion, so I'm not
> totally clean on what this affinity concept is about.  But as I
> understand it, these "slave" resources cannot be treated like an
> ordinary resource in in the reg property.  That means an older client
> will potentially misinterpret "reg" because it doesn't know about
> "affinity".

Not really, "reg" still complies with the current DT bindings. Affinity
is there to associate a reg property to a "master" but the reg property
definition does not change. I do not think backward compatibility is a
problem per-se here.

> Worse, again, if I've understood correctly, resources with different
> "masters" are essentially in different logical address spaces.  "reg"
> properties should always sit in the logical address space representing
> the parent node's bus.  Different address spaces could also have
> different address sizes, which would really complicate parsing "reg".

I think we need to post what we have, it is really complex to explain
the issue without a concrete example. To cut a long story short I
would not say that the resources sit in different address spaces, it is
that we need to associate those address ranges with specific bus masters.

We have to have a way to say:

"Address range 0x80001000 - 0x80001fff is used to programme the control
registers associated with the port connected to master X".

When a CPU wants to programme a control port for a specific master, it
needs to know what address range should be programmed.

I mentioned "resources" instead of addresses since the problem we are
having is the same when it comes to map IRQs to set of CPUS. We need
to associate a resource (IRQ or address) to a set of cpus (or more in
general, masters).

We will be posting the code we have soon, this will simplify the discussion.

> Now, a device tree extension I've considered before for other reasons
> might also be relevant to your case.  That is to add a new "bus-reg"
> property that has a meaning similar to "reg" but contains (phandle,
> address, size) tuples instead of just (address, size) tuples.  The
> phandle in each case would represent the node in whose address space
> this resource appears.  This would generalize the "dcr-reg" and
> "scom-reg" properties we've sometimes used on ppc.

It might be used that way but I need an example to understand if it fits
the purpose. We still have to associate that resource to a set of
masters, so for that to work the address space pointed at by the phandle
must be capable of defining a master (or set of masters).

I am not sure the problem is identical to dcr-reg though.

> It's not clear if this essentially incompatible extension would be
> worth it, though.  The other way to represent these structures is to
> just split the node for this logical device into different pieces
> under each address space it has resources on.  These pieces are then
> linked together with (binding specific) phandle pointers to each
> other.

I think that could be feasible too and we thought about that.

Code coming, let's restart the discussion then to see how we can move
forward.

Thank you very much,
Lorenzo