[OpenPower-Firmware] Undocumented SCOM registers and DIMM addressing

Fri Feb 12 02:48:43 AEDT 2021

Sorry for the delay, it took awhile to make sure the data could be
released.  It turns out that these registers were accidentally not included
in the register documents that IBM released.

Here are the documents.

POWER9  Processor Registers Specification - Vol 1-3
https://ibm.ent.box.com/s/ddcdl3g0otdzyiajhkfe3jjh2oy5p3mt
https://ibm.ent.box.com/s/gcg7o0sgke0cdqqw2z9pc9xc7zgjj1wu
https://ibm.ent.box.com/s/flt3hs6eiwd9glq3yzzff0flnup2j7p0
You get there from the IBM OpenPOWER Portal:

https://www-355.ibm.com/systems/power/openpower/tgcmDocumentRepository.xhtml?aliasId=POWER9_Sforza

If you look under "Power9Processor - Sforza Module" you will see " POWER9
Processor Registers Specification".  If you go to the links within Read and
Download it brings you to the PDFs that are within the box links above.

Note that Raptor has some copies on their wiki too -
https://wiki.raptorcs.com/wiki/Category:Documentation

Here is the description of the missing registers until the documents get
updated (which will likely take awhile).
(See attached file: MCP0XLT_regs.txt)

I talked to our memory team and here is the answer they gave for your
specific questions.

According to the JEDEC spec, BG1 is supported for 16Gb DRAM that use x4 or
x8 DRAM widths.  BG1 is not supported for 16Gb DRAM that are a x16 DRAM
width.  Currently, the code has plug rules in place that will fail for
anything other x4 or x8, meaning that a x16 part will be deconfigured in
step 7 (see plug_rules.C).  The code will not support x16 parts and would
need to be updated. Additionally x16 is not officially supported by the
Nimbus hardware. If you are trying to add in x16 support, you will have an
uphill battle against both the code and the hardware.
As another note: 0b10010 is the LSB for the port mapping.
(See attached file: 16Gb Addressing.png)

--
Dan Crowell
Senior Software Engineer - Power Systems Enablement Firmware
IBM Rochester: t/l 553-2987
dcrowell at us.ibm.com

From:	Krystian Hebel <krystian.hebel at 3mdeb.com>
To:	openpower-firmware at lists.ozlabs.org
Cc:	firmware at 3mdeb.com
Date:	01/28/2021 02:42 PM
Subject:	[EXTERNAL] Re: [OpenPower-Firmware] Undocumented SCOM registers
            and DIMM addressing
Sent by:	"OpenPower-Firmware" <openpower-firmware-bounces
            +dcrowell=us.ibm.com at lists.ozlabs.org>

Update: I have found out that the way MCAs are mapped is calculated in
istep 7.4 and put into appropriate registers in 14.5. All other
questions are still open.

On 19.01.2021 15:22, Krystian Hebel wrote:
> Hello,
>
> I'm analyzing memory initialization code as a part of preparation for
> coreboot port for Talos II. I have encountered a few undocumented
> registers, all of them (so far) have SCOM addresses 0x0501082X. In the
> part of code I am analyzing, the first write to those registers
> happens in istep 13.8, but as it is done by an initfile, I didn't pay
> too much attention and assumed that this is just the default,
> non-configurable value.
>
> Registers from the same group appear again in 13.13 for setting up the
> translation between controller addresses and DIMM addresses. This time
> they are at least named by constants: MCA_MBA_MCP0XLT{0,1,2}. Field
> names can also be found in header files, but without any explanation
> as to what they exactly configure. I haven't analyzed further code yet
> so can't tell if this is the last place where they are accessed.
>
> I have decoded, more or less, which port address is assigned to which
> DIMM bit and under what circumstances, but there are still some
> unanswered questions, or assumptions I would like to confirm.
> Nevertheless, it would be nice to have a proper documentation of all
> pieces required for the initialization of the platform.
>
> How many port address bits there are? Is it either 40 or 56 bits, as
> mentioned in section 10.2 of POWER9 Processor User's Manual? How are
> port addresses from all ports (MCAs) mapped to the address space seen
> by CPU?
>
> Which bit numbers are more significant? I assume lower port address
> bit number is more significant, seeing that the highest numbers are
> always allocated, and lower only if needed. On the other hand, if that
> is the case, wouldn't assigning higher number/less significant address
> to D-bit improve performance due to interleaving? Assuming that both
> DIMMs have the same size to avoid holes, of course.
>
> I assume that bits which are not encoded in `enum xlate_bit_maps` are
> hardwired to column bits 0-3 and row bits 0-14. There is also
> `PORT_ADDRESS_22` in enumeration that isn't used anywhere in the code,
> is it also hardwired? Also, why some of the map fields in registers
> are 5 bits, while other are only 3 bits long? Given that the registers
> are sparsely filled (at least from what can be decoded from the header
> files) I don't think this is done just to pack more data in one register.
>
> I have also noticed that both BG bits are mapped, even for 16Gb
> devices, which according to DDR4 specification have only two bank
> groups, so they should use just BG0. Was this an overlook or is there
> any purpose for mapping both bank group bits?
>
> I hope that wasn't too much questions for one email.
>
> Best regards,
>
--
Krystian Hebel
Firmware Engineer
https://3mdeb.com
  | @3mdeb_com

_______________________________________________
OpenPower-Firmware mailing list
OpenPower-Firmware at lists.ozlabs.org
https://lists.ozlabs.org/listinfo/openpower-firmware

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ozlabs.org/pipermail/openpower-firmware/attachments/20210211/d32c96dc/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: graycol.gif
Type: image/gif
Size: 105 bytes
Desc: not available
URL: <http://lists.ozlabs.org/pipermail/openpower-firmware/attachments/20210211/d32c96dc/attachment-0001.gif>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: MCP0XLT_regs.txt
URL: <http://lists.ozlabs.org/pipermail/openpower-firmware/attachments/20210211/d32c96dc/attachment-0001.txt>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 16Gb Addressing.png
Type: image/png
Size: 12088 bytes
Desc: not available
URL: <http://lists.ozlabs.org/pipermail/openpower-firmware/attachments/20210211/d32c96dc/attachment-0001.png>