<html><body><p><font size="2">Steven J. Munroe<br>Linux on Power Toolchain Architect<br>IBM Corporation, Linux Technology Center<br></font><br><br><tt><font size="2">"Linuxppc-users" <linuxppc-users-bounces+sjmunroe=us.ibm.com@lists.ozlabs.org> wrote on 06/27/2017 09:08:05 AM:<br><br>> From: Frederic DUMOULIN <frederic.dumoulin@mipsology.com></font></tt><br><tt><font size="2">> To: linuxppc-users@lists.ozlabs.org</font></tt><br><tt><font size="2">> Date: 06/27/2017 09:10 AM</font></tt><br><tt><font size="2">> Subject: Re: [Linuxppc-users] 256 bit aligned variables</font></tt><br><tt><font size="2">> Sent by: "Linuxppc-users" <linuxppc-users-bounces<br>> +sjmunroe=us.ibm.com@lists.ozlabs.org></font></tt><br><tt><font size="2">> <br>> Hi,<br>> <br>> Thanks for you answers.<br>> I've got new questions below.<br></font></tt><br><tt><font size="2">> On 23/06/2017 03:07, Steve Munroe wrote:</font></tt><br><tt><font size="2">> "Linuxppc-users" <linuxppc-users-bounces+sjmunroe=us.ibm.com@lists.ozlabs.org><br>> wrote on 06/22/2017 01:19:15 AM:<br>> <br>> > From: Frederic DUMOULIN <frederic.dumoulin@mipsology.com><br>> > To: linuxppc-users@lists.ozlabs.org<br>> > Date: 06/22/2017 05:02 PM<br>> > Subject: [Linuxppc-users] 256 bit aligned variables<br>> > Sent by: "Linuxppc-users" <linuxppc-users-bounces<br>> > +sjmunroe=us.ibm.com@lists.ozlabs.org><br>> > <br>> > Hi,<br>> > <br>> > I'd like to manage 256 bit aligned variables.<br>> > So I use gcc's vector_size attribute like in the following example:<br>> > <br>> PowerISA does not have 256-bit vectors, yet.<br>> <br>> It does have 128-bit vectors, with lots of vector registers (64 of <br>> them) and multiple vector units executing in parallel.<br>> <br>> POWER8 <br>> - 4 load units, 2 stores units (these are paired for vector)<br>> - Two symmetric fixed-point units (FXU)<br>> – Four floating-point units (FPU), implemented as two 2-way SIMD <br>> operations for double- and single-<br>> precision.<br>> – Two VMX execution units capable of executing simple FX, permute, <br>> complex FX, and 4-way SIMD<br>> single-precision floating-point operations<br>> – One Crypto unit<br>> <br>> Two 16-byte loads and one 16-byte store operation are supported for <br>> VMX and VSX operations per cycle.<br>> And<br>> Two Vector Double operation per cycle (4 X double FMAs per cycle)<br>> And<br>> Two Vector float / permute / Integer / Logic operations per cycle.<br>> <br>> Think RISC not CISC. <br>> <br>> > typedef long long uint256_t __attribute__ ((vector_size (8 * sizeof <br>> > (uint32_t))));<br>> > int main (void)<br>> > {<br>> > volatile uint256_t inVar256, outVar256;<br>> > outVar256 = inVar256;<br>> > }<br>> > <br>> <br>> GCC has native support for vector_size (16). Larger multiples are <br>> handled as arrays of vector_size(16).<br>> <br>> As such there is no real advantage for alignment greater then 16-bytes. </font></tt><br><tt><font size="2">> The 256 bit alignment is not used for performance issue, we use it <br>> to simplify the address decoding in the FPGA in charge of the <br>> management of the local bus on our PCIe board.<br>> <br>> So, if I well understood, it's not possible to guarantee a 256 bit <br>> access (1 address + 8 words) on ppc but it's possible to have 128 <br>> bit accesses (1 address + 4 words), it's correct ?<br>> Will it be the case if I manipulate uint128_t variables ? with <br>> typedef long long uint128_t __attribute__ ((vector_size (4 * sizeof<br>> (uint32_t)))); </font></tt><br><tt><font size="2">> </font></tt><br><tt><font size="2">You can use the aligned attribute, as in:</font></tt><br><br><tt>aligned (</tt><tt><i>alignment</i></tt><tt>)</tt><ul><ul>This attribute specifies a minimum alignment for the variable or structure field, measured in bytes. For example, the declaration:
<p><tt>int x __attribute__ ((aligned (32))) = 0;<br></tt></ul></ul><tt><font size="2">this works for static (BSS) storage . I am not sure about automatic (stack) as the ABI only requires (aligned (16)). </font></tt><br><br><tt><font size="2">But if you are working in IO-space you are likely be aligning the pointer yourself. If the pointer is aligned the accesses will be aligned.</font></tt><br><tt><font size="2"><br>> Also with the inherent instruction parallelism of the POWER8, you <br>> are already getting effective 256-bits of vector operations per cycle.<br>> </font></tt><br><tt><font size="2">Yes there can be multiple vector instructions dispatch and issue per cycle but the details of read/write ports at the various cache levels are specific to the processor.</font></tt><br><br><tt><font size="2">And also cache-coherent and cache-inhibited storage behave differently.</font></tt><br><br><tt><font size="2">Best to read the relevant sections of the "</font></tt><tt><font size="2">POWER8 Processor User’s Manual for the Single-Chip Module" yourself.</font></tt><br><br><tt><font size="2">The information portal: </font></tt><a href="https://www-355.ibm.com/systems/power/openpower/"><tt><font size="2">https://www-355.ibm.com/systems/power/openpower/</font></tt></a><br><br><tt><font size="2">Under power8: </font></tt><a href="https://www-355.ibm.com/systems/power/openpower/tgcmDocumentRepository.xhtml?aliasId=POWER8"><tt><font size="2">https://www-355.ibm.com/systems/power/openpower/tgcmDocumentRepository.xhtml?aliasId=POWER8</font></tt></a><br><br><tt><font size="2">and</font></tt><br><br><tt><font size="2">Power8 with NVidia: </font></tt><a href="https://www-355.ibm.com/systems/power/openpower/tgcmDocumentRepository.xhtml?aliasId=POWER8_with_NVIDIA_NVLink"><tt><font size="2">https://www-355.ibm.com/systems/power/openpower/tgcmDocumentRepository.xhtml?aliasId=POWER8_with_NVIDIA_NVLink</font></tt></a><br><br><br><tt><font size="2">There a similar links for power9 and the PowerISA under this portal.<br>> <br>> > On x86_64, I use -mavx so gcc generates AVX instructions.<br>> > Disassembling the object file, I've got:<br>> > outVar256 = inVar256;<br>> > 13: c5 fd 6f 45 b0 vmovdqa -0x50(%rbp),%ymm0<br>> > 18: c5 fd 7f 45 d0 vmovdqa %ymm0,-0x30(%rbp)<br>> > <br>> > On ppc (cross-compiling with: /opt/at10.0/bin/powerpc64le-linux-gnu-gcc <br>> > -I. -g -O -mcpu=powerpc64le -mvsx -c main.c -o main.o), I've got:<br>> > outVar256 = inVar256;<br>> > 1c: 40 00 40 39 li r10,64<br>> > 20: 98 56 89 7d lxvd2x vs12,r9,r10<br>> > 24: 10 00 40 39 li r10,16<br>> > 28: 50 00 00 39 li r8,80<br>> > 2c: 98 46 09 7c lxvd2x vs0,r9,r8<br>> > 30: 98 4f 80 7d stxvd2x vs12,0,r9<br>> > 34: 98 57 09 7c stxvd2x vs0,r9,r10<br>> > 38: 98 4e 80 7d lxvd2x vs12,0,r9<br>> > 3c: 98 56 09 7c lxvd2x vs0,r9,r10<br>> > 40: 20 00 40 39 li r10,32<br>> > 44: 98 57 89 7d stxvd2x vs12,r9,r10<br>> > 48: 30 00 40 39 li r10,48<br>> > 4c: 98 57 09 7c stxvd2x vs0,r9,r10<br>> > <br>> <br>> Again think RISC not CISC. The compiler broke the 256-bit vector <br>> into pairs of 128-bit vector operations.<br>> <br>> > I'm new on ppc and not familiar with ppc instructions but it doesn't <br>> > seem to be SIMD instructions.<br>> <br>> Those (lxvd2x/stxvd2x) are 128-bit SIMD load/store operations.<br>> <br>> > How can I generate 256 bit aligned instructions ?<br>> > <br>> You don't need more then 128-bit alignment and you will get <br>> effective 256-bit vector operation without this.<br>> <br>> > For more details, the goal is to drive a PCIe board and to generate 256 <br>> > bit aligned memcpy from host memory to PCIe bus.<br>> > <br>> You should look at the GLIBC memcpy_power8 implementation for <br>> example of how to maximize memory bandwidth</font></tt><br><tt><font size="2">> For the performance, I use write-combining by remapping the PCIe <br>> address space with ioremap_wc () in my kernel device driver.<br>> Is there any similar mechanism ?<br>> <br>> <br>> Thanks,<br>> Fred<br></font></tt><br><tt><font size="2">> <br>> Steven J. Munroe<br>> Linux on Power Toolchain Architect<br>> IBM Corporation, Linux Technology Center<br></font></tt><br><tt><font size="2">> _______________________________________________<br>> Linuxppc-users mailing list<br>> Linuxppc-users@lists.ozlabs.org<br>> <a href="https://lists.ozlabs.org/listinfo/linuxppc-users">https://lists.ozlabs.org/listinfo/linuxppc-users</a><br></font></tt><BR>
</body></html>