<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"><meta name="Generator" content="Microsoft Word 14 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";
color:black;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman","serif";
color:black;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";
color:black;}
span.EmailStyle18
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:windowtext;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;
color:black;}
span.EmailStyle21
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style></head><body bgcolor="white" lang="EN-US" link="blue" vlink="purple"><div class="WordSection1"><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Hi Brian,</span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Thanks for looking at this! I will check dmesg for the same. </span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Mike</span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span></p><div><div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0in 0in 0in"><p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext"> Brian King [mailto:<a href="mailto:brking@linux.ibm.com">brking@linux.ibm.com</a>] <br><b>Sent:</b> Wednesday, May 30, 2018 12:06 PM<br><b>To:</b> Mike Bieker; <a href="mailto:linuxppc-users@lists.ozlabs.org">linuxppc-users@lists.ozlabs.org</a><br><b>Subject:</b> Re: [Linuxppc-users] Fedora 28-1.1 taking 30 seconds to discover/enable PCIe adapter after link disable/enable</span></p></div></div><p class="MsoNormal"> </p><div><p class="MsoNormal">Hi Mike,<br><br>When I try that on a Power 9 system of mine, the act of doing the link disable results in the PHB going into EEH<br>state, which is essentially the PHB going into a frozen state due to an unexpected error of some sort. Lots of things<br>can cause this - bad DMA address, PCIe link errors, etc. In this case its the act of disabling the link. <br>If you check dmesg, my guess is that you will see errors related to EEH. The kernel will then attempt to <br>recover from this state. In fact, what I see on my system, is I don't even need to clear the link disable state,<br>as the act of going through EEH recovery in the kernel ends up clearing it.<br><br>Thanks,<br><br>Brian<br><br>On 05/30/2018 11:00 AM, Mike Bieker wrote:</p></div><blockquote style="margin-top:5.0pt;margin-bottom:5.0pt"><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">On x86 system, discovering/enabling a PCIe adapter after PCIe link disable/enable takes less than a second. However, on Power Systems it takes 30 seconds or more.</span></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""> </span></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">Here is the process we are using to test:</span></p><p style="margin:0in;margin-bottom:.0001pt"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">1) Boot system and verify that link is up between IBM Root Port and our Atlas PCIe Gen4x16 switch with no errors – ‘lspci –s 034:01:00.0 –vvv’</span></p><p style="margin:0in;margin-bottom:.0001pt"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">2) Set Link Disable bit (Bit 4) in PCIe Link Control register of Root Port - ‘setpci –s 034:00:00.0 58.w=0018’. </span></p><p style="margin:0in;margin-bottom:.0001pt"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">3) Verify that link is disabled between Root and Atlas – ‘setpci –s 034:00:00.0 58.w’ should show that link disable bit is set. Can also execute ‘lspci’ and see that link is down between Root Port and Atlas.</span></p><p style="margin:0in;margin-bottom:.0001pt"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">4) Clear Link Disable bit in PCIe Link Control register of Root Port – ‘setpci –s 034:00:00.0 58.w=0008’</span></p><p style="margin:0in;margin-bottom:.0001pt"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">5) Wait 5 seconds - ‘sleep 5’</span></p><p style="margin:0in;margin-bottom:.0001pt"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">6) Check that that link between Root Port and Atlas is enabled and at proper rate and width (Gen4x16) – ‘lspci –s 034:01:00.0 –vvv’. This is where error occurs because link is not up. If I keep trying lspci, after 30 to 60 seconds the port returns valid data. Why does Fedora on Power Systems take so long to link up and discover the adapter after link disable/enable?</span></p><p style="margin:0in;margin-bottom:.0001pt"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""> </span></p><p style="margin:0in;margin-bottom:.0001pt"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">Thanks,</span></p><p style="margin:0in;margin-bottom:.0001pt"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">Mike</span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""> </span></p><p class="MsoNormal"><br><br><br></p><pre>_______________________________________________</pre><pre>Linuxppc-users mailing list</pre><pre><a href="mailto:Linuxppc-users@lists.ozlabs.org">Linuxppc-users@lists.ozlabs.org</a></pre><pre><a href="https://lists.ozlabs.org/listinfo/linuxppc-users">https://lists.ozlabs.org/listinfo/linuxppc-users</a></pre></blockquote><p> </p><div><p class="MsoNormal">-- <br>Brian King<br>Power Linux I/O<br>IBM Linux Technology Center</p><div><pre><span style="font-family:"Arial","sans-serif""> </span></pre></div></div></div></body></html>