<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<div class="moz-cite-prefix">Hi Mike,<br>
<br>
When I try that on a Power 9 system of mine, the act of doing the
link disable results in the PHB going into EEH<br>
state, which is essentially the PHB going into a frozen state due
to an unexpected error of some sort. Lots of things<br>
can cause this - bad DMA address, PCIe link errors, etc. In this
case its the act of disabling the link. <br>
If you check dmesg, my guess is that you will see errors related
to EEH. The kernel will then attempt to <br>
recover from this state. In fact, what I see on my system, is I
don't even need to clear the link disable state,<br>
as the act of going through EEH recovery in the kernel ends up
clearing it.<br>
<br>
Thanks,<br>
<br>
Brian<br>
<br>
On 05/30/2018 11:00 AM, Mike Bieker wrote:<br>
</div>
<blockquote type="cite"
cite="mid:157e130f4c282cdb6bbf1682c2a9624f@mail.gmail.com">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 14 (filtered
medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
<div class="WordSection1">
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Arial","sans-serif"">On
x86 system, discovering/enabling a PCIe adapter after PCIe
link disable/enable takes less than a second. However, on
Power Systems it takes 30 seconds or more.</span></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Arial","sans-serif""> </span></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Arial","sans-serif"">Here
is the process we are using to test:</span></p>
<p style="margin:0in;margin-bottom:.0001pt"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif"">1)
Boot system and verify that link is up between IBM
Root Port and our Atlas PCIe Gen4x16 switch with no errors –
‘lspci –s 034:01:00.0 –vvv’</span></p>
<p style="margin:0in;margin-bottom:.0001pt"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif"">2)
Set Link Disable bit (Bit 4) in PCIe Link Control
register of Root Port - ‘setpci –s 034:00:00.0 58.w=0018’.
</span></p>
<p style="margin:0in;margin-bottom:.0001pt"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif"">3)
Verify that link is disabled between Root and Atlas –
‘setpci –s 034:00:00.0 58.w’ should show that link disable
bit is set. Can also execute ‘lspci’ and see that link is
down between Root Port and Atlas.</span></p>
<p style="margin:0in;margin-bottom:.0001pt"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif"">4)
Clear Link Disable bit in PCIe Link Control register
of Root Port – ‘setpci –s 034:00:00.0 58.w=0008’</span></p>
<p style="margin:0in;margin-bottom:.0001pt"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif"">5)
Wait 5 seconds - ‘sleep 5’</span></p>
<p style="margin:0in;margin-bottom:.0001pt"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif"">6)
Check that that link between Root Port and Atlas is
enabled and at proper rate and width (Gen4x16) – ‘lspci –s
034:01:00.0 –vvv’. This is where error occurs because link
is not up. If I keep trying lspci, after 30 to 60 seconds
the port returns valid data. Why does Fedora on Power
Systems take so long to link up and discover the adapter
after link disable/enable?</span></p>
<p style="margin:0in;margin-bottom:.0001pt"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif""> </span></p>
<p style="margin:0in;margin-bottom:.0001pt"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif"">Thanks,</span></p>
<p style="margin:0in;margin-bottom:.0001pt"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif"">Mike</span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif""> </span></p>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Linuxppc-users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Linuxppc-users@lists.ozlabs.org">Linuxppc-users@lists.ozlabs.org</a>
<a class="moz-txt-link-freetext" href="https://lists.ozlabs.org/listinfo/linuxppc-users">https://lists.ozlabs.org/listinfo/linuxppc-users</a>
</pre>
</blockquote>
<p><br>
</p>
<div class="moz-signature">-- <br>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
<title></title>
Brian King<br>
Power Linux I/O<br>
IBM Linux Technology Center<br>
<div class="moz-signature">
<pre><font face="Arial, sans-serif">
</font></pre>
</div>
</div>
</body>
</html>