<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:st1="urn:schemas-microsoft-com:office:smarttags" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv=Content-Type content="text/html; charset=us-ascii">
<meta name=Generator content="Microsoft Word 11 (filtered medium)">
<o:SmartTagType namespaceuri="urn:schemas-microsoft-com:office:smarttags"
name="City"/>
<o:SmartTagType namespaceuri="urn:schemas-microsoft-com:office:smarttags"
name="place"/>
<o:SmartTagType namespaceuri="urn:schemas-microsoft-com:office:smarttags"
name="PersonName"/>
<!--[if !mso]>
<style>
st1\:*{behavior:url(#default#ieooui) }
</style>
<![endif]-->
<style>
<!--
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman";}
a:link, span.MsoHyperlink
{color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{color:purple;
text-decoration:underline;}
p.MsoPlainText, li.MsoPlainText, div.MsoPlainText
{margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:Arial;
color:navy;}
@page Section1
{size:8.5in 11.0in;
margin:1.0in 77.95pt 1.0in 77.95pt;}
div.Section1
{page:Section1;}
-->
</style>
</head>
<body lang=EN-US link=blue vlink=purple>
<div class=Section1>
<p class=MsoPlainText><font size=2 color=navy face=Arial><span
style='font-size:10.0pt;font-family:Arial;color:navy'><see middle post><o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'><o:p> </o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>>> -----Original Message-----<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>>> From: Kumar Gala [mailto:galak@kernel.crashing.org]<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>>> Sent: Thursday, May 21, 2009 9:13 AM<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>>> To: Morrison, Tom<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>>> Cc: <st1:PersonName w:st="on">linuxppc-dev@ozlabs.org</st1:PersonName>;
Young, Andrew; Brown, Jeff<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>>> Subject: Re: How to debug a hung multi-core system....<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>>> <o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>>> <o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>>> On May 20, 2009, at 6:17 PM, Morrison, Tom wrote:<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 color=navy face="Courier New"><span
style='font-size:10.0pt;color:navy'><o:p> </o:p></span></font></p>
<p class=MsoPlainText><b><i><font size=2 color=navy face=Arial><span
style='font-size:10.0pt;font-family:Arial;color:navy;font-weight:bold;
font-style:italic'>[Morrison, Tom] <o:p></o:p></span></font></i></b></p>
<p class=MsoPlainText><b><i><font size=2 color=navy face=Arial><span
style='font-size:10.0pt;font-family:Arial;color:navy;font-weight:bold;
font-style:italic'><o:p> </o:p></span></font></i></b></p>
<p class=MsoPlainText><b><i><font size=2 color=navy face=Arial><span
style='font-size:10.0pt;font-family:Arial;color:navy;font-weight:bold;
font-style:italic'><snip some verbose explanations><o:p></o:p></span></font></i></b></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'> <o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>>> ><o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>>> > Core 1 seems to be Idle loop - happily doing
nothing<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>>> > (and not servicing
TCP and/or the console)...<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>>> ><o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>>> > Core 0 seems to be 'stuck' at the
"InstructionStorage"<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>>> > Exception. And it
seems to be going 'nowhere' fast<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>>> ><o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>>> > SRR0 seems to point to this same spot (0xc00006C0)<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>>> > SRR1 value is 0x00021200<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>>> ><o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>>> > I am at a loss to see how the kernel (and/or our kernel
BSP)<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>>> > cause this exception, and I am even more of a loss on
figuring<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>>> > out an application could cause this exception...<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>>> <o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>>> This is a bit odd as we shouldn't see an ISI from 0xc00006C0.<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>>> <o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>>> Are you able to single step Core0? Can you dump the
contents of the<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>>> TLBs on Core0<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 color=navy face="Courier New"><span
style='font-size:10.0pt;color:navy'><o:p> </o:p></span></font></p>
<p class=MsoPlainText><b><i><font size=2 color=navy face=Arial><span
style='font-size:10.0pt;font-family:Arial;color:navy;font-weight:bold;
font-style:italic'>[Morrison, Tom] <o:p></o:p></span></font></i></b></p>
<p class=MsoPlainText><b><i><font size=2 color=navy face=Arial><span
style='font-size:10.0pt;font-family:Arial;color:navy;font-weight:bold;
font-style:italic'><o:p> </o:p></span></font></i></b></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>[Morrison, Tom] <o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'><o:p> </o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'><snip some of verbose explanation><o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'><o:p> </o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>Yes, very odd...<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'><o:p> </o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>And I am able to get TLB entries from the core that is in <o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>Instruction Storage Exception, I made<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 color=navy face="Courier New"><span
style='font-size:10.0pt;color:navy'><o:p> </o:p></span></font></p>
<p class=MsoPlainText><b><i><font size=2 color=navy face=Arial><span
style='font-size:10.0pt;font-family:Arial;color:navy;font-weight:bold;
font-style:italic'>[Morrison, Tom] <o:p></o:p></span></font></i></b></p>
<p class=MsoPlainText><font size=2 color=navy face=Arial><span
style='font-size:10.0pt;font-family:Arial;color:navy'>>BKM>tat<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 color=navy face=Arial><span
style='font-size:10.0pt;font-family:Arial;color:navy'><o:p> </o:p></span></font></p>
<p class=MsoPlainText><font size=2 color=navy face=Arial><span
style='font-size:10.0pt;font-family:Arial;color:navy'>Entry
EPN RPN
TID TMASK WIMGE TSIZ U0:3 X0:1
PID TS PROT SHEN <st1:City w:st="on">UR</st1:City>
UW UX SR <st1:place w:st="on">SW
SX</st1:place> TIDZ VAL<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 color=navy face=Arial><span
style='font-size:10.0pt;font-family:Arial;color:navy'><o:p> </o:p></span></font></p>
<p class=MsoPlainText><font size=2 color=navy face=Arial><span
style='font-size:10.0pt;font-family:Arial;color:navy'>IT0
0000C000 00000000 00
000 0A
0 0 0
0 0 U P
D D D D D
D D I <o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 color=navy face=Arial><span
style='font-size:10.0pt;font-family:Arial;color:navy'>IT1
0000C000 00000000 00
000 0A
0 0 0
0 0 U P
D D D D
D D D I <o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 color=navy face=Arial><span
style='font-size:10.0pt;font-family:Arial;color:navy'>IT2
0000C000 00000000 00
000 0A
0 0 0
0 0 U P
D D D D
D D D I <o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 color=navy face=Arial><span
style='font-size:10.0pt;font-family:Arial;color:navy'>IT3
0000C000 00000000 00
000 0A
0 0 0
0 0 U P
D D D D
D D D I <o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 color=navy face=Arial><span
style='font-size:10.0pt;font-family:Arial;color:navy'>DT0
0011C000 00000000 00
000 06
0 0 0
0 0 U P
D D D D
D D D I <o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 color=navy face=Arial><span
style='font-size:10.0pt;font-family:Arial;color:navy'>DT1
D435C000 20000000 00
000 1E
0 0 0
0 0 U P
D D D D
D D D I <o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 color=navy face=Arial><span
style='font-size:10.0pt;font-family:Arial;color:navy'>DT2
0011C000 00000000 00
000 06
0 0 0
0 0 U P
D D D D
D D D I <o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 color=navy face=Arial><span
style='font-size:10.0pt;font-family:Arial;color:navy'>DT3
D435C000 20000000 00
000 1E
0 0 0
0 0 U P
D D D D
D D D I <o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 color=navy face=Arial><span
style='font-size:10.0pt;font-family:Arial;color:navy'>LT0
C0000000 00000000 00
0FF 04
9 0 0
0 0 P P
E E D E
E D D V <o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 color=navy face=Arial><span
style='font-size:10.0pt;font-family:Arial;color:navy'>LT1
D0000000 01000000 00
0FF 04
9 0 0
0 0 P P
E E D E
E D D V <o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 color=navy face=Arial><span
style='font-size:10.0pt;font-family:Arial;color:navy'>LT2
E0000000 02000000 00
0FF 04
9 0 0
0 0 P P
E E D E
E D D V <o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 color=navy face=Arial><span
style='font-size:10.0pt;font-family:Arial;color:navy'>LT3 39A40000
027FF700 0D 000
06 E A
3 0 1 U
S D D D
E E D D
I <o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 color=navy face=Arial><span
style='font-size:10.0pt;font-family:Arial;color:navy'>LT4
F924E000 7C054500 BA
000 0B
E 0 3
0 0 P S
E E D E
E D D V <o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 color=navy face=Arial><span
style='font-size:10.0pt;font-family:Arial;color:navy'>LT5 82A9F000 46664C00
FB 000
1A F 4
2 0 0
U S E E
D D E D
D I <o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 color=navy face=Arial><span
style='font-size:10.0pt;font-family:Arial;color:navy'>LT6
80000000 1F000000 F2
0FF 1D
9 B 3
0 0 U S
D E D E
E E D V <o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 color=navy face=Arial><span
style='font-size:10.0pt;font-family:Arial;color:navy'>LT7
64000000 1F000000 B3
07F 02
8 B 0
0 1 U S
D E D D
E E D V <o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 color=navy face=Arial><span
style='font-size:10.0pt;font-family:Arial;color:navy'>LT8
E5BF1000 995EA900 96
000 0C
D 8 0
0 1 U S
D E E E
E D D V <o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 color=navy face=Arial><span
style='font-size:10.0pt;font-family:Arial;color:navy'>LT9
7F3BF000 C6DF7300 DF
000 15
1 2 3
0 1 U S
E D D E
E E D I <o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 color=navy face=Arial><span
style='font-size:10.0pt;font-family:Arial;color:navy'>LT10
917C7000 EEA67F00 7F
000 17
C 5 3
0 1 P S
E E E E
E E D I <o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 color=navy face=Arial><span
style='font-size:10.0pt;font-family:Arial;color:navy'>LT11
6B000000 F5700000 BC
03F 04
7 D 0
0 1 P S
E E E E
E E D V <o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 color=navy face=Arial><span
style='font-size:10.0pt;font-family:Arial;color:navy'>LT12
712DB000 F1B59100 2A
000 19
C F 1
0 1 P S
E E E E
D E D V <o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 color=navy face=Arial><span
style='font-size:10.0pt;font-family:Arial;color:navy'>LT13 00000000
F0000000 7F 0FF
07 B 0
0 0 1 P
S D D E
E E E D
V <o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 color=navy face=Arial><span
style='font-size:10.0pt;font-family:Arial;color:navy'>LT14
A3000000 FDD00000 C5
03F 16
7 E 3
0 1 P S
E E E D
D E D V <o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 color=navy face=Arial><span
style='font-size:10.0pt;font-family:Arial;color:navy'>LT15 F7F00000
B0B80000 82 00F
1F 5 F
0 0 1
P P E E
D D D D
D V<o:p></o:p></span></font></p>
<p class=MsoPlainText><b><i><font size=2 color=navy face=Arial><span
style='font-size:10.0pt;font-family:Arial;color:navy;font-weight:bold;
font-style:italic'><o:p> </o:p></span></font></i></b></p>
<p class=MsoPlainText><font size=2 color=navy face="Courier New"><span
style='font-size:10.0pt;color:navy'>T</span></font>o answer your <font
color=navy><span style='color:navy'>2<sup>nd</sup> </span></font>question
– we have about 10 processes, and<o:p></o:p></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>about 60-70 threads total (30+ for the main processing process)...<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'><o:p> </o:p></span></font></p>
<p class=MsoPlainText><font size=2 color=navy face="Courier New"><span
style='font-size:10.0pt;color:navy'><o:p> </o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>>> > Anybody have any ideas - and/or ways to re-configure our<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>>> > setup to obtain more data? Or does this sound familiar to<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>>> > a bug somebody has already found in the kernel?<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>>> ><o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>>> > We are even having trouble defining a test program that
can<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>>> > cause (on purpose) the 'InstructionStorage' Exception
(does<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>>> > anybody have an simple 'c' (or ppc assembly) program that<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>>> > causes this exception (so we can run in user application
land<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>>> > and see if the symptoms are similar))?<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>>> ><o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>>> > Thank you in advance for any / all help you can
provide....<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>>> > because I am completely stumped on even how to proceed!<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>>> <o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>>> <o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>>> Is your application generating a lot of processes or have a
lot of<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>>> concurrent processes on the 8572?<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>>> <o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Courier New"><span style='font-size:
10.0pt'>>> - k<o:p></o:p></span></font></p>
</div>
</body>
</html>