Maple PPC970 kexec crash-dump problems

[Milton Miller]

On Feb 4, 2009, at 12:48 PM, Benjamin Walsh wrote:

> Hi Milton,
>
> I've tracked it down to the device tree passed to the second kernel 
> being screwed-up when patched by kexec-tools. Namely, it was creating 
> linux,usable-memory entries that were wrong, and the MMU 
> initialization hung when it failed allocating for the page tables. I 
> hacked the tool, and got passed that point in the init sequence, but 
> the very first IO mapped access fails, so the MMU doesn't seem to be 
> set up correctly.

I would need more details exactly what you think is wrong.

How does it fail?

If the first IO mapped access fails, then I would ask if you are using 
IOMMU.  It is quite possible that the dart iommu code needs to be 
modified to use the existing mapping table instead of allocating a new 
table, otherwise any existing mappings being used by inflight dma would 
fail and the that might cause mmio loads to wait for uncompletable dma 
writes.   Just a theory with the lack of information you gave me.


>
> Anyway, up to my question: is the crash dump (kdump) kernel supposed 
> to use the memory reserved for it by the first kernel for its working 
> memory ? e.g. On that board, I have 0->2GB and 4->6GB for a total of 
> 4GB of RAM. Let's say I reserve 128M at 32M, that's 0x2000000->0xa000000. 
> Is the second kernel supposed to use
>
> (0x2000000+<kernel size>) -> 0xa000000
>
> for its memory pool and leave everything else:
>
> 0->0x2000000, 0xa000000 -> 80000000, 0x100000000 -> 0x180000000
>
> as memory that is from the first kernel, used to debug it ?


Yes, but that is not quite how the device tree is formed.

The second kernel will also use the interrupt vector area at address 0. 
  Therefore that is saved as the backup region in purgatory to the 
address allocated in the kdump region.  The device tree is then 
created with linux,usable-memory regions extending the kdump region 
back to 0, and a reserve entry marking the area as reserved.

In addition, the device tree gets the memory backing tce tables for 
pseries smp mode.  It may need the page with the dart table, marked 
reserved, so that the table gets added to the linear map -- except it 
should be mapped cache inhibited so that may not work either.

>
> Basically, I am trying to figure out if I patched the tool correctly.
>
> Thanks,
> Ben
>
> On Sat, Jan 24, 2009 at 2:52 AM, Milton Miller <miltonm at bga.com> wrote:
>> On Sat Jan 24 at 07:59:47 EST in 2009, Benjamin Walsh wrote:
>>> I am trying to use kexec with a crash dump kernel on a Maple board 
>>> (Motorola
>>>  ATCA6101 to be precise). This board is running a two-CPU PPC970FX. 
>>> I am
>>>  running a 2.6.27-10 kernel and have tried both older kexec-tools 
>>> and the
>>>  newest ones. I have tried SMP and non-SMP kernels.
>>
>>  Once you start the second cpu it is likly executing instructions 
>> somewhere.
>>
>>  Priory to 2.6.27 you had to compile a fixxed offset kerenl to run 
>> kdump.  With 2.6.27 that option was removed and replaced with teh 
>> relocatable kerenl.  However, becasue of the way linux interacts with 
>> open firmware, the kernel will still move itself to 0 unless a 
>> specific flag is set.   The location of the flag was changed twice 
>> during the merge process, and the patches for kexec-tools were not 
>> made until early this year.
>>
>>
>>> Using kexec -l to fast boot works correctly. However, loading a 
>>> crash dump
>>>  kernel and triggering a crash via echo c > /proc/sysrq-trigger 
>>> simply hangs
>>>  the board. I have traced the sequence down to after the call to
>>>  kexec_copy_flush(), when the CPU returns to real-address mode (bl
>>>  real_mode). At this point I have no further debugging information.
>>
>>
>>> Two things could help me:
>>>
>>>  - Getting the fix if this is a known issue and a fix exists. I have 
>>> looked
>>>  at recent patches and nothing lept to mind, mostly relocatable 
>>> kernel
>>>  support.
>>
>>  That is a major change.
>>
>>  That said, I don't know if anyone has tested kexec panic beyond 
>> pseries for 64 bit powerpc.
>>
>>  I know Paul originally prototyped the relocatable patch on a 
>> powermac, but I dont' know what if any smp testing he performed.   
>> And you said you are actualy on maple not a powermac, so the startup 
>> issues are different.
>>
>>
>>> - Obtaining the address of the serial port @3f8 in real mode. The 
>>> init
>>>  sequence with udbg ON says that the physical address of the port is
>>>  0xf40003f8; however, setting it up in poll mode and trying to stuff
>>>  characters in the tx buffer doesn't produce anything.
>>
>>  Ah yes.  In real mode you can only talk to cacheable memory without 
>> implementation specific assistance.  However, if you look in the 
>> kernel for the maple early udbg support, you will find the code you 
>> need to talk to that serial port in real mode.
>>
>>
>>>
>>>  Has anyone recently tried to use the serial port in real mode ?
>>>
>>>  Thanks for any help.
>>>
>>>  Ben
>>
>>  Hope this gets you started.  I wrote a lot of the kernel code, but I 
>> had the advantage of external jtag access to the processor to see 
>> where it when ended up when it went astray.
>>
>>  milton
>>