debug problems on ppc 83xx target due to changed struct task_struct

[Holger Brunck]

On 12/08/16 18:09, Dave Hansen wrote:
> On 08/12/2016 08:47 AM, Holger Brunck wrote:
>> On 12/08/16 17:14, Dave Hansen wrote:
>>> On 08/12/2016 07:50 AM, Holger Brunck wrote:
>>>> When I try to debug our multithreaded userspace application with gdb I  get
>>>> stuck when trying to single step code.
>>>
>>> Can you clarify "stuck"?  Like the instructions don't advance?  Have you
>>> been able to find a root cause for this?
>>
>> the behaviour is slightly different on the kernel versions. So my setup is a
>> remote debug session via gdbserver.
>>
>> After connecting to the gdbserver I set a break point and start to run my
>> program. When hitting the breakpoint I try to single step. With stuck I mean
>> that the connection to the gdbserver is broken and I can't control my debug
>> session anymore while the application is not continuing.
> 
> Could you try debugging locally with gdb?  It would be nice to take all
> the stuff involved with remote debugging out of the picture.
> 

I tried this but unfortunately the error only occurs while remote debugging.
Locally with gdb everything works fine. BTW we double-checked with a 85xx ppc
target which is also 32-bit and it ends up with the same behaviour.

I was also investigating where I have to move the line in the struct task_struct
and it turns out to be like this (diff to 4.7 kernel):

diff --git a/include/linux/sched.h b/include/linux/sched.h
index 253538f..4868874 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -1655,7 +1655,9 @@ struct task_struct {
        struct signal_struct *signal;
        struct sighand_struct *sighand;

+       // struct thread_struct thread;   // until here everything is fine
        sigset_t blocked, real_blocked;
+       struct thread_struct thread;      // from here it's broken
        sigset_t saved_sigmask; /* restored if set_restore_sigmask() was used */
        struct sigpending pending;

@@ -1919,7 +1921,6 @@ struct task_struct {
        struct task_struct *oom_reaper_list;
 #endif
 /* CPU-specific state of this task */
-       struct thread_struct thread;
 /*

So it's in the area where some signal information are stored, which makes sense
because this is highly used in case of gdb debugging.

> Have you tried turning on a bunch of kernel debugging (SLAB/SLUB
> debugging, pagealloc debug, lockdep, etc...)?  If something is getting
> corrupted, those tend to catch it.
>

I switched on some memory debugging features but didn't get suspicious output.
To make the situation even more weird after enabling FTRACE in the kernel to
trace some signal code the error disappeared.

> 
> Is the process still alive at the point that the remote debugger stops
> responding?  What is it doing at that point?
> 

the process is still alive. The state of the process, it's threads and the
gdbserver is like this:

Bad case after a single step:
 73    73 TS       -   0  19   0  0.3 S    sigsuspend     gdbserver
 74    74 TS       -   0  19   0  0.0 tl+  ptrace_stop    infra_pbec83xx_
 74    77 IDL      0   -  19   0  0.0 tl+  ptrace_stop    TR_Task
 74    78 IDL      0   -  19   0  0.0 tl+  ptrace_stop    TR_Timeout
 74    79 TS       -   0  19   0  0.0 tl+  poll_schedule_ timed_msg
 74    80 IDL      0   -  19   0  0.0 tl+  ptrace_stop    stimuli
 74    81 TS       -  -5  24   0  0.0 t<l+ ptrace_stop    timer0Dflt
 74    82 TS       - -19  38   0  0.0 t<l+ futex_wait_que timerUpd0
 74    83 TS       - -19  38   0  0.0 t<l+ timerfd_read   timerClk
 74    84 TS       - -19  38   0  0.0 t<l+ ptrace_stop    b/beatWDogRefr

Good case after a single step:
 76    76 TS       -   0  19   0  4.0 S    poll_schedule_ gdbserver
 77    77 TS       -   0  19   0  0.0 tl   ptrace_stop    infra_pbec83xx_
 77    84 IDL      0   -  19   0  0.0 tl   ptrace_stop    TR_Task
 77    85 IDL      0   -  19   0  0.0 tl   ptrace_stop    TR_Timeout
 77    86 TS       -   0  19   0  0.0 tl   ptrace_stop    timed_msg
 77    87 IDL      0   -  19   0  0.0 tl   ptrace_stop    stimuli
 77    88 TS       -  -5  24   0  0.0 t<l  ptrace_stop    timer0Dflt
 77    89 TS       - -19  38   0  0.0 t<l  ptrace_stop    timerUpd0
 77    90 TS       - -19  38   0  0.0 t<l  ptrace_stop    timerClk
 77    91 TS       - -19  38   0  0.0 t<l  ptrace_stop    b/beatWDogRefr

So in the error case only some threads are at ptrace_stop, while all of them
should be after a single step with the gdb. So it's somewhere in the signal
handling between kernel and gdbserver.

Best regards
Holger Brunck