<div dir="ltr"><div><div><div><div><div><div><div>Hello again,<br></div>Yes, it's really hard to give the exact context in an email I tried to explain as much as I could. I know it sounds very abstract, apologies for that.<br>><span class="gmail-im"><font size="2"> "the call chain itself is broken as soon as I switch to the destination", what exactly does that mean? <br></font></span></div><span class="gmail-im"><font size="2">I meant that I wasnt able to see the frames of still alive function currently in the stack</font></span><span class="gmail-im"></span>. As you guessed Dr. <span class="gmail-im"><font size="2">Weigand, I debug using gdb and tell this by doing a backtrace. The prgram doesnt crash but it's missing some frames on the way back to "main".<br><br>></font></span><span class="gmail-im"><font size="2">"the callers don't have their frames on
the stack yet" -- note that in the ELFv2 ABI, there is an area of 32
bytes at the very bottom of each frame including the back chain pointer,
the LR save area, and the TOC save area. This area formally belongs to
the *caller* but is not actually used by the caller -- the caller just
allocates it on behalf of functions it calls, which are free to use that
space; all contents of that area are maintained and used by the called
function. So if you set up only the frame of the callee, but do not
properly initialize this area which is formally part of the caller's
frame, then the callee may not function correctly (in particular when it
attempts to return).<br></font></span></div><span class="gmail-im"><font size="2">Yes backchain point was also set correctly. Since we aid the compiler by LLVM's metadata, we collected all information needed for resuming the execution on the destination architecture.<br><br>></font></span><span class="gmail-im"><font size="2">- Talking about the back chain field,
this is something that other ABIs don't have in this form. This field
links each frame to its caller's frame, so everything that copies or
recreates stack frames must of course also recompute new values for the
back chain fields (just like you apparently do for SP and FP). Do you do
that? I'm wondering because you point out that you force usage of an
FP, but on Power this is usually not necessary since we already have the
back chain that can be used for those purposes you usually use the FP
for on other ABIs.<br></font></span></div><span class="gmail-im"><font size="2">Yes, we enforced usage of FP </font></span><span class="gmail-im"><font size="2"><span class="gmail-im"><font size="2"> by simply passing </font></span><span class="gmail-st">-fno-<em>omit</em>-<em>frame</em>-<em>pointer. </em>This w</span>as a design decision</font></span><span class="gmail-st"><em> s</em>imply because we have other architectures such as x86 and arm and as youu also pointed out they use fp. And we wanted to generalize the code to support all architectures we have as much as possible (well this might not be the best idea)<br><br>></span><font size="2">This may seem pretty basic, but have you considered sigsetjmp() and siglongjmp()?<br></font></div><font size="2">No actually I haven't. But thanks for pointing this out Gerrit. We might consider this if it'll make things easier.<br><br><br></font></div><font size="2">I have an update. We resolved the issue. One thing I noticed was fp wasnt correctly set after resuming the execution on the destination. It was already pointing to the top of the stack like sp but the execution starts from the beginning of the function so the old fp needs to be saved on the stack first. This is fixed, but still the frames were missing. Also there was something wrong with how the frames were walked back. There were 2 frames with the same return address, one is a should-not-exist frame. With these we were able to solve the issue. Still I'm confused with how this code was working on other architectures properly.<br><br><br></font></div><font size="2">Thank you very much for helping me out although what I could provide you as information was not enough!<br></font></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Sep 20, 2017 at 4:50 PM, Gerrit Huizenga [Notes] <span dir="ltr"><<a href="mailto:gerrit@us.ibm.com" target="_blank">gerrit@us.ibm.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><p><font size="2">This may seem pretty basic, but have you considered sigsetjmp() and siglongjmp()?</font><br><font size="2"><br><br>gerrit<br><br>--<br>"Only those who will risk going too far can possibly find out how far one can go." ~ T. S. Eliot<br><br>Gerrit Huizenga, STSM<br>Power Open Source Ecosystem Lead<br><a href="mailto:gerrit@us.ibm.com" target="_blank">gerrit@us.ibm.com</a></font><br><br><img src="cid:1__=07BB0B32DFCFF7508f9e8a93df938690918c07B@" alt="Inactive hide details for "Ulrich Weigand" ---09/20/2017 07:07:52 AM---Hi Buse, I'm not sure I complete follow what exactly you" width="16" border="0" height="16"><font size="2" color="#424282">"Ulrich Weigand" ---09/20/2017 07:07:52 AM---Hi Buse, I'm not sure I complete follow what exactly you're doing. In particular,</font><br><br><font size="2" color="#5F5F5F">From: </font><font size="2">"Ulrich Weigand" <<a href="mailto:Ulrich.Weigand@de.ibm.com" target="_blank">Ulrich.Weigand@de.ibm.com</a>></font><br><font size="2" color="#5F5F5F">To: </font><font size="2">Buse Yilmaz <<a href="mailto:busey@vt.edu" target="_blank">busey@vt.edu</a>></font><br><font size="2" color="#5F5F5F">Cc: </font><font size="2"><a href="mailto:linuxppc-users@lists.ozlabs.org" target="_blank">linuxppc-users@lists.ozlabs.<wbr>org</a></font><br><font size="2" color="#5F5F5F">Date: </font><font size="2">09/20/2017 07:07 AM</font><br><font size="2" color="#5F5F5F">Subject: </font><font size="2">Re: [Linuxppc-users] reasigning fp breaks the call chain</font><br><font size="2" color="#5F5F5F">Sent by: </font><font size="2">"Linuxppc-users" <linuxppc-users-bounces+<wbr>gerrit=<a href="mailto:us.ibm.com@lists.ozlabs.org" target="_blank">us.ibm.com@lists.<wbr>ozlabs.org</a>></font><br></p><hr style="color:#8091a5" width="100%" size="2" noshade align="left"><span class=""><br><br><br><font size="2">Hi Buse,</font><br><font size="2"><br>I'm not sure I complete follow what exactly you're doing. In particular, when you write "the call chain itself is broken as soon as I switch to the destination", what exactly does that mean? What specifically is "broken" and what are the symptoms of that breakage? Is it that the "backtrace" command in GDB doesn't show what you expect, is it that when you start executing code the function return crashes or doesn't return to the caller you expect, or what?</font><br><font size="2"><br>Not knowing in more detail what you're doing, just some observations that make me suspicious:</font><br><font size="2"><br>- "the callers don't have their frames on the stack yet" -- note that in the ELFv2 ABI, there is an area of 32 bytes at the very bottom of each frame including the back chain pointer, the LR save area, and the TOC save area. This area formally belongs to the *caller* but is not actually used by the caller -- the caller just allocates it on behalf of functions it calls, which are free to use that space; all contents of that area are maintained and used by the called function. So if you set up only the frame of the callee, but do not properly initialize this area which is formally part of the caller's frame, then the callee may not function correctly (in particular when it attempts to return).</font><br><font size="2"><br>- Talking about the back chain field, this is something that other ABIs don't have in this form. This field links each frame to its caller's frame, so everything that copies or recreates stack frames must of course also recompute new values for the back chain fields (just like you apparently do for SP and FP). Do you do that? I'm wondering because you point out that you force usage of an FP, but on Power this is usually not necessary since we already have the back chain that can be used for those purposes you usually use the FP for on other ABIs.</font><br><font size="2"><br><br>Mit freundlichen Gruessen / Best Regards<br><br>Ulrich Weigand<br><br>-- <br>Dr. Ulrich Weigand | Phone: <a href="tel:+49%207031%20163727" value="+497031163727" target="_blank">+49-7031/16-3727</a><br>STSM, GNU/Linux compilers and toolchain<br>IBM Deutschland Research & Development GmbH<br>Vorsitzende des Aufsichtsrats: Martina Koederitz | Geschäftsführung: Dirk Wittkopp<br>Sitz der Gesellschaft: Böblingen | Registergericht: Amtsgericht Stuttgart, HRB 243294</font><br><br><img src="cid:1__=07BB0B32DFCFF7508f9e8a93df938690918c07B@" alt="Inactive hide details for Buse Yilmaz ---19.09.2017 19:08:06---Hello, I'm working on a project that does migration between mach" width="16" height="16"><font size="2" color="#424282">Buse Yilmaz ---19.09.2017 19:08:06---Hello, I'm working on a project that does migration between machines with</font><br><font size="2" color="#5F5F5F"><br>From: </font><font size="2">Buse Yilmaz <<a href="mailto:busey@vt.edu" target="_blank">busey@vt.edu</a>></font><font size="2" color="#5F5F5F"><br>To: </font><font size="2"><a href="mailto:linuxppc-users@lists.ozlabs.org" target="_blank">linuxppc-users@lists.ozlabs.<wbr>org</a></font><font size="2" color="#5F5F5F"><br>Date: </font><font size="2">19.09.2017 19:08</font><font size="2" color="#5F5F5F"><br>Subject: </font><font size="2">[Linuxppc-users] reasigning fp breaks the call chain</font><font size="2" color="#5F5F5F"><br>Sent by: </font><font size="2">"Linuxppc-users" <linuxppc-users-bounces+<wbr>uweigand=<a href="mailto:de.ibm.com@lists.ozlabs.org" target="_blank">de.ibm.com@lists.<wbr>ozlabs.org</a>></font><br><hr width="100%" size="2" noshade align="left"><br><br><br></span><span class="">Hello,<br>I'm working on a project that does migration between machines with different ISAs (currently x86_64, Aarch64 and PowerPC64). The migration is done with a compiler and runtime support based LLVM (3.7) that does stack transformation. It generates binaries for all ISAs and resumes the execution on the migrated architecture. For this purpose we record the registers and walk the call chain to record any other information needed such as callee-saved registers, live values and addresses that FP and SP point to, CFA, TOC...etc. We enforce the usage of an FP. Then create the same call chain on the destination architecture.<br><br>To test our stack transformation first we try it on the same architecture assuming we do a migration from an architecture with ISA x to the machine with the same ISA.. We get an architecture say PowerPC, divide its stack into 2 and walk the call chain on the upper partuntil the leaf fuction is hit, then switch to the lower part assuming this is the destination architecture and rewrite the frames here.<br><br>To resume the execution when we switch to the lower part of the stack, we jump to the beginning of the leaf function and attach FP and SP accordingly (we already know the whole register values of this function as well as its frame size) and load the register set with correct values.<br><br>We're able to walk the chain up on the destination and create all the call frames, however the call chain itself is broken as soon as I switch to the destination. To be more precise it's broken when I move the FP to point to SP on the destination stack (this is how LLVM does it, FP points to the top of the stack ust as SP does). So I'm left with some frames missing, no crashes but the execution is not correctly performed.<br><br>I assume that the backchain is broken on destination since we resume starting from the leaf function, at this point the callers don't have their frames on the stack yet. <br><br>I wonder if creating frames on the destination in the reverse order (a.k.a like a normal execution would do, filling the stack with frames starting from the caller not the callee.<br><br>I'm looking forward your help. apologies for what I have described being very abstract and long. <br><br>P.S. We observe this behavior on neither x86 nor ARM.<br><br>Thank you!<br><br><br><br>-- <br></span><span class="">Buse<tt><font size="2">__________________________<wbr>_____________________<br>Linuxppc-users mailing list<br><a href="mailto:Linuxppc-users@lists.ozlabs.org" target="_blank">Linuxppc-users@lists.ozlabs.<wbr>org</a></font></tt><tt><u><font size="2" color="#0000FF"><br></font></u></tt><a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.ozlabs.org_listinfo_linuxppc-2Dusers&d=DwMFAw&c=jf_iaSHvJObTbx-siA1ZOg&r=ZzfGBIaWDnwaVH6wsZg3mjvpGLeR7_C0DMvJty2qEhU&m=k-iJ7S-MAA0HA2vH-rPfBU_MmoRmlQMGxIV34mTAbBE&s=V7qn-4LRlgxQZ439jMNn7KAdjnHOCFGndfzMgPJgtZY&e=" target="_blank"><tt><u><font size="2" color="#0000FF">https://lists.ozlabs.org/<wbr>listinfo/linuxppc-users</font></u></tt></a><br><br><br><tt><font size="2">______________________________<wbr>_________________<br>Linuxppc-users mailing list<br><a href="mailto:Linuxppc-users@lists.ozlabs.org" target="_blank">Linuxppc-users@lists.ozlabs.<wbr>org</a><br></font></tt></span><tt><font size="2"><a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.ozlabs.org_listinfo_linuxppc-2Dusers&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=ZzfGBIaWDnwaVH6wsZg3mjvpGLeR7_C0DMvJty2qEhU&m=k-iJ7S-MAA0HA2vH-rPfBU_MmoRmlQMGxIV34mTAbBE&s=V7qn-4LRlgxQZ439jMNn7KAdjnHOCFGndfzMgPJgtZY&e=" target="_blank">https://urldefense.proofpoint.<wbr>com/v2/url?u=https-3A__lists.<wbr>ozlabs.org_listinfo_linuxppc-<wbr>2Dusers&d=DwIGaQ&c=jf_<wbr>iaSHvJObTbx-siA1ZOg&r=<wbr>ZzfGBIaWDnwaVH6wsZg3mjvpGLeR7_<wbr>C0DMvJty2qEhU&m=k-iJ7S-<wbr>MAA0HA2vH-rPfBU_<wbr>MmoRmlQMGxIV34mTAbBE&s=V7qn-<wbr>4LRlgxQZ439jMNn7KAdjnHOCFGndfz<wbr>MgPJgtZY&e=</a></font></tt><tt><font size="2"> <br></font></tt><br><br>
<p></p></div>
</blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">Buse</div></div>
</div>