<html><head><meta http-equiv="Content-Type" content="text/html; charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On 21-Oct-2021, at 11:03 PM, Nathan Lynch <<a href="mailto:nathanl@linux.ibm.com" class="">nathanl@linux.ibm.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><meta charset="UTF-8" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">Nicholas Piggin <</span><a href="mailto:npiggin@gmail.com" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">npiggin@gmail.com</a><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">> writes:</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><blockquote type="cite" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">Excerpts from Athira Rajeev's message of July 11, 2021 10:25 pm:<br class=""><blockquote type="cite" class="">During Live Partition Migration (LPM), it is observed that perf<br class="">counter values reports zero post migration completion. However<br class="">'perf stat' with workload continues to show counts post migration<br class="">since PMU gets disabled/enabled during sched switches. But incase<br class="">of system/cpu wide monitoring, zero counts were reported with 'perf<br class="">stat' after migration completion.<br class=""><br class="">Example:<br class="">./perf stat -e r1001e -I 1000<br class=""> time counts unit events<br class=""> 1.001010437 22,137,414 r1001e<br class=""> 2.002495447 15,455,821 r1001e<br class=""><<>> As seen in next below logs, the counter values shows zero<br class=""> after migration is completed.<br class=""><<>><br class=""> 86.142535370 129,392,333,440 r1001e<br class=""> 87.144714617 0 r1001e<br class=""> 88.146526636 0 r1001e<br class=""> 89.148085029 0 r1001e<br class=""><br class="">Here PMU is enabled during start of perf session and counter<br class="">values are read at intervals. Counters are only disabled at the<br class="">end of session. The powerpc mobility code presently does not handle<br class="">disabling and enabling back of PMU counters during partition<br class="">migration. Also since the PMU register values are not saved/restored<br class="">during migration, PMU registers like Monitor Mode Control Register 0<br class="">(MMCR0), Monitor Mode Control Register 1 (MMCR1) will not contain<br class="">the value it was programmed with. Hence PMU counters will not be<br class="">enabled correctly post migration.<br class=""><br class="">Fix this in mobility code by handling disabling and enabling of<br class="">PMU in all cpu's before and after migration. Patch introduces two<br class="">functions 'mobility_pmu_disable' and 'mobility_pmu_enable'.<br class="">mobility_pmu_disable() is called before the processor threads goes<br class="">to suspend state so as to disable the PMU counters. And disable is<br class="">done only if there are any active events running on that cpu.<br class="">mobility_pmu_enable() is called after the processor threads are<br class="">back online to enable back the PMU counters.<br class=""><br class="">Since the performance Monitor counters ( PMCs) are not<br class="">saved/restored during LPM, results in PMC value being zero and the<br class="">'event->hw.prev_count' being non-zero value. This causes problem<br class=""></blockquote><br class="">Interesting. Are they defined to not be migrated, or may not be<span class="Apple-converted-space"> </span><br class="">migrated?<br class=""></blockquote><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">PAPR may be silent on this... at least I haven't found anything yet. But</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">I'm not very familiar with perf counters.</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">How much assurance do we have that hardware events we've programmed on</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">the source can be reliably re-enabled on the destination, with the same</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">semantics? Aren't there some model-specific counters that don't make</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">sense to handle this way?</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><blockquote type="cite" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><blockquote type="cite" class="">diff --git a/arch/powerpc/include/asm/rtas.h b/arch/powerpc/include/asm/rtas.h<br class="">index 9dc97d2..cea72d7 100644<br class="">--- a/arch/powerpc/include/asm/rtas.h<br class="">+++ b/arch/powerpc/include/asm/rtas.h<br class="">@@ -380,5 +380,13 @@ static inline void rtas_initialize(void) { }<br class="">static inline void read_24x7_sys_info(void) { }<br class="">#endif<br class=""><br class="">+#ifdef CONFIG_PPC_PERF_CTRS<br class="">+void mobility_pmu_disable(void);<br class="">+void mobility_pmu_enable(void);<br class="">+#else<br class="">+static inline void mobility_pmu_disable(void) { }<br class="">+static inline void mobility_pmu_enable(void) { }<br class="">+#endif<br class="">+<br class="">#endif /* __KERNEL__ */<br class="">#endif /* _POWERPC_RTAS_H */<br class=""></blockquote><br class="">It's not implemented in rtas, maybe consider putting this into a perf<span class="Apple-converted-space"> </span><br class="">header?<br class=""></blockquote><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">+1</span></div></blockquote></div><br class=""><div class="">Sure, <span style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);" class="">I will move this to perf_event header file</span></div><div class=""><span style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);" class=""><br class=""></span></div><div class=""><span style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);" class="">Thanks</span></div><div class=""><span style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);" class="">Athira</span></div></body></html>