[PATCH v2] powerpc/xics,icp-opal: Fix CPPR setting for icp-opal

[Balbir Singh]

CPPR (Current Processor Priority Register) emulation on icp-opal
uses a single priority in the backend and that can cause CPU
hotplug to be affected when we try to send an IPI to it.

The fix is in migrate_irqs_away, the fix does the following:

1. It moves the setting of CPPR to after all IRQ migration
   is complete
2. In icp-opal, we ignore masking the CPPR when the CPPR
is set to default priority, anything else we just pass
through

Right now this fix is designed with backporting to stable
in mind

We'll need a newer version of this later when we merge the
DEFAULT and IPI priorities to the same value in the kernel.

Fixes: d74361881f0d ("powerpc/xics: Add ICP OPAL backend")
Cc: stable at vger.kernel.org (v4.8+)

Testing

1. I've tested this fix on a virtual partition running under
   kvm
2. Gautham and Vaidy have done some testing on an system that
   uses the OPAL backend
3. I've also tested this on a system with a native XICS controller

Suggested-by: Michael Ellerman <mpe at ellerman.id.au>
Reported-by: Vaidyanathan Srinivasan <svaidy at linux.vnet.ibm.com>
Tested-by: Vaidyanathan Srinivasan <svaidy at linux.vnet.ibm.com>
Signed-off-by: Balbir Singh <bsingharora at gmail.com>
---
 arch/powerpc/sysdev/xics/icp-opal.c    | 12 ++++++++++++
 arch/powerpc/sysdev/xics/xics-common.c | 19 ++++++++++++++++---
 2 files changed, 28 insertions(+), 3 deletions(-)

diff --git a/arch/powerpc/sysdev/xics/icp-opal.c b/arch/powerpc/sysdev/xics/icp-opal.c
index f9670ea..77cdb36 100644
--- a/arch/powerpc/sysdev/xics/icp-opal.c
+++ b/arch/powerpc/sysdev/xics/icp-opal.c
@@ -91,6 +91,18 @@ static unsigned int icp_opal_get_irq(void)
 
 static void icp_opal_set_cpu_priority(unsigned char cppr)
 {
+	/*
+	 * Mask only IPI's and not external interrupts
+	 * XIVE emulation of XICS exposed as an OPAL
+	 * interface uses just one priority level for
+	 * both external and IPI's. By dropping default
+	 * priority to lowest, we leave all interrupts
+	 * on, assuming the only caller is migrate_irqs_away
+	 * at the time of CPU hotplug
+	 */
+	if (cppr == DEFAULT_PRIORITY)
+		cppr = LOWEST_PRIORITY;
+
 	xics_set_base_cppr(cppr);
 	opal_int_set_cppr(cppr);
 	iosync();
diff --git a/arch/powerpc/sysdev/xics/xics-common.c b/arch/powerpc/sysdev/xics/xics-common.c
index 69d858e..f025320 100644
--- a/arch/powerpc/sysdev/xics/xics-common.c
+++ b/arch/powerpc/sysdev/xics/xics-common.c
@@ -20,6 +20,7 @@
 #include <linux/of.h>
 #include <linux/slab.h>
 #include <linux/spinlock.h>
+#include <linux/delay.h>
 
 #include <asm/prom.h>
 #include <asm/io.h>
@@ -198,9 +199,6 @@ void xics_migrate_irqs_away(void)
 	/* Remove ourselves from the global interrupt queue */
 	xics_set_cpu_giq(xics_default_distrib_server, 0);
 
-	/* Allow IPIs again... */
-	icp_ops->set_priority(DEFAULT_PRIORITY);
-
 	for_each_irq_desc(virq, desc) {
 		struct irq_chip *chip;
 		long server;
@@ -255,6 +253,21 @@ void xics_migrate_irqs_away(void)
 unlock:
 		raw_spin_unlock_irqrestore(&desc->lock, flags);
 	}
+
+	/*
+	 * Allow sufficient time to drop any inflight IRQ's
+	 */
+	mdelay(1);
+
+	/*
+	 * Allow IPIs again. This is done at the very end, after
+	 * migrating all interrupts, the expectation is that we'll
+	 * only get woken up by an IPI interrupt beyond this point.
+	 * For this to work, we have a hack when icp_ops are OPAL
+	 * ops.
+	 */
+	icp_ops->set_priority(DEFAULT_PRIORITY);
+
 }
 #endif /* CONFIG_HOTPLUG_CPU */
 
-- 
2.9.3