[Skiboot] [PATCH 2/2] hw/phb3: Ensure PQ bits are cleared in the IVC when masking IRQ

[Michael Neuling]

When we mask an interrupt, we may race with another interrupt coming
in from the hardware.  If this occurs, the P and/or Q bit may end up
being set but we never EOI/clear them.  This could result in a lost
interrupt or the next interrupt that comes in after re-enabling never
being presented.

This patch ensures that when masking an interrupt, any pending P/Q
bits are cleared.

This fixes a bug seen with some CAPI workloads which have lots of
interrupt masking at the same time as high interrupt load.  The fix is
not specific to CAPI though.

Signed-off-by: Michael Neuling <mikey at neuling.org>
---
 hw/phb3.c | 32 ++++++++++++++++++++++++++------
 1 file changed, 26 insertions(+), 6 deletions(-)

diff --git a/hw/phb3.c b/hw/phb3.c
index a11f84e..c5c6fe3 100644
--- a/hw/phb3.c
+++ b/hw/phb3.c
@@ -1698,7 +1698,7 @@ static int64_t phb3_msi_set_xive(void *data,
 {
 	struct phb3 *p = data;
 	uint32_t chip, index;
-	uint64_t *cache, ive_num, data64, m_server, m_prio;
+	uint64_t *cache, ive_num, data64, m_server, m_prio, ivc;
 	uint32_t *ive;
 
 	chip = p8_irq_to_chip(isn);
@@ -1743,14 +1743,34 @@ static int64_t phb3_msi_set_xive(void *data,
 	*ive = (m_server << 8) | m_prio;
 	out_be64(p->regs + PHB_IVC_UPDATE, data64);
 
-	/*
-	 * Handle Q bit if we're going to enable the interrupt.
-	 * The OS should make sure the interrupt handler has
-	 * been installed already.
-	 */
 	if (prio != 0xff) {
+		/*
+		 * Handle Q bit if we're going to enable the
+		 * interrupt.  The OS should make sure the interrupt
+		 * handler has been installed already.
+		 */
 		if (phb3_pci_msi_check_q(p, ive_num))
 			phb3_pci_msi_flush_ive(p, ive_num);
+	} else {
+		/* Read from random PHB reg to force flush */
+		in_be64(p->regs + PHB_IVC_UPDATE);
+
+		/* Order with subsequent read of Q */
+		sync();
+
+		/* Clear P, Q and Gen, preserve PE# */
+		ive[1] &= 0x0000ffff;
+
+		/*
+		 * Update the IVC with a match against the old gen
+		 * count. No need to worry about racing with P being
+		 * set in the cache since IRQ is masked at this point.
+		 */
+		ivc = SETFIELD(PHB_IVC_UPDATE_SID, 0ul, ive_num) |
+			PHB_IVC_UPDATE_ENABLE_P |
+			PHB_IVC_UPDATE_ENABLE_Q |
+			PHB_IVC_UPDATE_ENABLE_GEN;
+		out_be64(p->regs + PHB_IVC_UPDATE, ivc);
 	}
 
 	return OPAL_SUCCESS;
-- 
2.5.0