[Skiboot] [PATCH v2 09/15] opal/hmi: Fix soft lockups during TOD errors

[Mahesh J Salgaonkar]

From: Mahesh Salgaonkar <mahesh at linux.vnet.ibm.com>

There are some TOD errors which do not affect working of TOD and TB. They
stay in valid state. Hence we don't need rendez vous for TOD errors that
does not affect TB working.

TOD errors that affects TOD/TB will report a global error on TFMR[44]
alongwith bit 51, and they will go in rendez vous path as expected.

But the TOD errors that does not affect TB register sets only TFMR bit 51.
The TFMR bit 51 is cleared when any single thread clears the TOD error.
Once cleared, the bit 51 is reflected to all the cores on that chip. Any
thread that reads the TFMR register after the error is cleared will see
TFMR bit 51 reset. Hence the threads that see TFMR[51]=1, falls through
rendez-vous path and threads that see TFMR[51]=0, returns doing
nothing. This ends up in a soft lockups in host kernel.

This patch fixes this issue by not considering TOD interrupt (TFMR[51])
as a core-global error and hence avoiding rendez-vous path completely.
Instead threads that see TFMR[51]=1 will now take different path that
just do the TOD error recovery.

Signed-off-by: Mahesh Salgaonkar <mahesh at linux.vnet.ibm.com>
---
 core/hmi.c        |   16 +++++++++++++++-
 hw/chiptod.c      |   14 ++++++++++++--
 include/chiptod.h |    1 +
 3 files changed, 28 insertions(+), 3 deletions(-)

diff --git a/core/hmi.c b/core/hmi.c
index eadb75be4..d9dd83c62 100644
--- a/core/hmi.c
+++ b/core/hmi.c
@@ -204,7 +204,7 @@
 	(SPR_TFMR_TBST_CORRUPT | SPR_TFMR_TB_MISSING_SYNC |	\
 	 SPR_TFMR_TB_MISSING_STEP | SPR_TFMR_FW_CONTROL_ERR |	\
 	 SPR_TFMR_TFMR_CORRUPT | SPR_TFMR_TB_RESIDUE_ERR |	\
-	 SPR_TFMR_HDEC_PARITY_ERROR | SPR_TFMR_CHIP_TOD_INTERRUPT)
+	 SPR_TFMR_HDEC_PARITY_ERROR)
 
 /* TFMR "thread" errors  */
 #define SPR_TFMR_THREAD_ERRORS \
@@ -1101,6 +1101,20 @@ static int handle_tfac_errors(uint64_t hmer, struct OpalHMIEvent *hmi_evt,
 		recover2 = handle_all_core_tfac_error(tfmr, out_flags);
 		if (recover != 0)
 			recover = recover2;
+	} else if (tfmr & SPR_TFMR_CHIP_TOD_INTERRUPT) {
+		int recover2;
+
+		/*
+		 * There are some TOD errors which do not affect working of
+		 * TOD and TB. They stay in valid state. Hence we don't need
+		 * rendez vous.
+		 *
+		 * TOD errors that affects TOD/TB will report a global error
+		 * on TFMR alongwith bit 51, and they will go in rendez vous.
+		 */
+		recover2 = chiptod_recover_tod_errors();
+		if (recover != 0)
+			recover = recover2;
 	} else if (this_cpu()->tb_invalid) {
 		/* This shouldn't happen, TB is invalid and no global error
 		 * was reported. We just return for now assuming one will
diff --git a/hw/chiptod.c b/hw/chiptod.c
index f6ef9a469..33d553956 100644
--- a/hw/chiptod.c
+++ b/hw/chiptod.c
@@ -970,7 +970,7 @@ bool chiptod_wakeup_resync(void)
 	return false;
 }
 
-static int chiptod_recover_tod_errors(void)
+static int __chiptod_recover_tod_errors(void)
 {
 	uint64_t terr;
 	uint64_t treset = 0;
@@ -1026,6 +1026,16 @@ static int chiptod_recover_tod_errors(void)
 	return 1;
 }
 
+int chiptod_recover_tod_errors(void)
+{
+	int rc;
+
+	lock(&chiptod_lock);
+	rc = __chiptod_recover_tod_errors();
+	unlock(&chiptod_lock);
+	return rc;
+}
+
 static int32_t chiptod_get_active_master(void)
 {
 	if (current_topology < 0)
@@ -1550,7 +1560,7 @@ int chiptod_recover_tb_errors(bool *out_resynced)
 	 * Bit 33 of TOD error register indicates sync check error.
 	 */
 	if (tfmr & SPR_TFMR_CHIP_TOD_INTERRUPT)
-		rc = chiptod_recover_tod_errors();
+		rc = __chiptod_recover_tod_errors();
 
 	/* Check if TB is running. If not then we need to get it running. */
 	if (!(tfmr & SPR_TFMR_TB_VALID)) {
diff --git a/include/chiptod.h b/include/chiptod.h
index 667e6fd83..5860e34d2 100644
--- a/include/chiptod.h
+++ b/include/chiptod.h
@@ -38,5 +38,6 @@ extern void chiptod_reset_tb(void);
 extern bool chiptod_adjust_topology(enum chiptod_topology topo, bool enable);
 extern bool chiptod_capp_timebase_sync(unsigned int chip_id, uint32_t tfmr_addr,
 				       uint32_t tb_addr, uint32_t offset);
+extern int chiptod_recover_tod_errors(void);
 
 #endif /* __CHIPTOD_H */