[PATCH] powerpc/rtas: Introduce rtas_get_sensor_nonblocking() for pci hotplug driver.
Mahesh Salgaonkar
mahesh at linux.ibm.com
Mon Nov 29 19:58:53 AEDT 2021
When certain PHB HW failure causes phyp to recover PHB, it marks the PE
state as temporarily unavailable until recovery is complete. This also
triggers an EEH handler in Linux which needs to notify drivers, and perform
recovery. But before notifying the driver about the pci error it uses
get_adapter_state()->get-sesnor-state() operation of the hotplug_slot to
determine if the slot contains a device or not. if the slot is empty, the
recovery is skipped entirely.
However on certain PHB failures, the rtas call get-sesnor-state() returns
extended busy error (9902) until PHB is recovered by phyp. Once PHB is
recovered, the get-sensor-state() returns success with correct presence
status. The rtas call interface rtas_get_sensor() loops over the rtas call
on extended delay return code (9902) until the return value is either
success (0) or error (-1). This causes the EEH handler to get stuck for ~6
seconds before it could notify that the pci error has been detected and
stop any active operations. Hence with running I/O traffic, during this 6
seconds, the network driver continues its operation and hits a timeout
(netdev watchdog). On timeouts, network driver go into ffdc capture mode
and reset path assuming the PCI device is in fatal condition. This
sometimes causes EEH recovery to fail. This impacts the ssh connection and
leads to the system being inaccessible.
------------
[52732.244731] DEBUG: ibm_read_slot_reset_state2()
[52732.244762] DEBUG: ret = 0, rets[0]=5, rets[1]=1, rets[2]=4000, rets[3]=>
[52732.244798] DEBUG: in eeh_slot_presence_check
[52732.244804] DEBUG: error state check
[52732.244807] DEBUG: Is slot hotpluggable
[52732.244810] DEBUG: hotpluggable ops ?
[52732.244953] DEBUG: Calling ops->get_adapter_status
[52732.244958] DEBUG: calling rpaphp_get_sensor_state
[52736.564262] ------------[ cut here ]------------
[52736.564299] NETDEV WATCHDOG: enP64p1s0f3 (tg3): transmit queue 0 timed o>
[52736.564324] WARNING: CPU: 1442 PID: 0 at net/sched/sch_generic.c:478 dev>
[...]
[52736.564505] NIP [c000000000c32368] dev_watchdog+0x438/0x440
[52736.564513] LR [c000000000c32364] dev_watchdog+0x434/0x440
------------
Fix this issue by introducing a new rtas_get_sensor_nonblocking() that does
not get blocked on BUSY condition and returns immediately with error. Use
this function in pseries pci hotplug driver which can return an error if
slot presence state can not be detected immediately. Please note that only
in certain PHB failures, the slot presence check returns BUSY condition. In
normal cases it returns immediately with a correct presence state value.
Hence this change has no impact on normal pci dlpar operations.
We could use rtas_get_sensor_fast() variant, but it thorws WARN_ON on BUSY
condition. The rtas_get_sensor_nonblocking() suppresses WARN_ON.
Signed-off-by: Mahesh Salgaonkar <mahesh at linux.ibm.com>
---
This is an alternate approach to fix the EEH issue instead of delaying slot
presence check proposed at
https://lists.ozlabs.org/pipermail/linuxppc-dev/2021-November/236956.html
Also refer:
https://lists.ozlabs.org/pipermail/linuxppc-dev/2021-November/237027.html
---
arch/powerpc/include/asm/rtas.h | 1 +
arch/powerpc/kernel/rtas.c | 19 ++++++++++++++++---
drivers/pci/hotplug/rpaphp_pci.c | 8 ++++----
3 files changed, 21 insertions(+), 7 deletions(-)
diff --git a/arch/powerpc/include/asm/rtas.h b/arch/powerpc/include/asm/rtas.h
index 9dc97d2f9d27e..d8e8befb1c193 100644
--- a/arch/powerpc/include/asm/rtas.h
+++ b/arch/powerpc/include/asm/rtas.h
@@ -250,6 +250,7 @@ extern void rtas_os_term(char *str);
void rtas_activate_firmware(void);
extern int rtas_get_sensor(int sensor, int index, int *state);
extern int rtas_get_sensor_fast(int sensor, int index, int *state);
+int rtas_get_sensor_nonblocking(int sensor, int index, int *state);
extern int rtas_get_power_level(int powerdomain, int *level);
extern int rtas_set_power_level(int powerdomain, int level, int *setlevel);
extern bool rtas_indicator_present(int token, int *maxindex);
diff --git a/arch/powerpc/kernel/rtas.c b/arch/powerpc/kernel/rtas.c
index ac61e226c9af6..fd5aa3bbd46c5 100644
--- a/arch/powerpc/kernel/rtas.c
+++ b/arch/powerpc/kernel/rtas.c
@@ -609,7 +609,8 @@ int rtas_get_sensor(int sensor, int index, int *state)
}
EXPORT_SYMBOL(rtas_get_sensor);
-int rtas_get_sensor_fast(int sensor, int index, int *state)
+static int
+__rtas_get_sensor(int sensor, int index, int *state, bool warn_on)
{
int token = rtas_token("get-sensor-state");
int rc;
@@ -618,14 +619,26 @@ int rtas_get_sensor_fast(int sensor, int index, int *state)
return -ENOENT;
rc = rtas_call(token, 2, 2, state, sensor, index);
- WARN_ON(rc == RTAS_BUSY || (rc >= RTAS_EXTENDED_DELAY_MIN &&
- rc <= RTAS_EXTENDED_DELAY_MAX));
+ WARN_ON(warn_on &&
+ (rc == RTAS_BUSY || (rc >= RTAS_EXTENDED_DELAY_MIN &&
+ rc <= RTAS_EXTENDED_DELAY_MAX)));
if (rc < 0)
return rtas_error_rc(rc);
return rc;
}
+int rtas_get_sensor_fast(int sensor, int index, int *state)
+{
+ return __rtas_get_sensor(sensor, index, state, true);
+}
+
+int rtas_get_sensor_nonblocking(int sensor, int index, int *state)
+{
+ return __rtas_get_sensor(sensor, index, state, false);
+}
+EXPORT_SYMBOL(rtas_get_sensor_nonblocking);
+
bool rtas_indicator_present(int token, int *maxindex)
{
int proplen, count, i;
diff --git a/drivers/pci/hotplug/rpaphp_pci.c b/drivers/pci/hotplug/rpaphp_pci.c
index c380bdacd1466..8a7d681254ce9 100644
--- a/drivers/pci/hotplug/rpaphp_pci.c
+++ b/drivers/pci/hotplug/rpaphp_pci.c
@@ -23,7 +23,7 @@ int rpaphp_get_sensor_state(struct slot *slot, int *state)
int rc;
int setlevel;
- rc = rtas_get_sensor(DR_ENTITY_SENSE, slot->index, state);
+ rc = rtas_get_sensor_nonblocking(DR_ENTITY_SENSE, slot->index, state);
if (rc < 0) {
if (rc == -EFAULT || rc == -EEXIST) {
@@ -38,10 +38,10 @@ int rpaphp_get_sensor_state(struct slot *slot, int *state)
if (rc < 0) {
dbg("%s: power on slot[%s] failed rc=%d.\n",
__func__, slot->name, rc);
- } else {
- rc = rtas_get_sensor(DR_ENTITY_SENSE,
- slot->index, state);
+ return rc;
}
+ rc = rtas_get_sensor_nonblocking(DR_ENTITY_SENSE,
+ slot->index, state);
} else if (rc == -ENODEV)
info("%s: slot is unusable\n", __func__);
else
More information about the Linuxppc-dev
mailing list