[Issue Report] Kernel panic while running "openssl engine" within cryptodev kernel module inserted
Neal Liu
neal_liu at aspeedtech.com
Wed Feb 16 20:09:13 AEDT 2022
Hi all,
I would like to report an issue about OpenBMC + OpenSSL + Cryptodev-linux combination running on Aspeed ast2600 platform.
Kernel panic occurs while running "openssl engine" command. Will put more detail logs below.
After digging in more, I figure out an workaround to avoid this issue.
I'm wondering whether this bug is related to gcc compiler or not?
Did anybody met this issue before?
Any suggestion is appreciated.
Workaround:
- Force enable CONFIG_GCC_PLUGINS config in Linux-5.15
Here are the detail steps to reproduce the issue:
1. Get latest OpenBMC source
* $ git clone git at github.com:openbmc/openbmc.git
2. Setup ast2600 target
* $ . setup evb-ast2600
3. Apply below patches to support openssl and cryptodev-linux kernel modules
* Linux defconfig
diff --git a/meta-aspeed/recipes-kernel/linux/linux-aspeed/aspeed-g6/defconfig b/meta-aspeed/recipes-kernel/linux/linux-aspeed/aspeed-g6/defconfig
index 451afd81a..d4b0ba4d8 100644
--- a/meta-aspeed/recipes-kernel/linux/linux-aspeed/aspeed-g6/defconfig
+++ b/meta-aspeed/recipes-kernel/linux/linux-aspeed/aspeed-g6/defconfig
@@ -289,3 +289,4 @@ CONFIG_DEBUG_LIST=y
CONFIG_FUNCTION_TRACER=y
CONFIG_DEBUG_USER=y
# CONFIG_RUNTIME_TESTING_MENU is not set
+CONFIG_MODULES=y
* Use hw engine
diff --git a/meta-phosphor/recipes-connectivity/openssl/openssl_%.bbappend b/meta-phosphor/recipes-connectivity/openssl/openssl_%.bbappend
index 0581dcd63..c51b8ecde 100644
--- a/meta-phosphor/recipes-connectivity/openssl/openssl_%.bbappend
+++ b/meta-phosphor/recipes-connectivity/openssl/openssl_%.bbappend
@@ -1,5 +1,6 @@
# General config settings.
-EXTRA_OECONF:append:class-target = " shared no-hw no-err no-psk no-srp "
+EXTRA_OECONF:append:class-target = " shared no-err no-psk no-srp no-dynamic-engine "
# Disable SSL (keep TLS only).
EXTRA_OECONF:append:class-target = " no-ssl2 no-ssl3 "
* Install libcrypto & libssl & openssl stuffs
diff --git a/meta-phosphor/recipes-phosphor/images/obmc-phosphor-image.bb b/meta-phosphor/recipes-phosphor/images/obmc-phosphor-image.bb
index 494b06c7c..4fa2efcad 100644
--- a/meta-phosphor/recipes-phosphor/images/obmc-phosphor-image.bb
+++ b/meta-phosphor/recipes-phosphor/images/obmc-phosphor-image.bb
@@ -52,3 +52,12 @@ ROOTFS_POSTPROCESS_COMMAND += "remove_etc_version ; "
# The shadow recipe provides the binaries(like useradd, usermod) needed by the
# phosphor-user-manager.
ROOTFS_RO_UNNEEDED:remove = "shadow"
+
+IMAGE_INSTALL:append = " \
+ libcrypto \
+ libssl \
+ openssl \
+ openssl-bin \
+ openssl-conf \
+ openssl-engines \
+ "
* Enable cryptodev-linux
diff --git a/poky/meta/recipes-connectivity/openssl/openssl_3.0.1.bb b/poky/meta/recipes-connectivity/openssl/openssl_3.0.1.bb
index 7727ec43e..15fd68e8a 100644
--- a/poky/meta/recipes-connectivity/openssl/openssl_3.0.1.bb
+++ b/poky/meta/recipes-connectivity/openssl/openssl_3.0.1.bb
@@ -27,6 +27,7 @@ MULTILIB_SCRIPTS = "${PN}-bin:${bindir}/c_rehash"
PACKAGECONFIG ?= ""
PACKAGECONFIG:class-native = ""
PACKAGECONFIG:class-nativesdk = ""
+PACKAGECONFIG:class-target = "cryptodev-linux"
PACKAGECONFIG[cryptodev-linux] = "enable-devcryptoeng,disable-devcryptoeng,cryptodev-linux,,cryptodev-module"
PACKAGECONFIG[no-tls1] = "no-tls1"
4. Build obmc-phosphor-image & running on ast2600
5. Insert cryptodev kernel module
* $ modprobe cryptodev
i. [ 136.485187] cryptodev: loading out-of-tree module taints kernel.
ii. [ 136.494384] cryptodev: driver 1.12 loaded.
6. Running openssl engine to check hw engine is supported
* $ openssl engine
7. Kernel panic
root at ast2600-default:~# openssl engine
[ 165.288203] Kernel panic - not syncing: stack-protector: Kernel stack is corrupted in: cryptodev_ioctl+0xc60/0xd30 [cryptodev]
[ 165.300953] CPU: 0 PID: 501 Comm: openssl Tainted: G O 5.15.0-dirty-0e311ef17334 #1
[ 165.310961] Hardware name: Generic DT based system
[ 165.316307] Backtrace:
[ 165.319038] [<80bc7708>] (dump_backtrace) from [<80bc7950>] (show_stack+0x20/0x24)
[ 165.327507] r7:80d665f8 r6:8100e768 r5:60000093 r4:80d7a02c
[ 165.333819] [<80bc7930>] (show_stack) from [<80bcc48c>] (dump_stack_lvl+0x48/0x54)
[ 165.342279] [<80bcc444>] (dump_stack_lvl) from [<80bcc4b0>] (dump_stack+0x18/0x1c)
[ 165.350738] r5:00000000 r4:811003a8
[ 165.354722] [<80bcc498>] (dump_stack) from [<80bc7d48>] (panic+0x108/0x330)
[ 165.362504] [<80bc7c40>] (panic) from [<80bd516c>] (rcu_dynticks_inc+0x0/0x44)
[ 165.370579] r3:81538d80 r2:861e7200 r1:7f0024c0 r0:80d665f8
[ 165.376893] r7:8486c000
[ 165.379714] [<80bd5150>] (__stack_chk_fail) from [<7f0024c0>] (cryptodev_ioctl+0xc60/0xd30 [cryptodev])
[ 165.390223] [<7f001860>] (cryptodev_ioctl [cryptodev]) from [<80309f44>] (sys_ioctl+0x570/0xc44)
[ 165.400055] r10:00000036 r9:00000003 r8:84b19f00 r7:76f15c60 r6:7ed8f728 r5:84b19f00
[ 165.408792] r4:8004636a
[ 165.411613] [<803099d4>] (sys_ioctl) from [<80100060>] (ret_fast_syscall+0x0/0x48)
[ 165.420071] Exception stack(0x8486dfa8 to 0x8486dff0)
[ 165.425712] dfa0: 76f15b60 76f14824 00000003 8004636a 76f15c60 00a82808
[ 165.434842] dfc0: 76f15b60 76f14824 7ed8f728 00000036 00000003 76d30e5c 7ed8fde0 00000000
[ 165.443969] dfe0: 76f14944 7ed8f684 76e2d758 76cae67c
[ 165.449609] r10:00000036 r9:8486c000 r8:80100244 r7:00000036 r6:7ed8f728 r5:76f14824
[ 165.458337] r4:76f15b60
[ 165.461168] CPU1: stopping
[ 165.464202] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G O 5.15.0-dirty-0e311ef17334 #1
[ 165.474205] Hardware name: Generic DT based system
[ 165.479551] Backtrace:
[ 165.482283] [<80bc7708>] (dump_backtrace) from [<80bc7950>] (show_stack+0x20/0x24)
[ 165.490749] r7:81418e40 r6:00000001 r5:600f0193 r4:80d7a02c
[ 165.497061] [<80bc7930>] (show_stack) from [<80bcc48c>] (dump_stack_lvl+0x48/0x54)
[ 165.505519] [<80bcc444>] (dump_stack_lvl) from [<80bcc4b0>] (dump_stack+0x18/0x1c)
[ 165.513979] r5:81006654 r4:811002c8
[ 165.517963] [<80bcc498>] (dump_stack) from [<8010ff64>] (do_handle_IPI+0x2c8/0x2fc)
[ 165.526511] [<8010fc9c>] (do_handle_IPI) from [<8010ffc0>] (ipi_handler+0x28/0x30)
[ 165.534971] r9:81552000 r8:bc80200c r7:81418e40 r6:00000014 r5:81006654 r4:814da200
[ 165.543610] [<8010ff98>] (ipi_handler) from [<80187a6c>] (handle_percpu_devid_irq+0x8c/0x1e0)
[ 165.553135] [<801879e0>] (handle_percpu_devid_irq) from [<80180e80>] (handle_domain_irq+0x6c/0x88)
[ 165.563150] r7:00000004 r6:00000000 r5:00000000 r4:80f85030
[ 165.569461] [<80180e14>] (handle_domain_irq) from [<80101230>] (gic_handle_irq+0x78/0x8c)
[ 165.578600] r7:80f8503c r6:bc802000 r5:81553f38 r4:81006654
[ 165.584911] [<801011b8>] (gic_handle_irq) from [<80100afc>] (__irq_svc+0x5c/0x78)
[ 165.593269] Exception stack(0x81553f38 to 0x81553f80)
[ 165.598904] 3f20: 000a49a8 00000000
[ 165.608035] 3f40: 00000001 8011c4e0 81552000 00000001 81005f50 81005f90 810ced6f 80d69660
[ 165.617165] 3f60: 00000000 81553f94 81553f98 81553f88 80108df0 80108df4 600f0013 ffffffff
[ 165.626294] r9:81552000 r8:810ced6f r7:81553f6c r6:ffffffff r5:600f0013 r4:80108df4
[ 165.634932] [<80108dac>] (arch_cpu_idle) from [<80bdd378>] (default_idle_call+0x38/0xc8)
[ 165.643972] [<80bdd340>] (default_idle_call) from [<8015df78>] (do_idle+0xd8/0x144)
[ 165.652526] [<8015dea0>] (do_idle) from [<8015e314>] (cpu_startup_entry+0x28/0x2c)
[ 165.660988] r9:410fc075 r8:8000406a r7:81553ff8 r6:10c0387d r5:00000001 r4:00000096
[ 165.669627] [<8015e2ec>] (cpu_startup_entry) from [<80110530>] (secondary_start_kernel+0x160/0x184)
[ 165.679732] [<801103d0>] (secondary_start_kernel) from [<801016f0>] (__enable_mmu+0x0/0x30)
[ 165.689062] r5:00000051 r4:8156806a
Other information:
root at ast2600-default:~# openssl version
OpenSSL 1.1.1g 21 Apr 2020
root at ast2600-default:~# modinfo cryptodev
filename: /lib/modules/5.15.0-dirty-0e311ef17334/extra/cryptodev.ko
license: GPL
description: CryptoDev driver
author: Nikos Mavrogiannopoulos <nmav at gnutls.org>
depends:
name: cryptodev
vermagic: 5.15.0-dirty-0e311ef17334 SMP mod_unload ARMv7 p2v8
parm: cryptodev_verbosity:0: normal, 1: verbose, 2: debug (int)
Thanks
Best Regards,
-Neal
More information about the openbmc
mailing list