OpenBMC - Support NVMe drive health monitoring

Tue Apr 11 17:06:08 AEST 2023

Thanks Hao,

This looks very promising.
I would appreciate some more details as I am new to OpenBMC.

  1.  If I understand correctly, in order to run this tool as OOB, it needs to be compiled for OpenBMC platform and somehow be included in the image.
     *   If so, can you please guide me how to do that?
     *   Can this tool be tested on QEMU or RaspberryPi (running OpenBMC image)?
     *   I found on OpenBMC the meta-openembedded/meta-oe/recipes-bsp/nvme-cli/nvme-cli_1.13.bb but it looks like it is pointing to an old version of the repo (commit f0e9569df9289d6ee55ba2c23615cc7c73a9b088 from Oct 21, 2020). Should I change that?
     *   It looks like this cli tool also needs an updated (latest) version of libnvme. How to make sure the OpenBMC image is built with latest version of this library?
  2.  When the cli tool is used for OOB, it needs to have mctp:<net>,<eid>[:ctrl-id].

How do I obtain this information (net, eid, ctrl-id)?

Is there a tool that runs auto discovery and will give a map of the devices it finds? (kind of like nvme list)?

Thanks again,
Kind regards,
Lior.

From: Hao Jiang <jianghao at google.com>
Sent: Monday, April 10, 2023 7:28 PM
To: Lior Weintraub <liorw at pliops.com>
Cc: Andrew Jeffery <andrew at aj.id.au>; Ed Tanous <ed at tanous.net>; openbmc at lists.ozlabs.org; rashmica at linux.ibm.com
Subject: Re: OpenBMC - Support NVMe drive health monitoring

CAUTION: External Sender
We have upstreamed the support for libnvme-mi in the opensource nvme-cli tool. https://github.com/linux-nvme/nvme-cli

The cli tool has only nvme admin command set support, since it is co-used for inband and oob.

For the MI cmd set, we have a really simple demo tool, but it doesn't have all MI CMD (it is a demo only :) )
https://github.com/linux-nvme/libnvme/blob/master/examples/mi-mctp.c

On Sat, Apr 8, 2023 at 2:42 AM Lior Weintraub <liorw at pliops.com<mailto:liorw at pliops.com>> wrote:
Thanks Hao,

Do you know if there is a tool (e.g. ipmitool) that can be used to generate a basic NVMe MI command?

For example, the NVMe MI Revision 1.1 spec defines a "Management Interface Command Set" that can be used to:

Opcode                Command

00h                        Read NVMe-MI Data Structure

01h                        NVM Subsystem Health Status Poll

02h                        Controller Health Status Poll

03h                        Configuration Set

04h                        Configuration Get

05h                        VPD Read

06h                        VPD Write

07h                        Reset

08h                        SES Receive

09h                        SES Send

0Ah                        Management Endpoint Buffer Read

0Bh                        Management Endpoint Buffer Write

0Ch to BFh          Reserved

C0h to FFh           Vendor specific

If we had a tool that can send such a command it would help with the development of this feature.

The idea is to start with some manually triggered commands.

Thanks,

Lior.

From: Hao Jiang <jianghao at google.com<mailto:jianghao at google.com>>
Sent: Friday, April 7, 2023 8:40 PM
To: Lior Weintraub <liorw at pliops.com<mailto:liorw at pliops.com>>
Cc: Andrew Jeffery <andrew at aj.id.au<mailto:andrew at aj.id.au>>; Ed Tanous <ed at tanous.net<mailto:ed at tanous.net>>; openbmc at lists.ozlabs.org<mailto:openbmc at lists.ozlabs.org>; rashmica at linux.ibm.com<mailto:rashmica at linux.ibm.com>
Subject: Re: OpenBMC - Support NVMe drive health monitoring

CAUTION: External Sender
Hi Lior and Andrew,

The goal of enhancing nvmesensor with nvme-mi is to fulfill the full stack management of NVMe devices, including health monitoring but beyond. An example could be SSD firmware update and security protocol transportation.

The MCTP support has improved compared to two years ago. The MCTP I2C binding has been upstreamed to linux kernel and there is an mctp daemon for openbmc for the MCTP management.

On Fri, Apr 7, 2023 at 5:35 AM Lior Weintraub <liorw at pliops.com<mailto:liorw at pliops.com>> wrote:

Thanks Andrew,

I wish I had the experience to help with the review process 😊

I have more than 2 decades experience with bare metal embedded systems but no experience with Embedded Linux\Yocto nor OpenBMC (nor Gerrit).

It's all new to me and came to realize that many guides \ documentations I find on the web are old and doesn't work anymore on OpenBMC (e.g. using the oe-init-build-env vs setup script).

I am happy to assist and contribute but I need some guidance.

I have only 2 weeks experience with OpenBMC.

I have succeeded to build few platforms and run them (e.g. qemuarm and RaspberryPi4).

I watched many youtube videos on OpenBMC which gave me a good overall understanding but when it comes to start coding or adding new stuff I am lost.

If we start our development with NVMe-MI basic management command as a starting point, what steps would you recommend we need to take?

Can we use the PaspberryPi4 for this initial implementation?

We have a basic OpenBMC image on this RaspPi and we have it's I2C connected to our PCIe card (which is using a Desktop as the host PC).

(BTW, the i2c-dev module is not loaded automatically and we still haven't figured out why. As a workaround, we load the module manually with modprobe i2c-dev on the OpenBMC UART terminal)

Thanks,

Lior.

-----Original Message-----
From: Andrew Jeffery <andrew at aj.id.au<mailto:andrew at aj.id.au>>
Sent: Friday, April 7, 2023 2:46 PM
To: Lior Weintraub <liorw at pliops.com<mailto:liorw at pliops.com>>
Cc: Ed Tanous <ed at tanous.net<mailto:ed at tanous.net>>; openbmc at lists.ozlabs.org<mailto:openbmc at lists.ozlabs.org>; rashmica at linux.ibm.com<mailto:rashmica at linux.ibm.com>; jianghao at google.com<mailto:jianghao at google.com>
Subject: Re: OpenBMC - Support NVMe drive health monitoring

CAUTION: External Sender

On Fri, 7 Apr 2023, at 19:50, Lior Weintraub wrote:

> Hi Jet,

>

> My name is Lior Weintraub and I face exactly the same task that you

> handled 2 years ago 😊

> I found this mailing list thread on OpenBmc:

> https://lists.ozlabs.org/pipermail/openbmc/2020-November/024012.html

>

> Can you please advice about the current status of this issue?

> I need to add OOB support to our NVMe device (supporting NVMe MI ->

> MCTP with SMBus binding).

> For this task we plan to buy the AST2600 development

> kit<https://portwell.com/solutions/openBMC.php> and use the OpenBMC

> for our development and testing.

Currently OpenBMC only supports NVMe-MI via the basic management command. See phosphor-nvme or nvmesensor from dbus-sensors. We're working on migrating the OpenBMC MCTP stack to the AF_MCTP sockets that are now provided by Linux.

Please help review and test Rashmica's work in Gerrit to speed that process along:

https://gerrit.openbmc.org/q/topic:AF_MCTP

Along with Hao Jiang's work to exploit libnvme-mi in nvmesensor:

https://gerrit.openbmc.org/q/topic:nvme-mi

Andrew

--
Hao

--
Hao
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ozlabs.org/pipermail/openbmc/attachments/20230411/f2f45a08/attachment-0001.htm>