OpenBMC - Support NVMe drive health monitoring

Wed Apr 12 07:50:48 AEST 2023

> If so, can you please guide me how to do that?

The prerequisite for nvme OOB is the mctp layer, including:
1. mctp linux kernel - starting from 5.15
2. libmctp
3. mctpd
+ Jeremy as the author of MCTP.

> Can this tool be tested on QEMU or RaspberryPi (running OpenBMC image)?
I am not sure if you want to emulate a nvme mctp device in QEMU.
RaspberryPi is doable but you need to rework the raiser for I2C from
PCIe.

> I found on OpenBMC the meta-openembedded/meta-oe/recipes-bsp/nvme-cli/nvme-cli_1.13.bb but it looks like it is pointing to an old version of the repo (commit f0e9569df9289d6ee55ba2c23615cc7c73a9b088 from Oct 21, 2020). Should I change that?
I have an internal recipe for nvme-cli 2.1.2, we can upstream it, but
you are welcome to do it yourself.

> How do I obtain this information (net, eid, ctrl-id)?
MCTP layer should tell you

> Is there a tool that runs auto discovery and will give a map of the devices it finds? (kind of like nvme list)?
That is mctpd. https://github.com/CodeConstruct/mctp


On Tue, Apr 11, 2023 at 12:06 AM Lior Weintraub <liorw at pliops.com> wrote:
>
> Thanks Hao,
>
>
>
> This looks very promising.
>
> I would appreciate some more details as I am new to OpenBMC.
>
> If I understand correctly, in order to run this tool as OOB, it needs to be compiled for OpenBMC platform and somehow be included in the image.
>
> If so, can you please guide me how to do that?
> Can this tool be tested on QEMU or RaspberryPi (running OpenBMC image)?
> I found on OpenBMC the meta-openembedded/meta-oe/recipes-bsp/nvme-cli/nvme-cli_1.13.bb but it looks like it is pointing to an old version of the repo (commit f0e9569df9289d6ee55ba2c23615cc7c73a9b088 from Oct 21, 2020). Should I change that?
> It looks like this cli tool also needs an updated (latest) version of libnvme. How to make sure the OpenBMC image is built with latest version of this library?
>
> When the cli tool is used for OOB, it needs to have mctp:<net>,<eid>[:ctrl-id].
>
> How do I obtain this information (net, eid, ctrl-id)?
>
> Is there a tool that runs auto discovery and will give a map of the devices it finds? (kind of like nvme list)?
>
>
>
> Thanks again,
>
> Kind regards,
>
> Lior.
>
>
>
> From: Hao Jiang <jianghao at google.com>
> Sent: Monday, April 10, 2023 7:28 PM
> To: Lior Weintraub <liorw at pliops.com>
> Cc: Andrew Jeffery <andrew at aj.id.au>; Ed Tanous <ed at tanous.net>; openbmc at lists.ozlabs.org; rashmica at linux.ibm.com
> Subject: Re: OpenBMC - Support NVMe drive health monitoring
>
>
>
> CAUTION: External Sender
>
> We have upstreamed the support for libnvme-mi in the opensource nvme-cli tool. https://github.com/linux-nvme/nvme-cli
>
> The cli tool has only nvme admin command set support, since it is co-used for inband and oob.
>
>
>
> For the MI cmd set, we have a really simple demo tool, but it doesn't have all MI CMD (it is a demo only :) )
> https://github.com/linux-nvme/libnvme/blob/master/examples/mi-mctp.c
>
>
>
> On Sat, Apr 8, 2023 at 2:42 AM Lior Weintraub <liorw at pliops.com> wrote:
>
> Thanks Hao,
>
>
>
> Do you know if there is a tool (e.g. ipmitool) that can be used to generate a basic NVMe MI command?
>
> For example, the NVMe MI Revision 1.1 spec defines a "Management Interface Command Set" that can be used to:
>
> Opcode                Command
>
> 00h                        Read NVMe-MI Data Structure
>
> 01h                        NVM Subsystem Health Status Poll
>
> 02h                        Controller Health Status Poll
>
> 03h                        Configuration Set
>
> 04h                        Configuration Get
>
> 05h                        VPD Read
>
> 06h                        VPD Write
>
> 07h                        Reset
>
> 08h                        SES Receive
>
> 09h                        SES Send
>
> 0Ah                        Management Endpoint Buffer Read
>
> 0Bh                        Management Endpoint Buffer Write
>
> 0Ch to BFh          Reserved
>
> C0h to FFh           Vendor specific
>
>
>
> If we had a tool that can send such a command it would help with the development of this feature.
>
> The idea is to start with some manually triggered commands.
>
>
>
> Thanks,
>
> Lior.
>
>
>
>
>
> From: Hao Jiang <jianghao at google.com>
> Sent: Friday, April 7, 2023 8:40 PM
> To: Lior Weintraub <liorw at pliops.com>
> Cc: Andrew Jeffery <andrew at aj.id.au>; Ed Tanous <ed at tanous.net>; openbmc at lists.ozlabs.org; rashmica at linux.ibm.com
> Subject: Re: OpenBMC - Support NVMe drive health monitoring
>
>
>
> CAUTION: External Sender
>
> Hi Lior and Andrew,
>
> The goal of enhancing nvmesensor with nvme-mi is to fulfill the full stack management of NVMe devices, including health monitoring but beyond. An example could be SSD firmware update and security protocol transportation.
>
> The MCTP support has improved compared to two years ago. The MCTP I2C binding has been upstreamed to linux kernel and there is an mctp daemon for openbmc for the MCTP management.
>
>
>
> On Fri, Apr 7, 2023 at 5:35 AM Lior Weintraub <liorw at pliops.com> wrote:
>
> Thanks Andrew,
>
>
>
> I wish I had the experience to help with the review process 😊
>
> I have more than 2 decades experience with bare metal embedded systems but no experience with Embedded Linux\Yocto nor OpenBMC (nor Gerrit).
>
> It's all new to me and came to realize that many guides \ documentations I find on the web are old and doesn't work anymore on OpenBMC (e.g. using the oe-init-build-env vs setup script).
>
>
>
> I am happy to assist and contribute but I need some guidance.
>
> I have only 2 weeks experience with OpenBMC.
>
> I have succeeded to build few platforms and run them (e.g. qemuarm and RaspberryPi4).
>
> I watched many youtube videos on OpenBMC which gave me a good overall understanding but when it comes to start coding or adding new stuff I am lost.
>
>
>
> If we start our development with NVMe-MI basic management command as a starting point, what steps would you recommend we need to take?
>
> Can we use the PaspberryPi4 for this initial implementation?
>
> We have a basic OpenBMC image on this RaspPi and we have it's I2C connected to our PCIe card (which is using a Desktop as the host PC).
>
> (BTW, the i2c-dev module is not loaded automatically and we still haven't figured out why. As a workaround, we load the module manually with modprobe i2c-dev on the OpenBMC UART terminal)
>
>
>
> Thanks,
>
> Lior.
>
>
>
> -----Original Message-----
> From: Andrew Jeffery <andrew at aj.id.au>
> Sent: Friday, April 7, 2023 2:46 PM
> To: Lior Weintraub <liorw at pliops.com>
> Cc: Ed Tanous <ed at tanous.net>; openbmc at lists.ozlabs.org; rashmica at linux.ibm.com; jianghao at google.com
> Subject: Re: OpenBMC - Support NVMe drive health monitoring
>
>
>
> CAUTION: External Sender
>
>
>
> On Fri, 7 Apr 2023, at 19:50, Lior Weintraub wrote:
>
> > Hi Jet,
>
> >
>
> > My name is Lior Weintraub and I face exactly the same task that you
>
> > handled 2 years ago 😊
>
> > I found this mailing list thread on OpenBmc:
>
> > https://lists.ozlabs.org/pipermail/openbmc/2020-November/024012.html
>
> >
>
> > Can you please advice about the current status of this issue?
>
> > I need to add OOB support to our NVMe device (supporting NVMe MI ->
>
> > MCTP with SMBus binding).
>
> > For this task we plan to buy the AST2600 development
>
> > kit<https://portwell.com/solutions/openBMC.php> and use the OpenBMC
>
> > for our development and testing.
>
>
>
> Currently OpenBMC only supports NVMe-MI via the basic management command. See phosphor-nvme or nvmesensor from dbus-sensors. We're working on migrating the OpenBMC MCTP stack to the AF_MCTP sockets that are now provided by Linux.
>
>
>
> Please help review and test Rashmica's work in Gerrit to speed that process along:
>
>
>
> https://gerrit.openbmc.org/q/topic:AF_MCTP
>
>
>
> Along with Hao Jiang's work to exploit libnvme-mi in nvmesensor:
>
>
>
> https://gerrit.openbmc.org/q/topic:nvme-mi
>
>
>
> Andrew
>
>
>
>
> --
>
> Hao
>
>
>
>
> --
>
> Hao


-- 
Hao