LAN Cable insertion/removal

[Johnathan Mantey]

LAN/NCSI experts,

I'm looking for some guidance to correctly report LAN state when RJ45
cables are inserted/removed from the NIC's monitored by the BMC.

Background:

 1. Intel uses one NIC that is dedicated to the BMC (1Gib phy, eth0)
 2. There is another LAN channel managed by the BMC via NCSI (100Mib
    NCSI, eth1)
 3. Prior generations of Intel servers correctly report the presence of
    an active LAN link upon insertion/removal
 4. Prior generations correctly update /sys/class/net/eth(x)/carrier
    based on cable insertion
 5. LAN carrier state is propagated to the IPMI system, and is logged
    via IPMI events.

OpenBMC:

 1. The dedicated NIC (aka eth0) correctly propagates
    /sys/class/net/eth0/carrier state to the kernel
 2. The eth0 carrier state does not propagate to Redfish/IPMI. I believe
    this to be a problem related to DBus update.
 3. The NCSI NIC (aka eth1) does not propagate the cable presence to
    /sys/class/net/eth1, which I believe to be invalid behavior.
 4. The eth1 carrier state does not propagate to Redfish/IPMI. I believe
    this is a two-fold problem with DBus update, and kernel update.
 5. LAN carrier state is not propagated to IPMI or Redfish, and is not
    logged.

I've already pushed a change to handle synchronous enable/disable
actions via DBus. Unfortunately it's only half the solution. I've been
trying to work my way through the FTGMAC100 driver code, and the
phosphor-networkd code in an effort to tie the async event into the DBus
space.  So far without success.

Desired results:

 1. Redfish GET /redfish/v1/Managers/bmc/EthernetInterfaces/eth0 (eth1)
    correctly reports the InterfaceEnabled state when a cable is
    pulled/inserted.
 2. Redfish receives an event that permits the state change to be logged.
 3. IPMI correctly reports the link status (up/down)
 4. ip link correctly reports the link status (up/down).

Does anyone on the list have some recommendations?



Details of my investigation for the strong of heart:

  * Synchronous NIC Enable/Disable change

     1. https://gerrit.openbmc-project.xyz/c/openbmc/phosphor-networkd/+/26696
     2. https://gerrit.openbmc-project.xyz/c/openbmc/phosphor-dbus-interfaces/+/26694
     3. https://gerrit.openbmc-project.xyz/c/openbmc/bmcweb/+/26693

  * FTGMAC driver: (kernel-source/drivers/net/ethernet/faraday/ftgmac100.c)

        This code looks suspect, as it uses "speed" to determine if the
link is up/down.  For the Intel system the NCSI channel is always
running 100Mibps, so does this really work?

        /* Link is down, do nothing else */
        if (!new_speed)
            return;

        Likewise, this code announces the link is "down", which seems
would cause the kernel to alter the state of
/sys/class/net/eth1/carrier.  Insertion of the cable does not cause a
corresponding "link is up" message:

        static void ftgmac100_ncsi_handler(struct ncsi_dev *nd)
        {
            if (unlikely(nd->state != ncsi_dev_state_functional))
            return;

           netdev_dbg(nd->dev, "NCSI interface %s\n",
           nd->link_up ? "up" : "down");
                }

  * PHY driver (kernel-source/drivers/net/phy/phy.c and phylink.c)

        The PHY driver for eth0 seems to be working correctly. Removal
shows the link going down, and the /sys/class/net/eth0/carrier file
changing state from 1->0. Insertion works correctly changing the state
from 0->1.  What isn't present is some action to phosphor-network or
DBus indicating a link state change.

  * ip link

        The "ip link" command always returns
"<BROADCAST,MULTICAST,UP,LOWER_UP>" for eth1 (NCSI) channel.

-- 
Johnathan Mantey
Senior Software Engineer
*azad te**chnology partners*
Contributing to Technology Innovation since 1992
Phone: (503) 712-6764
Email: johnathanx.mantey at intel.com <mailto:johnathanx.mantey at intel.com>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ozlabs.org/pipermail/openbmc/attachments/20191202/b3301c5e/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: OpenPGP digital signature
URL: <http://lists.ozlabs.org/pipermail/openbmc/attachments/20191202/b3301c5e/attachment.sig>