[External] ` NETDEV WATCHDOG: eth0 (ftgmac100): transmit queue 0 timed out ` error in qemu

[健张]

Add a wiki to record this issue and for your reference.

https://github.com/zhangjian3032/qemu/wiki/NETDEV-WATCHDOG:-eth0-(ftgmac100):-transmit-queue-0-timed-ou--error-in-qemu,-cause-net-device-start-fail


On Fri, Apr 29, 2022 at 1:48 PM Cédric Le Goater <clg at kaod.org> wrote:
>
> On 4/28/22 15:55, 健张 wrote:
> > Hi C;
> >
> > That’s awesome , It's okay to use bridge,
> >
> >
> > Here is my command line, 2 devices are good.
> >
> > ./qemu/build/install/bin/qemu-system-arm -M g220a-bmc -nographic -drive file=image-bmc,format=raw,if=mtd -net nic,netdev=net0, -netdev bridge,id=net0,helper=./qemu/build/install/libexec/qemu-bridge-helper,br=virbr0, -net nic,netdev=net1, -netdev bridge,id=net1,helper=./qemu/build/install/libexec/qemu-bridge-helper,br=virbr0
>
> Perfect !
>
> Something I would really like to work is the usb-net device.
>
> Thanks,
>
> C.
>
> >> -net nic,macaddr=....,netdev=net0 \
> >> -netdev bridge,id=net0,helper=/usr/libexec/qemu-bridge-helper,br=virbr0
> >
> >
> >
> >> 2022年4月28日 19:31，张健 <zhangjian.3032 at bytedance.com <mailto:zhangjian.3032 at bytedance.com>> 写道：
> >>
> >>
> >>
> >> From: "Cédric Le Goater"<clg at kaod.org <mailto:clg at kaod.org>>
> >> Date: Thu, Apr 28, 2022, 19:23
> >> Subject: [External] Re: ` NETDEV WATCHDOG: eth0 (ftgmac100): transmit queue 0 timed out ` error in qemu
> >> To: "张健"<zhangjian.3032 at bytedance.com <mailto:zhangjian.3032 at bytedance.com>>, "Joel Stanley"<joel at jms.id.au <mailto:joel at jms.id.au>>
> >> Hello C;
> >>
> >> [ Adding Joel who has some experience on NCSI vs. QEMU ]
> >>
> >> On 4/28/22 12:29, 张健 wrote:
> >> > Hi C;
> >> > Excuse me to ask you a qemu question.
> >>
> >> It would help if you provided the QEMU command line. Not all NICs are
> >> connected to the QEMU machine.
> >>
> >> It's my command line
> >> qemu-system-arm -M g220a-bmc -nographic -drive file=image-bmc,format=raw,if=mtd -net nic,model=ftgmac100,netdev=netdev1 -netdev user,id=netdev1,hostfwd=udp::5623-:623, -net nic,model=ftgmac100,netdev=netdev2 -netdev user,id=netdev2,hostfwd=:127.0.0.1:4222-:22,hostfwd=:127.0.0.1:3443-:443,hostfwd=udp::3623-:623
> >>
> >> > According to my development needs, I need to reconfigure the network configuration every time I start.
> >> >
> >> > The order is
> >> >
> >> > 1. Power on.
> >> > 2. systemd-networkd start
> >> > 3. update network config
> >> > 4. systemctl reload systemd-networkd
> >> >
> >> > It looks fine after booting for a long time. I manually execution step 3 and 4.>
> >> > But I hit an issue in qemu when I make step 3/4 in Start-up process(use a systemd script)
> >> >
> >> > There is always an error log like below .
> >> > [ 155.772381] ftgmac100 1e660000.ethernet eth0: NCSI Channel 0 timed out!
> >>
> >> That could be the issue. QEMU doesn't really model NCSI. There is
> >> only a simple responder in slirp.
> >>
> >> Also at startup, OpenBMC does a few things that could load the CPU
> >> which would slow down very much the overall machine.
> >>
> >>
> >> > [ 159.132481] ------------[ cut here ]------------
> >> > [ 159.133035] WARNING: CPU: 0 PID: 207 at net/sched/sch_generic.c:477 dev_watchdog+0x2a8/0x2c4
> >> > [ 159.133543] NETDEV WATCHDOG: eth0 (ftgmac100): transmit queue 0 timed out
> >> > [ 159.134143] CPU: 0 PID: 207 Comm: dropbearkey Not tainted 5.15.34-0626498-dirty-006ed35 #1
> >> > [ 159.134748] Hardware name: Generic DT based system
> >> > [ 159.135241] Backtrace:
> >> > [ 159.135637] [<807fbe0c>] (dump_backtrace) from [<807fc004>] (show_stack+0x20/0x24)
> >> > [ 159.136240] r7:00000009 r6:000001dd r5:806b0850 r4:80a06be8
> >> > [ 159.136587] [<807fbfe4>] (show_stack) from [<808063e4>] (dump_stack+0x28/0x30)
> >> > [ 159.136983] [<808063bc>] (dump_stack) from [<80115fe4>] (__warn+0xe4/0x150)
> >> > [ 159.137359] r5:806b0850 r4:80a75280
> >> >
> >> > Before this error log I reload systemd-networkd, will cause network device start fail.
> >> > But this error don't appear on physical machines.
> >>
> >> They are faster and they have a real NCSI backend.
> >>
> >> > Thus , I add a dirty delay `sleep 20`, reload after timeout. It will be fine.
> >>
> >> ok. It looks like a load issue then.
> >>
> >> > Do you know how to fix `NETDEV WATCHDOG: eth0 (ftgmac100): transmit queue 0 timed out` in qemu?
> >> > or
> >> > Can BMC know if it boots from a qemu virtual machine or a physical machine?
> >>
> >> not really and it is not the goal. we try to tune things from outside,
> >> in QEMU.
> >>
> >> okay, i think so.
> >>
> >> > Network start fail log:
> >> > Jan 01 00:01:28 g220a systemd[1]: Starting Network Configuration...
> >> > Jan 01 00:01:54 g220a systemd-networkd[360]: bond1: netdev ready
> >> > Jan 01 00:01:55 g220a systemd-networkd[360]: lo: Link UP
> >> > Jan 01 00:01:55 g220a systemd-networkd[360]: lo: Gained carrier
> >> > Jan 01 00:01:56 g220a systemd-networkd[360]: Enumeration completed
> >> > Jan 01 00:01:57 g220a systemd[1]: Started Network Configuration.
> >> > Jan 01 00:02:00 g220a systemd-networkd[360]: eth1: Link UP
> >> > Jan 01 00:02:01 g220a systemd-networkd[360]: eth1: Gained carrier
> >> > Jan 01 00:02:01 g220a systemd-networkd[360]: eth0: Link UP
> >> > Jan 01 00:02:01 g220a systemd-networkd[360]: eth0: Gained carrier
> >> > Jan 01 00:02:02 g220a systemd-networkd[360]: bond1: Link UP
> >> > Jan 01 00:02:02 g220a systemd-networkd[360]: bond1: Gained carrier
> >> > Jan 01 00:02:04 g220a systemd-networkd[360]: bond1: DHCPv4 address 10.0.2.15/24 , gateway 10.0.2.2 acquired from 10.0.2.2
> >>
> >> Ah. this is bonding two devices. I never tried that. Joel, have you ?
> >>
> >> > Jan 01 00:02:04 g220a systemd-networkd[360]: bond1: Gained IPv6LL
> >> > Apr 28 10:24:11 g220a systemd-networkd[360]: eth1: Reconfiguring with /etc/systemd/network/00-bmc-eth1.network.
> >> > Apr 28 10:24:11 g220a systemd-networkd[360]: eth0: Reconfiguring with /etc/systemd/network/00-bmc-eth0.network.
> >> > Apr 28 10:24:13 g220a systemd-networkd[360]: eth1: Link DOWN
> >> > Apr 28 10:24:13 g220a systemd-networkd[360]: eth1: Lost carrier
> >> > Apr 28 10:24:13 g220a systemd-networkd[360]: eth0: Link DOWN
> >> > Apr 28 10:24:13 g220a systemd-networkd[360]: eth0: Lost carrier
> >> > Apr 28 10:24:38 g220a systemd-networkd[360]: eth1: Failed to sync link information: Connection timed out
> >> > Apr 28 10:24:38 g220a systemd-networkd[360]: eth1: Failed
> >> > Apr 28 10:24:39 g220a systemd-networkd[360]: eth0: Failed to sync link information: Connection timed out
> >> > Apr 28 10:24:39 g220a systemd-networkd[360]: eth0: Failed
> >> >
> >> > Then, start manually, also fail.
> >> > ip link set eth0 up
> >> > [ 308.820798] ftgmac100 1e660000.ethernet eth0: NCSI: No channel found to configure!
> >> > ip: SIOCSIFFLAGS: No such device
> >>
> >>
> >> Just to eliminate the cause, have you tried with a libvirt bridge ?
> >>
> >> -net nic,macaddr=....,netdev=net0 \
> >> -netdev bridge,id=net0,helper=/usr/libexec/qemu-bridge-helper,br=virbr0
> >>
> >> nop, i'll have a try
> >>
> >> Thanks,
> >>
> >> C.
> >
>