Connection issue in OpenBMC image
George Keishing
gkeishin at in.ibm.com
Mon Oct 5 20:29:03 AEDT 2020
Jayashree,
Couple places where it checks for the SSH to BMC and SOL as well, that
should catch those in your HW test CI environment
Example:
https://github.com/openbmc/openbmc-test-automation/blob/master/test_lists/HW_CI#L2
Test_SSH_And_IPMI_Connections
https://github.com/openbmc/openbmc-test-automation/blob/master/test_lists/HW_CI#L4
-i Verify_Redfish_Host_PowerOn
-i Verify_Redfish_Host_PowerOff
Here SOL connection done as part of the above setup/teardown
https://github.com/openbmc/openbmc-test-automation/blob/master/redfish/systems/test_power_operations.robot#L87
https://github.com/openbmc/openbmc-test-automation/blob/master/redfish/systems/test_power_operations.robot#L95
Those tests are currently running in upstream community HW Jenkins test CI.
So it would have caught those SSH related for generic ports in general and
you can use similar in your environment.
Thanks and Regards,
George Keishing
From: Jayashree D <jayashree-d at hcl.com>
To: "openbmc at lists.ozlabs.org" <openbmc at lists.ozlabs.org>
Cc: George Keishing <gkeishin at in.ibm.com>, "geissonator at yahoo.com"
<geissonator at yahoo.com>, Vijay Khemka <vijaykhemka at fb.com>,
Konstantin Klubnichkin <kitsok at yandex-team.ru>
Date: 05-10-2020 02:09 PM
Subject: [EXTERNAL] RE: Connection issue in OpenBMC image
Sent by: "openbmc" <openbmc-bounces
+gkeishin=in.ibm.com at lists.ozlabs.org>
Classification: HCL Internal
Regarding SSH connection, an issue has been created in openbmc and I also
see others having this same issue.
>From the comments, I have run "dropbear -E -p 5022" in the target
(UART-console) and tried to connect the target using "ssh -p 5022 <ip>" and
ssh connection established.
But, reboot and systemctl commands hangs.
Issue -
https://github.com/openbmc/openbmc/issues/3701
root at tiogapass:~# dropbear -E -p 5022
[348] Jan 01 00:06:48 Failed loading /etc/dropbear/dropbear_dss_host_key
[348] Jan 01 00:06:48 Failed loading /etc/dropbear/dropbear_ecdsa_host_key
[348] Jan 01 00:06:48 Failed
loading /etc/dropbear/dropbear_ed25519_host_key
[349] Jan 01 00:06:48 Running in background
[root at odc ~]# ssh -p 5022 root at 10.0.128.108 root at 10.0.128.108's password:
root at tiogapass:~#
Hi George,
We are facing connection issue in accessing the target after flashing the
latest image.
In openbmc-test-automation, whether any test cases are present in CI to
identify these issues ?
Please let us know your comments on this.
Regards,
Jayashree
-----Original Message-----
From: Jayashree D
Sent: Friday, September 25, 2020 10:29 AM
To: openbmc at lists.ozlabs.org
Cc: Konstantin Klubnichkin <kitsok at yandex-team.ru>; Vijay Khemka
<vijaykhemka at fb.com>; geissonator at yahoo.com; joel at jms.id.au; Patrick
Williams <patrick at stwcx.xyz>
Subject: RE: Connection issue in OpenBMC image
Classification: HCL Internal
Hi Team,
In the latest openbmc build, after image upgradation in the target, not
able to connect the target through SSH but able to ping the IP Address.
After analysing the latest commits, reverted the below commit in the latest
build and checked by flashing the image. Now the target is connecting
through SSH. Please help us on fixing this issue.
Commit Link -
https://github.com/openbmc/openbmc/commit/635e0e4637e40ba03f69204265427550fd404f4c
Observation on UART-console after flashing latest image without any
changes:
1. reboot command is not working.
2. systemctl status <service_name> is not providing any status. ( Failed to
get properties: Connection timed out) 3. I tried "ssh -vvv <ip>" and logs
are attached for working and non-working image.
4. From controller, I tried to upgrade image using redfish and image is
being copied and following logs shown.
root at tiogapass:~# journalctl | grep image Jan 01 00:00:37 tiogapass
phosphor-image-updater[246]: Error in mapper GetSubTreePath Jan 01 10:43:59
tiogapass phosphor-image-updater[246]: BMC image activating - BMC reboots
are disabled.
5. Using Rest API command,
[root at odc ]# curl -k -H "X-Auth-Token: $token" -H "Content-Type:
application/json" -X PUT -d
'{"data":"xyz.openbmc_project.Software.Activation.RequestedActivations.Active"}'
https://$%7Bbmc%7D/xyz/openbmc_project/software/a77348be/attr/RequestedActivation
{
"data": {
"description": "org.freedesktop.DBus.Error.NoReply"
},
"message": "Method call timed out",
"status": "error"
}
Regards,
Jayashree
-----Original Message-----
From: Jayashree D
Sent: Friday, September 18, 2020 3:18 PM
To: Patrick Williams <patrick at stwcx.xyz>; openbmc at lists.ozlabs.org
Cc: Konstantin Klubnichkin <kitsok at yandex-team.ru>
Subject: RE: Connection issue in OpenBMC image
Classification: HCL Internal
Hello Patrick,
I saw the post about dropbear, but that commit was updated on July16 and my
target is connecting till August last week image. I don't think that will
be an issue. Also on working image, I tried with 'ssh -vvv ' and I got
below information.
OpenSSH_7.4p1, OpenSSL 1.0.2k-fips 26 Jan 2017
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 58: Applying options for *
debug2: resolving "10.0.128.108" port 22
debug2: ssh_connect_direct: needpriv 0
debug1: Connecting to 10.0.128.108 [10.0.128.108] port 22.
debug1: Connection established.
debug1: permanently_set_uid: 0/0
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_rsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.4
debug1: Remote protocol version 2.0, remote software version
dropbear_2020.80
debug1: no match: dropbear_2020.80
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to 10.0.128.108:22 as 'root'
debug3: hostkeys_foreach: reading file "/root/.ssh/known_hosts"
debug3: record_hostkey: found key type RSA in
file /root/.ssh/known_hosts:68
debug3: load_hostkeys: loaded 1 keys from 10.0.128.108
debug3: order_hostkeyalgs: prefer hostkeyalgs:
ssh-rsa-cert-v01 at openssh.com,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms:
curve25519-sha256,curve25519-sha256 at libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1,ext-info-c
debug2: host key algorithms:
ssh-rsa-cert-v01 at openssh.com,rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256-cert-v01 at openssh.com,ecdsa-sha2-nistp384-cert-v01 at openssh.com,ecdsa-sha2-nistp521-cert-v01 at openssh.com,ssh-ed25519-cert-v01 at openssh.com,ssh-dss-cert-v01 at openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,ssh-dss
debug2: ciphers ctos:
chacha20-poly1305 at openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm at openssh.com,aes256-gcm at openssh.com,aes128-cbc,aes192-cbc,aes256-cbc
debug2: ciphers stoc:
chacha20-poly1305 at openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm at openssh.com,aes256-gcm at openssh.com,aes128-cbc,aes192-cbc,aes256-cbc
debug2: MACs ctos:
umac-64-etm at openssh.com,umac-128-etm at openssh.com,hmac-sha2-256-etm at openssh.com,hmac-sha2-512-etm at openssh.com,hmac-sha1-etm at openssh.com,umac-64 at openssh.com,umac-128 at openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc:
umac-64-etm at openssh.com,umac-128-etm at openssh.com,hmac-sha2-256-etm at openssh.com,hmac-sha2-512-etm at openssh.com,hmac-sha1-etm at openssh.com,umac-64 at openssh.com,umac-128 at openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib at openssh.com,zlib
debug2: compression stoc: none,zlib at openssh.com,zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms:
curve25519-sha256,curve25519-sha256 at libssh.org,ecdh-sha2-nistp521,ecdh-sha2-nistp384,ecdh-sha2-nistp256,diffie-hellman-group14-sha256,kexguess2 at matt.ucc.asn.au
debug2: host key algorithms: rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: chacha20-poly1305 at openssh.com,aes128-ctr,aes256-ctr
debug2: ciphers stoc: chacha20-poly1305 at openssh.com,aes128-ctr,aes256-ctr
debug2: MACs ctos: hmac-sha1,hmac-sha2-256
debug2: MACs stoc: hmac-sha1,hmac-sha2-256
debug2: compression ctos: zlib at openssh.com,none
debug2: compression stoc: zlib at openssh.com,none
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: rsa-sha2-256
debug1: kex: server->client cipher: chacha20-poly1305 at openssh.com MAC:
<implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305 at openssh.com MAC:
<implicit> compression: none
debug1: kex: curve25519-sha256 need=64 dh_need=64
debug1: kex: curve25519-sha256 need=64 dh_need=64
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: Server host key: ssh-rsa SHA256:3WwhPmIIxzrw0
+cm/0vN3hifY4kh9sJhClVNw6zrJ7Y
debug3: hostkeys_foreach: reading file "/root/.ssh/known_hosts"
debug3: record_hostkey: found key type RSA in
file /root/.ssh/known_hosts:68
debug3: load_hostkeys: loaded 1 keys from 10.0.128.108
debug1: Host '10.0.128.108' is known and matches the RSA host key.
debug1: Found key in /root/.ssh/known_hosts:68
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey after 134217728 blocks
debug2: key: /root/.ssh/id_rsa (0x558ea3ad3640), agent
debug2: key: /root/.ssh/id_rsa ((nil))
debug2: key: /root/.ssh/id_dsa ((nil))
debug2: key: /root/.ssh/id_ecdsa ((nil))
debug2: key: /root/.ssh/id_ed25519 ((nil))
debug3: send packet: type 5
debug3: receive packet: type 7
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info:
server-sig-algs=<ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,rsa-sha2-256,ssh-rsa,ssh-dss>
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,password
debug3: start over, passed a different list publickey,password
debug3: preferred
gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /root/.ssh/id_rsa
debug3: send_pubkey_test
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,password
debug1: Trying private key: /root/.ssh/id_rsa
debug3: sign_and_send_pubkey: RSA SHA256:YfteufmWUV8W7EQEycZ
+38skgUWGDTYFHw93a7SwwLM
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,password
debug1: Trying private key: /root/.ssh/id_dsa
debug3: no such identity: /root/.ssh/id_dsa: No such file or directory
debug1: Trying private key: /root/.ssh/id_ecdsa
debug3: no such identity: /root/.ssh/id_ecdsa: No such file or directory
debug1: Trying private key: /root/.ssh/id_ed25519
debug3: no such identity: /root/.ssh/id_ed25519: No such file or directory
debug2: we did not send a packet, disable method
debug3: authmethod_lookup password
debug3: remaining preferred: ,password
debug3: authmethod_is_enabled password
debug1: Next authentication method: password
In non-working image, the logs are stopped after below lines and it is not
providing any errors.
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.4
Also one more observation in UART-Console, after flashing latest image.
1. reboot command is not working.
2. systemctl status <service_name> is not providing any status. ( Failed to
get properties: Connection timed out) 3. I can able to ping the ip address
but scp is not working.
Thanks,
Jayashree
-----Original Message-----
From: Patrick Williams <patrick at stwcx.xyz>
Sent: Thursday, September 17, 2020 9:16 PM
To: Jayashree D <jayashree-d at hcl.com>
Cc: Konstantin Klubnichkin <kitsok at yandex-team.ru>;
openbmc at lists.ozlabs.org
Subject: Re: Connection issue in OpenBMC image
Hello Jayashree,
I saw an output `ssh -v` from you earlier, but there really wasn't any
useful information there. It looked like the connection was being made and
keys were exchanged and then the log just stopped abruptly. This tells me
it likely isn't a networking issue but an issue in the handshake between
the ssh-client (your computer) and ssh-server (dropbear). You can continue
to add '-v' parameters up to `ssh -vvv` and you'll get increasingly more
information.
Joseph Reynolds recently posted a reminder about dropbear disabling weak
ciphers[1]. Is it possible that your client is using an old cipher?
On Wed, Sep 16, 2020 at 11:35:28AM +0000, Jayashree D wrote:
> root at tiogapass:~# journalctl | grep drop
...
> Jan 01 00:15:28 tiogapass systemd[1]:
dropbear at 0-10.0.128.108:22-10.0.0.1:51810.service: Succeeded.
> Jan 01 00:15:44 tiogapass dropbear[2753]: Child connection from
> ::ffff:10.0.0.1:51944 Jan 01 00:15:50 tiogapass dropbear[2753]: PAM
> password auth succeeded for 'root' from ::ffff:10.0.0.1:51944
This looks like a valid connection was established.
> 15.09.2020, 16:12, "Jayashree D" <jayashree-d at hcl.com<
mailto:jayashree-d at hcl.com>>:
>
> OpenSSH_7.4p1, OpenSSL 1.0.2k-fips 26 Jan 2017
> debug1: Reading configuration data /etc/ssh/ssh_config
> debug1: /etc/ssh/ssh_config line 58: Applying options for *
> debug1: Connecting to 10.0.128.108 [10.0.128.108] port 22.
> debug1: Connection established.
> debug1: permanently_set_uid: 0/0
> debug1: key_load_public: No such file or directory
> debug1: identity file /root/.ssh/id_rsa type -1
> debug1: key_load_public: No such file or directory
> debug1: identity file /root/.ssh/id_rsa-cert type -1
> debug1: key_load_public: No such file or directory
> debug1: identity file /root/.ssh/id_dsa type -1
> debug1: key_load_public: No such file or directory
> debug1: identity file /root/.ssh/id_dsa-cert type -1
> debug1: key_load_public: No such file or directory
> debug1: identity file /root/.ssh/id_ecdsa type -1
> debug1: key_load_public: No such file or directory
> debug1: identity file /root/.ssh/id_ecdsa-cert type -1
> debug1: key_load_public: No such file or directory
> debug1: identity file /root/.ssh/id_ed25519 type -1
> debug1: key_load_public: No such file or directory
> debug1: identity file /root/.ssh/id_ed25519-cert type -1
> debug1: Enabling compatibility mode for protocol 2.0
> debug1: Local version string SSH-2.0-OpenSSH_7.4
This is the log that also looks like a good connection. Identity files
were attempted to be exchanged. Version strings were exchanged. And then
the log just abruptly stops. Was the connection dropped? Is it hung?
1.
https://lists.ozlabs.org/pipermail/openbmc/2020-September/023071.html
--
Patrick Williams
::DISCLAIMER::
________________________________
The contents of this e-mail and any attachment(s) are confidential and
intended for the named recipient(s) only. E-mail transmission is not
guaranteed to be secure or error-free as information could be intercepted,
corrupted, lost, destroyed, arrive late or incomplete, or may contain
viruses in transmission. The e mail and its contents (with or without
referred errors) shall therefore not attach any liability on the originator
or HCL or its affiliates. Views or opinions, if any, presented in this
email are solely those of the author and may not necessarily reflect the
views or opinions of HCL or its affiliates. Any form of reproduction,
dissemination, copying, disclosure, modification, distribution and / or
publication of this message without the prior written consent of authorized
representative of HCL is strictly prohibited. If you have received this
email in error please delete it and notify the sender immediately. Before
opening any email and/or attachments, please check them for viruses and
other defects.
________________________________
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ozlabs.org/pipermail/openbmc/attachments/20201005/d432fe72/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: graycol.gif
Type: image/gif
Size: 105 bytes
Desc: not available
URL: <http://lists.ozlabs.org/pipermail/openbmc/attachments/20201005/d432fe72/attachment-0001.gif>
More information about the openbmc
mailing list