<html><body><p><font size="2">Jayashree,</font><br><br><font size="2">Couple places where it checks for the SSH to BMC and SOL as well, that should catch those in your HW test CI environment</font><br><br><font size="2">Example:</font><br><br><a href="https://github.com/openbmc/openbmc-test-automation/blob/master/test_lists/HW_CI#L2"><font size="2">https://github.com/openbmc/openbmc-test-automation/blob/master/test_lists/HW_CI#L2</font></a><br><font color="#24292E" face="Consolas">Test_SSH_And_IPMI_Connections</font> <br><br><a href="https://github.com/openbmc/openbmc-test-automation/blob/master/test_lists/HW_CI#L4"><font size="2">https://github.com/openbmc/openbmc-test-automation/blob/master/test_lists/HW_CI#L4</font></a><br><font color="#24292E" face="Consolas">-i Verify_Redfish_Host_PowerOn</font> <br><font color="#24292E" face="Consolas">-i Verify_Redfish_Host_PowerOff</font> <br><font size="2"><br>Here SOL connection done as part of the above setup/teardown<br></font><a href="https://github.com/openbmc/openbmc-test-automation/blob/master/redfish/systems/test_power_operations.robot#L87"><font size="2">https://github.com/openbmc/openbmc-test-automation/blob/master/redfish/systems/test_power_operations.robot#L87</font></a><br><a href="https://github.com/openbmc/openbmc-test-automation/blob/master/redfish/systems/test_power_operations.robot#L95"><font size="2">https://github.com/openbmc/openbmc-test-automation/blob/master/redfish/systems/test_power_operations.robot#L95</font></a><font size="2"><br><br></font><br><font size="2">Those tests are currently running in upstream community HW Jenkins test CI. So it would have caught those SSH related for generic ports in general and you can use similar in your environment.</font><br><br><b><font size="2" color="#0000FF">Thanks and Regards,</font></b><br><font size="2" color="#0000FF"> George Keishing</font><br><font size="2" color="#0000FF"> </font><br><br><br><img width="16" height="16" src="cid:1__=8FBB0F6BDFA0C9758f9e8a93df938690918c8FB@" border="0" alt="Inactive hide details for Jayashree D ---05-10-2020 02:09:58 PM---Classification: HCL Internal Regarding SSH connection, an iss"><font size="2" color="#424282">Jayashree D ---05-10-2020 02:09:58 PM---Classification: HCL Internal Regarding SSH connection, an issue has been created in openbmc and I al</font><br><br><font size="2" color="#5F5F5F">From: </font><font size="2">Jayashree D <jayashree-d@hcl.com></font><br><font size="2" color="#5F5F5F">To: </font><font size="2">"openbmc@lists.ozlabs.org" <openbmc@lists.ozlabs.org></font><br><font size="2" color="#5F5F5F">Cc: </font><font size="2">George Keishing <gkeishin@in.ibm.com>, "geissonator@yahoo.com" <geissonator@yahoo.com>, Vijay Khemka <vijaykhemka@fb.com>, Konstantin Klubnichkin <kitsok@yandex-team.ru></font><br><font size="2" color="#5F5F5F">Date: </font><font size="2">05-10-2020 02:09 PM</font><br><font size="2" color="#5F5F5F">Subject: </font><font size="2">[EXTERNAL] RE: Connection issue in OpenBMC image</font><br><font size="2" color="#5F5F5F">Sent by: </font><font size="2">"openbmc" <openbmc-bounces+gkeishin=in.ibm.com@lists.ozlabs.org></font><br><hr width="100%" size="2" align="left" noshade style="color:#8091A5; "><br><br><br><tt><font size="2">Classification: HCL Internal<br><br>Regarding SSH connection, an issue has been created in openbmc and I also see others having this same issue.<br>From the comments, I have run "dropbear -E -p 5022" in the target (UART-console) and tried to connect the target using "ssh -p 5022 <ip>" and ssh connection established.<br>But, reboot and systemctl commands hangs.<br><br>Issue - </font></tt><tt><font size="2"><a href="https://github.com/openbmc/openbmc/issues/3701">https://github.com/openbmc/openbmc/issues/3701</a></font></tt><tt><font size="2"> <br><br>root@tiogapass:~# dropbear -E -p 5022<br>[348] Jan 01 00:06:48 Failed loading /etc/dropbear/dropbear_dss_host_key<br>[348] Jan 01 00:06:48 Failed loading /etc/dropbear/dropbear_ecdsa_host_key<br>[348] Jan 01 00:06:48 Failed loading /etc/dropbear/dropbear_ed25519_host_key<br>[349] Jan 01 00:06:48 Running in background<br><br>[root@odc ~]# ssh -p 5022 root@10.0.128.108 root@10.0.128.108's password:<br>root@tiogapass:~#<br><br>Hi George,<br><br>We are facing connection issue in accessing the target after flashing the latest image.<br>In openbmc-test-automation, whether any test cases are present in CI to identify these issues ?<br>Please let us know your comments on this.<br><br>Regards,<br>Jayashree<br><br>-----Original Message-----<br>From: Jayashree D<br>Sent: Friday, September 25, 2020 10:29 AM<br>To: openbmc@lists.ozlabs.org<br>Cc: Konstantin Klubnichkin <kitsok@yandex-team.ru>; Vijay Khemka <vijaykhemka@fb.com>; geissonator@yahoo.com; joel@jms.id.au; Patrick Williams <patrick@stwcx.xyz><br>Subject: RE: Connection issue in OpenBMC image<br><br>Classification: HCL Internal<br><br>Hi Team,<br><br>In the latest openbmc build, after image upgradation in the target, not able to connect the target through SSH but able to ping the IP Address.<br><br>After analysing the latest commits, reverted the below commit in the latest build and checked by flashing the image. Now the target is connecting through SSH. Please help us on fixing this issue.<br><br>Commit Link - </font></tt><tt><font size="2"><a href="https://github.com/openbmc/openbmc/commit/635e0e4637e40ba03f69204265427550fd404f4c">https://github.com/openbmc/openbmc/commit/635e0e4637e40ba03f69204265427550fd404f4c</a></font></tt><tt><font size="2"> <br><br><br>Observation on UART-console after flashing latest image without any changes:<br><br>1. reboot command is not working.<br>2. systemctl status <service_name> is not providing any status. ( Failed to get properties: Connection timed out) 3. I tried "ssh -vvv <ip>" and logs are attached for working and non-working image.<br>4. From controller, I tried to upgrade image using redfish and image is being copied and following logs shown.<br>root@tiogapass:~# journalctl | grep image Jan 01 00:00:37 tiogapass phosphor-image-updater[246]: Error in mapper GetSubTreePath Jan 01 10:43:59 tiogapass phosphor-image-updater[246]: BMC image activating - BMC reboots are disabled.<br><br>5. Using Rest API command,<br><br>[root@odc ]# curl -k -H "X-Auth-Token: $token" -H "Content-Type: application/json" -X PUT -d '{"data":"xyz.openbmc_project.Software.Activation.RequestedActivations.Active"}' </font></tt><tt><font size="2"><a href="https://$%7Bbmc%7D/xyz/openbmc_project/software/a77348be/attr/RequestedActivation">https://$%7Bbmc%7D/xyz/openbmc_project/software/a77348be/attr/RequestedActivation</a></font></tt><tt><font size="2"> <br>{<br> "data": {<br> "description": "org.freedesktop.DBus.Error.NoReply"<br> },<br> "message": "Method call timed out",<br> "status": "error"<br>}<br><br><br>Regards,<br>Jayashree<br><br>-----Original Message-----<br>From: Jayashree D<br>Sent: Friday, September 18, 2020 3:18 PM<br>To: Patrick Williams <patrick@stwcx.xyz>; openbmc@lists.ozlabs.org<br>Cc: Konstantin Klubnichkin <kitsok@yandex-team.ru><br>Subject: RE: Connection issue in OpenBMC image<br><br>Classification: HCL Internal<br><br>Hello Patrick,<br><br>I saw the post about dropbear, but that commit was updated on July16 and my target is connecting till August last week image. I don't think that will be an issue. Also on working image, I tried with 'ssh -vvv ' and I got below information.<br><br>OpenSSH_7.4p1, OpenSSL 1.0.2k-fips 26 Jan 2017<br>debug1: Reading configuration data /etc/ssh/ssh_config<br>debug1: /etc/ssh/ssh_config line 58: Applying options for *<br>debug2: resolving "10.0.128.108" port 22<br>debug2: ssh_connect_direct: needpriv 0<br>debug1: Connecting to 10.0.128.108 [10.0.128.108] port 22.<br>debug1: Connection established.<br>debug1: permanently_set_uid: 0/0<br>debug1: key_load_public: No such file or directory<br>debug1: identity file /root/.ssh/id_rsa type -1<br>debug1: key_load_public: No such file or directory<br>debug1: identity file /root/.ssh/id_rsa-cert type -1<br>debug1: key_load_public: No such file or directory<br>debug1: identity file /root/.ssh/id_dsa type -1<br>debug1: key_load_public: No such file or directory<br>debug1: identity file /root/.ssh/id_dsa-cert type -1<br>debug1: key_load_public: No such file or directory<br>debug1: identity file /root/.ssh/id_ecdsa type -1<br>debug1: key_load_public: No such file or directory<br>debug1: identity file /root/.ssh/id_ecdsa-cert type -1<br>debug1: key_load_public: No such file or directory<br>debug1: identity file /root/.ssh/id_ed25519 type -1<br>debug1: key_load_public: No such file or directory<br>debug1: identity file /root/.ssh/id_ed25519-cert type -1<br>debug1: Enabling compatibility mode for protocol 2.0<br>debug1: Local version string SSH-2.0-OpenSSH_7.4<br>debug1: Remote protocol version 2.0, remote software version dropbear_2020.80<br>debug1: no match: dropbear_2020.80<br>debug2: fd 3 setting O_NONBLOCK<br>debug1: Authenticating to 10.0.128.108:22 as 'root'<br>debug3: hostkeys_foreach: reading file "/root/.ssh/known_hosts"<br>debug3: record_hostkey: found key type RSA in file /root/.ssh/known_hosts:68<br>debug3: load_hostkeys: loaded 1 keys from 10.0.128.108<br>debug3: order_hostkeyalgs: prefer hostkeyalgs: ssh-rsa-cert-v01@openssh.com,rsa-sha2-512,rsa-sha2-256,ssh-rsa<br>debug3: send packet: type 20<br>debug1: SSH2_MSG_KEXINIT sent<br>debug3: receive packet: type 20<br>debug1: SSH2_MSG_KEXINIT received<br>debug2: local client KEXINIT proposal<br>debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1,ext-info-c<br>debug2: host key algorithms: ssh-rsa-cert-v01@openssh.com,rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,ssh-dss<br>debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc<br>debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc<br>debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1<br>debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1<br>debug2: compression ctos: none,zlib@openssh.com,zlib<br>debug2: compression stoc: none,zlib@openssh.com,zlib<br>debug2: languages ctos:<br>debug2: languages stoc:<br>debug2: first_kex_follows 0<br>debug2: reserved 0<br>debug2: peer server KEXINIT proposal<br>debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp521,ecdh-sha2-nistp384,ecdh-sha2-nistp256,diffie-hellman-group14-sha256,kexguess2@matt.ucc.asn.au<br>debug2: host key algorithms: rsa-sha2-256,ssh-rsa<br>debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes256-ctr<br>debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes256-ctr<br>debug2: MACs ctos: hmac-sha1,hmac-sha2-256<br>debug2: MACs stoc: hmac-sha1,hmac-sha2-256<br>debug2: compression ctos: zlib@openssh.com,none<br>debug2: compression stoc: zlib@openssh.com,none<br>debug2: languages ctos:<br>debug2: languages stoc:<br>debug2: first_kex_follows 0<br>debug2: reserved 0<br>debug1: kex: algorithm: curve25519-sha256<br>debug1: kex: host key algorithm: rsa-sha2-256<br>debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none<br>debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none<br>debug1: kex: curve25519-sha256 need=64 dh_need=64<br>debug1: kex: curve25519-sha256 need=64 dh_need=64<br>debug3: send packet: type 30<br>debug1: expecting SSH2_MSG_KEX_ECDH_REPLY<br>debug3: receive packet: type 31<br>debug1: Server host key: ssh-rsa SHA256:3WwhPmIIxzrw0+cm/0vN3hifY4kh9sJhClVNw6zrJ7Y<br>debug3: hostkeys_foreach: reading file "/root/.ssh/known_hosts"<br>debug3: record_hostkey: found key type RSA in file /root/.ssh/known_hosts:68<br>debug3: load_hostkeys: loaded 1 keys from 10.0.128.108<br>debug1: Host '10.0.128.108' is known and matches the RSA host key.<br>debug1: Found key in /root/.ssh/known_hosts:68<br>debug3: send packet: type 21<br>debug2: set_newkeys: mode 1<br>debug1: rekey after 134217728 blocks<br>debug1: SSH2_MSG_NEWKEYS sent<br>debug1: expecting SSH2_MSG_NEWKEYS<br>debug3: receive packet: type 21<br>debug1: SSH2_MSG_NEWKEYS received<br>debug2: set_newkeys: mode 0<br>debug1: rekey after 134217728 blocks<br>debug2: key: /root/.ssh/id_rsa (0x558ea3ad3640), agent<br>debug2: key: /root/.ssh/id_rsa ((nil))<br>debug2: key: /root/.ssh/id_dsa ((nil))<br>debug2: key: /root/.ssh/id_ecdsa ((nil))<br>debug2: key: /root/.ssh/id_ed25519 ((nil))<br>debug3: send packet: type 5<br>debug3: receive packet: type 7<br>debug1: SSH2_MSG_EXT_INFO received<br>debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,rsa-sha2-256,ssh-rsa,ssh-dss><br>debug3: receive packet: type 6<br>debug2: service_accept: ssh-userauth<br>debug1: SSH2_MSG_SERVICE_ACCEPT received<br>debug3: send packet: type 50<br>debug3: receive packet: type 51<br>debug1: Authentications that can continue: publickey,password<br>debug3: start over, passed a different list publickey,password<br>debug3: preferred gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,password<br>debug3: authmethod_lookup publickey<br>debug3: remaining preferred: keyboard-interactive,password<br>debug3: authmethod_is_enabled publickey<br>debug1: Next authentication method: publickey<br>debug1: Offering RSA public key: /root/.ssh/id_rsa<br>debug3: send_pubkey_test<br>debug3: send packet: type 50<br>debug2: we sent a publickey packet, wait for reply<br>debug3: receive packet: type 51<br>debug1: Authentications that can continue: publickey,password<br>debug1: Trying private key: /root/.ssh/id_rsa<br>debug3: sign_and_send_pubkey: RSA SHA256:YfteufmWUV8W7EQEycZ+38skgUWGDTYFHw93a7SwwLM<br>debug3: send packet: type 50<br>debug2: we sent a publickey packet, wait for reply<br>debug3: receive packet: type 51<br>debug1: Authentications that can continue: publickey,password<br>debug1: Trying private key: /root/.ssh/id_dsa<br>debug3: no such identity: /root/.ssh/id_dsa: No such file or directory<br>debug1: Trying private key: /root/.ssh/id_ecdsa<br>debug3: no such identity: /root/.ssh/id_ecdsa: No such file or directory<br>debug1: Trying private key: /root/.ssh/id_ed25519<br>debug3: no such identity: /root/.ssh/id_ed25519: No such file or directory<br>debug2: we did not send a packet, disable method<br>debug3: authmethod_lookup password<br>debug3: remaining preferred: ,password<br>debug3: authmethod_is_enabled password<br>debug1: Next authentication method: password<br><br><br>In non-working image, the logs are stopped after below lines and it is not providing any errors.<br><br>debug1: Enabling compatibility mode for protocol 2.0<br>debug1: Local version string SSH-2.0-OpenSSH_7.4<br><br>Also one more observation in UART-Console, after flashing latest image.<br><br>1. reboot command is not working.<br>2. systemctl status <service_name> is not providing any status. ( Failed to get properties: Connection timed out) 3. I can able to ping the ip address but scp is not working.<br><br>Thanks,<br>Jayashree<br><br><br>-----Original Message-----<br>From: Patrick Williams <patrick@stwcx.xyz><br>Sent: Thursday, September 17, 2020 9:16 PM<br>To: Jayashree D <jayashree-d@hcl.com><br>Cc: Konstantin Klubnichkin <kitsok@yandex-team.ru>; openbmc@lists.ozlabs.org<br>Subject: Re: Connection issue in OpenBMC image<br><br>Hello Jayashree,<br><br>I saw an output `ssh -v` from you earlier, but there really wasn't any useful information there. It looked like the connection was being made and keys were exchanged and then the log just stopped abruptly. This tells me it likely isn't a networking issue but an issue in the handshake between the ssh-client (your computer) and ssh-server (dropbear). You can continue to add '-v' parameters up to `ssh -vvv` and you'll get increasingly more information.<br><br>Joseph Reynolds recently posted a reminder about dropbear disabling weak ciphers[1]. Is it possible that your client is using an old cipher?<br><br>On Wed, Sep 16, 2020 at 11:35:28AM +0000, Jayashree D wrote:<br>> root@tiogapass:~# journalctl | grep drop<br>...<br>> Jan 01 00:15:28 tiogapass systemd[1]: dropbear@0-10.0.128.108:22-10.0.0.1:51810.service: Succeeded.<br>> Jan 01 00:15:44 tiogapass dropbear[2753]: Child connection from<br>> ::ffff:10.0.0.1:51944 Jan 01 00:15:50 tiogapass dropbear[2753]: PAM<br>> password auth succeeded for 'root' from ::ffff:10.0.0.1:51944<br><br>This looks like a valid connection was established.<br><br>> 15.09.2020, 16:12, "Jayashree D" <jayashree-d@hcl.com<</font></tt><tt><font size="2"><a href="mailto:jayashree-d@hcl.com">mailto:jayashree-d@hcl.com</a></font></tt><tt><font size="2">>>:<br>><br>> OpenSSH_7.4p1, OpenSSL 1.0.2k-fips 26 Jan 2017<br>> debug1: Reading configuration data /etc/ssh/ssh_config<br>> debug1: /etc/ssh/ssh_config line 58: Applying options for *<br>> debug1: Connecting to 10.0.128.108 [10.0.128.108] port 22.<br>> debug1: Connection established.<br>> debug1: permanently_set_uid: 0/0<br>> debug1: key_load_public: No such file or directory<br>> debug1: identity file /root/.ssh/id_rsa type -1<br>> debug1: key_load_public: No such file or directory<br>> debug1: identity file /root/.ssh/id_rsa-cert type -1<br>> debug1: key_load_public: No such file or directory<br>> debug1: identity file /root/.ssh/id_dsa type -1<br>> debug1: key_load_public: No such file or directory<br>> debug1: identity file /root/.ssh/id_dsa-cert type -1<br>> debug1: key_load_public: No such file or directory<br>> debug1: identity file /root/.ssh/id_ecdsa type -1<br>> debug1: key_load_public: No such file or directory<br>> debug1: identity file /root/.ssh/id_ecdsa-cert type -1<br>> debug1: key_load_public: No such file or directory<br>> debug1: identity file /root/.ssh/id_ed25519 type -1<br>> debug1: key_load_public: No such file or directory<br>> debug1: identity file /root/.ssh/id_ed25519-cert type -1<br>> debug1: Enabling compatibility mode for protocol 2.0<br>> debug1: Local version string SSH-2.0-OpenSSH_7.4<br><br>This is the log that also looks like a good connection. Identity files were attempted to be exchanged. Version strings were exchanged. And then the log just abruptly stops. Was the connection dropped? Is it hung?<br><br>1. </font></tt><tt><font size="2"><a href="https://lists.ozlabs.org/pipermail/openbmc/2020-September/023071.html">https://lists.ozlabs.org/pipermail/openbmc/2020-September/023071.html</a></font></tt><tt><font size="2"> <br><br>--<br>Patrick Williams<br>::DISCLAIMER::<br>________________________________<br>The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only. E-mail transmission is not guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or may contain viruses in transmission. The e mail and its contents (with or without referred errors) shall therefore not attach any liability on the originator or HCL or its affiliates. Views or opinions, if any, presented in this email are solely those of the author and may not necessarily reflect the views or opinions of HCL or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification, distribution and / or publication of this message without the prior written consent of authorized representative of HCL is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately. Before opening any email and/or attachments, please check them for viruses and other defects.<br>________________________________<br><br></font></tt><br><br><BR>
</body></html>