Security Working Group - Wednesday July 22 - results

[Ed Tanous]

First off, if you are finding certificates are expiring in a
production environment, you need to rethink your certificate
replacement strategy.  The BMC cannot be the primary entity in charge
of certificate replacement and revocation, and the fact that this is
happening points to a greater organizational problem than the bmc can
cover.

One thing to note;  At one point, I had talked through how to
prototype ACME protocol replacement of certificates automatically, so,
given an ACME server on the network, the BMC could essentially
automatically provision itself and keep its certs up to date.  If
someone wanted to run with that, it might reduce some of the pain here
(and be extremely cool).

On Thu, Jul 23, 2020 at 7:24 AM Joseph Reynolds <jrey at linux.ibm.com> wrote:
>
>
> 4 Question: If BMCWeb finds an unusable HTTPS site identity certificate,
> it DELETES it and self-generates one.  This has caused problems for
> certificates that are not valid until a future date. In general, what
> certificate management support should we have for BMCWeb?  What is needed?

This is not how bmcweb was intended to work, and I had explicitly
tested that it doesn't have this behavior (prior to it being moved
over to certificate manager).  The only time it previously would
replace a certificate was if the cert file didn't exist, or was
completely unreadable.  I suspect the right thing here is to bisect
the commit that broke this behavior (I'll bet it's easy to find) and
get a bug filed with the author so they can either fix it or revert
the patch.

This one smells awfully suspicious: 082f28fd2588fd9fcd9452ad38234ce875319163.

> ANSWER:
> There were two discussion threads: The BMC’s notion of time of day
> (TOD), and how BMCWeb should handle certificates.
>
> Does the BMC have a battery backed TOD clock?  Depends on BMC model.
> Can it validate if it has access to its NTP server (when configured)?
> Does the BMC know if its time was set correctly?
>
> How does the BMC know if the BMC has the correct time?  Have a BMC flag
> that says, “Look like the BMC TOD clock is not working.”  Does the BMC
> know if we got a good time from an  NTP server?  Can we read the GPS
> signal?  What is the industry solution?
> Should the BMC store its idea of what date it is?  So it can report if
> the time changes significantly.  Or will this lead to a bigger problem?
> Is it better/simpler to check for TOD = beginning-of-era-1/1/1970?  →
> start an email thread

The above is all asking the wrong question: "Can we determine if the
certificate is valid?"  This is irrelevant, the question is: "Should
we ever be replacing a user provided certificate with one generated on
the BMC."  The answer previously has been no.  In almost all cases the
user provided certificate, even an expired one, will still be better
than one the BMC self signs.  Between having an invalid certificate
chain, and an invalid date, I'll take the invalid date every time.

It should be noted, most browsers (in my testing) seem to ignore the
HTTP date header entirely, so the BMC doesn't even need the correct
time to set up a proper encryption channel.

>
> BMCWeb configuration?  Configure option: delete cert and generate
> self-signed -vs- use defective certificate.  What is the purpose of
> deleting the unusable cert?

This question needs answered.  I suspect this was unintentional, and
someone simply used some bad openssl code to attempt to verify the
cert against the (possibly non existent) chain without realizing this
requirement.

>  Should “out of date” not be part of the
> “unusable” definition? ⇒ Ideas: 1. If bmcweb finds a usable cert but is
> out of date, that cert can still be used.  2. Leave the defective
> certificate (do not delete it) and log an error.

A lot of BMCs don't have a dedicated RTC, and rely on other systems
(like the PCH or NTP) to get the correct time.  bmcweb needs to come
up long before the PCH or NTP (both of which are also optional) so as
a general rule, using these for valid time is a non-starter.  I could
see logging an error _if_ you know time is valid, but I'm not sure how
a bmc could know that.

>
> The group consensus was that BMCWeb should treat its HTTPS site identity
> certificate like this:
> 1 certificate is perfectly good - Use the certificate
> 2 certificate is good but expired or not yet valid - Use the certificate
> and log a warning

Add "and we know time has been set appropriately".  Also, be careful
with not yet valid.  I don't know how openssl chains handle clock skew
between certificate generator and client.  If the BMC has a time
that's 1 minute fast, is the certificate "not yet valid"?

> 3 certificate is missing or bad format or algorithm too old - Use
> another certificate or self-generate a certificate (and log that action)

We shouldn't be replacing certificates unless it's completely missing
(ie, we're on a first boot or a factory reset) and wouldn't be
switching certificates on the fly.  If the algorithm is too old, the
user is free to replace it with their own, which is the right
procedure anyway.  We also have no idea if the user has added this
public certificate to their local root store, so replacing it out of
the blue might look like an attack.  Also, we have no idea if the
users' client systems support the new crypto format.  We could be
unintentionally DOSing them if, say, we turn on a new keytype or size
and their clients don't support that key.

> There are no cases where BMCWeb should delete any certificate.

I think I know what you mean, but to be clear, we should delete the
old certificates when they're replaced.  We should delete root trust
store certificates when requested, ect.  I don't think we can say "no
cases" here.