Metrics vs Logging, Continued
Michael.E.Brown at dell.com
Michael.E.Brown at dell.com
Sat Dec 23 05:02:18 AEDT 2017
Prometheus metrics is fundamentally a pull model, not a push model. If you have a pull model, it greatly simplifies the dependencies:
- Make them compile-time selectable on the daemon side, compile them out on builds where you don’t need them.
- startup dependencies are easier: you don’t have to worry about error cases where the metrics daemon doesn't start
- Pull metrics internally or externally (daemons listen on 127.0.0.1, optionally reverse proxy that through your web service).
- different metrics servers can poll at different rates for different endpoints. Push model is more of a one-size-fits-all. Push model also means that each daemon needs to know the destinations for each push
- Optionally run the metrics server or not depending on configuration.
- Pull model naturally self-limits in performance-limited cases... you don’t have a thundering herd of daemons trying to push metrics. In case metrics server gets loaded it will naturally slow down polls to backend daemons.
- We can write a metrics server to poll daemons and present dbus/etc endpoints to expose those metrics for the things we think should be part of our API.
All that being said, I would be open to designing a dbus api that is similar to the Prometheus api. Boiling down the types of metrics is a useful abstraction, and I think Prometheus has this basically right: counters, gauges, histograms, etc.
But what I think would be pretty nice is if you could point graphana/Prometheus towards every BMC on your network to get nice graphs of temp, fan speeds, etc.
From: Patrick Venture [mailto:venture at google.com]
Sent: Wednesday, December 20, 2017 2:11 PM
To: Brown, Michael E <Michael_E_Brown at Dell.com>
Cc: OpenBMC Maillist <openbmc at lists.ozlabs.org>; Brad Bishop <bradleyb at fuzziesquirrel.com>
Subject: Re: Metrics vs Logging, Continued
On Wed, Dec 20, 2017 at 12:01 PM, Patrick Venture <venture at google.com> wrote:
> On Wed, Dec 20, 2017 at 11:07 AM, <Michael.E.Brown at dell.com> wrote:
>> So what do you mean by "something similar on the inside"? Do you have references?
> I was just indicating that this use of http to export metrics is
> something with which I'm familiar.
>> And what do you mean by "not well-suited for an embedded platform"? What metrics are you using to base this opinion on?
> I'm going from design metrics. Every daemon now will need a thread to
> provide the information to anyone who asks. So that's X daemons
> needing new threads to handle the requests. That's my understanding
> of how this works when you add the library to your daemon.
I'm reading through the c++ library for this to see whether it pushes data to some central metric server, which would be preferable design-wise.
>> -----Original Message-----
>> From: Patrick Venture [mailto:venture at google.com]
>> Sent: Monday, December 18, 2017 12:17 PM
>> To: Brown, Michael E <Michael_E_Brown at Dell.com>
>> Cc: OpenBMC Maillist <openbmc at lists.ozlabs.org>; Brad Bishop
>> <bradleyb at fuzziesquirrel.com>
>> Subject: Re: Metrics vs Logging, Continued
>> So, we use something similar on the inside, but it's not well-suited for an embedded platform.
>> On Fri, Dec 15, 2017 at 11:06 AM, <Michael.E.Brown at dell.com> wrote:
>>> Things like Prometheus and graphana already exist and have fairly standardized metrics gathering interfaces. Why re-invent the wheel here? I would advocate for Prometheus for metrics gathering.
>>> Quick primer on Prometheus: each process exposes an HTTP endpoint that reports metrics in a standardized JSON format. The process can expose counters, gauges, histograms or summaries. The server side is responsible for polling the various clients at whatever interval is desired and can format results graphically and over a bunch of clients. Each process can instrument itself, there are c++ client libraries to do this already. Then we can aggregate them out with a route from the main HTTP server.
>>> All of the things in your metrics list fall neatly into types of data that Prometheus format handles well.
>>> -----Original Message-----
>>> From: openbmc
>>> [mailto:openbmc-bounces+michael.e.brown=dell.com at lists.ozlabs.org]
>>> On Behalf Of Patrick Venture
>>> Sent: Tuesday, December 5, 2017 10:51 AM
>>> To: OpenBMC Maillist <openbmc at lists.ozlabs.org>; Brad Bishop
>>> <bradleyb at fuzziesquirrel.com>
>>> Subject: Metrics vs Logging, Continued
>>> Logging being something separate from metrics -- I've been toying around with different approaches to allowing userspace metrics collection and distribution. There are likely better ways, and I think I saw a message on chat about a metrics library that could be used. -- but I've mostly been following email.
>>> I was thinking this morning of a couple methods, some y'all might like (one where the daemon owns it, one where the metric owner owns it):
>>> 1) Each daemon can be responsible for exporting onto dbus some objects with a well-defined path that are of a metric type that has a value and the daemon that owns it is therefore responsible for maintaining it. to collect the metrics, one must grab the subtree for the starting point and trace out all the different metrics and get the values from their owners. and reports that up somehow. -- the somehow could be several IPMI packets. or several IPMI packets containing a protobuf (similarly to the flash access approach proposed by Brendan).
>>> The upside to the free-form text and paths is you could parse it out to figure out what was each thing.
>>> 2) Each daemon that wants to track a metric creates a metric object in another daemon (via dbus calls) and then periodically updates that value. then the information can be reported in the way described above similarly, except the owner of the dbus objects would be the one daemon and one bus, etc. This implementation requires a lot more dbus traffic to maintain the values. However, in situations where one doesn't want to manage their own dbus object for this, they can just make one dbus call to update their value based on whatever mechanism they use for timing this and they can store the metrics internally in their daemon however they please. Another upside to this is that it'd be straightforward to add to the current set of daemons without needing to restructure anything. Also, depending on the metric itself, it may not be something updated all that frequently. For many, I foresee updating on non-critical failures, or interesting failures -- for instance, how often the ipmi daemon's reply is rejected by the btbridge daemon.
>>> Approach #2 could be rolled into a couple library calls as well, very easily such that they don't even know the internals of the tracking...
>>> I like and don't like the free-form text naming of the metrics, because obviously they can be human-readable. Another approach might be to assign them human readable names and IDs, similarly to sensors so that you can read back the name for a metric once, and then in the future cache it, making subsequent requests smaller.
>>> Obvious downside to both implementations (although #2 has an easy mitigation), if the daemon with the internal state crashes the metrics are lost, when it comes back up all the metrics are 0. If the metrics are owned by another daemon, then the library calls to set up the metrics tracking could check if the metric already exists, and use that value to start with -- then you only have to care about that one daemon crashing. It could periodically write the values down and then read them on start-up to persist these values. However, you might want the values to not persist... I imagine I wouldn't, however, something like boot count would...
>>> There are specific things that the host wants to know, that really fall into metrics over logging:
>>> 1) BMCs boot count
>>> 2) i2c ioctl failure count (which bus/device/reg: count)
>>> 3) Specific sensor requests (reading, writing)
>>> 4) Fan control failsafe mode count, how often it's falling into
>>> failsafe mode
>>> 5) How often the ipmi daemon's reply to the btbridge daemon fails.
>>> Given some feedback on this, I'll write up a design and the use-cases it's trying to address.
More information about the openbmc